4.1.0

4.1.0 (2023-01-25)

Features

• add ibis.get_backend function (2d27df8)
• add py.typed to allow mypy to type check packages that use ibis (765d42e)
• api: add ibis.set_backend function (e7fabaf)
• api: add selectors for easier selection of columns (306bc88)
• bigquery: add JS UDF support (e74328b)
• bigquery: add SQL UDF support (db24173)
• bigquery: add to_pyarrow method (30157c5)
• bigquery: implement bitwise operations (55b69b1)
• bigquery: implement ops.Typeof (b219919)
• bigquery: implement ops.ZeroIfNull (f4c5607)
• bigquery: implement struct literal (c5f2a1d)
• clickhouse: properly support native boolean types (31cc7ba)
• common: add support for annotating with coercible types (ae4a415)
• common: make frozendict truly immutable (1c25213)
• common: support annotations with typing.Literal (6f89f0b)
• common: support generic mapping and sequence type annotations (ddc6603)
• dask: support connect() with no arguments (67eed42)
• datatype: add optional timestamp scale parameter (a38115a)
• datatypes: add as_struct method to convert schemas to structs (64be7b1)
• duckdb: add read_json function for consuming newline-delimited JSON files (65e65c1)
• mssql: add a bunch of missing types (c698d35)
• mssql: implement inference for DATETIME2 and DATETIMEOFFSET (aa9f151)
• nicer repr for Backend.tables (0d319ca)
• pandas: support connect() with no arguments (78cbbdd)
• polars: allow ibis.polars.connect() to function without any arguments (d653a07)
• polars: handle casting to scaled timestamps (099d1ec)
• postgres: add Map(string, string) support via the built-in HSTORE extension (f968f8f)
• pyarrow: support conversion to pyarrow map and struct types (54a4557)
• snowflake: add more array operations (8d8bb70)
• snowflake: add more map operations (7ae6e25)
• snowflake: any/all/notany/notall reductions (ba1af5e)
• snowflake: bitwise reductions (5aba997)
• snowflake: date from ymd (035f856)
• snowflake: fix array slicing (bd7af2a)
• snowflake: implement ArrayCollect (c425f68)
• snowflake: implement NthValue (0dca57c)
• snowflake: implement ops.Arbitrary (45f4f05)
• snowflake: implement ops.StructColumn (41698ed)
• snowflake: implement StringSplit (e6acc09)
• snowflake: implement StructField and struct literals (286a5c3)
• snowflake: implement TimestampFromUNIX (314637d)
• snowflake: implement TimestampFromYMDHMS (1eba8be)
• snowflake: implement typeof operation (029499c)
• snowflake: implement exists/not exists (7c8363b)
• snowflake: implement extract millisecond (3292e91)
• snowflake: make literal maps and params work (dd759d3)
• snowflake: regex extract, search and replace (9c82179)
• snowflake: string to timestamp (095ded6)
• sqlite: implement _get_schema_using_query in SQLite backend (7ff84c8)
• trino: compile timestamp types with scale (67683d3)
• trino: enable ops.ExistsSubquery and ops.NotExistsSubquery (9b9b315)
• trino: map parameters (53bd910)
• ux: improve error message when column is not found (b527506)

Bug Fixes

• backend: read the default backend setting in _default_backend (11252af)
• bigquery: move connection logic to do_connect (42f2106)
• bigquery: remove invalid operations from registry (911a080)
• bigquery: resolve deprecation warnings for StructType and Schema (c9e7078)
• clickhouse: fix position call (702de5d)
• correctly visualize array type (26b0b3f)
• deps: make sure pyarrow is not an implicit dependency (10373f4)
• duckdb: make read_csv on URLs work (9e61816)
• duckdb: only try to load extensions when necessary for csv (c77bde7)
• duckdb: remove invalid operations from registry (ba2ec59)
• fallback to default backend with to_pyarrow/to_pyarrow_batches (a1a6902)
• impala: remove broken alias elision (32b120f)
• ir: error for order_by on nonexistent column (57b1dd8)
• ir: ops.Where output shape should consider all arguments (6f87064)
• mssql: infer bit as boolean everywhere (24f9d7c)
• mssql: pull nullability from column information (490f8b4)
• mysql: fix mysql query schema inference (12f6438)
• polars: remove non-working Binary and Decimal literal inference (0482d15)
• postgres: use permanent views to avoid connection pool defeat (49a4991)
• pyspark: fix substring constant translation (40d2072)
• set ops: raise if no tables passed to set operations (bf4bdde)
• snowflake: bring back bitwise operations (260facd)
• snowflake: don't always insert a cast (ee8817b)
• snowflake: implement working TimestampNow (42d95b0)
• snowflake: make sqlalchemy 2.0 compatible (8071255)
• snowflake: re-enable ops.TableArrayView (a1ad2b7)
• snowflake: remove invalid operations from registry (2831559)
• sql: add typeof test and bring back implementations (7dc5356)
• sqlalchemy: 2.0 compatibility (837a736)
• sqlalchemy: fix view creation with select stmts that have bind parameters (d760e69)
• sqlalchemy: handle correlated exists sanely (efa42bd)
• sqlalchemy: handle generic geography/geometry by name instead of geotype (23c35e1)
• sqlalchemy: use exec_driver_sql in view teardown (2599c9b)
• sqlalchemy: use the backend's compiler instead of AlchemyCompiler (9f4ff54)
• sql: fix broken call to ibis.map (045edc7)
• sqlite: interpolate pathlib.Path correctly in attach (0415bd3)
• trino: ensure connecting works with trino 0.321 (07cee38)
• trino: remove invalid operations from registry (665265c)
• ux: remove extra trailing newline in expression repr (ee6d58a)

Documentation

• add BigQuery backend docs (09d8995)
• add streamlit app for showing the backend operation matrix (3228f64)
• allow deselecting geospatial ops in backend support matrix (012da8c)
• api: document more public expression APIs (337018f)
• backend-info: prevent app from trying install duckdb extensions (3d94082)
• clean up gen_matrix.py after adding streamlit app (deb80f2)
• duckdb: add to_pyarrow_batches documentation (ec1ffce)
• embed streamlit operation matrix app to docs (469a50d)
• make firefox render the proper iframe height (ff1d4dc)
• publish raw data for operation matrix (62e68da)
• re-order when to download test data (8ce8c16)
• release: update breaking changes in the release notes for 4.0.0 (4e91401)
• remove trailing parenthesis (4294397)
• update ibis-version-4.0.0-release.md (f6701df)
• update links to contributing guides (da615e4)

Refactors

• bigquery: explicite disallow INT64 in JS UDF (fb33bf9)
• datatype: add custom sqlalchemy nested types for backend differentiation (dec70f5)
• datatype: introduce to_sqla_type dispatching on dialect (a8bbc00)
• datatypes: remove Geography and Geometry types in favor of GeoSpatial (d44978c)
• datatype: use a mapping to store StructType fields rather than names and types tuples (ff34c7b)
• dtypes: expose nbytes property for integer and floating point datatypes (ccf80fd)
• duckdb: remove .raw_sql call (abc939e)
• duckdb: use sqlalchemy-views to reduce string hacking (c162750)
• ir: remove UnnamedMarker (dd352b1)
• postgres: use a bindparam for metadata queries (b6b4669)
• remove empty unused file (9d63fd6)
• schema: use a mapping to store Schema fields rather than names and types tuples (318179a)
• simplify _find_backend implementation (60f1a1b)
• snowflake: remove unnecessary parse_json call in ops.StructField impl (9e80231)
• snowflake: remove unnecessary casting (271554c)
• snowflake: use unary instead of fixed_arity(..., 1) (4a1c7c9)
• sqlalchemy: clean up quoting implementation (506ce01)
• sqlalchemy: generalize handling of failed type inference (b0f4e4c)
• sqlalchemy: move _get_schema_using_query to base class (296cd7d)
• sqlalchemy: remove the need for deferred columns (e4011aa)
• sqlalchemy: remove use of deprecated isnot (4ec53a4)
• sqlalchemy: use exec_driver_sql everywhere (e8f96b6)
• sql: finally remove _CorrelatedRefCheck (f49e429)

Deprecations

• api: deprecate .to_projection in favor of .as_table (7706a86)
• api: deprecate get_column/s in favor of __getitem__/__getattr__ syntax (e6372e2)
• ir: schedule DatabaseTable.change_name for removal (e4bae26)
• schema: schedule Schema.delete() and Schema.append() for removal (45ac9a9)