2024 Changelog

​

1️⃣ Amazon Data Firehose ingestion

• Zero-code setup with just a few clicks
• Real-time ingestion without batch jobs
• Automatic JSON flattening and column mapping
• Built-in authentication for Firehose integration

​

2️⃣ Time Range Picker UI component

• Relative and absolute time ranges
• Date-to-now selections
• Custom time period options
• Themeable UI

​

3️⃣ Segment event ingestion

• Real-time event processing
• Automatic JSON property flattening
• Batch and individual event support
• Real-time transformations

​

4️⃣ Data Grid UI component

• Built-in pagination
• Ordering and filtering
• Automatic column name formatting
• Built-in drawer for displaying row or JSON values
• Fully themable and customizable

​

5️⃣ DynamoDB → Propel’s ClickHouse

• Seamless integration with DynamoDB streams
• Real-time replication of inserts, updates, and deletes
• No infrastructure management needed
• No need to pay for expensive ETL tools ❤️

​

Improvements and fixes

​

Management API

• Added CREATE, MODIFY and GET Environment operations to the Management API with new ENVIRONMENT_ADMIN scope
• Added Delete Environment mutation
• Improved hardening of the Copy Job functionality

​

Serverless ClickHouse

• Added support for SHOW commands including:
  • SHOW DATABASES
  • SHOW TABLES
  • SHOW COLUMNS
  • SHOW INDEXES
  • SHOW CREATE DATABASE
  • SHOW CREATE TABLE
  • SHOW FUNCTIONS
  • SHOW ENGINES
• Added support for system tables including:
  • system.databases
  • system.tables
  • system.columns
  • system.aggregate_function_combinators
  • system.formats
  • system.functions
  • system.licenses
  • system.settings
  • system.settings_changed
  • system.table_engines
  • system.table_functions
  • system.time_zones
• Improved PopSQL compatibility:
  • Switch to date_time_output_format = 'simple' when Faraday user agent is detected
  • Added support for HTTP GET requests with queries in body
• Enhanced timezone handling:
  • Added timezone parameters to Propel functions
  • Improved timezone management for Time Series API date/time boundaries
• Fixed bug where tables were added without “propel” database qualifier
• Changed currentDatabase() to return “propel”
• Added support for database query parameter and x-clickhouse-database header
• Added support for wait_end_of_query parameter used by Python clients
• Improved authentication handling for missing X-ClickHouse-Key header
• Added support for client_protocol_version in ClickHouse HTTP endpoint

​

Console

• Introduced new DataGrid component for Preview Data and SQL console
• Moved ClickHouse Data Pools to Federated tables section

​

1️⃣ ClickHouse SQL interface

echo 'SELECT 1' | \
curl https://clickhouse.us-east-2.propeldata.com:8443 \
  --user "$APPLICATION_ID:$APPLICATION_SECRET" \
  --data-binary @-

​

2️⃣ New docs

• Serverless ClickHouse: Connection details, supported SQL syntax, SQL reference, functions, and everything you need to know to manage, transform and query your data.
• Ingestion: Guides on ingesting data from a variety of sources to Propel, including Webhooks, events streams, and data warehouses.
• Query APIs: Guides and examples for using the SQL and Query APIs for common analytics use cases and data visualizations.
• Embeddable UIs: Guides and documentation for embedding Propel’s visualization components into your applications.

​

3️⃣ Server-side authentication

curl -X POST https://api.us-east-2.propeldata.com/graphql \
-u $APPLICATION_ID:$APPLICATION_SECRET \
-H "Content-Type: application/json" \
-d '{"query": ...'

​

4️⃣ New GroupBy UI component

​

Improvements and fixes

​

Serverless ClickHouse

• Added support for authentication using X-ClickHouse-User and X-ClickHouse-Key headers.
• Fixed issues with Materialized Views and Copy Jobs that contain common table expressions (CTEs).

​

Ingestion

• Fixed an issue that ocassionally prevented Webhook syncs from progressing beyond the PENDING state.

​

Console

• Fixed issue with GitHub login authentication.
• Added new Console start page and insights section.
• For SSO-enabled accounts, the “Invite Users” button is now disabled and displays a message explaining that user management must be done through the SSO provider.
• Added ability to modify the default timestamp column in Data Pools.
• Added SQL dialect selector to the SQL Console, allowing users to switch between ClickHouse and PostgreSQL syntax

​

API

• Exposed the SQL dialect in the SQL API.
• Fixed a case where the Metric Report API was not returning correct results.

​

Embeddable UI

• Fixed type definitions for chartConfigProps in the <TimeSeries> component.
• Removed unintended Storybook environment variable exports.
• Added groupBy functionality to <TimeSeries> and <FilterProvider> components for dynamic data grouping.
• Standardized border radius styling across all components.
• Updated <Leaderboard> component to properly sync with time range from <FilterProvider>.
• Corrected type definition for legendPosition prop in <PieChart> component.
• Added customizable accentColors prop to <Leaderboard> component for visual styling.

​

Improvements

​

Ingestion

• Improved Webhook Data Pool performance: Increased request per second (RPS) and concurrent request capacity 4x, significantly reducing 429 responses.
• The Webhook Data Pool now accepts payloads up to 1 MiB.

​

Console

• Enhanced account security: Users can now set up Multi-Factor Authentication (MFA) for their accounts.
• Improved user experience: The last selected Propeller for the Data Pool’s Preview Data will be remembered and displayed for future sessions.
• Updated console for deprecated API Scope: Applications without prior access to DATA_POOL_READ, DATA_POOL_STATS, METRIC_READ, and METRIC_STATS scopes will no longer be able to set these scopes as they are now deprecated

​

API

• Deprecated DATA_POOL_READ, DATA_POOL_STATS, METRIC_READ, and METRIC_STATS API scopes.
• Expanded DATA_POOL_QUERY, METRIC_QUERY API scopes to allow seeing the schema for the Data Pools and columns they have access to.
• Improved the resiliency of copy jobs for large tables, reducing errors and preventing jobs from getting stuck. Enhanced error messages to provide more informative feedback when jobs fail.
• Propellers no longer have a limit on maximum bytes to read making them more powerful.
• Materialize View creation now tests the SQL query prior to creating the Materialized View. This protects developers from creating Materialized Views with incorrect SQL. I handles the case when queries take a long time for large datasets gracefully.

​

SQL

• Enhanced JOIN clause functionality:
  • Added support for multiple column expressions using AND and OR operators
  • Expanded operator support beyond = to include >, <, LIKE, and IN among others.
• Increased query result size limit from 500KB to 10MB, allowing for larger data retrieval
• New ClickHouse functions for enhanced string manipulation and searching:
  • Position and Search functions like:- position, locate, positionCaseInsensitive, positionUTF8, positionCaseInsensitiveUTF8- multiSearch functions for various use cases (e.g., AllPositions, FirstPosition, FirstIndex)
  • Pattern Matching:- match, REGEXP, multiMatchAny, multiMatchAnyIndex, multiMatchAllIndices- Fuzzy matching: multiFuzzyMatchAny, multiFuzzyMatchAnyIndex, multiFuzzyMatchAllIndices
  • Extraction and Comparison:- extract, extractAll, extractAllGroupsHorizontal, extractAllGroupsVertical- like, notLike, ilike, notILike functions
  • N-gram and Substring Operations:- ngramDistance, ngramSearch (with case-sensitive and UTF8 variants)- countSubstrings, countMatches (with case-insensitive options)
  • Specialized String Functions:- regexpExtract, hasSubsequence, hasToken (with various options for case sensitivity and UTF8)

​

Embeddable UI (0.11.1)

• New layout components: <Container>, <Flex>, <Grid> , and <Card>.
• New typography components: <Text> and <Heading> .
• New <Tabs> component for tab-based layouts with support for Card components as tabs.
• Added importable theme colors: Users can now import specific colors (e.g., gray) from "@propeldata/ui-kit/colors" for consistent styling across the application.
• <TimeRangePicker> now integrates seamlessly with <FilterProvider> for improved data filtering across components.
• Added new <TimeGrainPicker> component that also integrated seamlessly with the <FilterProvider> .
• Improved React compatibility: All components are now compatible with React Server Components (RSC) and exported as client-side components, enhancing performance and flexibility.
• Enhanced debugging capabilities: Components now log prop mismatch errors, facilitating easier troubleshooting and development.
• Added groupBy functionality to the <TimeSeries> component, enabling data grouping and more flexible visualizations.
• Enhanced the <SimpleFilter> component with clearable functionality. Developers can use the disableClearable prop to turn off this feature if needed.
• Improved color customization: Replaced accentColor with accentColors to provide more versatile theming options.

​

Terraform (v1.3.4)

• Enhanced flexibility in Data Pool configuration: Added support for explicitly setting empty values for partition_by, order_by, and primary_key fields. Users can now use the syntax [""] to define these fields as empty when needed. This improvement allows for more precise control over Data Pool settings.

​

Fixes

​

Console

• Resolved issues with Google sign-in for accepting invitations.
• Fixed a bug that prevented sign-ups for users who had previously registered but not verified their email.
• Fixed a bug that caused the SQL Console to attempt to send a Propeller for Applications resulting in an error.

​

API

• The API now returns a NOT_FOUND error when the requested resource doesn’t exist. This applies to Materialized Views, Copy Jobs, and Data Pool access policies, improving error handling and user experience.

​

Embeddable UI

• Fixed inconsistent border radius across components for improved visual coherence.
• Aligned <SimpleFilter> component styles with select-based components like the <TimeRangePicker> for a more uniform user interface.

​

Terraform (v1.3.4)

• Fixed an issue where order_by, partition_by, and primary_key columns were being applied in an incorrect order when defining Table Settings during Data Pool creation. We replaced Set with List to ensure order is preserved in fields where it’s critical.

​

Improvements

​

Console

• Developers can now sign up and sign in using their Google account.
• Developers can now sign up and sign in using their GitHub account.

​

SQL

• Enhanced support for ClickHouse and PostgreSQL array functions.
• Introduced new ClickHouse SQL functions:
  • arrayJoin: Allows for the expansion of arrays into separate rows.
  • JSONExtractArrayRaw: Extracts an array from a JSON string.
  • JSONExtractKeys: Retrieves keys from a JSON object.
  • JSONArrayLength: Determines the length of a JSON array.
• Added ClickHouse tuple manipulation functions:
  • tupleNames: Returns the names of tuple elements.
  • tupleElement: Extracts a specific element from a tuple.
• Added ClickHouse geospatial functions for advanced geographical data processing.

​

New Features

​

Up to 47% price reduction for your queries

​

User management in Console

​

Fixes

​

Console

• SQL Playground will not show an error when the query is missing a FROM statement.
• We fixed an issue where the Snowflake Data Pool creation flow was incorrectly sending ver param for MergeTree tables, causing creation to fail.
• Customers will not see cached tables after running an introspection on any data source.

​

Ingestion

• We fixed an issue with new Snowflake Data Pools on ReplacingMergeTree that were not supporting Re-sync functionality, the FINAL clause was not being correctly added to the underlying ClickHouse queries, and _propel_is_deleted filtering wasn’t working in some cases.

​

API

• We fixed an issue when creating and modifying Applications that was not allowing nullable values for the unique name and description fields. Now unique name and description fields are optional as defined in the GraphQL schema.

​

Improvements

​

Console

• The SQL Console and Materialized View creation flow now supports formatting the current query.
• When creating a Materialized View with an existing Data Pool, developers will only see Data Pools with a compatible schema.
• Customers will be able to enable/disable access control on Snowflake Data Pool creation.
• You can now copy Metric name and ID from the Metric list page.

​

Terraform

• Developers can now manage (create, delete, update) Applications via Terraform.

​

Materialized Views

​

Rockset Migration Service

​

Customizable Table Engine and Sorting Key for all Data Pools

​

Fixes

​

Console

• The “Operations” tab for all Data Pool types.
• All Data Pool types can now have a customizable Propeller in the “Preview Data” section.

​

Improvements

​

Terraform

• The resource for Kafka Data Pools is now available.
• The resource for ClickHouse Data Pools is now available.
• The resource for Materialized View is now available.
• The Data Pool resource now supports table settings.
• The Unique ID field is now deprecated in the Data Pool resource.
• The Tenant ID field is now deprecated in the Data Pool resource.
• The timestamp field is now Optional in the Data Pool terraform.

​

API

• The API ADMIN scope is sufficient for Applications to get and list other Applications, but it does not allow them to fetch other Application secrets. This simplifies Terraforming of Data Pool Access Policies, which previously failed when the Terraform Application had only ADMIN scope.
• The ClickHouse Data Source API is now public.
• You can now modify the timestamp field in the Data Pool API.
• The uniqueId and tenantId fields are now deprecated in the Webhook Connection Settings object

​

New Features

​

Expanded SQL function support

​

New Console Navigation

​

GraphQL Schema Explorer

​

Fixes

​

Ingestion

• We resolved an issue with the S3 Data Pool that caused a persistent “CONNECTING” state when customers used an “s3://” or “https://” prefix in the URL for their table path, rather than a relative path within the bucket. Now, if the provided “s3://” or “https://” URL points to the named bucket, the URL’s path is used as the relative path into the bucket. Invalid table paths provided by the customer will now be synchronously rejected.

​

Console

• We fixed the SQL Console to respect the Propeller selection.
• We fixed the GraphQL Playground to respect the Propeller selection.
• When creating a Data Pool, we fixed the case when a timestamp is not selected.
• In the API Playground, we fixed leaderboard dimensions clearing for ClickHouse Data Pools.
• In the API Playground, we fixed the leaderboard table view width and height.
• In the API Playground, we fixed the time dimension not pre-populating.
• In the SQL Console, we fixed result cell wrapping.
• Data Pools created by Materialized Views now see the “Operations” tab.

​

Improvements

​

SQL

• The SQL interface now properly handles identifier quoting.
• We now support CASE statements in SQL.

​

API

• The query APIs now support SQL-style filters. Everywhere you could provide filters, you can now provide filterSql. For example, see the filterSql parameter in the counter API.
• Data Pool Access Policies can now be defined with the new filterSql parameter that supports SQL-style filters.

​

Console

• The API Playground will now populate filter values using the Top Values API.
• The API Playground now includes an “All time” option for the relative time range.
• The API Playground has a new layout for TimeSeries, Counter, and Leaderboard APIs.
• The API Playground now supports setting a “Time Dimension” that is different from the Data Pool’s default timestamp.
• The API Playground will now show the Data Pool selection first.
• The API Playground will now only show the “Existing Metrics” for the selected Data Pool.
• The SQL Console has a new improved layout.
• The SQL Console has new improved SQL syntax validation.
• The SQL Console will now show long result values in a tooltip when they are trimmed.
• The Data Pool list page will not break if the sync activity query fails.
• When creating and updating Amazon S3 credentials, the path and bucket will be trimmed to prevent white spaces from being accidentally entered.

​

New features

​

ClickHouse Data Pool beta

​

GraphQL Playground

• GraphQL schema autocompletion
• Code examples
• Access token generator

​

Fixes

• Fixed AND/OR logic bug in SQL.
• Fixed typo in default Webhook Data pool name.
• Free trial plan no longer expires incorrectly.
• Fixed case-insensitivity in Materialized Views.
• Fix automatic syncing of HTTP Data Sources which contain multiple HTTP Data Pools.
• Fixed the column _propel_synced_at that was incorrectly set for some Webhook Data Pools, resulting in out-of-range values.
• Fixed columns length validation for Kafka on Data Pool creation.
• Fixed a race condition while re-assigning Access Policies.

​

Improvements

• You can now create Data Pools with ClickHouse types via the API.
• Implemented improvements to our access token API that reduced/eliminated the HTTP 500 errors that some customers experienced.
• Added support for AggregatingMergeTree table engine for Data Pools via the API.
• The Webhook Data Pool now returns HTTP 413 Content Too Large error if the payload exceeds 1,048,320 bytes or has more than 500 events.
• Add support for read-only filterSql with a SQL representation of the filters.
• Raised the GraphQL aliases limit to 250.
• Improved the case insensitivity matching for identifiers in SQL.
• Added support lists of strings and lists of numbers in Postgres SQL.
• Added support NOT IN and AT TIME ZONE operators in SQL.
• Added support column auto aliasing in SQL.
• Added support for unary expressions in SQL.
• Added support for CURRENT_DATE in SQL.

​

⚡️ 10x faster ingestion for the Webhook Data Pool

​

🛝 SQL Playground

• Code examples: easily integrate queries into your code. Get code examples for querying data using SQL via the API, cURL, and JavaScript.
• “Query as” selector: simulate querying as your app. Different apps have unique data query permissions set through Access Policies. The Playground allows you to test queries as a specific app.
• “Propeller” selector: experiment with query speeds. Test different query speeds to optimize performance and cost for your data application.

​

✈️ Airbyte destination

​

Fixes

• Customers will be able to change the URL in their ClickHouse credentials on creation flow in the Console.
• Customers will not see stale data when modifying credentials for ClickHouse and Kafka in the Console.
• Signed in customers will be redirected from /login to dashboard.
• In the GraphQL Playground, customers will be able to clear the variables input and run queries with no variables.
• Fixed a bug when parsing table names that contain aliases in SQL.
• Fixed a bug where table aliases where being lowercased in SQL.

​

Improvements

• By setting the disable_partial_success=true query parameter, you can ensure that, if any individual event in a batch of events fails validation, the entire request will fail. For example: https://webhooks.us-east-2.propeldata.com/v1/WHK00000000000000000000000000?disable_partial_success=true
• Added SUBSTRING function to SQL.
• Added support for extracting parts from a timestamp in SQL.
• Raised SQL response size limit to 2 MB.
• The PostgreSQL interface now supports extended queries.

​

Kafka Data Pool

​

🆓 New Generous Free Tier

​

䷰ Schema evolution: Add a column to Data Pool operation

​

🚚 Batch delete and update operation

​

🪵 React UI Kit logging controls

​

Fivetran preview

​

Bring your own ClickHouse preview

​

Fixes

• Fix timezone argument on toStartOfWeek, toStartOfMonth, and toStartOfYear SQL functions.
• Fixed login loop for accounts with Okta integration.
• Fixed environments dropdown in Console in the new Data Pool page.
• Fixed the Preview Data section of the Console where customers can now change time range and page size when the query results in a time out or an error.

​

Improvements

• Add support for timestamps without timezones
• Support NOW() and CURRENT_DATE functions in SQL.
• Support INTERVAL in SQL.
• An alternative timestamp can be supplied to TimeRangeInput when querying.
• Customers can view TableSettings (engine, partitionBy, primaryKey, and orderBy) for their Data Pools via the API.
• Allow creating Data Pools (including Webhook Data Pools) without timestamp via the API.
• Allow setting TableSettings (engine, partitionBy, primaryKey, and orderBy) when creating a Data Pool via the API.
• In the Console, customers can have a different environment in multiple tabs without losing the last selected state.
• In the Console, Customers will see the processedRecords instead of newRecords in the Processed Records column for the Syncs table.
• Customers can now change the sort and timestamp column in the Console in the Preview Data tab.

​

🔎 SQL API

​

🐘 PostgreSQL-compatible SQL interface

​

🗄️ Data Grid API

​

📌 Records by ID API

​

🧮 Top Values API

​

💚💙💜 UI Kit themes

​

🔓 UI Kit Access Token Provider

​

⏳ UI Kit Filter component

​

🍰 UI Kit Pie Chart component

​

🪝 UI Kit Query Hooks

​

Fixes

• Fix timezone argument on toStartOfWeek, toStartOfMonth, and toStartOfYear SQL functions.
• Fixed login loop for accounts with Okta integration.

​

Improvements

• Support LIKE and NOT LIKE filter operators in SQL and the GraphQL API.
• Support TO_TIMESTAMP function in SQL.
• Support CAST function in SQL.
• Mark the tenant ID field in the Data Pool as deprecated. No longer needed with the new Access Policies.
• Support WITH statements in the SQL API.
• Support UNION statements in the SQL API.
• New data_pool:read scope to list Data Pools and their schemas.
• We made timeRange optional in GraphQL API.
• The dimensions stats API has now been deprecated and replaced with the Top Values API.