Plugin 'Ready-for-production' Checklist

5:00 min
Expert

Congratulations on completing (the first version of) your Luzmo plugin connector development! Before you launch it in production, please take a moment to ensure your plugin implementation aligns perfectly with our specifications and follows the best practices. To make this process easy for you, we've put together a comprehensive checklist below. Let's dive in and make sure your plugin is primed for success 🚀

Some of the checks below only apply to plugin specification 3.0.0 (more info here), and might require an optional feature to be enabled. These checks are indicated in a similar way as this remark, and list the specific requirement(s).
Hide plugin spec v3.0.0 checks

General checklist

Reset General checkboxes
  • We highly recommend storing any secrets in a secure way, for example as environment variables.
  • We highly recommend shortly caching schema results in the plugin, using (hashed) 'X-Property-<Property>' authentication header value(s) as cache keys. These are often heavy database queries with rarely any changes. See this Academy article for a more detailed explanation.
  • If your plugin does not cache data, you can always make use of our built-in "query results" caching mechanism (see this Academy article), or even further accelerate your plugin dataset(s) using WARP (see this Academy article).
  • If actual source data or query results are cached in the plugin, we highly recommend ensuring the cache scope is defined using the following properties:
    • The (hashed) authentication property value(s) as passed along as 'X-Property-<Property>' header(s)
    • The metadata passed along as user.metadata in the request body
    • (In case of query results caching) The actual query fired to your data source

/authorize endpoint checklist

Reset '/authorize' checkboxes
  • If the plugin does not, each request send to the plugin will be returned with a 404 Not found HTTP response. This will be interpreted as your plugin authenticating the request, and thus allowing the user to see and query the available dataset(s) in Luzmo as exposed via the POST /datasets endpoint.
  • The plugin secret, as exposed in the Plugin configuration in Luzmo, will be sent to each endpoint of the plugin as 'X-Secret' header in the request! You should use this to verify that the request is coming from Luzmo. You can find more information about plugin secrets in this Academy article.
  • This Academy article contains more information about authentication possibilities in a plugin.
  • Based on the authentication properties defined when adding the Plugin to Luzmo. These properties will be passed along as headers, together with the appropriate values as filled in when connecting to your plugin. The same headers will be passed along with each request to the /datasets and /query endpoint of your plugin.
  • Any other HTTP code returned is interpreted as the plugin not authorizing the request, and thus the user is presented a connection error message.

/datasets endpoint checklist

Reset '/datasets' checkboxes
  • The /datasets endpoint is required to be implemented by the plugin, and should return a list of available datasets exposed by the plugin.
  • The plugin secret, as exposed in the Plugin configuration in Luzmo, will be sent to each endpoint of the plugin as 'X-Secret' header in the request! You should use this to verify that the request is coming from Luzmo. You can find more information about plugin secrets in this Academy article.
  • Luzmo greatly simplifies column data types by only using three simple types:
    • "numeric"
    • "datetime"
    • "hierarchy"
    The type will influence which aggregations and filter expressions you can use to query the column (and how that is requested from the /query endpoint). You should always map the source data types to one of these three types in the response of the /datasets endpoint!
  • Within Luzmo, you can optionally set some columns to special data types like "currency", "duration", etc. More information about these special data types in this Academy article.

This check is only applicable to plugins that implement plugin specification 3.0.0.

  • In order to handle timezones correctly, Luzmo needs to know if the source column is stored as a "date" (Luzmo will never apply timezone shifts on "date" columns), or as a "datetime" (Luzmo will only apply timezone shifts for time-level responses). A checkbox below verifies that your plugin's /query endpoint correctly applies timezone shifts when expected.
  • For your date(time) columns, your plugin can specify a "subtype" property in the response set to either "date" or "datetime". If not specified, Luzmo will assume the column is of subtype "datetime", but you can always change this afterwards in the Luzmo dataset itself.
  • Your plugin can omit the "subtype" property for hierarchy and numeric columns, or set it to null.
  • The plugin is expected to return a payload of a list of dataset objects, with 'Content-Type': 'application/json'. You can see an example here.

This check is only applicable to plugins that implement plugin specification 3.0.0.

  • As a performance improvement for schema retrieval when adding datasets, your plugin can optionally return only the table identifier and names in the response (omitting the "columns" property in the response body) when no list of table identifier is specified in the "ids" property of the request body. Optionally, you can also further limit search results by using the "search" property in the request body.
  • Your plugin can also immediately return all columns of all tables in their respective "columns" property, if necessary.

This check is only applicable to plugins that implement plugin specification 3.0.0.

  • As a performance improvement for schema retrieval when adding datasets, we will send a second request to the /datasets endpoint to return the table(s) and column(s) schema of those that were selected (based on the "id" property in the request).
  • Your plugin can also (again) return all columns of all tables, if necessary.

/query endpoint checklist

Reset '/query' checkboxes
  • The /query endpoint is required to be implemented by the plugin, and should return data based on the request properties.
  • The plugin secret, as exposed in the Plugin configuration in Luzmo, will be sent to each endpoint of the plugin as 'X-Secret' header in the request! You should use this to verify that the request is coming from Luzmo. You can find more information about plugin secrets in this Academy article.
  • The id property indicates from which dataset Luzmo expects data to be returned.
  • Successful queries are expected to return a 200 HTTP response with a payload with 'Content-Type': 'application/json': the payload consists of an array of arrays, where each inner array represents a row of data.
  • You should always return data for a specific column with the same (Luzmo) data type as specified in your plugin's /datasets endpoint.
  • You should of course always ensure that the source data types are appropriately respected when querying data, while respecting Luzmo data types when returning data!
  • 'Hierarchy' columns are expected to be returned as string values (e.g. "Text value").
  • 'Datetime' columns are expected to be returned as string values with RFC3339 format (e.g. "2020-12-31T01:23:45.678Z").
  • 'Numeric' columns are expected to be returned as integer or float values (e.g. 1.25647).
  • Empty / null row values are supported by Luzmo; the plugin is expected to return them as null values in the response.
  • It might be necessary to ensure your plugin can correctly handle any request timeouts from the underlying data source, or any rate limits applied by it. It could be interesting to implement an intelligent retry mechanism to avoid an immediate query failure upon such occurrences.
  • You can return error messages by returning a non-200 HTTP response with a similar 'Content-Type': 'application/json' payload like below:
    {
      "type": {
        "code": 500,
        "description": "Internal Server Error"
      },
      "message": "<message_displayed_upon_query_failed>"
    }
  • A pushdown-enabled plugin is expected to only return (aggregated) data for the columns that are specified in the columns property of the request, in the same order. Your plugin should also support querying the same column multiple times.
  • A pushdown-enabled plugin is only expected to return aggregated data for the columns requested when the options.pushdown property in the request is true. Otherwise, Luzmo expects the plugin to return row-level data for the columns requested.
  • A pushdown-enabled plugin can be requested to return data for date(time) columns on one or more of the following levels:
    • "year"
    • "quarter"
    • "month"
    • "week"
    • "day"
    • "hour"
    • "minute"
    • "second"
    • "millisecond"
  • When grouping on date levels "day","week", "month", "quarter" and "year", we expect the plugin to apply the appropriate timezone shift as indicated by the options.timezone_id property (this is a timezone id from the IANA timezone database). We do still expect UTC RFC3339 date string values to be returned.
    When e.g. grouping a value "2023-01-01T00:00:00.000Z" on "day" level (assuming the column is of subtype "datetime"), we expect the following response:
    • For timezone "Etc/UTC", we expect a value "2023-01-01T00:00:00.000Z" to be returned.
    • For timezone "Japan" (i.e. "UTC+9"), we expect a value "2023-01-01T00:00:00.000Z" to be returned.
    • For timezone "US/Hawaii" (i.e. "UTC-10"), we expect a value "2022-12-31T00:00:00.000Z" to be returned (i.e. the previous day, due to timezone shift).
  • For date levels lower than "day" (i.e. "hour", "minute", etc.), Luzmo will apply the appropriate timezone shift after data retrieval (the plugin is expected to query and return UTC RFC3339 date string values).
    When e.g. grouping a value "2023-01-01T00:00:00.000Z" on "hour" level (assuming the column is of subtype "datetime"), we expect the response to always be the same for any timezone: the source UTC value "2023-01-01T00:00:00.000Z".
  • Within Luzmo, you can make use of several aggregation types to aggregate column data in a specific way: this Academy article provides more information on the different aggregation types we support. Note that not all these aggregation types are (currently) pushed down towards a plugin.
  • A pushdown-enabled plugin can be requested to return aggregated data for one or more columns with the following aggregation types:
    • "sum"
    • "count"
    • "min"
    • "max"
  • A pushdown-enabled plugin can be requested to return a "count" of column_id "*": this should count all rows of the (filtered) dataset.

This check is only applicable to plugins that implement plugin specification 3.0.0, and have enabled the property "supports_distinctcount": true.

  • A pushdown-enabled plugin can be requested to return filtered data with filters using one or more of the following expressions:
    • "="
    • "><"
    • ">="
    • ">"
    • "<"
    • "<="
    • "is null"
    • "is not null"
    • "in"
    • "not in"
    You can find a list of filters to support for each Luzmo data type in this Developer docs guide.
  • A filter value will always be included in an array (except for "is null" and "is not null", which do not provide a value). When querying data from your source, you should ensure that filters are applied according to the specified filter expression and the source data type (e.g. use boolean filter values on boolean columns, instead of numeric/text values).
  • SQL is a bit different in handling null values than Luzmo, as comparisons to null are tri-state. E.g.:
    • A filter expression "in" with values ['a', null] should result in a query like '<column_name>' IN ('a') OR '<column_name>' IS NULL. This ensures that possible null values are appropriately returned by the query.
    • A filter expression "not in" with values ['a'] should result in a query like '<column_name>' NOT IN ('a') OR '<column_name>' IS NULL. This ensures that possible null values are appropriately returned by the query.
    • A filter expression "not in" with values ['a', null] should result in a query like '<column_name>' NOT IN ('a') AND '<column_name>' IS NOT NULL. This ensures that possible null values are excluded by the query.

This check is only applicable to plugins that implement plugin specification 3.0.0, and have enabled the property "supports_like": true.

  • If enabled, the "ilike" and "not ilike" filter expressions in the /query request will expect your plugin to filter the referenced column(s) to only values that do (not) contain a specific text value.
  • The filter is expected to be applied as a case-insensitive (i.e. filtering to "ABC" or "abc" should return the same result)
  • The filter uses an UTF-8 collation
  • "%" is used as a wildcard character at the beginning and/or end of the text value to filter to
  • Luzmo filter expressions like "contains", "starts with", "ends with", etc. will result in a "ilike" or "not ilike" Plugin filter expression.

This check is only applicable to plugins that implement plugin specification 3.0.0, and have enabled the property "supports_join": true.

  • If enabled, the "join" property in the /query request will indicate which relations exist in Luzmo between the referenced datasets in the query. Next to that, each column reference will contain both a dataset_id (canonical dataset identifier) and a column_id (canonical column identifier) to ensure uniquely identifiable columns.
  • Joins can be either of type "lookup" or type "exists":
    • "lookup" joins retrieve exactly zero or one row from the lookup dataset, based on the join criteria. The "criteria" property for these join types will list the "many"->"one" links that have been defined in Luzmo between two datasets (i.e. the first dataset listed in each criterion is the "many" dataset, the second dataset is the "one" dataset).
    • "exists" joins retrieve only rows from the lookup dataset, where the "many" dataset has at least one matching row, based on the join criteria. The "criteria" property for these join types will list the "one"<-"many" links that have been defined in Luzmo between two datasets (i.e. the first dataset listed in each criterion is the "one" dataset, the second dataset is the "many" dataset).
  • Joins can be pushed down on aggregations, group-bys, filters, etc. Similarly, optional "plugin specification 3.0.0" features like "Distinct count" and "Order and Limit" could also reference linked datasets' columns.
  • Join chains (e.g. "Many" -> "One" <- "Many" -> "One") should be supported by the plugin.

This check is only applicable to plugins that implement plugin specification 3.0.0, and have enabled the property "supports_order_limit": true.

  • If enabled, the "order" list property in the /query request will indicate on which column(s) the result should be sorted, as well as in which order and direction.
  • Secondly, the "limit" property in the request will indicate which offset should be applied on the resultset (in the "limit.offset" property), as well as how much rows should be returned from there (in the "limit.by" property).

OAuth2 Authentication checklist

Reset OAuth2 checkboxes

These checks are only useful for plugins that use OAuth2 authentication.

  • We expect a payload of 'Content-Type': 'application/json' and the property "auth_url". This URL will be used by Luzmo to redirect the user to start the OAuth2 connect, and is expected to be of a similar format as the following example:
    https://www.facebook.com/v3.0/dialog/oauth?client_id=<your_OAuth2_application_id>&redirect_uri=https%3A%2F%2Fapp.luzmo.com%2Fstart%2Fconnect%2F<your_Luzmo_plugin_slug>
  • We will call the /exchange endpoint after the user has authorized your OAuth2 application (you should validate the 'X-Secret' header to ensure the request is coming from us, see this Academy article). In the JSON body of the request, we will include a code property from the authorization, which you can now exchange in your plugin for an OAuth2 identifier, access token, and refresh token. The access and refresh token are expected to be returned as a JSON response in the access_token and refresh_token properties:
    {
      "access_token": "<received_oauth2_access_token>",
      "refresh_token": "<received_oauth2_refresh_token>"
    }
  • The OAuth2 Refresh token will be passed along with each request as 'X-token' header.

Luzmo setup checklist

Reset Luzmo setup checkboxes
  • The Base URL should be a publicly reachable HTTPS URL, which does not end with a "/". Luzmo will send requests to the appropriate endpoints using the base URL.
  • Personal plugins can only be seen in the "Add datasets" modal by your Luzmo user, while Organization plugins can be seen by any user within your Organization. You can always share datasets added from a personal plugin with any user within your Organization.
  • You could also request a review by Luzmo, in case you'd like to expose your plugin connector to all Luzmo users.
  • This property will control whether or not aggregations are pushed down towards the data source. For a Basic plugin, this property should be disabled!
  • You can choose between either "No properties" (i.e. no authentication), "OAuth2", or "Custom properties". Each of these custom properties will be shown as input fields to a user who can connect through your plugin. Your plugin's endpoints will receive these properties as 'X-Property-<Property>' headers in each request.
  • We highly recommend using the latest plugin protocol to ensure you can benefit from any improvements made.
  • These elements will be shown in the "Add datasets" modal with the plugin connector.


That's it, you've successfully managed to create a production-ready plugin! Time to enjoy the result by creating some dashboards on your first dataset(s) 🎉

pluginbasic pluginpushdown-enabledcustom connectorpushdown plugin

Need more information?

Do you still have questions? Let us know how we can help.
Send us feedback!

Related Articles

2.1 How to add a new dataset
1:41
2.8 Deleting datasets in Luzmo
0:35
Adding and sharing your first dataset(s)
1:00
Change the synchronization frequency for webservices datasets
0:40
Connect a database by whitelisting our IP addresses
0:15
Connecting and sharing your data source(s)
1:00
Edit the credentials of your database
3:00
How to connect my data?
1:00
Local file upload
1:00
Making a BigQuery Connection
2:00
Making a Clickhouse Connection
2:00
Making a Connection to an ElasticSearch Cluster
2:00
Making a Connection to a SAP HANA Database
2:00
Making a Connection to Oracle
1:30
Making a Databricks Connection
Making a Google Analytics Connection
1:10
Making a Google Drive Connection
2:00
Making a MariaDB Connection
2:00
Making a MongoDB Connection
2:00
Making a MySQL Connection
2:00
Making an AWS Athena Connection
2:00
Making an Azure Synapse connection
1:30
Making a Panoply Connection
2:00
Making a PostgreSQL Connection
2:00
Making a Redshift Connection
2:00
Making a Snowflake Connection
2:00
Making a SQL Server Connection
2:00
Making a Trino or Presto connection
3:00
Plugin 'Ready-for-production' Checklist
5:00
SQL datasets
2:40
Writing your own SQL queries
2:00