Skip to content

Add ability for user to set create_table and table_partition when saving data to QDB #90

@newskooler

Description

@newskooler

Hi,

I have a big issue when using the QDB python script to save a pd.DataFrame which has the same schema each time, but I don't know what the schema is ahead of time (ie. when I create the table). This is because I use it to save thousands of tables.

I use the below code to save new and old data.
If the table exists - it appends to it. If it does not, it creates it and appends to it.

import pandas as pd
from questdb.ingress import Sender

df = pd.DataFrame({
    'id': pd.Categorical(['toronto1', 'paris3']),
    'temperature': [20.0, 21.0],
    'humidity': [0.5, 0.6],
    'timestamp': pd.to_datetime(['2021-01-01', '2021-01-02'])})

conf = f'http::addr=localhost:9000;'
with Sender.from_conf(conf) as sender:
    sender.dataframe(df, table_name='sensors', at='timestamp')

This causes a few issue:

  1. upon creation it always creates a partition by DAY

Feature request: add a parameter which is by default DAY but can be changed by the user to be MONTH, NONE or other partition frequency

  1. It silently creates a table by default, even if one does not exist.
    Feature request: add a parameter which is by default True (to create a table) but can be also set to False such that if a table does not exist, it will not create one.

This is very large pain point to use this py-questdb-client in a more powerful way. For daya which is daily, monthly or even annual, it creates a large burden.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions