Skip to content

Instantly share code, notes, and snippets.

@jeremyyeo
Last active January 27, 2023 01:36
Show Gist options
  • Save jeremyyeo/4c5fb6db98a0cc3c876e718f80bb3420 to your computer and use it in GitHub Desktop.
Save jeremyyeo/4c5fb6db98a0cc3c876e718f80bb3420 to your computer and use it in GitHub Desktop.

Revisions

  1. jeremyyeo revised this gist Jan 27, 2023. 1 changed file with 17 additions and 3 deletions.
    20 changes: 17 additions & 3 deletions README.md
    Original file line number Diff line number Diff line change
    @@ -65,22 +65,36 @@ create or replace transient table development.dbt_jyeo.unpack as

    The magic is in:

    ```
    ```jinja
    {%- set column_name = var(key | int, key) %}
    ```

    Where we make `key` into an integer - because if not, it is a string and if you look at our vars, there is no variable named `"1"` (the string) but rather there is a variable named `1` (the integer). As you can tell - it's not a super good idea to have integers as variable names. Basically by doing:

    ```
    ```yaml
    vars:
    1: foo
    2: bar
    ```
    We've constructed a dictionary:
    We've constructed a `vars` dictionary:

    ```Python
    vars = {1: "foo", 2: "bar"} # NOT THE SAME AS vars = {"1": "foo", "2": "bar"}
    vars[1] # 'foo'
    vars["1"] # KeyError: '1'
    ```

    Knowing this, we could stringify our vars declaration:

    ```yaml
    vars:
    "1": foo
    "2": bar
    ```

    And then do the following instead:

    ```jinja
    {%- set column_name = var(key, key) %}
    ```
  2. jeremyyeo revised this gist Jan 27, 2023. 1 changed file with 12 additions and 6 deletions.
    18 changes: 12 additions & 6 deletions README.md
    Original file line number Diff line number Diff line change
    @@ -69,12 +69,18 @@ The magic is in:
    {%- set column_name = var(key | int, key) %}
    ```

    Where we make `key` into an integer - because if not, it is a string and if you look at our vars, there is no variable named `"1"` (the string) but rather there is a variable named `1` (the integer). As you can tell - it's not a super good idea to have integers as variable names... just like how you would not write Python like so:
    Where we make `key` into an integer - because if not, it is a string and if you look at our vars, there is no variable named `"1"` (the string) but rather there is a variable named `1` (the integer). As you can tell - it's not a super good idea to have integers as variable names. Basically by doing:

    ```Python
    1 = 'foo'
    2 = 'bar'
    1 + 2
    ```
    vars:
    1: foo
    2: bar
    ```

    We've constructed a dictionary:

    ^ Pretty clear this is not advisable :)
    ```Python
    vars = {1: "foo", 2: "bar"} # NOT THE SAME AS vars = {"1": "foo", "2": "bar"}
    vars[1] # 'foo'
    vars["1"] # KeyError: '1'
    ```
  3. jeremyyeo revised this gist Jan 27, 2023. 1 changed file with 16 additions and 0 deletions.
    16 changes: 16 additions & 0 deletions README.md
    Original file line number Diff line number Diff line change
    @@ -62,3 +62,19 @@ create or replace transient table development.dbt_jyeo.unpack as
    );
    01:25:55 SQL status: SUCCESS 1 in 1.32 seconds
    ```

    The magic is in:

    ```
    {%- set column_name = var(key | int, key) %}
    ```

    Where we make `key` into an integer - because if not, it is a string and if you look at our vars, there is no variable named `"1"` (the string) but rather there is a variable named `1` (the integer). As you can tell - it's not a super good idea to have integers as variable names... just like how you would not write Python like so:

    ```Python
    1 = 'foo'
    2 = 'bar'
    1 + 2
    ```

    ^ Pretty clear this is not advisable :)
  4. jeremyyeo created this gist Jan 27, 2023.
    64 changes: 64 additions & 0 deletions README.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,64 @@
    ## Extracting Snowflake variant keys and mapping them to dbt vars so we can use them as column names

    1. Create a table with variant/json data:

    ```sql
    create or replace table development.dbt_jyeo.tbl_with_json as (
    select object_construct('my_key',object_construct('1', 'foo', '2', 'bar', '3', 'baz')) as variant_col
    );
    ```

    ![image](https://user-images.githubusercontent.com/8496748/214987282-ab7011b7-9b53-4534-a0d2-eee7d2e669b7.png)

    We will use this table to create a dbt model.

    2. Add vars to dbt_project.yml:

    ```yaml
    # dbt_project.yml
    vars:
    1: foo
    2: bar
    ```
    > Note that it's probably not the best idea to have integers as variable names here.
    3. Add our dbt model with all the necessary components:
    ```sql
    -- models/unpack.sql
    {%- set query_keys -%}
    select k.key
    from development.dbt_jyeo.tbl_with_json t,
    lateral flatten(input => t.variant_col:my_key) k;
    {%- endset -%}

    {%- set results = run_query(query_keys) -%}

    {%- if execute -%}
    {%- set res = results.columns[0].values() -%}
    {%- else -%}
    {%- set res = [] -%}
    {%- endif -%}

    select
    {%- for key in res %}
    {%- set column_name = var(key | int, key) %}
    variant_col['my_key']['{{ key }}'] as col_{{ column_name }} {%- if not loop.last -%},{%- endif -%}
    {%- endfor %}
    from development.dbt_jyeo.tbl_with_json
    ```

    4. Run the dbt model and inspect the logs (or compile the model):

    ```
    01:25:54 On model.my_dbt_project.unpack: /* {"app": "dbt", "dbt_version": "1.3.2", "profile_name": "user", "target_name": "dev", "node_id": "model.my_dbt_project.unpack"} */
    create or replace transient table development.dbt_jyeo.unpack as
    (select
    variant_col['my_key']['1'] as col_foo,
    variant_col['my_key']['2'] as col_bar,
    variant_col['my_key']['3'] as col_3
    from development.dbt_jyeo.tbl_with_json
    );
    01:25:55 SQL status: SUCCESS 1 in 1.32 seconds
    ```