Experiment with replacing Symbol with Bytes in function names, enum variants, struct field names

Using the following three examples as a way to test an unit enum variant:
#![no_std]
use soroban_sdk::{contractimpl, Env, RawVal};

pub struct Contract;

#[contractimpl]
impl Contract {
    pub fn f(_env: Env) -> RawVal {
        // 186 bytes
        RawVal::VOID
    }
}

#![no_std]
use soroban_sdk::{contractimpl, Env, RawVal};

pub struct Contract;

#[contractimpl]
impl Contract {
    pub fn f(env: Env) -> RawVal {
        // 234 bytes
        use soroban_sdk::{symbol, IntoVal};
        let v = (symbol!("soroban00"),);
        v.into_val(&env)
    }
}

#![no_std]
use soroban_sdk::{contractimpl, Env, RawVal};

pub struct Contract;

#[contractimpl]
impl Contract {
    pub fn f(env: Env) -> RawVal {
        // 278 bytes
        use soroban_sdk::{Bytes, IntoVal};
        let v = (Bytes::from_slice(&env, b"soroban00"),);
        v.into_val(&env)
    }
}

This doesn't seem particularly great, however I did observe that this code size growth wasn't so bad when adding multiple uses of the same field name.
But we're likely to have a better alternative to Symbol in the new constant-byte type that @graydon is exploring, so I'll wait any further investigation till that shows up. Some conversation about that is here: https://discord.com/channels/897514728459468821/1042914710019903498/1043251976969920594


Drawing on the web is very slow

It looks like drawing with wasm is way slower than it needs to be. I'm using tiny-skia and it takes 14ms just to present a 1000x1000 picture:

doing this I could draw way faster:
#[cfg(target_arch = "wasm32")]
fn draw_buffer_web(win: &Window, pixmap: &Pixmap) {
    use wasm_bindgen::prelude::*;
    web_sys::console::log_1(&"draw_buffer_web".into());
    let canvas = get_a_canvas(win);
    let ctx: web_sys::CanvasRenderingContext2d = canvas
        .get_context("2d")
        .expect("Failed to get 2d context")
        .expect("Failed to get 2d context")
        .dyn_into()
        .expect("Failed to convert to CanvasRenderingContext2d");

    let width = pixmap.width();
    let clamped = wasm_bindgen::Clamped(pixmap.data());
    let image = web_sys::ImageData::new_with_u8_clamped_array(clamped, width)
        .expect("Failed to create image data");
    ctx.put_image_data(&image, 0.0, 0.0)
        .expect("Failed to put image data");

    fn get_a_canvas(win: &Window) -> web_sys::HtmlCanvasElement {
        use winit::platform::web::WindowExtWebSys;
        win.canvas().expect("Failed to get canvas")
    }
}

It looks like the current present_with_damage  function iterates throught the whole buffer and makes a copy each time. Wouldn't it be possible to just take in some buffer and present it with no copies or individual byte manipulation?

I assume this is a release build?
One problem with the web backend is that softbuffer uses BGRX color, which is converted to RGBA in the web backend. Softbuffer needs a format API to be able to use a different pixel format without conversion (https://github.com/rust-windowing/softbuffer/issues/98). Though any application that only supports rendering to one format will need conversion on some platforms, if there's not one format that's supported everywhere.
There may be other ways to improve the web backend, but its the backend I'm personally least familiar with.

Yes, it is a release build. Hum. I had not noticed that it was using BGRX, I thought it was just using a u32 instead of 4 bytes of RGBA. It is wierd that its taking so long, even to iterate 1M values should not be that bad. I will see if I can find a way to improve it a bit.

Just took off all the skips and takes on the iterators and it ran in 1.5ms the present function.

If I remember correctly, the reason why we do this in the first place is because we want to avoid converting pixels we won't draw. If conversion wouldn't be necessary, we can remove the whole thing here not even needing union_damage() anymore.
So I would rather wait for #98 before doing any major refactoring here, but I'm happy to look at a PR improving the status quo.


Gradle Javadoc documentation missing methods

The Javadoc for StandardJavadocDocletOptions contained a setTagletPath(java.util.List) in 3.4:
https://docs.gradle.org/3.4/javadoc/org/gradle/external/javadoc/StandardJavadocDocletOptions.html#setTagletPath(java.util.List)
But in 3.5, it doesn't:
https://docs.gradle.org/3.5/javadoc/org/gradle/external/javadoc/StandardJavadocDocletOptions.html
This method wasn't removed, but it has never had any Javadoc:
https://github.com/gradle/gradle/blob/master/subprojects/language-java/src/main/java/org/gradle/external/javadoc/StandardJavadocDocletOptions.java#L941
I'm not sure what changed between 3.4 and 3.5.  Maybe the minor version of Java 7 we usually build with? /cc @wolfs @donat
Since the Javadoc is the source of truth for our public APIs, this is a serious problem.  @eriwen We should fix this before 4.1 final and regenerate the docs for 3.5, 4.0 and 4.0.1.

I just noticed that it's worse than that.  The inherited methods are missing and the methods that do exist, don't have a "full" entry like you see in 3.4.

It's not a general problem with our Javadoc. So it is not a serious problem as feared.
It is only with the javadoc of StandardJavadocDocletOptions. Everything is actually there in the page sources of 3.4.
The problem is that https://github.com/gradle/gradle/commit/4a091a4d727c87974724118d745de99a59e31bb7 moved Javadoc from private fields to public methods in StandardJavadocDocletOptions.java. So it actually added more Javadoc. Unfortunately, one of these doc blogs contains an unescaped <title>. This breaks the rendering of the page.

Can't we fail the build on invalid Javadoc?

Seems like we could build our Javadoc with Java 8 in some test build: http://blog.joda.org/2014/02/turning-off-doclint-in-jdk-8-javadoc.html

Yes. We just need to turn it on before... 🙈
https://github.com/gradle/gradle/blob/3c89e4ce7312ece6670e83b1979fcbe43847c9d3/subprojects/docs/docs.gradle#L429-L429

Awesome, much relieved.

The Javadoc fix/improvement for StandardJavadocDocletOptions  and CoreJavadocOptions are now on release.
As for activating doclint(Java 8): I activated it now (not yet on CI) for the categories syntax and reference. I also would like to activate it for html, but that fails on our autoTested attribute in
 blocks (error: unknown attribute: autoTested).
I fixed all warnings and errors by running locally, including the html warnings except for the autoTested thing. It actually found a few things that were formatted wrong. The fixes are on release as well.

Thanks @jjohannes -- could we change
 into something supported like
?
If doclint supported HTML5, we could use data-*, but we have to use HTML4.

@big-guy Yes we could. Good idea. We just detect it like this:
file.text.eachMatch(/(?ms).*?
(.*?)<\/pre>(.*?)/)

Done: https://github.com/gradle/gradle/commit/907a4e78989a1d7115020535ea0c0fc313d65b20

Java 8 javadoc is now part of Sanity Check (https://github.com/gradle/gradle/commit/fd3486cde0040289e81903d0ec4971b82e64f0d4) executing the syntax, html and reference checks.

Awesome, thanks @jjohannes


Merge data freezes indefenetely

Environment
[tool.poetry]
name = "code"
version = "0.1.0"
description = ""
authors = ["Edgar Bahilo<EMAIL_ADDRESS>readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"
black = "^23.11.0"
ruff = "^0.1.6"
pytest = "^7.4.3"
pre-commit = "^3.5.0"
deltalake = "^0.13.0"
polars = "^0.19.17"
azure-identity = "^1.15.0"
azure-storage-blob = "^12.19.0"
pandas = "^2.1.3"
ipdb = "^0.13.13"
fsspec = "^2023.10.0"
adlfs = "^2023.10.0"
azure-keyvault-secrets = "^4.7.0"
pyarrow = "^14.0.1"
python-dotenv = "^1.0.0"
azure-storage-file-datalake = "^12.14.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[tool.black]
skip-string-normalization = true
line-length = 120

[tool.ruff]
# Same as Black.
line-length = 120

exclude = [
    "jupyter_notebook_config.py",
    ".bzr",
    ".direnv",
    ".eggs",
    ".git",
    ".git-rewrite",
    ".hg",
    ".mypy_cache",
    ".nox",
    ".pants.d",
    ".pytype",
    ".ruff_cache",
    ".svn",
    ".tox",
    ".venv",
    "__pypackages__",
    "_build",
    "buck-out",
    "build",
    "dist",
    "node_modules",
    "venv"]

select = [
    "E",  # pycodestyle errors (settings from FastAPI, thanks, @tiangolo!)
    "W",  # pycodestyle warnings
    "F",  # pyflakes
    "I",  # isort
    "C",  # flake8-comprehensions
    "B",  # flake8-bugbear
]
ignore = [
    "E501",  # line too long, handled by black
    "C901",  # too complex
]

[tool.ruff.isort]
order-by-type = true
relative-imports-order = "closest-to-furthest"
extra-standard-library = ["typing"]
section-order = ["future", "standard-library", "third-party", "first-party", "local-folder"]
known-first-party = []

Delta-rs version:
0.13.0
Binding:
Python bindings
Environment:

Cloud provider: Azure
OS:
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Bug
What happened:
Hi,
Im trying to ramp up the knoweldge of the team in transactional datalakes. I have used delta before with pyspark with more or less proper success but in my current company we do not have yet the use case for spark (as we are not processing not even TB of data for our projects) and polars/duckDB seems a good compromise. I have created the following demo code which performs the following:

Unzip CSV data from a set of URLs.
Convert the CSV to .parquet and send it to a landing zone.
Read the parquet data from the landing zone.
Convert the parquet to delta and write it in an append only mode.
Make upserts into a silver table.

The code is as follows:
import logging
import os
from datetime import datetime
from io import BytesIO
from zipfile import ZipFile

import polars as pl
import requests
from azure.identity import AzureCliCredential
from azure.storage.filedatalake import DataLakeServiceClient
from deltalake import DeltaTable
from dotenv import load_dotenv

load_dotenv('../.env')
# Set the logging level for all azure-* libraries
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger('azure')
logger.setLevel(logging.ERROR)
logger_normal = logging.getLogger(__name__)

DOWNLOAD_URIS = [
    "https://divvy-tripdata.s3.amazonaws.com/Divvy_Trips_2018_Q4.zip",
    "https://divvy-tripdata.s3.amazonaws.com/Divvy_Trips_2019_Q1.zip",
    "https://divvy-tripdata.s3.amazonaws.com/Divvy_Trips_2019_Q2.zip",
    "https://divvy-tripdata.s3.amazonaws.com/Divvy_Trips_2019_Q3.zip",
    "https://divvy-tripdata.s3.amazonaws.com/Divvy_Trips_2019_Q4.zip",
    "https://divvy-tripdata.s3.amazonaws.com/Divvy_Trips_2020_Q1.zip",
]

LANDING_ZONE_PATH = os.getenv('LANDING_ZONE_PATH')
ACCOUNT_NAME = os.getenv('STORAGE_ACCOUNT_NAME')
BRONZE_CONTAINER = os.getenv('STAGING_PATH')
SILVER_CONTAINER = os.getenv('HISTORICAL_PATH')
GOLD_CONTAINER = os.getenv('DW_PATH')

class ETLPipeline:
    """
    Mock class that simulates a modulith to process from landing to bronze, silver and gold.
    It takes the assumption that we process one file at a time for simplification (mostly because
    I did not want to write asyncio calls and deal with the connections in the ADLSGenClient)
    """

    def __init__(self) -> None:
        self.adlsgen2_client = DataLakeServiceClient(
            account_url=f"https://{ACCOUNT_NAME}.dfs.core.windows.net/", credential=AzureCliCredential()
        )
        self.bronze_client = self.adlsgen2_client.get_directory_client(file_system=LANDING_ZONE_PATH, directory='/')

    def upload_to_landing(self, uri: str):
        """_summary_

        Args:
            uri (str): _description_
        """

        try:
            response = requests.get(url=uri)
            response.raise_for_status()
            with ZipFile(BytesIO(response.content), 'r') as zip_ref:
                # Assuming there is only one file in the zip archive
                file_name = zip_ref.namelist()[0]
                # Read the CSV file into a polars dataframe
                raw_df = pl.read_csv(zip_ref.open(file_name).read())

                # Save the DataFrame as a parquet file in blob
                file_name = f"{file_name.lower().split('.csv')[0]}.parquet"
                path_name = f"../data/input/{file_name}"
                raw_df.write_parquet(path_name, use_pyarrow=True, compression='lz4')

                file_client = self.bronze_client.get_file_client(file_name)

                with open(file=path_name, mode="rb") as data:
                    file_client.upload_data(data, overwrite=True)
                logger_normal.info(f"{file_name} uploaded to adlsgen2")
                self.file_name = file_name
        except requests.exceptions.HTTPError as e:
            logger_normal.error(f"Failed to download {uri}. HTTP error occurred: {e}")
        except Exception as e:
            logger_normal.error(f"An error occurred processing {uri}: {e}")

    def raw_to_bronze(self):
        try:
            storage_options_raw = {"account_name": ACCOUNT_NAME, "anon": False}
            storage_options_raw_delta = {"account_name": ACCOUNT_NAME, "use_azure_cli": "True"}
            logger_normal.info(f"Reading {self.file_name}")
            df = pl.read_parquet(
                source=f'abfs://{LANDING_ZONE_PATH}/{self.file_name}', storage_options=storage_options_raw
            )
            df = df.with_columns(insertion_date=datetime.now())
            # df = df.with_columns([(pl.col("start_time").str.to_datetime().dt.strftime("%Y-%m").alias("monthdate"))])
            logger_normal.info(f"Converting {self.file_name} to delta")
            df.write_delta(
                target=f'abfs://{BRONZE_CONTAINER}/',
                mode='append',
                storage_options=storage_options_raw_delta
                # delta_write_options={"partition_by": ['monthdate']}
            )
            # bronze_df = DeltaTable(
            #         table_uri=f'abfs://{BRONZE_CONTAINER}/', storage_options=storage_options_raw_delta)
            # bronze_df.optimize.z_order(['trip_id'])
        except Exception as e:
            logger_normal.error(e)
            logger_normal.error(f"Failed to conver to delta {self.file_name}")

    def bronze_to_silver(self):
        try:
            storage_options_raw_delta = {"account_name": ACCOUNT_NAME, "use_azure_cli": "True"}
            # source
            bronze_df = pl.read_delta(source=f'abfs://{BRONZE_CONTAINER}/', storage_options=storage_options_raw_delta)
            # target
            silver_check = self._table_checker(container=SILVER_CONTAINER, options=storage_options_raw_delta)
            if silver_check:
                bronze_df = DeltaTable(
                    table_uri=f'abfs://{BRONZE_CONTAINER}/', storage_options=storage_options_raw_delta
                ).to_pyarrow_table()
                logger_normal.info("Merging new data into silver")
                silver_df = DeltaTable(
                    table_uri=f'abfs://{SILVER_CONTAINER}/', storage_options=storage_options_raw_delta
                )
                (
                    silver_df.merge(
                        source=bronze_df, predicate="s.trip_id = t.trip_id", source_alias="s", target_alias="t"
                    )
                    .when_matched_update_all()
                    .when_not_matched_insert_all()
                    .execute()
                )
                silver_df.optimize.z_order(['trip_id'])
            else:
                logger_normal.info("Because silver table is empty we save the first bronze file as silver")
                bronze_df.write_delta(
                    target=f'abfs://{SILVER_CONTAINER}/', mode='overwrite', storage_options=storage_options_raw_delta
                )

        except Exception as e:
            logger_normal.error(e)
            logger_normal.error(f"Failed to merge {self.file_name}")

    def silver_to_gold(self):
        return True

    def _table_checker(self, container, options):
        try:
            delta_table = DeltaTable(table_uri=f"abfs://{container}/", storage_options=options)
            logger_normal.info(f"Delta table version is {delta_table.version()}")
            table_exist = True
            logger_normal.info(f"Delta Table Exists in {container}")
        except Exception as e:
            logger_normal.error(e)
            table_exist = False
        return table_exist

if __name__ == "__main__":
    etl_workflow = ETLPipeline()
    for uri in DOWNLOAD_URIS:
        etl_workflow.upload_to_landing(uri=uri)
        etl_workflow.raw_to_bronze()
        etl_workflow.bronze_to_silver()

When running the code:

The merge seems not to work.
I am aware that I am not using bloom filters and partitions but I am running this in a pretty beefy machine (112RAM and 16 cores). Is this normal?
When looking to the storage account I am only able to see the first file:

And no additional commits seem to be running.
What you expected to happen:
I would expect either a warning or something that indicates why this is working so slowly.
BR
E

@edgBR are you able to build and compile the latest python release to see if that will make it run?
In the main branch there are some improvements for the merge operation, which you can use in the next python release

Hi @ion-elgreco
Where I can find the instructions for compilation?
Do I need rust installed?
Also would it be possible for you to reproduce locally?

Hi @ion-elgreco
I have cloned and installed the version based on: https://github.com/delta-io/delta-rs/commit/bca00ae0540a452cc465ab0c3d292fef90d8fda0 (last commit)
Unfortunately I am not able to build it properly.
I am getting the following error:
error: could not compile `arrow-data` (lib)

Caused by:
  process didn't exit successfully: `/home/azureuser/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rustc --crate-name arrow_data --edition=2021 /home/azureuser/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-data-48.0.1/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=225 --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 --cfg 'feature="ffi"' -C metadata=3bd98a31750d62d1 -C extra-filename=-3bd98a31750d62d1 --out-dir /mnt/batch/tasks/shared/LS_root/mounts/clusters/poetryebrhpc/code/Users/edgar.bahilo/delta-rs/target/debug/deps -L dependency=/mnt/batch/tasks/shared/LS_root/mounts/clusters/poetryebrhpc/code/Users/edgar.bahilo/delta-rs/target/debug/deps --extern arrow_buffer=/mnt/batch/tasks/shared/LS_root/mounts/clusters/poetryebrhpc/code/Users/edgar.bahilo/delta-rs/target/debug/deps/libarrow_buffer-fa7d385f97e94f64.rmeta --extern arrow_schema=/mnt/batch/tasks/shared/LS_root/mounts/clusters/poetryebrhpc/code/Users/edgar.bahilo/delta-rs/target/debug/deps/libarrow_schema-75a9e732fa8d2b13.rmeta --extern half=/mnt/batch/tasks/shared/LS_root/mounts/clusters/poetryebrhpc/code/Users/edgar.bahilo/delta-rs/target/debug/deps/libhalf-c4c9d4c9e1f59e7e.rmeta --extern num=/mnt/batch/tasks/shared/LS_root/mounts/clusters/poetryebrhpc/code/Users/edgar.bahilo/delta-rs/target/debug/deps/libnum-bde8b76108b30fd4.rmeta --cap-lints allow` (signal: 7, SIGBUS: access to undefined memory)

Is there a way you can try to reproduce?
I will not try to run the same code but using local storage instead of ADLSGen2 to see if the problem is also reproducible there.
BR
E

Hi @ion-elgreco
I have reproduced locally and unfortunately the merge still freezes:
import logging
import os
from datetime import datetime
from io import BytesIO
from zipfile import ZipFile
import pdb

import polars as pl
from polars.io.delta import _convert_pa_schema_to_delta
import requests
from azure.identity import AzureCliCredential
from azure.storage.filedatalake import DataLakeServiceClient
from deltalake import DeltaTable, write_deltalake
from dotenv import load_dotenv

load_dotenv('../.env')
# Set the logging level for all azure-* libraries
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger('azure')
logger.setLevel(logging.ERROR)
logger_normal = logging.getLogger(__name__)

DOWNLOAD_URIS = [
    "https://divvy-tripdata.s3.amazonaws.com/Divvy_Trips_2018_Q4.zip",
    "https://divvy-tripdata.s3.amazonaws.com/Divvy_Trips_2019_Q1.zip",
    "https://divvy-tripdata.s3.amazonaws.com/Divvy_Trips_2019_Q2.zip",
    "https://divvy-tripdata.s3.amazonaws.com/Divvy_Trips_2019_Q3.zip",
    "https://divvy-tripdata.s3.amazonaws.com/Divvy_Trips_2019_Q4.zip",
    "https://divvy-tripdata.s3.amazonaws.com/Divvy_Trips_2020_Q1.zip",
]

LANDING_ZONE_PATH = os.getenv('LANDING_ZONE_PATH')
ACCOUNT_NAME = os.getenv('STORAGE_ACCOUNT_NAME')
BRONZE_CONTAINER = os.getenv('STAGING_PATH')
SILVER_CONTAINER = os.getenv('HISTORICAL_PATH')
GOLD_CONTAINER = os.getenv('DW_PATH')

class ETLPipeline:
    """
    Mock class that simulates a modulith to process from landing to bronze, silver and gold.
    It takes the assumption that we process one file at a time for simplification (mostly because
    I did not want to write asyncio calls and deal with the connections in the ADLSGenClient)
    """

    def __init__(self) -> None:
        self.adlsgen2_client = DataLakeServiceClient(
            account_url=f"https://{ACCOUNT_NAME}.dfs.core.windows.net/", credential=AzureCliCredential()
        )
        self.bronze_client = self.adlsgen2_client.get_directory_client(file_system=LANDING_ZONE_PATH, directory='/')

    def upload_to_landing(self, uri: str):
        """_summary_

        Args:
            uri (str): _description_
        """

        try:
            response = requests.get(url=uri)
            response.raise_for_status()
            with ZipFile(BytesIO(response.content), 'r') as zip_ref:
                # Assuming there is only one file in the zip archive
                file_name = zip_ref.namelist()[0]
                # Read the CSV file into a polars dataframe
                raw_df = pl.read_csv(zip_ref.open(file_name).read())

                # Save the DataFrame as a parquet file in blob
                file_name = f"{file_name.lower().split('.csv')[0]}.parquet"
                path_name = f"../data/input/{file_name}"
                raw_df.write_parquet(path_name, use_pyarrow=True, compression='lz4')

                # file_client = self.bronze_client.get_file_client(file_name)

                # with open(file=path_name, mode="rb") as data:
                #     file_client.upload_data(data, overwrite=True)
                # logger_normal.info(f"{file_name} uploaded to adlsgen2")
                self.file_name = file_name
        except requests.exceptions.HTTPError as e:
            logger_normal.error(f"Failed to download {uri}. HTTP error occurred: {e}")
        except Exception as e:
            logger_normal.error(f"An error occurred processing {uri}: {e}")

    def raw_to_bronze(self):
        try:
            # storage_options_raw = {"account_name": ACCOUNT_NAME, "anon": False}
            # storage_options_raw_delta = {"account_name": ACCOUNT_NAME, "use_azure_cli": "True"}
            logger_normal.info(f"Reading {self.file_name}")
            # df = pl.read_parquet(
            #     source=f'abfss://{LANDING_ZONE_PATH}/{self.file_name}', storage_options=storage_options_raw
            # )
            df = pl.read_parquet(
                source=f'../data/input/{self.file_name}'
            )
            #df = df.with_columns(insertion_date=datetime.now())
            #df = df.with_columns([(pl.col("start_time").str.to_datetime().dt.strftime("%Y-%m").alias("monthdate"))])
            logger_normal.info(f"Converting {self.file_name} to delta")
            # df.write_delta(
            #     target=f'abfss://{BRONZE_CONTAINER}/',
            #     mode='append',
            #     storage_options=storage_options_raw_delta
            #     # delta_write_options={"partition_by": ['monthdate']}
            # )
            df.write_delta(target="/tmp/bronze",
                           mode='append')
            # bronze_df = DeltaTable(
            #         table_uri=f'abfs://{BRONZE_CONTAINER}/', storage_options=storage_options_raw_delta)
            # bronze_df.optimize.z_order(['trip_id'])
            # bronze_df = DeltaTable(
            #         table_uri="tmp/bronze")
            # bronze_df.optimize.z_order(['trip_id'])
        except Exception as e:
            logger_normal.error(e)
            logger_normal.error(f"Failed to conver to delta {self.file_name}")

    def bronze_to_silver(self):
        try:
            #storage_options_raw_delta = {"account_name": ACCOUNT_NAME, "use_azure_cli": "True"}
            # source
            #bronze_df = pl.read_delta(source=f'abfss://{BRONZE_CONTAINER}/', storage_options=storage_options_raw_delta)
            bronze_df = DeltaTable(
                    table_uri="/tmp/bronze"
                ).to_pyarrow_table()
            # target
            #silver_check = self._table_checker(container=SILVER_CONTAINER, options=storage_options_raw_delta)
            silver_check = self._table_checker_local(container=SILVER_CONTAINER)
            if silver_check:
                # bronze_df = DeltaTable(
                #     table_uri=f'abfss://{BRONZE_CONTAINER}/', storage_options=storage_options_raw_delta
                # ).to_pyarrow_table()

                logger_normal.info("Merging new data into silver")
                silver_df = DeltaTable(
                    table_uri="/tmp/silver"
                )
                (
                    silver_df.merge(
                        source=bronze_df, predicate="s.trip_id = t.trip_id", source_alias="s", target_alias="t"
                    )
                    .when_matched_update_all()
                    .when_not_matched_insert_all()
                    .execute()
                )
                logger_normal.info("Optimizing by Z order")
                # silver_df.optimize.z_order(['trip_id'])
            else:
                logger_normal.info("Because silver table is empty we save the first bronze file as silver")
                # bronze_df.write_delta(
                #     target=f'abfss://{SILVER_CONTAINER}/', mode='append', storage_options=storage_options_raw_delta
                # )
                write_deltalake(
                table_or_uri="/tmp/silver",
                data=bronze_df,
                mode='append')
        except Exception as e:
            logger_normal.error(e)
            logger_normal.error(f"Failed to merge {self.file_name}")

    def silver_to_gold(self):
        return True

    def _table_checker(self, container, options):
        try:
            delta_table = DeltaTable(table_uri=f"abfss://{container}/", storage_options=options)
            logger_normal.info(f"Delta table version is {delta_table.version()}")
            table_exist = True
            logger_normal.info(f"Delta Table Exists in {container}")
        except Exception as e:
            logger_normal.error(e)
            table_exist = False
        return table_exist

    def _table_checker_local(self, container):
        try:
            delta_table = DeltaTable(table_uri=f"/tmp/{container}")
            logger_normal.info(f"Delta table version is {delta_table.version()}")
            table_exist = True
            logger_normal.info(f"Delta Table Exists in {container}")
        except Exception as e:
            logger_normal.error(e)
            table_exist = False
        return table_exist

if __name__ == "__main__":
    etl_workflow = ETLPipeline()
    for uri in DOWNLOAD_URIS:
        etl_workflow.upload_to_landing(uri=uri)
        etl_workflow.raw_to_bronze()
        etl_workflow.bronze_to_silver()

@edgBR that's unfortunate, can you make a minimal reproducible example with fake data?

Hi @ion-elgreco
I am now using the examples from here: https://delta.io/blog/2023-10-22-delta-rs-python-v0.12.0/ and running:
from deltalake import DeltaTable, write_deltalake
from datetime import datetime
import polars as pl
from polars.io.delta import _convert_pa_schema_to_delta

def execute():

    df = pl.DataFrame(
        {
            "sales_order_id": ["1000", "1001", "1002", "1003"],
            "product": ["bike", "scooter", "car", "motorcycle"],
            "order_date": [
                datetime(2023, 1, 1),
                datetime(2023, 1, 5),
                datetime(2023, 1, 10),
                datetime(2023, 2, 1),
            ],
            "sales_price": [120.25, 2400, 32000, 9000],
            "paid_by_customer": [True, False, False, True],
        }
    )
    print(df)

    df.write_delta("sales_orders", mode="append")

    new_data = pl.DataFrame(
        {
            "sales_order_id": ["1002", "1004"],
            "product": ["car", "car"],
            "order_date": [datetime(2023, 1, 10), datetime(2023, 2, 5)],
            "sales_price": [30000.0, 40000.0],
            "paid_by_customer": [True, True],
        }
    )

    dt = DeltaTable("sales_orders")
    source = new_data.to_arrow()
    delta_schema = _convert_pa_schema_to_delta(source.schema)
    source = source.cast(delta_schema)

    (
        dt.merge(
            source=source,
            predicate="s.sales_order_id = t.sales_order_id",
            source_alias="s",
            target_alias="t",
        )
        .when_matched_update_all()
        .when_not_matched_insert_all()
        .execute()
    )

if __name__ == "__main__":
    execute()

Error is now:
Traceback (most recent call last):
  File "/anaconda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/anaconda/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/poetryebrhpc/code/Users/edgar.bahilo/delta-lake-examples/code/minimal.py", line 56, in <module>
    execute()
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/poetryebrhpc/code/Users/edgar.bahilo/delta-lake-examples/code/minimal.py", line 25, in execute
    df.write_delta("sales_orders", mode="append")
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/poetryebrhpc/code/Users/edgar.bahilo/delta-lake-examples/.venv/lib/python3.10/site-packages/polars/dataframe/frame.py", line 3655, in write_delta
    write_deltalake(
  File "/mnt/batch/tasks/shared/LS_root/mounts/clusters/poetryebrhpc/code/Users/edgar.bahilo/delta-lake-examples/.venv/lib/python3.10/site-packages/deltalake/writer.py", line 372, in write_deltalake
    _write_new_deltalake(
OSError: Generic DeltaLocalObjectStore error: Operation not supported (os error 95)

@edgBR where are you writing to? Seems like a mounted storage.

Hi,
As I am using AML (Azure Machine Learning) the folders are over a similar thing to an NFS storage.
When appending /tmp/ to the paths I get now another error:
    metrics = self.table._table.merge_execute(
_internal.DeltaError: Generic DeltaTable error: Execution error: Fail to build join indices in NestedLoopJoinExec, error:Arrow error: Invalid argument error: Invalid comparison operation: LargeUtf8 == Utf8

@edgBR with which deltalake version? With version 0.14, I can run this fine.

Yes, there were some build failures with 0.14 so not everything got released to pypi and docs are not updated yet. I am looking into that now.


Victor VosMottor: Watch cowàrd

Victor VosMottor requests the watch of the watch_keyword cowàrd. See the MS search here and the Stack Exchange search in text, in URLs, and in code.

cowàrd has been seen in 0 true positives, 0 false positives, and 0 NAAs.

Rejected by tripleee in Charcoal HQ. Reason: 'no matches, somewhat FP-prone'


Documentation!!!

Hey!!!
Thanks for this awesome project. Can you please provide some more documentation and its features?
Is this project only a proxy server? Or can it also get the access tokens using the email:password? I think not, but can I use your chatgpt-to-api project to achieve that functionality combined with this project?

https://github.com/acheong08/ChatGPT-to-API can work alone without a proxy (given clean IP)

Or can it also get the access tokens using the email:password?

https://github.com/acheong08/OpenAIAuth
There is a repo for everything

chatgpttoapi  | Location: begin
chatgpttoapi  | Status code: 0
chatgpttoapi  | Details:
chatgpttoapi  | Embedded error: Get "https://chat.openai.com/api/auth/csrf": x509: certificate signed by unknown authority
getting this while running from docker. tried installing certificates through dockerfile but didn't work.
RUN apk update &&
apk add --no-cache openssl ca-certificates &&
wget -O /usr/local/share/ca-certificates/RootCA.crt https://letsencrypt.org/certs/isrgrootx1.pem.txt &&
echo -n | openssl s_client -servername chat.openai.com -connect chat.openai.com:443 2>/dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /usr/local/share/ca-certificates/OpenAI.crt &&
update-ca-certificates

idk how to fix in docker. Just run it outside of container


BLAKE3

Dear @rhash,
BLAKE3 seems to be capable to replace the whole zoo of MD5/SHA1/2/3/Blake2 flavors to check files for malicious changes, while xxHash 128 is a relevant CRC32 replacement for catching accidental changes (e.g. network transmission errors and data rot). This consensus is maturing as we speak. (cc: @Cyan4973)
Alas, b3sum, the official CLI implementation of BLAKE3, has a serious drawback related to memory management and its devs are far from fixing it (cc: @veorq). So until now I’ve been fortunate to use the advantages of this algorithm mainly thanks to GUI apps such as RapidCRC by @ov2 and HashCheck fork by @idrassi. I believe a proper CLI implementation would benefit the community.
Be kind to add BLAKE3 (and sooner or later xxHash 128) to RHash, please.

Non-optimized and public domain https://github.com/michaelforney/blake3-tiny

@sergeevabc
There's a third GUI implementation - TurboSFV (which, incidentally, also has a neat CLI version). It's pretty fast, I think the fastest out of three.

Btw, I have done some tests, and according to my results TurboSFV cli version is x2 faster than RapidCRC, and about 40% faster than HashCheck. Not trying to promote turbo software, which is not free.

Still looks like Blake3 most perspective and fast to check files

Also vote for blake3 implementation


fix: Upload/download parameter rename

Closes #2508
Trying to improve the docstrings and the name of the arguments a little bit
Changed <session>.download() second argument local_file_name to local_directory, as this argument specifies which local directory to write the file to
Note: We don't have any tests for the second argument of the <session>.upload() and <session>.download() methods

Closes #2508
Trying to improve the docstrings and the name of the arguments a little bit
Changed <session>.download() second argument local_file_name to local_directory, as this argument specifies which local directory to write the file to
Note: We don't have any tests for the second argument of the <session>.upload() and <session>.download() methods

🥇


Add support for Zola

This project could become the Computer Language Benchmarks Game of static site generators.

That's the hope! 😄
I appreciate your contributions here. I started out with a few of the more popular SSG choices, and am trying to build and maintain a platform that makes it relatively easy for maintainers of SSGs to add their own projects.
I'm working through a few issues at the moment that are getting in the way. But I'm not going to personally add any more SSGs. You're welcome to contribute a PR with Zola.


Call to route behind proxy adds port 0 to URL

Laravel Version
10.17.0
PHP Version
8.2.8
Database Driver & Version
No response
Description
My setup is Laravel Sail with Caddy added for local https support.
If I set $proxies = '*' in TrustProxies.php, URLs generated by URL::route() is getting port 0 appended.

If I don't set the value for $proxies, I always get Invalid Signature error
Steps To Reproduce

Setup Laravel Sail
Add Caddy as a docker service
Set $proxies to '*' in TrustProxies.php
Generate route via dynamic domain URL pointing to localhost (i.e. *.localhost.referralplus.dev)

Hey there,
While this may be a legitimate issue, can you first try posting your problem or question on one of the support channels below? If this issue can be definitively identified as a bug, feel free to open up a new issue with a link to the original one and we'll gladly help you out.

Laracasts Forums
Laravel.io Forums
StackOverflow
Discord
Larachat
IRC

Thanks!

I was having the same issue with caddy, fixed by adding the following to the TrustedProxies::class middleware:
    protected $proxies = ['<IP_ADDRESS>', '*'];

For disclosure, only tested this solution in the local environment so far. I haven't tested on a production environment yet.
Thanks to @gergonzalez for the tip.
Jeff.


Bootstrap-generator - generate initial migration based on existing schema

Can you add several examples.

Smth like
tarantool> require('migrations').bootstrap({url='localhost:3301', file='./00-bootstrap-schema.lua'})
---
- |-
box.schema.create_space('test')
box.space.test.create_index('primary')

A helper that traverses existing instance and prepares .lua containing relevat ddl statements.

https://stackoverflow.com/questions/1884758/generate-ddl-programmatically-on-postgresql


[Govt Accessibility] charts on ONS website and screen reader software

Hello
I recently published this article on the ONS website and I am doing as much as I can to make it as accessible as possible.
I have written guidance on how to make data vis as accessible as possible before (for the analysis function). With charts our guidance has been to put the alternative text description below the chart in the body text of the page so more people can access it. I have done this (as best I can) on this publication.
When we tested this approach with screen reader software users we found that “hiding” the chart from screen reader software completely was the best way to go (by leaving the alt text field blank or otherwise marking it as decorative) – then the screen reader software would read the chart title, skip over the chart image and go straight to the description.
(This is all laid out in the Analysis Function data vis: charts guidance and elearning if you’re interested)
But – my problem is when publishing on the ONS website I cannot put the chart description directly below the chart image. The source information, embed code and data download button all have to come first (for now, I am trying to work on this!).
And the chart is published within an iframe  - which gets announced.
At the moment it isn’t great because when I test this structure with NVDA screen reader it reads the titles on the page, then announces the iframe, then reads the title again, the reads the embed code stuff, then reads the title again. So the title gets read out three times!
I have some control over what gets read out after the in iframe is announced – at the moment the publishing team are just pasting in the title again – but they could paste in something else, or nothing at all.
I can do nothing about the embed code as this is hard coded and I am having trouble getting anything changed that is hard coded in (although I am trying my best!)
So, my question is – what should get read out once the iframe is announced:
Option 1: nothing – the iframe should just be announced
Option 2: leave the title there
Option 3: something like “chart description follows data download button” – something that describes where the chart description is?
Any and all advice or thoughts are welcome – sorry this is a bit of a long one!
Thanks,
Hannah
Hannah Thomas
Principal statistician for ethnic group, national identity, language and religion statistics
Office for National Statistics - Swyddfa Ystadegau Gwladol
@statshan
<EMAIL_ADDRESS>I worked reduced hours on Wednesdays and Fridays

Forward to Pete Broad

@JNATION85 did you want me to forward this to someone else in Data Vis? I can't remember the names you said other than Pete


Ticker channel missing messages?

I subscribe to the 'tickers' websocket channel.
I use the  received messages to build my own candlesticks. When I compare my calcultated candlesticks with the ones requested directly from the api, I realize that something is wrong. The open, close, max, min values are ok, but my calculated volume value is nearly always lower than the one from the gdax candlesticks. However, sometimes it is correct. It is like there were missing messages.
From the gdax web, I understand that the tickers channel should not have missing messages.
Any help on this?

I've been thinking about it.
The ticker channel "It batches updates in case of cascading matches, greatly reducing bandwidth requirements." Maybe the key is that the "last_size" on the messages is not the total size, just the size of the last match in a batch of matches. If so, i can't build a candlestick with the tickers channel data.
Any suggestion of how could I calculate my own candlesticks (ohlcv) from the gdax socket? I am a bit lost.

Solved.
If I want to get all volumes I should subscribe the matches channel instead of the tickers channel.


make module config file

there is now a modules setting in the application_settings file to determine which modules are shown on the website. settings are saved to ->/json/module_settings.json


SafeJSON exits early if first character is a codepoint > 127

This is probably a bug in the check:
https://github.com/Dyalog/HttpCommand/blob/31d9a947629ce62971bce9a5171644ac920dcfa1/source/HttpCommand.dyalog#L1267
The question is if this is necessary at all. Is UTF-8 encoding not enough?

Fixed in f5d082e07fbcd752604b5410511afe072c694bef