Support partitioned writes

Feature Request / Improvement
Support partitioned writes
So I think we want to tackle the static overwrite first, and then we can compute the predicate for the dynamic overwrite to support that. We can come up with a separate API. I haven't really thought this trough, and we can still change this.
I think the most important steps are the breakdown of the work. There is a lot involved, but luckily we already get the test suite from the full overwrite.
Steps I can see:

[ ] Extend the summary generation to support partitioned writes (here in Java)
[ ] Add support for the append files.

How are we going to fan out the writing of the data. We have an Arrow table, what is an efficient way to compute the partitions and scale out the work. For example, are we going to sort the table on the partition column and do a full pass through it? Or are we going to compute all the affected partitions, and then scale out?

[ ] Add support for static overwrites
[ ] Add support for dynamic overwrites

Other things on my mind:

In Iceberg it can be that some files are still on an older partitioning, we should make sure that we handle those correctly based on the that we provide.
How to handle delete files; it might be that the delete files become unrelated because the affected datafiles are replaced. We could first ignore this.

The good part:

In PyIceberg we're first going to ignore the fast-appends (this is when you create a new manifest, and add it to the manifest list). Instead we'll just take the existing manifest(s) and rewrite it into a single new manifest which makes it a bit easier to reason about the snapshot (and therefore the snapshot summaries). The reason is that this caused quite a few bugs in Java, and it can be added always on a later moment.

Hi @Fokko and Iceberg community, I and @syun64  are continuing working on testing the write capability in Write support pr. We are excited about it as it will help us with our use case a lot.
Our use case also includes overwriting partitions of tables. So I am highly interested in opportunities to contribute to this issue. Would it be alright for me to start working on this issue, based on the Write support pr if no one else has already begun?

Hey @jqin61 Thanks for replying here. I'm not aware of the fact that anyone already started on this. It would be great if you can take a stab at it 🚀

In Iceberg it can be that some files are still on an older partitioning, we should make sure that we handle those correctly based on the that we provide.

It seems Spark's iceberg support has such overwrite behaviors under schema evolution:

dynamic overwrite: data files generated from old partition spec will not be replaced even if some of the records match the overwriting data
static overwrite with PARTITION values specified: same as above
static overwrite without PARTITION values: all data is deleted regardless of what partition specs they conform to.

As Fokko mentioned, we need to make sure in the implementation we use the latest partition spec_id when overwriting partitions so that the data in the old partition spec is not touched.

How are we going to fan out the writing of the data. We have an Arrow table, what is an efficient way to compute the partitions and scale out the work. For example, are we going to sort the table on the partition column and do a full pass through it? Or are we going to compute all the affected partitions, and then scale out?

It just comes to me that when spark writes to iceberg, it requires the input dataframe to be sorted by the partition value otherwise an error will be raised during writing. Do we want to take the same assumption for pyiceberg?
If not, if we have to use arrow.compute.filter() to extract each partition before serialization, it seems a global sort before the filter() on the entire table is unnecessary since the filter makes no assumption of the array organization? It would be helpful if we have an API which uses one scan to bucket sort an arrow array into partitions and return buckets (partitions) as a list of arrow arrays.

I currently see two approaches:

First get the unique partitions, and then filter for each of the partitions the relevant data. It is nice that we know the partition upfront, but the downside is that we need to do a pass through the dataframe to get the relevant rows for that partition.
Sort the entire dataframe on the partition columns, which is expensive, but we need to do this only once. Once it is sorted, we can just do a single pass through the dataframe. If we want to scale out, we can split out the rows: [0, n/2], [n/2, 2].

I'm not sure what is the best. I think the first one works better if you have few partitions, and the latter one is more efficient when you have many partitions.

To extract the partitions by filter(), would it be helpful if we firstly build an API in pyarrow which does a full scan of the array and bucket-sorts it into partitions and returns buckets (partitions) as a list of arrow arrays? These arrays could be further passed as input to writing jobs which are executed in a multi-threading way.

Starting with the API is always a great idea. My only concern is that we make sure that we don't take copies of the data, since that might blow up the memory quite quickly.
Hope this helps!

@jqin61 I did some more thinking over the weekend, and I think that the approach that you suggested is the most flexible. I forgot about the sort-order that we also want to add at some point. Then we would need to sort twice 👎

Based on the existing discussion, there are 3 major possible directions for detecting partitions and writing each partition in a multi-threaded way to maximize I/O. It seems there isn’t any approach simple enough that we could purely leverage the existing Pyarrow APIs in Pyiceberg. I list and compare these approaches for discussion purpose:
Filter Out Partitions
As Fokko suggested, we could filter the table to get partitions before writing but we will need an API on Arrow to get unique partition values (e.g. extend compute.unique() from array/scalar to table).
partitions: list[dict] = pyarrow.compute.unique(arrow_table)

With it, we could filter the table to get partitions and provide them as inputs to concurrent jobs.
arrow_table = pa.table({
    'column1': ['A', 'B', 'A', 'C', 'B'],
    'column2': [1, 2, 1, 3, 2],
    'column3': ['X', 'Y', 'X', 'Z', 'Y']
})
partition_keys = table.select(['column1', 'column2'])
# The existing unique does not have support on table, we need to create API on Arrow side.
partitions: list[dict] = pyarrow.compute.unique(partition_keys)
# = [
    {'column1': 'A', 'column2': 1},
    {'column1': 'B', 'column2': 2},
    {'column1': 'C', 'column3': 3}
]

def filter_and_write_table(partition, index):
    # Create a boolean mask for rows that match the criteria
    mask = pc.and_(*(pc.equal(table[col], val) for col, val in partition.items()))

    # Filter the table
    filtered_table = table.filter(mask)

    # Write the filtered table to a Parquet file
    parquet_file = f'output_partition_{index}.parquet'
    pq.write_table(filtered_table, parquet_file)

with ThreadPoolExecutor() as executor:
    for i, partition in enumerate(partitions):
        executor.submit(filter_and_write_table, partition, i)

Sort and Single-direction Writing
As Fokko suggested, we could sort the table first. We then slice the table and do a one-direction scan for each slice to write out partitioned files.
Suppose we have such an arrow API that takes a sorted table, scans through it, creates a new file whenever encountering a row with a new partition value, and raises an error if it encounters a row with a partition value it already passes, just like how spark writes to an iceberg table.
def write_table_partitions(sorted_table, partition_columns, dir)

Then we could do
partition_columns = ['column1', 'column2']
sorted_table = table.sort_by([('column1', 'ascending'), ('column2', 'ascending')])
directory_path = '/path/to/output/directory'

# Break down the sorted table to slices with zero-copy
slices = slice_table(sorted_table, slice_options)

# Call the API
with ThreadPoolExecutor() as executor:
    # Submit tasks to the executor
    for i, partition in enumerate(partitions):
        executor.submit(write_table_partitions, sorted_table, partition_columns, dir)

Bucketing
We could create an arrow API to return the partitioned tables/record batches based on the inputs of a table and alist of partition columns in a way that the algorithm does a full scan of the arrow table in O(table_length) time and bucket-sorts it and creates a table/record batch for each bucket:
table_partitions = pyarrow.compute.partition(arrow_table, partition_columns)

We could write each batch:
def write_table_to_parquet(table, directory_path, file_suffix):
    # Construct the full path for the Parquet file
    file_path = os.path.join(directory_path, f'record_batch_{file_suffix}.parquet')

    # Write the table to a Parquet file
    pq.write_table(table, file_path)

with ThreadPoolExecutor() as executor:
    for i, partition_as_table in enumerate(table_partitions):
        executor.submit(write_table_to_parquet, partition_as_table, directory_path, i)

As Fokko pointed out, the filter method will not be efficient if there are many partitions - the filter takes O(table_length) time and although each thread can filter on its own, on a single node, the execution will be O(table_length * number_of_partitions) for all the jobs. Technically we only need one same scan to get all the buckets.
It seems the sort method is not as efficient compared to the bucketing method because the relative order of partitions does not matter, so a general sort algorithm on the partition column might be overkill (compared with bucketing).
I feel like all 3 directions require some implementation on Arrow itself (I did not find any approach simple enough that we could purely leverage the existing Pyarrow APIs to implement any of the direction). And I want to get opinions on whether pursuing arrow API level utilities smell good. Thank you!
Specifically, for the third direction of bucketing and returning materialized tables/batches, since Arrow has dataset.write_dataset() which supports partition-respected writing, I did some reading to see how it partitions and whether we could leverage anything from it.
https://github.com/apache/arrow/blob/main/cpp/src/arrow/dataset/partition.cc#L118 is where the partition happens. The partition algorithm is a full scan with bucket sort leveraging  Grouper class utilities in arrow's compute component. Specifically:
1.Grouper.consume() initiates the groups based on the presenting partition columns
https://github.com/apache/arrow/blob/55afcf0450aa2b611e78335bdbfd77e55ae3bc9f/cpp/src/arrow/compute/row/grouper.cc#L422
2.Grouper.MakeGroupings() builds a ListArray where each list represents a partition and each element in the list represents the row_id of the original table.
https://github.com/apache/arrow/blob/55afcf0450aa2b611e78335bdbfd77e55ae3bc9f/cpp/src/arrow/compute/row/grouper.cc#L886C45-L886C58
3.Grouper.ApplyGroupings() efficiently converts the grouped representation of row_ids into actual rows.
https://github.com/apache/arrow/blob/55afcf0450aa2b611e78335bdbfd77e55ae3bc9f/cpp/src/arrow/compute/row/grouper.cc#L875
Other than being used in the dataset writing, Grouper from Arrow's compute component is used to support other exposed compute APIs such as aggregation functions. At the end of the day, what we want (in order to support Pyiceberg's partitioned write) is an API that returns record batches/tables based on an input table and an input partition scheme, so maybe we could expose such a new API under compute leveraging Grouper.

@jqin61 just wondering if we can use this directly
https://arrow.apache.org/docs/python/generated/pyarrow.dataset.partitioning.html

@jqin61 just wondering if we can use this directly https://arrow.apache.org/docs/python/generated/pyarrow.dataset.partitioning.html

Thank you Ashish! I overlooked it, as you mention, we could just use write_dataset() with specified args of partitioning  base_nametemplate to write out the partitioned datafiles as iceberg needs.

@Fokko @jqin61 I am also interested in this to move forward as we also deal with lot of write involves partitions. Happy to collaborate on to this.
For write_dataset() we might need to look if we need to add field meta data at parquet in terms of field-id etc while committing data files.

@Fokko @jqin61 I am also interested in this to move forward as we also deal with lot of write involves partitions. Happy to collaborate on to this. For write_dataset() we might need to look if we need to add field meta data at parquet in terms of field-id etc while committing data files.

Yes - I think we learned this from our earlier attempts: https://github.com/apache/iceberg-python/pull/41/files/1398a2fb01341087a1334482db84a193843a2362#r1427302782
As @jqin61 pointed out in a previous PR, adding these to the schema should output parquet files with the correct field_id.

@Fokko @jqin61
Today I tried basic example on partition write
from pyiceberg.io.pyarrow import schema_to_pyarrow
import pyarrow as pa
from pyarrow import parquet as pq
data = {'key': ['001', '001', '002', '002'],
'value_1': [10, 20, 100, 200],
'value_2': ['a', 'b', 'a', 'b']}
my_partitioning = pa.dataset.partitioning(pa.schema([pa.field("key", pa.string())]), flavor='hive')
TABLE_SCHEMA = Schema(
NestedField(field_id=1, name="key", field_type=StringType(), required=False),
NestedField(field_id=2, name="value_1", field_type=StringType(), required=False),
NestedField(field_id=3, name="value_2", field_type=StringType(), required=False),
)
schema = schema_to_pyarrow(TABLE_SCHEMA)
patbl = pa.Table.from_pydict(data)
pq.write_to_dataset(patbl,'partitioned_data',partitioning=my_partitioning,schema=schema)
If I don't use schema in write it works fine. But if I pass the schema  create schema = schema_to_pyarrow(TABLE_SCHEMA)
It fails with
ArrowTypeError: Item has schema
key: string
value_1: int64
value_2: string
which does not match expected schema
key: string
-- field metadata --
PARQUET:field_id: '1'
value_1: string
-- field metadata --
PARQUET:field_id: '2'
value_2: string
-- field metadata --
PARQUET:field_id: '3'
I also tried the parquet write the way we are doing currenlty
writer = pq.ParquetWriter("test", schema=schema, version="1.0")
writer.write_table(patbl)
ValueError: Table schema does not match schema used to create file:
table:
key: string
value_1: int64
value_2: string vs.
file:
key: string
-- field metadata --
PARQUET:field_id: '1'
value_1: string
-- field metadata --
PARQUET:field_id: '2'
value_2: string
-- field metadata --
PARQUET:field_id: '3
Do we do any other transformation for the schema before we write in current write support.

Hey @jqin61
Thanks for the elaborate post, and sorry for my slow reply. I did want to take the time to write a good answer.
Probably the following statement needs another map step:
partitions: list[dict] = pyarrow.compute.unique(arrow_table)

The above is true for an identity partition, but often we take truncate the month, day or hour from a field, and use that as a partition. Another example is the bucketing partition where we hash the field, and determine in which bucket it will fall.
With regard of utilizing the Arrow primitives that are already there. I think that's a great idea, we just have to make sure that they are flexible enough for Iceberg. There are a couple of questions that pop into my mind:

Can we support all Icebergs partition strategies, such as bucketing, truncating etc.
Are we able to extract the metrics similar that we do for non-partitioned writes.

@asheeshgarg Thanks for giving it a try. Looking at the schema, there is a discrapency. The test-data that you generate has value_1 as an int64, and the table expects a string. I think the error is correct here.

@Fokko thanks for pointing out the mismatch it worked. After modifying the datatype it worked.

@Fokko Thank you! These 2 points of supporting hidden partitioning and extracting metrics efficiently during writing are very insightful!
For using pyarrow.dataset.write_dataset(), its behavior removed the partition columns in the written-out parquet files. I think this is the deal breaker for using write_dataset(). So either we extend pyarrow.dataset.write_dataset() or fall back to the arrow API direction.
Some findings during chasing a solution of dataset.write_dataset():

pyarrow.dataset.partitioning() only supports static values in the column, so we might need on Iceberg to add a transformed column. (this column will be dropped into the directory when write_dataset() uses hive partitioning)
write_dataset() takes in a customized callable to collect file paths and file metadata when files are written and  when we create DataFile object this metadata could be reused:

visited_paths = []
metadata_list = []
def file_visitor(written_file):
    visited_paths.append(written_file.path)
    metadata_list .append(written_file.metadata)

Right, as @jqin61 mentioned, if we only had to support Transformed Partitions, we could have employed some hack to add partition column to the dataset, which gets consumed by write_dataset API when we pass the column in pyarrow.dataset.partitioning.
But we can't apply the same hack with Identity Partitions, where the HIVE partition scheme on the file path shares the same name as the partition column that needs to be persisted into the data file. Arrow does not allow two columns to share the same name, and this hack will lead to an exception on write_dataset.
So it sounds like we might be running out of options in using the existing APIs...
If we are in agreement that we need a new PyArrow API to optimally bucket sort the partitions and produce partitioned pyarrow tables or record batches to pass into WriteTask, do we see any value in introducing a simpler PyIceberg feature in the interim, where write_file can support partitioned tables as long as the provided arrow_table only has a single partition of data?
I think introducing this first would have two upsides:

We decouple the work of supporting writes to partitioned table (like handling partitions in file paths on write, adding partition metadata to manifests) with the work of optimally sorting and bucketing an arrow table into target partitions
If a user really needs to break down their in memory pyarrow table into partitions, they can do so, using existing methods to filter on the partition column and producing a new pyarrow.Table. This isn't optimal, especially if they have many partitions within the in-memory table, and is precisely the reason why @jqin61 is investigating the different options in bucket sorting by partition within Arrow/Arrow Datasets.

Maybe another approach we could take if we want to use existing PyArrow functions is:

table.sort_by (all partitions)
figure out the row index for each permutation of partition groups by taking another pass through the table
Use table.slice(index, length) with indexes we generated above to write out the tables using List[WriteTask] in write_file

If there was an existing PyArrow API that gave us the outcome of (1) + (2) in one pass, it would have been the most optimal, but it seems like there isn't... so I think taking just one more pass to find the indices is maybe not the worst idea.
We could also argue that (1) should be a requirement that we check on the provided PyArrow table, rather than running the sort within the PyIceberg API.
Please let me know your thoughts!

@jqin61 I have also seen this behavior pyarrow.dataset.write_dataset(), its behavior removes the partition columns in the written-out parquet files.
@syun64 above approach look reasonable to me.
It would have been ideal if the partition write we can be done directly using arrow dataset API and use meta data based hidden partitioning using Pyiceberg API. But we need to do good amount of lift in order to that. Haven't seen support for bucket partitioning.
I think we can add write directly using the Pyarrow API as suggested above.

@Fokko @syun64 @syun64 another option I can think is use polars to do it simple example below with hashing and partitioning sorting in partition. Where all the partition is handle by rust layer in Polars and we write parquet based on arrow table returned.
Not sure if we want to add it as dependency? We can do custom transforms like hours etc we have in iceberg as well easily.
import pyarrow as pa
import pyarrow.compute as pc
import polars as pl
t = pa.table({'strings':["A", "A", "B", "A"],'ints':[2, 1, 3, 4]})
df=pl.from_arrow(t)
N = 2
tables=(df.with_columns([
(pl.col("strings").hash() % N).alias("partition_id")
]).partition_by("partition_id"))
for tbl in tables:
print(tbl.to_arrow().sort_by("ints"))

@jqin61 and I discussed this a great deal offline, and we just wanted to follow up on step (2). If we wanted to use existing PyArrow functions, I think we could use a 2 pass algorithm to figure out the row index of each permutation of partition groups on a partition-sorted pyarrow table:
import pyarrow as pa
import pyarrow.dataset
import pyarrow.compute

# pyarrow table, which is already sorted by partitions 'year' and 'animals'
pylist = [{'year': 2021, 'animals': 'Bear'}, {'year': 2021, 'animals': 'Bear'}, {'year': 2021, 'animals': 'Centipede'}, {'year': 2022, 'animals': 'Cat'}, {'year': 2022, 'animals': 'Flamingo'},{'year': 2023, 'animals': 'Dog'}]
table = pa.Table.from_pylist(pylist)

# assert that the table is sorted by checking sort_indices are in order
pa.compute.sort_indices(table, sort_keys=[('year', "ascending"), ("animals", "ascending")])
<pyarrow.lib.UInt64Array object at 0x7fe9b0f4c340>
[
  0,
  1,
  2,
  3,
  4,
  5
]

# then sort the same list of partitions in opposite order, and check the indices to figure out the offsets and lengths of each partition group. If a larger number comes before a smaller index, that's the starting offset of the partition group. the number of consecutive number of indices that are in correct ascending order is the length of that partition group.
pa.compute.sort_indices(table, sort_keys=[('year', "descending"), ("animals", "descending")])
<pyarrow.lib.UInt64Array object at 0x7fe9b0f3ff40>
[
  5,
  4,
  3,
  2,
  0,
  1
]

# from above, we get the following partition group:
partition_slices = [(0, 2), (1, 1), (2, 1), (3, 1), (4, 1), (5, 1)]

for offset, length in partition_slices:
    write_file(iceberg_table, iter([WriteTask(write_uuid, next(counter), table.slice(offset, length))]))

Then, how do we handle transformed partitions? I think going back to the previous idea, we could create intermediate helper columns to generate the transformed partition values in order to use them for sorting. We can keep track of these columns and ensure that we drop the column after we use the above algorithm to split up the table by partition slices.
Regardless of whether we choose to support sorting within the PyIceberg write API or have it as a requirement, maybe we can create a helper function that takes the PartitionSpec of the iceberg table and the pyarrow table and makes sure that the table is sortedByPartition by using the above method.

The Design Document on data file writes that was discussed during the monthly sync. The document summarizes all of the approaches discussed above

I have an incoming PR with working code samples that conform to the design above and cover identity transform + append as the first step of supporting partitioned write. During implementation, I find if the partition column has nulls the code will break. This issue is the same with the existing write where append() or overwrite() would break for any arrow table with a column consisting only of nulls. So I opened Issue#348 to track separately.

Opened draft PR with working code samples (it supports partitioned append with identity transform for now): https://github.com/apache/iceberg-python/pull/353

Idea from @Fokko - support day/month/year transforms first

hi, do we have any way to write to partitioned table so far?

Curious as well as where you guys are standing on partitioned write.

Hey everyone, the support for partioned writes are coming along pretty nicely. We miss some of the transforms, such as the bucket transform. Most of the stuff is on the main branch waiting for a release.
If you want to check it out, you can also install it from source: https://py.iceberg.apache.org/contributing/

@Fokko I installed from source, but I'm hitting this error.
Does anyone have a minimal example of writing to a partitioned table?

Here's what I've been trying (sorry for long example, but thought the context would help)
iowa_sales_df = pcsv.read_csv("/Users/mritchie712/blackbird/demoData/Iowa_Liquor_Sales_20240607.csv")
date_column = pc.strptime(iowa_sales_df['date'], format='%m/%d/%Y', unit='us')

# Replace the old 'date' column with the new date column
iowa_sales_df = iowa_sales_df.set_column(iowa_sales_df.schema.get_field_index('date'), 'date', date_column)

iceberg_schema = Schema(
    NestedField(1, "invoice_item_number", StringType(), required=True),
    NestedField(2, "date", TimestampType(), required=True),
    NestedField(3, "store_number", LongType(), required=True),
    NestedField(4, "store_name", StringType(), required=True),
    NestedField(5, "address", StringType(), required=True),
    NestedField(6, "city", StringType(), required=True),
    NestedField(7, "zip_code", StringType(), required=True),
    NestedField(8, "store_location", StringType(), required=True),
    NestedField(9, "county_number", LongType(), required=True),
    NestedField(10, "county", StringType(), required=True),
    NestedField(11, "category", LongType(), required=True),
    NestedField(12, "category_name", StringType(), required=True),
    NestedField(13, "vendor_number", LongType(), required=True),
    NestedField(14, "vendor_name", StringType(), required=True),
    NestedField(15, "item_number", StringType(), required=True),
    NestedField(16, "item_description", StringType(), required=True),
    NestedField(17, "pack", LongType(), required=True),
    NestedField(18, "bottle_volume_ml", LongType(), required=True),
    NestedField(19, "state_bottle_cost", DoubleType(), required=True),
    NestedField(20, "state_bottle_retail", DoubleType(), required=True),
    NestedField(21, "bottles_sold", LongType(), required=True),
    NestedField(22, "sale_dollars", DoubleType(), required=True),
    NestedField(23, "volume_sold_liters", DoubleType(), required=True),
    NestedField(24, "volume_sold_gallons", DoubleType(), required=True)
)

from pyiceberg.partitioning import PartitionSpec, PartitionField
from pyiceberg.transforms import DayTransform
PARTITION_FIELD_ID_START = 1000

source_id = iowa_sales_df.schema.get_field_index('date')

partition_spec = PartitionSpec(
    PartitionField(
        source_id=source_id,  # ID of the "date" field in the schema (0-based index)
        field_id=PARTITION_FIELD_ID_START,  # Unique ID for the partition field, starting from 1000
        transform=DayTransform(),
        name="date_day"
    )
)

table = catalog.create_table(
    "default.iowa_liquor_sales",
    schema=iceberg_schema,
    partition_spec=partition_spec,
)

table.append(iowa_sales_df)
len(table.scan().to_arrow())

{
	"name": "ValueError",
	"message": "Cannot write to partitioned tables",
	"stack": "---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[17], line 1
----> 1 table.overwrite(iowa_sales_df)
      2 # table.append(iowa_sales_df)
      3 len(table.scan().to_arrow())

File ~/blackbird/notebooks/.venv/lib/python3.12/site-packages/pyiceberg/table/__init__.py:1094, in Table.overwrite(self, df, overwrite_filter)
   1091     raise NotImplementedError(\"Cannot overwrite a subset of a table\")
   1093 if len(self.spec().fields) > 0:
-> 1094     raise ValueError(\"Cannot write to partitioned tables\")
   1096 from pyiceberg.io.pyarrow import schema_to_pyarrow
   1098 _check_schema_compatible(self.schema(), other_schema=df.schema)

ValueError: Cannot write to partitioned tables"
}

update: I uninstalled pyiceberg and installed with:
pip install "git+https://github.com/apache/iceberg-python.git#egg=pyiceberg[s3fs,gcsfs,hive,sql-sqlite,duckdb,pandas]"

It's working now!

The PR says "Support partitioned writes", but on version 0.6.1 it still fails for me with ValueError: Cannot write to partitioned tables"
        NestedField(1, "city", StringType(), required=False),
        NestedField(2, "lat", DoubleType(), required=False),
        NestedField(3, "long", DoubleType(), required=False),
    )
    partition_spec = PartitionSpec(PartitionField(source_id=1, field_id=1000, transform=IdentityTransform(), name="id"), spec_id=1)

    table2 = catalog.create_table(
        "whatever.table_partition_spec",
        schema=schema,
        partition_spec = partition_spec
    )

    table2.append(df1)```

@Pilipets that's correct. The partitioned writes are not yet released. You can use it by installing directly from Github as @mike-luabase suggested above, or wait for 0.7.0 to be released, which will happen soon.

@Fokko , also note that pyiceberg doesn't write partition specification in the manifest files as spark-sql does, which results into parsing errors on the attempt to read Manifest files with Java SDK.
I don't know if it's bug or not but shared with you as a part of the iceberg community.
java.lang.IllegalArgumentException: Cannot parse partition spec fields, not an array: {"spec-id":0,"fields":[]}
	at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:220)
	at org.apache.iceberg.PartitionSpecParser.buildFromJsonFields(PartitionSpecParser.java:123)
...
	at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:111)
...
Cannot parse partition spec fields, not an array: {"spec-id":0,"fields":[]}

In particular, pyiceberg doesn't write "partition-spec". There is a workaround for this - request table metadata and use table specs, though

@Pilipets That looks like a bug, let me check if I can reproduce this. Thanks for sharing!

@Fokko , spark-sql can still read it, so maybe it's expected - i don't know (
In order to read it with Java SDK, i need to provide table.specs() to the ManifestFiles.read, which is actually empty {0=[]}. Without it, it fails to parse with the error I shared above.
For the same table created with no partitioning from spark-sql, I don't need to specify table.specs()
Snapshot snapshot = table.currentSnapshot();
FileIO fileIO = table.io();
for (ManifestFile manifest : snapshot.dataManifests(fileIO))
{
    ManifestReader<DataFile> reader = ManifestFiles.read(manifest, fileIO, table.specs());

First, create  table with no partitioning as below.
def catalog_sqlite(catalog_name: str, warehouse_path: str):
    props = {
        "uri": f"sqlite:///{warehouse_path}/sql-catalog2.db",
        "warehouse": f"file:{warehouse_path}",
    }
    catalog = SqlCatalog(catalog_name, **props)
    catalog.destroy_tables()
    catalog.create_tables()
    return catalog

catalog = catalog_sqlite("local", "/tmp/warehouse")

df = pa.Table.from_pylist(
    [
        {"city": "Amsterdam", "lat": 52.371807, "long": 4.896029},
        {"city": "San Francisco", "lat": 37.773972, "long": -122.431297},
        {"city": "Drachten", "lat": 53.11254, "long": 6.0989},
        {"city": "Paris", "lat": 48.864716, "long": 2.349014},
    ],
)

catalog.create_namespace("whatever")
table = catalog.create_table(
    "whatever.my_table",
    schema=df.schema,
)

table.append(df)

Thanks again for the code! The json that's being generated through the SQLCatalog/SQLite is:
{
    "location": "file:/tmp/whatever.db/my_table",
    "table-uuid": "d394d593-ed82-4c3b-8c58-ad3405965407",
    "last-updated-ms":<PHONE_NUMBER>213,
    "last-column-id": 3,
    "schemas": [
        {
            "type": "struct",
            "fields": [
                {
                    "id": 1,
                    "name": "city",
                    "type": "string",
                    "required": false
                },
                {
                    "id": 2,
                    "name": "lat",
                    "type": "double",
                    "required": false
                },
                {
                    "id": 3,
                    "name": "long",
                    "type": "double",
                    "required": false
                }
            ],
            "schema-id": 0,
            "identifier-field-ids": []
        }
    ],
    "current-schema-id": 0,
    "partition-specs": [
        {
            "spec-id": 0,
            "fields": []
        }
    ],
    "default-spec-id": 0,
    "last-partition-id": 1000,
    "properties": {},
    "current-snapshot-id":<PHONE_NUMBER>625458700,
    "snapshots": [
        {
            "snapshot-id":<PHONE_NUMBER>625458700,
            "sequence-number": 1,
            "timestamp-ms":<PHONE_NUMBER>213,
            "manifest-list": "file:/tmp/whatever.db/my_table/metadata/snap-2465573600625458552-0-c5284c9e-77bc-4a94-b412-1a3a9db4100d.avro",
            "summary": {
                "operation": "append",
                "added-files-size": "1656",
                "added-data-files": "1",
                "added-records": "4",
                "total-data-files": "1",
                "total-delete-files": "0",
                "total-records": "4",
                "total-files-size": "1656",
                "total-position-deletes": "0",
                "total-equality-deletes": "0"
            },
            "schema-id": 0
        }
    ],
    "snapshot-log": [
        {
            "snapshot-id":<PHONE_NUMBER>625458700,
            "timestamp-ms":<PHONE_NUMBER>213
        }
    ],
    "metadata-log": [],
    "sort-orders": [
        {
            "order-id": 0,
            "fields": []
        }
    ],
    "default-sort-order-id": 0,
    "refs": {
        "main": {
            "snapshot-id":<PHONE_NUMBER>625458700,
            "type": "branch"
        }
    },
    "format-version": 2,
    "last-sequence-number": 1
}

With SQLCatalog:
{
  "format-version" : 2,
  "table-uuid" : "70ed598e-01e2-48d2-98c4-33396b3b4bc4",
  "location" : "s3://warehouse/nyc/taxis",
  "last-sequence-number" : 0,
  "last-updated-ms" :<PHONE_NUMBER>513,
  "last-column-id" : 19,
  "current-schema-id" : 0,
  "schemas" : [ {
    "type" : "struct",
    "schema-id" : 0,
    "fields" : [ {
      "id" : 1,
      "name" : "VendorID",
      "required" : false,
      "type" : "long"
    }, {
      "id" : 2,
      "name" : "tpep_pickup_datetime",
      "required" : false,
      "type" : "timestamptz"
    }, {
      "id" : 3,
      "name" : "tpep_dropoff_datetime",
      "required" : false,
      "type" : "timestamptz"
    }, {
      "id" : 4,
      "name" : "passenger_count",
      "required" : false,
      "type" : "double"
    }, {
      "id" : 5,
      "name" : "trip_distance",
      "required" : false,
      "type" : "double"
    }, {
      "id" : 6,
      "name" : "RatecodeID",
      "required" : false,
      "type" : "double"
    }, {
      "id" : 7,
      "name" : "store_and_fwd_flag",
      "required" : false,
      "type" : "string"
    }, {
      "id" : 8,
      "name" : "PULocationID",
      "required" : false,
      "type" : "long"
    }, {
      "id" : 9,
      "name" : "DOLocationID",
      "required" : false,
      "type" : "long"
    }, {
      "id" : 10,
      "name" : "payment_type",
      "required" : false,
      "type" : "long"
    }, {
      "id" : 11,
      "name" : "fare_amount",
      "required" : false,
      "type" : "double"
    }, {
      "id" : 12,
      "name" : "extra",
      "required" : false,
      "type" : "double"
    }, {
      "id" : 13,
      "name" : "mta_tax",
      "required" : false,
      "type" : "double"
    }, {
      "id" : 14,
      "name" : "tip_amount",
      "required" : false,
      "type" : "double"
    }, {
      "id" : 15,
      "name" : "tolls_amount",
      "required" : false,
      "type" : "double"
    }, {
      "id" : 16,
      "name" : "improvement_surcharge",
      "required" : false,
      "type" : "double"
    }, {
      "id" : 17,
      "name" : "total_amount",
      "required" : false,
      "type" : "double"
    }, {
      "id" : 18,
      "name" : "congestion_surcharge",
      "required" : false,
      "type" : "double"
    }, {
      "id" : 19,
      "name" : "airport_fee",
      "required" : false,
      "type" : "double"
    } ]
  } ],
  "default-spec-id" : 0,
  "partition-specs" : [ {
    "spec-id" : 0,
    "fields" : [ ]
  } ],
  "last-partition-id" : 999,
  "default-sort-order-id" : 0,
  "sort-orders" : [ {
    "order-id" : 0,
    "fields" : [ ]
  } ],
  "properties" : {
    "owner" : "root",
    "write.parquet.compression-codec" : "zstd"
  },
  "current-snapshot-id" : -1,
  "refs" : { },
  "snapshots" : [ ],
  "statistics" : [ ],
  "partition-statistics" : [ ],
  "snapshot-log" : [ ],
  "metadata-log" : [ ]
}

I don't see any discrepancies with the sort-orders, but I can continue tomorrow. @adrianqin feel free to jump in here :)

"partition-spec" : [ ], is missing.
  static final String PARTITION_SPEC = "partition-spec";
Try generating table with spark-sql to compare and it will be there.
export AWS_ACCESS_KEY_ID="***"
export AWS_SECRET_ACCESS_KEY="***"
export AWS_REGION=us-west-2

spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.2_2.12:1.3.0,software.amazon.awssdk:bundle:2.20.147 \
    --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
    --conf spark.sql.defaultCatalog=local \
    --conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
    --conf spark.sql.catalog.local.warehouse=s3://hp-bucket/wh \
    --conf spark.sql.catalog.local.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog \
    --conf spark.sql.catalog.local.io-impl=org.apache.iceberg.aws.s3.S3FileIO

create database local.test_db; create table local.test_db.table (col int) using iceberg;
insert into local.test_db.table values (1), (2), (3); select * from local.test_db.table;

Thanks @Pilipets, again, really appreciate the examples.
So the difference is with the versions. I was generating V2 tables, and you have a V1 table (Java 1.3.0 has the default on V1, which should be changed later on.).

Yeah, I actually don't know if partition-spec is the culprit, but that's the difference that I spotted.
Below is full error stack.
java.lang.IllegalArgumentException: Cannot parse partition spec fields, not an array: {"spec-id":0,"fields":[]}
	at org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:220)
	at org.apache.iceberg.PartitionSpecParser.buildFromJsonFields(PartitionSpecParser.java:123)
	at org.apache.iceberg.PartitionSpecParser.fromJsonFields(PartitionSpecParser.java:114)
	at org.apache.iceberg.PartitionSpecParser.lambda$fromJsonFields$4(PartitionSpecParser.java:119)
	at org.apache.iceberg.util.JsonUtil.parse(JsonUtil.java:95)
	at org.apache.iceberg.PartitionSpecParser.fromJsonFields(PartitionSpecParser.java:119)
	at org.apache.iceberg.ManifestReader.readPartitionSpec(ManifestReader.java:130)
	at org.apache.iceberg.ManifestReader.<init>(ManifestReader.java:114)
	at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:131)
	at org.apache.iceberg.ManifestFiles.read(ManifestFiles.java:111

I should have gone deeper into the stacktrace, it looks like the partition-spec header is misformed.
Python:
➜  docker-spark-iceberg git:(main) avro-tools getmeta ~/Desktop/45c184f4-4550-4f6e-a99c-496d93671653-m0.avro | grep -i partition-spec
24/06/21 17:28:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
partition-spec {"spec-id":1,"fields":[{"name":"tpep_pickup_datetime_day","transform":"day","source-id":2,"field-id":1000}]}
partition-spec-id 1

Java:
➜  docker-spark-iceberg git:(main) avro-tools getmeta ~/Desktop/793ff997-d03b-4992-a477-2cf9dbe3cb6b-m0.avro | grep -i partition-spec
24/06/21 17:31:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
partition-spec-id 1
partition-spec [{"name":"tpep_pickup_datetime_day","transform":"day","source-id":2,"field-id":1000}]

Thanks again @Pilipets for raising this, I'll come up with a fix shortly.

@Fokko any expected timeline you can share on support for bucket transform? Is there a separate issue I can follow for that? Thanks for all the hard work so far!!

@RLashofRegas Sorry for the long wait, @sungwy has been working on adding a rust extension to efficiently run the bucketing transform 🥳  We're blocked on a release on the rust side of things, but we're working on a release there as well 👍


LCFS - Improve Number Filtering

As a user I want to be able to filter my tables by entering the start of a number.
For example:
Entering 32 in the filter should return:
32
32.1
32.17
322
It should not return:
132

Create a new number filter that applies a startsWith filter
Implement the startsWith filter on the backend for numbers

@dhaselhan Thank you, Daniel, for creating this ticket. Just to confirm, should this apply to all tables? Does it also include table row IDs?

This should be applied to all number/int/float columns in each grid. I don't believe we show rowIds in the ui, where can we filter them?

Thanks, Daniel, for the clarification.  Sorry for the confusion. I meant "Row ID" in the Audit Log table, but it's not the actual row ID. 👍

For testing purposes, these are the columns affected by this change:
BCeID:

Transactions (Units, Value per unit)

IDIR:

Transactions (Units, Value per unit)
Fuel Codes (Iteration, Carbon Intensity)
Administration > User Login History (User Login History ID)
Administration > Audit Log (Row ID)

Let me know if anything else needs to be included!


Set Up SAP Data Hub, Developer Edition 2.4

Tutorial URL: https://developers.sap.com/tutorials/datahub-docker-v2-setup.html
Please specify the step you are referring to
Step 3: Build the Docker image
The step asks to build the Dockerfile. But where does one download it from ?
Abhishek

Never mind, I found it in the fine print on https://www.sap.com/developer/trials-downloads/additional-downloads/sap-data-hub-developer-edition-15004.html
.


BVAL-655 getContainerElementTypes() -> getConstrainedContainerElement…

…Types(); adding example
https://hibernate.atlassian.net/browse/BVAL-655

Reworked and force-pushed the commit to address your comments, @gsmet!

Rebased and merged, thanks!


fix: 3/4 moon orientation

Thanks! Please can we stick with simple version naming though? Eg 0.02 vs 0.01.1? That's what we use for everything else.

Cool clock, especially the moon  :)

@gfwilliams I fixed the version number
@MaBecker in readme I gave credit for having basically copied your widget, hope this doesn't hurt you :-)

Great - thanks!


No Kernels Found Error

Whenever I try to open a notebook I get a warning that looks like the image below:

Steps to replicate:
install brew with curl
install pipx with brew
install euporie with pipx
pip install --user ipykernel
all of the above is done with the suggested command from my ~ dir
I also added all the suggested paths to my $PATH from each installation
How can I run code? Also I thought the whole point of pipx was to install all required packages in an isolated venv that the executable uses, so why am I even executing pip install --user ipykernel in the first place?

Okay I made some progress. So it looks like to install packages for the pipx environment one has to activate the venv and then run python -m pip install ipykernel (not just pip install, otherwise won't install for venv)
Now when I open a notebook file the editor immediately print some error lines which disappear instantly so I can't read them and then shows the editor (I can type in it but I can't execute code, nor is the menu bar visible so I just quit by closing the entire terminal window)

There's quite a lot going on here!
Euporie does not pull in any Jupyter kernels as dependencies - it's a Jupyter client, but you need to install a kernel yourself.
A Jupyter kernel does not have to be Python - there are kernels for many programming languages. ipykernel is a Jupyter kernel for the Python programming language. While it is technically possible to instally ipykernel inside your euporie pipx virtual-environment, it's almost certainly not what you want to do (you might want to uninstall it from the pipx virtual-environment). You probably want your Python kernel to be installed in your system-wide Python prefix.
You can do this by running the following (without any virtual environment activated):
pip install --upgrade ipykernel

You then need to register this ipykernel installation as your Python kernel of choice, which you can do by running the following:
python -m ipykernel install --user

Then you should be able to run code in euporie.

You can quit euporie by pressing Ctrl + q.

Also, iterm2 also doesn't allow for any keybinding to execute cells.

Ctrl+e and Ctrl+r should work.

Any ideas how to provide good debugging information?

If you run euporie-notebook with --log-level=debug --log-file=euporie.log, then debug info will be logged to the file euporie.log.
The main cause of the kernel related errors (as described in #75) should have been fixed by 3a5e39f7ccb762496c8d7f5728daf49ea5a1a213. I'll be making a new release with this pretty soon.

Should be fixed, please open a new issues if you're still experiencing problems with the latest version of euporie


Start-SPOSiteContentMove - More information would be very helpful

Currently looking into the possibility of automating moving SharePoint online sites through PowerShell, however due to lack of documentation, I am unable to assert whether this command could help me or not.
Since I have not test environment at my disposal, it's hard for me to conduct any sort of conclusive tests.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: ab43a7d9-79d4-5f40-f49f-1f90c2af1583
Version Independent ID: a54c713c-4995-30d6-c500-bd13152e8190
Content: Start-SPOSiteContentMove (sharepoint-online)
Content Source: sharepoint/sharepoint-ps/sharepoint-online/Start-SPOSiteContentMove.md
Service: sharepoint-powershell
GitHub Login: @techwriter40
Microsoft Alias: kirks

Hi @GaetanLongree how are you? thanks for your suggestion, could you please be more specific on the information you want to be in the documentation for that cmdlet?
Thanks

This dates back from when I was working on a project that needed an automated method of moving subsites between sites on SharePoint Online.
As my original description states, the documentation at the time was not complete and did not provide enough information on the function/features, scope and uses of this command.
Since I'm no longer working on the project, I can't elaborate further on the follow-up of this command uses for that specific use case.

Hi @GaetanLongree Exactly, documentation at the time was different, now it's more complete about function and features. Thanks!

@get-itips  Thank you very much for the contribution and sharing this explanation. @GaetanLongree
Hope this comment is helpful for you. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.


Popover state is always open

Describe the bug
The data-state of Popover.Trigger is always open and never closed.
Reproduction
https://stackblitz.com/edit/github-258g8x?file=src%2Froutes%2F%2Bpage.svelte
Logs
No response
System Info
System:
    OS: Linux 6.8 Ubuntu 22.04.5 LTS 22.04.5 LTS (Jammy Jellyfish)
    CPU: (16) x64 12th Gen Intel(R) Core(TM) i5-12500H
    Memory: 14.98 GB / 31.02 GB
    Container: Yes
    Shell: 5.8.1 - /usr/bin/zsh
  Binaries:
    Node: 22.11.0 - /run/user/1000/fnm_multishells/120790_1732637740412/bin/node
    npm: 10.9.0 - /run/user/1000/fnm_multishells/120790_1732637740412/bin/npm
    pnpm: 9.12.3 - /run/user/1000/fnm_multishells/120790_1732637740412/bin/pnpm
  Browsers:
    Brave Browser: <IP_ADDRESS>
    Chrome: 131.0.6778.85
    Chromium: 131.0.6778.85
  npmPackages:
    @sveltejs/kit: 2.8.1 => 2.8.1
    bits-ui: 0.21.16 => 0.21.16
    svelte: 5.2.2 => 5.2.2

Severity
annoyance

Fixed in @next. If someone wants to resolve this in main you're more than welcome to override that by placing the attribute after the melt builder spread

Going to close this as we're focusing on a stable 1.0 release for Bits UI which will use Svelte 5 and this issue is fixed in the next release.


v1

I've taken ownership of this package and made some updates:

Code adapted from latest mobx-react to be Preact specific.
Removed a few features to slim things down a bit and simplify the code. See README (although I'm open to putting these back)
Continue to support @connect from mobx-preact 0.x
I migrated the test suite to my preference, Jest, because I find it's built-in JSDom support makes it easy to work with Preact

Thanks @nightwolfz for transferring ownership

Great stuff @philmander !

Thanks @mweststrate , I only started out trying to add a few features to mobx-observer and ended up here!

Amazing work @philmander  !

Thanks @nightwolfz , but it was mainly just find and replace "react" --> "preact" !!!


Request for Student Discount or Affordable Pricing for TamaGUI Takeout

I'm Ayush Naphade, a second-year B.Tech student in Mechanical Engineering at Visvesvaraya National Institute of Technology, India. I recently completed my first React Native app and was looking into pre-built UI component libraries to streamline development. That's when I came across Tamagui and its TakeOut UI component template, which seems like a fantastic tool for quick and easy app development.
However, as a student working with a startup, budget is a concern for us. The pricing for TakeOut is a bit steep, and we were wondering if there are any options available for a free template or at least a discount . We're really excited about the potential of Tamagui for our project and would love to explore it further.
Any guidance or assistance you can provide would be greatly appreciated!
Thanks,
Ayush Naphade ~ @aayushnaphade

We have parity pricing that should show a banner on the bottom, but if you email us<EMAIL_ADDRESS>I can also send you a coupon.


Add an operator page to show application counts of all projects

What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #1511
Does this PR introduce a user-facing change?:

The operator now can see the number of application of projects

Nice
/lgtm


docs: Add python version support policy to plugin README.mds

Description
Closed #404
Development notes

Added the python version policy to the README.md files of each plugin
Added licence/contributing info to plugins where it was missing.

Checklist

[ ] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
[ ] Updated the documentation to reflect the code changes
[ ] Added a description of this change in the relevant RELEASE.md file
[ ] Added tests to cover my changes

@astrojuanlu Did you mean to already merge this? The builds were still running

😳 I enabled the automerge and in fact the gatekeeper is already green

unsure why this happened, but yeah I didn't intend to merge before running all the jobs, sorry 🙏🏽

😳 I enabled the automerge and in fact the gatekeeper is already green

Hmmm it looks like the merge gatekeeper only tracked part of the builds: https://github.com/kedro-org/kedro-plugins/actions/runs/6735475912/job/18308735307 🤔


"Please fix your bundle id" - Window too small

[REQUIRED] Please fill in the following fields:

Unity editor version: 2020.2.0b7
Firebase Unity SDK version: 6.16.0
Source you installed the SDK: UPM
Problematic Firebase Component: Other
Other Firebase Components in use: Core, Crashlytics
Additional SDKs you are using: Several
Platform you are using the Unity editor on: Windows
Platform you are targeting: Android
Scripting Runtime: IL2CPP

[REQUIRED] Please describe the issue here:
I frequently got the "Please fix your bundle id" after I entered play mode. Selecting my entry from the dropdown menu did not fix the window appearing.
What did fix it?
The window was too small.  Only by searching around here on Git I learned of that button.
So: Please give the "Please fix your bundle id" window a proper minimum size.

@drallcom3
The minimum size of this window is hardcoded and worked fine in previous versions of Unity.
Perhaps we should scale this dynamically.
Thank you for reporting this!
Shawn


wifi doesn't work

whenever I try to connect to wifi, nothing ever shows up

do you have a ethernet port? you can use it temporary to download wifi drivers

Closed due to no response in 10 days, assuming he has solved the issue.


Fixes: (xls/xlsx) Time fields rejected on import even though they are valid

Fixes #1313
Was receiving "Enter a valid time" errors for time fields because tablib is already parsing these fields into python datetime objects, then it tries to convert them again in the clean method. This was already done with same method for the DateWidget and DateTimeWidget classes.
I did not test the change on the DurationWidget. I assume it will have the same issue, but I've never used timedelta fields before.

Hi
Thanks for raising this and for supplying a fix.  Would you be able to look at adding tests to confirm the fix?

Thanks for adding tests - please can you look into the test failures?

To reproduce with Test app

Export Book objects as xlsx
Edit the spreadsheet and set 'published' format to time
Save the spreadsheet and re-import

Thanks - please can you squash the commits and reference the issue in the commit text.

Coverage increased (+0.003%) to 98.063% when pulling f888aeab8aa6debdda5f7a8464321cc1967da3d4 on gwarkentin:patch-1 into 6d15fe98fca87b7ed0c2f336591083ce5d1cd6a8 on django-import-export:main.

Sorry, for the delay. I made the original changes on github site and didn't have a chance to work on this properly till today. I've never used git outside of hosting homework assignments and am not sure how to squash commits. Seems like I would download the repo and make all of the changes at once before making a commit?
Is this completed or do still need to do anything?
Thanks

This is now completed, so no need to do anything.  I picked it up directly because I wasn't sure if you were able to complete, and I wanted it the main codebase.
It's easy to squash commits.  You have rebase your branch in your local repo, and then force push it to your remote repo.  Having squashed commits makes it easier to read the main log.  There's a short guide here.
Thanks again for submitting this issue and patch.


windows_exporter_options does not affect the install

When you specify the windows_exporter_options parameter when using this role the installer does not respect it/change the installation.
Desired state
specifying the role in a playbook like:
- role: windows_exporter
      windows_exporter_version: '0.18.1'
      windows_exporter_options: 'ENABLED_COLLECTORS=cpu,cs,logical_disk,memory,net,os,textfile,iis TEXTFILE_DIR=D:\metrics'

results in a installation that has the specific collectors enables, and the textfile directory set to the user specified location.
The readme would need to be updated to specify that you should use the MSI parameters as defined in the prometheus windows_exporter documentation here: https://github.com/prometheus-community/windows_exporter#installation

Nice catch. Thank you.


config via environment variable broken

[notting@nostromo: ~]$ tower-cli --version
Tower CLI 3.1.3
[notting@nostromo: ~]$ TOWER_HOST=foo.bar TOWER_USERNAME=cow TOWER_PASSWORD=foo TOWER_VERIFY_SSL=false tower-cli config

# Defaults.
username:
description_on: False
host: <IP_ADDRESS>
verbose: False
certificate:
format: human
color: True
password:
verify_ssl: True

Have not dug through to see when this last worked as documented.

Tried with 3.0.2 and replicated with the steps you gave. The content it echos is unchanged by the env variables. Ping @wwitzel3 so he knows.

The problem here is that documentation is not accurate enough. Because environment variables won't be included when using Tower CLI as a command-line tool. Under the hood, env variables only affect setting.runtime_values context manager, and that context manager is only used by tests and  ansible tower modules.
Will update docs to remove the confusion.

Is there any reason not to support it when using the CLI?

@wenottingham Because reading configuration from env variables has not been taken into consideration before, but we definitely can support it. Ping @AlanCoding for opinion on adding this feature.

This feature is very useful to use Tower CLI in automation tools. I use the environment variables to change Tower CLI configuration programmatically from Hubot scripts, for example changing the user, password and event hostname of Tower CLI depending on the chat user or task request to Hubot from Slack.


Solution for Issue#58

Issue Title:
resolve issue: #58
Type of change ☑
What sort of change have you made:

[x] New feature (non-breaking change which adds functionality)

Changes made:

[x] a popup for file save type (will save time in future file type additions as well)
[x] Currently available export types: PNG, JPG
[x] added a transparent background fill option

@thatsyogeshjjr Kindly add a working video of the changes that you made.

https://github.com/singodiyashubham87/Draw-it-out/assets/69074180/93c6106d-b309-469a-9a8e-33b7b903863e
The first image saved is saved over a transparent background (w/o the smiley) displaying ability of the solution to save images as transparent png(s). The next two images are saved over a white background in jpg and png formats.

@thatsyogeshjjr Kindly resolve the conflicts.

Wait,I have a better way to integrate it with the recent project changes, I'll be back with a new PR

@singodiyashubham87  I just made a new PR #139
The updates now integrate better with the newer UI


[Bug]: Rolling update feature doesn't work with expired container registry credentials

What happened?
I pushed a docker app from a private container registry. I tried to restart the app with cf restrart <my-app>  --strategy=rolling when the container registry secrets which failed living the app in an unhealthy state. I noticed that when I executed the restart command my container registry credentials where expired.
What you expected to happen
I would expect that the restart command fails but the app stays in healthy state.
Acceptance Criteria
GIVEN that I have a docker based app
WHEN I try to execute a command which supports --strategy=rolling with expired credentials
THEN I expected the command to fails but the app instance should stay in a healthy sate.
How to reproduce it (as minimally and precisely as possible)

cf push <my-app> --docker-image <docker-image-path>:latest --docker-username <my-user-name>
Invalidate your docker credentials/token
cf restart <my-app> --strategy=rolling

Anything else we need to know?
No response
Environment
This happens with https://github.com/cloudfoundry/korifi/releases/tag/v0.9.0

Rolling strategy is implemented using K8S rolling update capabilities of the scheduler. At the moment korifi uses statefulsets to run the app workloads. Rolling update for statefulsets stops the old instance before starting the new one, for ordering reasons. If the app has more than 1 instance this downtime can be avoitded.
Better yet, if we switch the implementation to use deployments instead we would not see downtime even with one instance. However doing this means we cannot supprot the CF_INSTANCE_INDEX env var. We have had many conversations historically about its importance, but no decisions have been taken.

I see, the downtime is not good and provides a different experience for app update operations from what Diego offers and it is worth to be discussed. Could you maybe provide a link to the that discussions?
Additionally, I would expect that at the end my instance is healthy again when the restart operation finishes which isn't the case. Apps with one instance will be down after the restart operation and apps with multiple instances will have at least one instance "down". Will this be mitigated if the switch to deployments is implemented?


Add config option for initial map position and zoom level

When I set up my own instance of brouter-web, I wanted to change the default map location closer to where I live. I had to spend a little bit of time looking through the source code to find where I could change this. I think making this a configuration option would make it easier for people to change the default map location in the future.

Thanks @rmsacks for the improvement!

I pushed 23b5439 to prevent current running servers to break due to outdated config.js.


add an execution to the test phase in the pom.xml

Fix #5
Signed-off-by: Karim Elghamry<EMAIL_ADDRESS>

The app should run upon calling mvn test


Shaped Tensor are not compatible with FastAPI

Initial Checks

[X] I have read and followed the docs and still think this is a bug

Description
NdArray['x'] is not compatible with fastAPI schema.
in a main.py file do:

from fastapi import FastAPI
from docarray.base_doc import DocArrayResponse, BaseDoc
from docarray.typing import NdArray
from pydantic import Field

class EmbeddingObject(BaseDoc):
    object: str = 'embedding'
    index: int = Field(description='The index of the embedding output, corresponding to the index in the list of inputs')
    embedding: NdArray['x'] = Field(description='The embedding of the texts', default=[])

app = FastAPI()

@app.post("/", response_model=EmbeddingObject, response_class=DocArrayResponse)
async def create_item() -> EmbeddingObject:
    ## call my fancy model to generate the embeddings
    return EmbeddingObject(embedding=[0.1, 0.2, 0.3], index=0)

then do uvicorn main:app
and from your browser try to access localhost:8000/docs
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/fastapi/applications.py", line 289, in __call__
    await super().__call__(scope, receive, send)
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/fastapi/applications.py", line 244, in openapi
    return JSONResponse(self.openapi())
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/fastapi/applications.py", line 217, in openapi
    self.openapi_schema = get_openapi(
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/fastapi/openapi/utils.py", line 458, in get_openapi
    field_mapping, definitions = get_definitions(
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/fastapi/_compat.py", line 458, in get_definitions
    return {}, get_model_definitions(
  File "/home/sami/.cache/pypoetry/virtualenvs/docarray-OI1ry5x2-py3.8/lib/python3.8/site-packages/fastapi/_compat.py", line 359, in get_model_definitions
    m_schema, m_definitions, m_nested_models = model_process_schema(
  File "pydantic/schema.py", line 582, in pydantic.schema.model_process_schema
  File "pydantic/schema.py", line 623, in pydantic.schema.model_type_schema
  File "pydantic/schema.py", line 251, in pydantic.schema.field_schema
  File "pydantic/schema.py", line 319, in pydantic.schema.get_field_schema_validations
  File "pydantic/schema.py", line 106, in pydantic.schema._apply_modify_schema
  File "./docarray/typing/tensor/abstract_tensor.py", line 282, in __modify_schema__
    reduce(mul, cls.__docarray_target_shape__, 1)
TypeError: '<=' not supported between instances of 'str' and 'int'

if I used NdArray instead of NdArray['x'] it is working tho.
Example Code
below

Python, DocArray & OS Version
0.38.1.dev1

Affected Components

[ ] Vector Database / Index
[ ] Representing
[X] Sending
[ ] storing
[ ] multi modal data type

what about NdArray[128]?

what about NdArray[128]?

this one is working


add message id

Should add MessageId to the email.
MessageId is a "Should" at RFC, but Sympa and for example GMail requires it.
To set the MessageId at MTA is not working as this would break DKIM/DMARC.
Mike

And... this PR is untested, as iam not familiar with golang... first time to write anything in go


Website documentation should be versioned.

As per our discussions here https://cloud-native.slack.com/archives/CL25937SP/p1585323322092300 and on Thanos Sync we agreed that we need improvement to our Hugo-based website that will allow exploring documentation part (docs/) per release version (and latest master) (:
Help wanted :hugs:
Maybe good as GSoC/Community project.
AC:

(docs/ should be versioned under https://thanos.io/getting-started.md/ on our website for each minor releases (release-* branches) and master.
Allow user to tell in URL path what version to show.
By default show docs from the latest release
It should be easy to add new releases.

Ideally should be part as some preprocessing before Hugo, maybe as Go command (:

Cortex is doing the same :hugs:  https://github.com/thanos-io/thanos/issues/2488

Hi!
We are trying to achieve something similar in Cortex. We have proposed to version specific pages to start with. We would sincerely appreciate feedback and suggestions on our proposal.

Cortex is doing the same 🤗 cortexproject/cortex#2349

Cool!

AFAIK @jaybatra26 is helping with proposal only. We are adding this project to Community Bridge. :hugs:

@bwplotka Are we planning to achieve the versioned documentation similar to k8s.io ? Or do you have something else in mind?

Rather like https://prometheus.io/docs/introduction/overview/ (:

Rather like https://prometheus.io/docs/introduction/overview/ (:

Nice, got it! I would to apply for the same😄

Wow! @bwplotka  This is the kind of issue I would love to contribute! Drafting my first PR as soon as possible

Hi @bwplotka! I am interested to get mentored for this issue for the Community Bridge program.
Thus, I have applied as a mentee on the Community Bridge website for this project.
Hoping to getting interviewed and selected. Looking forward to it.

@thisisobate worth to check what Cortex did: https://github.com/cortexproject/cortex/pull/2349


Support for Retrain the model on a device in MicroPython

Description:
I am currently working on a project using ESP32 and ESP32-S3 boards and utilizing the microlite library for running TFLite models in MicroPython. First of all, I would like to express my gratitude for this excellent library and its support for TFLite in MicroPython.
I am exploring ways to push the capabilities of the ESP32 board to its limits. As part of my project, I would like to investigate the possibility of training and retraining TFLite models in real-time using the microlite library.
Specifically, I am interested in the following capabilities:
Training TFLite models: Is it possible to utilize the microlite library to train TFLite models directly on the ESP32 board? This would involve updating the model weights and biases during runtime based on real-time data. It would be beneficial for scenarios where the model needs to adapt and improve its performance continuously.
I would appreciate any insights, guidance, or examples on how to achieve these capabilities using the microlite library. Additionally, if these features are not currently supported, I would be interested to know if there are any plans or possibilities of adding such functionality in future updates.
Thank you for your attention and support. I look forward to any feedback or suggestions regarding training and real-time retraining of TFLite models using the microlite library in MicroPython.
Here are some useful relevant resources

Tensorflow
Article 1
Article 2

The model file is stored on the filesystem so can be edited.   Its a Flatbuffer file: https://flatbuffers.dev/
Reference: https://www.tensorflow.org/lite/microcontrollers/build_convert#model_conversion
At the moment the model doesn't map to any Micropython type its just a byte array.
If you can find the c or c++ methods needed to enable rewriting the model weights something could be surfaced into the microlite api to allow calling from micropython.


Frost Caves Update..

Bumping for the new update.. Thanks!

Would love to see the update as well :)
Thanks for your hard work

ValheimTooler has been updated for the latest game update (version 0.209.10). Sorry for the very very big delay :(


"The SCEP server configuration is not supported" error

I have used a ubuntu server configed this, But when mobile device installing profile and request "scep?operation=GetCACert" url, it will response "The SCEP server configuration is not supported" error in device.
Any ideas?
Thanks

I'm having the same problem (almost 5 years later 😄). @grantchen were you able to fix it?


Error: Naming collision detected

when I start development server using this command:
react-native start
then I am getting error this:
Error building DepdendencyGraph:
Error: Naming collision detected: /home/awesomepankaj/Work/lifemakerProjects/workingLifemaker/LifeMaker/node_modules/react-native/node_modules/react-tools/src/shared/vendor/core/dom/focusNode.js collides with /home/awesomepankaj/Work/lifemakerProjects/workingLifemaker/LifeMaker/node_modules/react-native-modalbox/node_modules/react-native/node_modules/react-tools/src/shared/vendor/core/dom/focusNode.js
react-native version 0.13
Ubuntu 14.04

Delete the extra react-native folder in node_modules/react-native-modalbox/node_modules/.

thanks


Add a css style file and adjust html

Added a style.css file, referenced it in index.html. Adjusted css properties to style the page.


fakelib* SONAME fix

Hey everyone ! We have been using WI4MPI recently, and came accross a problem when using libraries with an RPATH.
To circumvent the RPATHs total ignorance of LD_LIBRARY_PATH, we have been preloading necessary libraries.
Despite preloading the corresponding fake library (in $WI4MPI/libexec/wi4mpi/fakelib*) , the dynamic linker would still load the original library, leading to crashes. This behaviour was unexpected, as a preloaded library is expected not to be reopened.
This is due to a mismatch in the SONAME field of the ELF header. On a library installed with WI4MPI v3.4.1:
$ readelf -d /packages/wi4mpi/v3.4.1/libexec/wi4mpi/fakelibMPICH/libmpifort.so.12 | grep -i soname
 0x000000000000000e (SONAME)             Library soname: [libmpifort.so.12.so]

This is due to a CMake quirk that when asked to create a shared object, a system-specific suffix is appended. On linux, that is the outstanding .so.
The following PR uses CMake target properties to fix this. As the version is now a property, and CMake requires every target name to be unique, the add_library call now uses a composite name ${LIB}${MAJOR}.
foreach(LIB ${LIST_FAKECXX})
    foreach(MAJOR ${LIST_FAKECXX_MAJOR})
        add_library(${LIB}${MAJOR} SHARED ${SOURCE_WI4MPI_EMPTY})
        set_target_properties(${LIB}${MAJOR} PROPERTIES OUTPUT_NAME ${LIB} SOVERSION ${MAJOR})
    endforeach(MAJOR)

CMake will then output a library with the following format: ${OUTPUT_NAME}.so.${SOVERSION}.
I changed the name of the affected variables and reformatted the modified lines. Please let me know if I can change anything for the sake of consistency within the project.

Hi, thank you for this PR.
We should merge, @perache @frogeb :-) !


aspnet-codegenerator nuget version still at 2.1.0-preview1-final

Since 2.1.0-rc1 is now released, shouldn't this package also be updated on nuget? Or is this now available via global tools?
https://www.nuget.org/packages/Microsoft.VisualStudio.Web.CodeGeneration.Tools/

Or is this now available via global tools?

@tura08 you are correct. This is now available as a global tool
dotnet-aspnet-codegenerator

Though I was missing something, thanks. I was able to successfully install it via:
dotnet tool install -g dotnet-aspnet-codegenerator --version 2.1.0-rc1-final.
Is this documented somewhere because I couldn't find references to it on docs.microsoft?

Closing since we have the most recent Scaffolding package versions available in NuGet.org


ISSNs contain extra punctuation

I noticed a minor issue when converting records containing ISSNs. For example, the second record from this batch of two records has a 490 field containing an ISSN:
    <marc:datafield tag="490" ind1="1" ind2=" ">
      <marc:subfield code="a">Jyväskylä studies in humanities,</marc:subfield>
      <marc:subfield code="x">1459-4323 ;</marc:subfield>
      <marc:subfield code="v">187</marc:subfield>
    </marc:datafield>

The ISSN gets converted into this BIBFRAME2 RDF/XML:
        <bf:identifiedBy>
          <bf:Issn>
            <rdf:value>1459-4323 ;</rdf:value>
          </bf:Issn>
        </bf:identifiedBy>

Notice that the rdf:value still contains the space and semicolon. I would expect these to be stripped as they are not really part of the ISSN code.

We will address this in our next update. Thanks!


Fix query appending latest to all urls

Description
Currently when switching tabs you will see latest appended to all urls strings
Our nextjs "rewrite" strategy is making it so that if you try to navigate to:
http://localhost:3000/assets/carbon-components/latest/accordion/design
it is getting redirected to:
http://localhost:3000/assets/carbon-components/latest/accordion/design/latest
Steps to reproduce
Nav between the different tabs:
http://localhost:3000/assets/carbon-components/latest/accordion#glance

Hey team! Please add your planning poker estimate with ZenHub @alisonjoseph @andreancardona @francinelucca @jdharvey-ibm

Could be fixed by blocking issue?


New UI can't display Chinese

The old UI can display Chinese,but the new one can't display Chinese even if I add '-language schinese'as a Startup item.
Could you solve the problem?
I use LINUX.

Hello @RyuueikenLiu, this issue is being tracked at https://github.com/ValveSoftware/csgo-osx-linux/issues/1730. Please use the csgo-osx-linux issue tracker for issues with the CS:GO game client.
Closing as a duplicate.


Develop

Hi Jason,
is working for me with:
client = DataSift::Client.new({ username: CONFIG["datasift_user_name"], api_key: CONFIG["datasift_api_key"], enable_ssl: true })
client.ingestion_service.ingest(source_id, File.open("list_of_hashes").read)

Thanks for the PR @giovannelli. We'll need some time to think about issues such as the best way to handle using the unversioned API, and the change of api_host for calls to the Ingestion API.

Ok, I'll use mine waiting for your release. However if you give me some hints on how you want to manage the api_host and api_version I'll be glad to do it.
Best

Superseded by #78


why not pylance show torch.sum 's docstring?

pytorch write most of their code in cpp, with many add_docstr(func, "some text") calls in torch/_torch_docs.py to supply enough docstring for their cpp function's usage, which could be print use help() function.
For example, as the picture show, I'd like to review torch.sum 's docstring, however pylance only show some stub hints, is there any method show the docstring in pylance?

that's because the pyi shipped with torch doesn't have doc string in them and we can't get those doc strings from native modules

since torch ship with its own pyi stubs, we can't do what we do for pandas. we either need to invent some new mechanism only for doc string or need to ask torch team to add doc string on their pyi stubs even if that is not the standard.

thanks for you explanation


A bit of spit and polish

A few things done

Mainly trying to address #32. Figured out it failed to install due to still keeping .pm extension, although I'm not really sure if that's the case. Not bad do change it anyway. There might other reasons why zef does not find it but it really beats me.
While I was at it, found #26 and #27. Once I got that working, tests failed because, well, they were incorrect. They are correct now, and don't fail. PR by @zoffixznet has been borgified into this PR too.

Since this is released directly in the ecosystem, do you want me to bump up the version?

I guess  :-)


ARROW-6564: [Python] Do not require pandas for invoking ChunkedArray.array

Follow-up on https://github.com/apache/arrow/pull/5402, to apply the same fix for ChunkedArray (@wesm was a bit quick with merging :))

@ursabot crossbow submit docker-python-3.6-nopandas

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@7f2d637). Click here to learn what that means.
The diff coverage is 9.09%.

@@            Coverage Diff            @@
##             master    #5497   +/-   ##
=========================================
  Coverage          ?   66.32%
=========================================
  Files             ?      508
  Lines             ?    70139
  Branches          ?        0
=========================================
  Hits              ?    46518
  Misses            ?    23621
  Partials          ?        0

Impacted Files
Coverage Δ

python/pyarrow/table.pxi
86.01% <0%> (ø)

python/pyarrow/tests/test_table.py
96.5% <16.66%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7f2d637...e6f5adc. Read the comment docs.


Fix typo in xsimd_avx512cd_register

Wow, this means we don't validate that as we should... let me investigate CI

I've applied your patch on top of #843 . This does solve some issue, but with proper testing, some new issues are unveiled :-) I think we can close this PR and keep fixing the other bug in #843


Add bearer token option (--token) to cli

This allows access to Google DICOMweb stores
as shown in the new cli example.

I recently added support for streaming data using chunked transfer encoding (see branch "feature/chunked-transfer-encoding"). As part of this effort, I also refactored the command line interface, such that the client is constructed in the main() function and then passed to other functions:
https://github.com/MGHComputationalPathology/dicomweb-client/blob/462f6a587af88bc220313f693edcb2570fc5373c/src/dicomweb_client/cli.py#L638-L645
This avoids having to construct the client in each function separately with the same set of passed arguments.
You may want to consider re-basing your changes onto this feature branch instead of the current master branch.

Probably it's easiest to wait until you merge your chunked transfer branch into master and then I'll rebase.  I do like the idea of unifying the client creation code.

Probably it's easiest to wait until you merge your chunked transfer branch into master and then I'll rebase. I do like the idea of unifying the client creation code.

I have already tested the chunked transfer branch with a dcm4chee-arc-light origin server but would like to add a couple of unit tests before merging it into master.
In addition, I would like to refactor the code a bit under the hood to allow streaming data sets directly to and from files on disk. However, this could wait and get addressed with a subsequent release.

@pieper would you mind testing the chunked transfer branch against the Google Healthcare API?

It was easier just to create a new PR.


Merge MONDO:0033004 and MONDO:0033004

Mondo term (ID and Label) MONDO:0033004 - polycystic kidney disease 4
Reason for deprecation Links to OMIM:263200 / PKHD1
Term to be merged with MONDO:0009889 - autosomal recessive polycystic kidney disease

Thank you!
I think there is a reason for not merging, AR is not equivalent to PKD4
https://omim.org/phenotypicSeries/PS173900
PKD5 is also AR, which justifies keeping the grouping
@nicolevasilevsky can you assign PKD5 as AR thanks

What is the motivation for deletion? There is a separate OMIM entry for PKD4 (263200) and PKD1 (173900)?

can you assign PKD5 as AR thanks
done on PR


fix jdkDate parser

mongo store Date object instead of String at databse
-related: #360

Signed-off-by: OscarQQ<EMAIL_ADDRESS>

Could you resolve the merge conflicts? Thanks

updated :)


Introduce LifecycleAwareProject

Part of https://github.com/gradle/gradle/issues/29187
This PR introduces an another inheritor of MutableStateAccessAwareProject inheritor - LifecycleAwareProject. This project become a wrapper of DefaultProject after any cross project access. During cross project access target project may be configured eagerly and we want to make sure appropriate lifecycle callback be invoked before.
Note, that similar wrapping is happening during crossproject access in IP mode, so in IP mode final structure is ProblemReportingProject(LifecycleAwareProject(DefaultProject)).
Since LifecycleAwareProject is using in a vintage mode, it highlighted more state-bearing nuances in Groovy, that wasn't considered in ProblemReportingProject in this PR:
As soon as these wrappers aren't persistent, they can't bear any state, because it will be dropped. But with Groovy it's not so easy to achieve it - in Groovy properties can be added to this bean without delegation, and we handle it with missing callbacks.
However, for hasProperty there is no support as a for getProperty/setProperty/invokeMethod. Conceptually, it means
val a = DefaultProject()
a.setProperty("foo", "bar")
val b = LifecycleAwareProject(a)
b.hasProperty("foo") // false

This PR solves the issue with special DynamicObject implementation, that delegates hasProperty invocations in the same manner this has property? -> no! mutable state accessed -> return delegate has property.

@bot-gradle test without pts

I've triggered the following builds with parameters: -DenablePredictiveTestSelection=false for you. Click here to see all build failures.

PullRequestFeedback build

@bot-gradle test this

@bot-gradle test this

I've triggered the following builds for you. Click here to see all build failures.

PullRequestFeedback build

@bot-gradle test without PTS

I've triggered the following builds with parameters: -DenablePredictiveTestSelection=false for you. Click here to see all build failures.

Pull Request Feedback (Trigger) (Check)


Accessibility: aria-label doesn't seem configurable

Hi there. Is there any way configure the aria-label of the input element? At the moment it is set to aria-label="Select date". For screen readers this overrides the label associated with the input via the "for" attribute. It would be good if the aria-label was optional and the text configurable.

Hi @dan-mu, that particular value is controller by the placeholder props. Specifically, it's startDatePlaceholderText/endDatePlaceholderText for the DRP and just placeholder for the SDP. You can set it to whatever you like, including probably undefined which would remove the tag.
Most other text you can change by overriding the values in the phrases prop. Something like:
import { DateRangePickerPhrases } from 'react-dates/lib/defaultPhrases';
...
const customPhrases = {
  ...DateRangePickerPhrases,
  calendarLabel: 'custom calendar label text',
};
return (
  <DateRangePicker
    phrases={customPhrases}
    ...
  />
);

Thanks majapw.
Does the placeholder text displayed have to be the same as the aria-label? It would be great if they could be individually set. E.g. Set the placeholder text displayed in the field to be "Select date" and the aria-label to be something else or blank. The purpose of aria-label is to make the screenreader read out extra guidance for those with visual impairments. This simple feature would vastly improve the accessibility of the tool.
Cheers,
Dan

@dan-mu that's the purpose of the placeholder too; it actually shouldn't be "select date" - that's something that should go in a label - it should be example input, and the identical input is what should be read out to VO users as well.

What is the status of this issue? I agree with @dan-mu, the placeholder shouldn't be the aria-label, and it should be possible to set them separately. The actual label is not read anymore by the screen reader and it just reads the description in my case: (MM-DD-YYYY). Which is not descriptive. According to https://www.w3.org/TR/wai-aria/#aria-labelledby

If the label text is visible on screen, authors SHOULD use aria-labelledby and SHOULD NOT use aria-label.

I'm also running into problems with the SingleDatePicker because of this issue. The ariaLabel prop that is provided doesn't seem to be respected. The placeholder text always takes precedent and the text you pass to ariaLabel always seems to be ignored.
https://github.com/airbnb/react-dates/blob/master/src/components/DateInput.jsx#L223


Minor command line app bug fixes

A few avdecccmdline app annoyances fixed here. Most significant commit is: https://github.com/ahogen/avdecc-lib/commit/43dd87e7cd93f722a127df25942c3e84129b57c6 where some command line segfaults were (based on my limited testing) fixed.
Critiques and comments welcome.

Needs to go to staging, not master... my bad.


Posts & Pages stats do not expand

Clicking on an individual page/post doesn’t go to the expanded stats for that particular page/post. It just goes nowhere.

In the screenshot, for instance, clicking “On the Importance…” and “Happy Birthday, Dad” does nothing.
Furthermore, if I click “View All” to see the full list, clicking on any of the individual pages/posts now takes me to the page/post itself, rather than the expanded stats.
Reported by @dennishhong

@kwonye is this on WPiOS 5.8 or 5.9 internal?

Actually this is broken for both builds - it's fixed with #362 but I haven't added the fixes yet to the WPiOS build.


Fail to build python script from nixpkgs as appimage

Hello, i used nix-appimage in the past and it was a joy! However i'm now failing when trying to build a weasyprint appimage. Here's the output:
$ nix bundle --bundler github:ralismark/nix-appimage nixpkgs#python311Packages.weasyprint
error:
       … while calling the 'derivationStrict' builtin

         at //builtin/derivation.nix:9:12: (source not available)

       … while evaluating derivation 'python3.11-weasyprint-59.0-x86_64.AppImage'
         whose name attribute is located at /nix/store/d5c6h5p16jg2rna7db5y4s3y19dmwvx6-source/pkgs/stdenv/generic/make-derivation.nix:278:7

       … while evaluating attribute 'buildCommand' of derivation 'python3.11-weasyprint-59.0-x86_64.AppImage'

         at /nix/store/d5c6h5p16jg2rna7db5y4s3y19dmwvx6-source/pkgs/build-support/trivial-builders.nix:73:14:

           72|     stdenv.mkDerivation ({
           73|       inherit buildCommand name;
             |              ^
           74|       passAsFile = [ "buildCommand" ]

       error: main program /nix/store/1z9djg8r09lfpri1mlzf2cbqi8hsmr19-python3.11-weasyprint-59.0/bin/python3.11-weasyprint does not exist

I guess it's because the executable is called weasyprint and not python3.11-weasyprint. I think it's related to #3. Alternatively, is there something to fix on the package itself to make that possible?
EDIT: Manually creating a symlink in the nix store (yes that's evil!) made it work. I'll use that workaround for the moment but it's not elegant ;)

The least hacky fix right now is to set the package's meta.mainProgram to the right path e.g. "/bin/weasyprint". But also yeah, doing this is annoying and not really possible for some packages, which is why I've been meaning to do #3.

I added ./bundle so this should be possible now:
$ ./bundle python311Packages.weasyprint weasyprint
$ ./weasyprint-x86_64.AppImage --version
WeasyPrint version 58.1

Let me know if this is still an issue.


Dockerfile for image creation

My current Dockerfile creates an image that is ~2GB on my machine, which is certainly better than the 7GB original take but still not nearly small enough (if it can be helped). It does not even include serving the frontend static content 😢
Ideally I just need an image that has the server executable to run and the static content to serve. I use sqlite, but ideally would mount a directory on the container to one on the host machine? I am not a docker wizard yet
Current server/Dockerfile (at time of writing):
FROM rust:1-alpine3.15
# https://github.com/rust-lang/docker-rust/issues/85
ENV RUSTFLAGS="-C target-feature=-crt-static"

RUN apk add --no-cache --verbose musl-dev build-base sqlite openssl-dev

WORKDIR /app/server
COPY ./src /app/server/src
COPY ./Cargo.lock /app/server/Cargo.lock
COPY ./Cargo.toml /app/server/Cargo.toml
COPY ./Rocket.toml /app/server/Rocket.toml
# TODO: what to do with frontend?

RUN cargo build --release
RUN strip target/release/stump

I've gotten it down to 39 MB(!), I still think this might be able to be optimized so I'll leave this issue open for the time being

is there any docker compose available for stump?

is there any docker compose available for stump?

When Stump has its first release I will be pushing a Docker image to Docker Hub, but until then unfortunately no. Docker images are for development only until that point, if you're interested in you can build it locally and follow relevant guide, but keep in mind Stump is still in pretty early stages (i.e. not all of the core features are completed or accessible).


Update Module Name to support WMF 5

Update Module Name to support WMF 5
WMF 5 doesn't support resources being named the same as a module.

Hi @TravisEz13, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

It looks like you're a Microsoft contributor (Travis Plunk). If you're full-time, we DON'T require a Contribution License Agreement. If you are a vendor, please DO sign the electronic Contribution License Agreement. It will take 2 minutes and there's no faxing! https://cla.microsoft.com.

TTYL, MSBOT;

overall looks fine too me, comments can be addressed later

looks good


Error codes not documented

The error codes are not documented within the open api specification.

There is a seperate documentation that includes some of the error codes. Example here : https://help.internetx.com/display/APIXMLEN/Domain+Create
I think The Open API documentation is not 100% suitable for such large APIs to document everything to the smallest detail. Therefore a separate documentation with extended information exists.

Just come across this issue again this time searching for another undocumented error code: "E0202002"
status code=400: {"stid":"20210326-app4-169456","status":{"code":"E0202002","text":"Zone konnte auf dem Nameserver nicht aktualisiert werden.","type":"ERROR"}}


[SPIKE] Add featureEnable attribute to generic addon plugin

Currently the generic addon plugin only installs the addon if the addon is already enabled on the hub. We want to add an option to enable (/disable?) features on the hub from the generic addon plugin.

First PR: https://github.com/stolostron/ocmplus.cm/pull/89

@hanqiuzh can you please update with recent improvements?

Currently refactoring to reflect new MCE and MCH API changes.

Complexities:

Waiting on completion of API rewrite from ACM's Install team (timeline independent from ours)

changes merged. closing


Forcing SSL for DeviseTokenAuth causes error 'new' could not be found

I am trying to enable ssl for authentication and registration.  I add the following to the production.rb environment to force ssl for the DeviseTokenAuth classes but get this error.  The full call stack is at the bottom. I'm guessing this is an error with my setup versus devise_token_auth but any help would be appreciated.
AbstractController::ActionNotFound (The action 'new' could not be found for DeviseTokenAuth::SessionsController):
environments/production.rb
config.to_prepare { DeviseTokenAuth::SessionsController.force_ssl }
config.to_prepare { DeviseTokenAuth::RegistrationsController.force_ssl }
config.to_prepare { DeviseTokenAuth::TokenValidationsController.force_ssl }

production.log
I, [2015-02-08T18:32:31.032943 #6890]  INFO -- : Started POST "/api/v1/auth/sign_in" for <IP_ADDRESS> at 2015-02-08 18:32:31 -0500
I, [2015-02-08T18:32:31.036738 #6890]  INFO -- : Processing by DeviseTokenAuth::SessionsController#create as JSON
I, [2015-02-08T18:32:31.036969 #6890]  INFO -- :   Parameters<EMAIL_ADDRESS>"password"=>"[FILTERED]", "remembe
r_me"=>0, "redirect_url"=>"http://savethesparkles.com/#/"<EMAIL_ADDRESS>"password"=>"[FILTERED]", "
remember_me"=>0, "redirect_url"=>"http://savethesparkles.com/#/"}}
I, [2015-02-08T18:32:31.038020 #6890]  INFO -- : Redirected to https://savethesparkles.com/api/v1/auth/sign_in
I, [2015-02-08T18:32:31.038269 #6890]  INFO -- : Filter chain halted as #<Proc:0x00000003a75168@/home/ubuntu/sandbox/sparkle/shared/bun
dle/ruby/1.9.1/gems/actionpack-4.1.4/lib/action_controller/metal/force_ssl.rb:65> rendered or redirected
I, [2015-02-08T18:32:31.038579 #6890]  INFO -- : Completed 301 Moved Permanently in 1ms (ActiveRecord: 0.0ms)
I, [2015-02-08T18:32:31.096425 #6890]  INFO -- : Started GET "/api/v1/auth/sign_in" for <IP_ADDRESS> at 2015-02-08 18:32:31 -0500
F, [2015-02-08T18:32:31.098640 #6890] FATAL -- :
AbstractController::ActionNotFound (The action 'new' could not be found for DeviseTokenAuth::SessionsController):
  actionpack (4.1.4) lib/abstract_controller/base.rb:131:in `process'
  actionview (4.1.4) lib/action_view/rendering.rb:30:in `process'
  actionpack (4.1.4) lib/action_controller/metal.rb:196:in `dispatch'
  actionpack (4.1.4) lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
  actionpack (4.1.4) lib/action_controller/metal.rb:232:in `block in action'
  actionpack (4.1.4) lib/action_dispatch/routing/route_set.rb:82:in `call'
  actionpack (4.1.4) lib/action_dispatch/routing/route_set.rb:82:in `dispatch'
  actionpack (4.1.4) lib/action_dispatch/routing/route_set.rb:50:in `call'
  actionpack (4.1.4) lib/action_dispatch/routing/mapper.rb:45:in `call'
  actionpack (4.1.4) lib/action_dispatch/journey/router.rb:71:in `block in call'
  actionpack (4.1.4) lib/action_dispatch/journey/router.rb:59:in `each'
  actionpack (4.1.4) lib/action_dispatch/journey/router.rb:59:in `call'

This gem doesn't use the new action.
It looks like force_ssl accepts an except param (docs are here). Can you try passing except: :new as an argument to the force_ssl method?

@lynndylanhurley why does mount_devise_token_auth_for write a route for the new action?

https://github.com/lynndylanhurley/devise_token_auth#why-are-the-new-routes-included-if-this-gem-doesnt-use-them
@milesmatthias - removing the new route would be a large, sustained effort for a near-zero benefit.

@lynndylanhurley I will try and get back to this this week to answer your question.  As a workaround I simply used ssl for all routes.
config.force_ssl = true

@lynndylanhurley that's cool, but instead of raising an exception when you get /auth/sign_in, shouldn't you return a message saying something like "Use post /auth/sign_in to sign in."
Having it raise an exception just makes the gem seem unfinished and doesn't guide the developer to what they should be doing with the unhelpful exception that gets raised.
I'll open a separate issue and send you a PR since this is slightly different than what this ticket talks about.

Hi there @sburke56 and @milesmatthias   ,
In an effort to cleanup this project and prioritize a bit, we're marking issues that haven't had any activity in a while with a "close-in-7-days" label. If we don't hear from you in about a week, we'll be closing this issue. Obviously feel free to re-open it at any time if it's the right time or this was done in error!
If you are still having the issue (especially if it's a bug report) please refer to our new Issue Template to provide some more details to help us solve it.
Hope all is well.


WIP: Allow window to be maximized

Allow window to be maximized

Btw, amazing project!!! I love it!!

This is amazing!!!! I'd love to have it merged. Though the moving the windows around thing is gonna be fixed. Overall great work!! Thank you for the help!

@PuruVJ now it gets fixed. Though it's a little bit hacky :p

Its perfect!!! Will review and merge soon.

The code looks really good to me. I wouldn't have been able to write this complex code. Great work!!
Thank you for contributing!

Hi, I'm sorry, I had to revert it back. Seems it didn't have the latest codebase when you started working on this PR. WOuld you mind doing a git pull origin main, and then making a PR? Really sorry for the inconvenience 😅😅

never mind, I got confused. Its perfectly fine