[Python] pyarrow.csv.write_csv crashes when writing tables containing FixedSizeBinaryArray

Describe the bug, including details regarding any error messages, version, and platform.
Suppose we have a large array of binary strings
import numpy as np
import pyarrow as pa
import pyarrow.csv as pcsv

nparr = np.frombuffer(np.random.randint(65, 91, int(4E8), 'u1'), 'S4')

We want to construct a FixedSizedBinaryArray and Table
fixedarr = pa.array(nparr, pa.binary(4))
fixedtable = pa.Table.from_arrays([fixedarr], names=['fixedsize'])

Alternatively, construct a BinaryArray and Table
binarr = pa.array(nparr, pa.binary())
bintable = pa.Table.from_arrays([binarr], names=['binary'])

Setting up the csv write option
csvoption = pcsv.WriteOptions(include_header=False, batch_size=2048, delimiter='|', quoting_style='none')

Writing the table containing BinaryArray works
pcsv.write_csv(bintable, 'binary.csv', write_options=csvoption)

But writing the table containing FixedSizedBinaryArray crashes after first batch_size
pcsv.write_csv(fixedtable, 'fixedsize.csv', write_options=csvoption)

Note both tables are compatible with pyarrow.parquet.write_table though.
Environment:
Windows 10
pyarrow 12.0.0
Python 3.10.10
Component(s)
Python

This appears to be a bug in the cast function that goes from FixedSizeBinaryArray to StringArray:
In BinaryToBinaryCastExec in scalar_cast_string.cc we have:
  if (input.offset == output->offset) {
    output->buffers[0] = input.GetBuffer(0);
  } else {
    ARROW_ASSIGN_OR_RAISE(
        output->buffers[0],
        arrow::internal::CopyBitmap(ctx->memory_pool(), input.buffers[0].data,
                                    input.offset, input.length));
  }

Unfortunately, input.buffers[0].data might be nullptr (since we can omit the validity bitmap if everything is valid) and that causes CopyBitmap to fail.
A full stack trace is:
#0  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:317
#1  0x00007fff711f26e7 in arrow::internal::TransferBitmap<(arrow::internal::TransferMode)0> (data=0x100 <error: Cannot access memory at address 0x100>, offset=2048, length=2048, dest_offset=0,
    dest=0x5555566193c0 "") at /home/pace/dev/arrow/cpp/src/arrow/util/bitmap_ops.cc:163
#2  0x00007fff711f2c0f in arrow::internal::TransferBitmap<(arrow::internal::TransferMode)0> (pool=0x7fff73311508 <arrow::global_state+8>, data=0x0, offset=2048, length=2048)
    at /home/pace/dev/arrow/cpp/src/arrow/util/bitmap_ops.cc:222
#3  0x00007fff711ee979 in arrow::internal::CopyBitmap (pool=0x7fff73311508 <arrow::global_state+8>, data=0x0, offset=2048, length=2048) at /home/pace/dev/arrow/cpp/src/arrow/util/bitmap_ops.cc:252
#4  0x00007fff7165b1c2 in arrow::compute::internal::(anonymous namespace)::BinaryToBinaryCastExec<arrow::StringType, arrow::FixedSizeBinaryType> (ctx=0x555556670288, batch=..., out=0x7fffffffc710)
    at /home/pace/dev/arrow/cpp/src/arrow/compute/kernels/scalar_cast_string.cc:341
#5  0x00007fff715369a1 in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::ExecuteNonSpans (this=0x5555564a37d0, listener=0x7fffffffc930) at /home/pace/dev/arrow/cpp/src/arrow/compute/exec.cc:920
#6  0x00007fff7153551c in arrow::compute::detail::(anonymous namespace)::ScalarExecutor::Execute (this=0x5555564a37d0, batch=..., listener=0x7fffffffc930)
    at /home/pace/dev/arrow/cpp/src/arrow/compute/exec.cc:810
#7  0x00007fff7157e88b in arrow::compute::detail::FunctionExecutorImpl::Execute (this=0x555556670260, args=std::vector of length 1, capacity 1 = {...}, passed_length=-1)
    at /home/pace/dev/arrow/cpp/src/arrow/compute/function.cc:276
#8  0x00007fff7157be03 in arrow::compute::(anonymous namespace)::ExecuteInternal (func=..., args=std::vector of length 1, capacity 1 = {...}, passed_length=-1, options=0x7fffffffcea0, ctx=0x7fffffffd020)
    at /home/pace/dev/arrow/cpp/src/arrow/compute/function.cc:341
#9  0x00007fff7157bf9e in arrow::compute::Function::Execute (this=0x5555568088e0, args=std::vector of length 1, capacity 1 = {...}, options=0x7fffffffcea0, ctx=0x7fffffffd020)
    at /home/pace/dev/arrow/cpp/src/arrow/compute/function.cc:348
#10 0x00007fff715229b7 in arrow::compute::internal::(anonymous namespace)::CastMetaFunction::ExecuteImpl (this=0x5555562ae610, args=std::vector of length 1, capacity 1 = {...}, options=0x7fffffffcea0,
    ctx=0x7fffffffd020) at /home/pace/dev/arrow/cpp/src/arrow/compute/cast.cc:124
#11 0x00007fff7157d37e in arrow::compute::MetaFunction::Execute (this=0x5555562ae610, args=std::vector of length 1, capacity 1 = {...}, options=0x7fffffffcea0, ctx=0x7fffffffd020)
    at /home/pace/dev/arrow/cpp/src/arrow/compute/function.cc:481
#12 0x00007fff7153a6ba in arrow::compute::CallFunction (func_name="cast", args=std::vector of length 1, capacity 1 = {...}, options=0x7fffffffcea0, ctx=0x7fffffffd020)
    at /home/pace/dev/arrow/cpp/src/arrow/compute/exec.cc:1369
#13 0x00007fff71523bbc in arrow::compute::Cast (value=..., options=..., ctx=0x7fffffffd020) at /home/pace/dev/arrow/cpp/src/arrow/compute/cast.cc:229
#14 0x00007fff71523d32 in arrow::compute::Cast (value=..., to_type=..., options=..., ctx=0x7fffffffd020) at /home/pace/dev/arrow/cpp/src/arrow/compute/cast.cc:236
#15 0x00007fff71523dfe in arrow::compute::Cast (value=..., to_type=..., options=..., ctx=0x7fffffffd020) at /home/pace/dev/arrow/cpp/src/arrow/compute/cast.cc:241
#16 0x00007fff71445a0d in arrow::csv::(anonymous namespace)::ColumnPopulator::UpdateRowLengths (this=0x55555673e540, data=..., row_lengths=0x555556a25bc0) at /home/pace/dev/arrow/cpp/src/arrow/csv/writer.cc:131
#17 0x00007fff714498bb in arrow::csv::(anonymous namespace)::CSVWriterImpl::TranslateMinimalBatch (this=0x555556642100, batch=...) at /home/pace/dev/arrow/cpp/src/arrow/csv/writer.cc:561
#18 0x00007fff7144888b in arrow::csv::(anonymous namespace)::CSVWriterImpl::WriteTable (this=0x555556642100, table=..., max_chunksize=-1) at /home/pace/dev/arrow/cpp/src/arrow/csv/writer.cc:485
#19 0x00007fff726d513a in arrow::ipc::RecordBatchWriter::WriteTable (this=0x555556642100, table=...) at /home/pace/dev/arrow/cpp/src/arrow/ipc/writer.cc:1030
#20 0x00007fff7144a028 in arrow::csv::WriteCSV (table=..., options=..., output=0x555556141990) at /home/pace/dev/arrow/cpp/src/arrow/csv/writer.cc:609
#21 0x00007fff6e61b5ae in __pyx_pw_7pyarrow_4_csv_7write_csv(_object*, _object* const*, long, _object*) () from /home/pace/dev/arrow/python/pyarrow/_csv.cpython-311-x86_64-linux-gnu.so
#22 0x000055555574bec1 in _PyObject_VectorcallTstate (kwnames=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=0x7fff6e759e50, tstate=0x555555ae0d98 <_PyRuntime+166328>)
    at /usr/local/src/conda/python-3.11.0/Include/internal/pycore_call.h:92
#23 PyObject_Vectorcall (callable=0x7fff6e759e50, args=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.11.0/Objects/call.c:299
#24 0x000055555573ec6c in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.11.0/Python/ceval.c:4772
#25 0x000055555573a1fb in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fb0020, tstate=0x555555ae0d98 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.0/Include/internal/pycore_ceval.h:73
#26 _PyEval_Vector (tstate=0x555555ae0d98 <_PyRuntime+166328>, func=0x7ffff6dd1f80, locals=<optimized out>, args=0x0, argcount=<optimized out>, kwnames=<optimized out>)
    at /usr/local/src/conda/python-3.11.0/Python/ceval.c:6428
#27 0x00005555558049af in PyEval_EvalCode (co=<optimized out>, globals=0x7ffff6df2d40, locals=<optimized out>) at /usr/local/src/conda/python-3.11.0/Python/ceval.c:1154
#28 0x0000555555827479 in run_eval_code_obj (tstate=0x555555ae0d98 <_PyRuntime+166328>, co=0x555555bb2900, globals=0x7ffff6df2d40, locals=0x7ffff6df2d40)
    at /usr/local/src/conda/python-3.11.0/Python/pythonrun.c:1714
#29 0x0000555555823724 in run_mod (mod=<optimized out>, filename=<optimized out>, globals=0x7ffff6df2d40, locals=0x7ffff6df2d40, flags=<optimized out>, arena=<optimized out>)
    at /usr/local/src/conda/python-3.11.0/Python/pythonrun.c:1735
#30 0x00005555558387d2 in pyrun_file (fp=fp@entry=0x555555b48c10, filename=filename@entry=0x7ffff6d53270, start=start@entry=257, globals=globals@entry=0x7ffff6df2d40, locals=locals@entry=0x7ffff6df2d40,
    closeit=closeit@entry=1, flags=0x7fffffffd888) at /usr/local/src/conda/python-3.11.0/Python/pythonrun.c:1630
#31 0x000055555583812f in _PyRun_SimpleFileObject (fp=0x555555b48c10, filename=0x7ffff6d53270, closeit=1, flags=0x7fffffffd888) at /usr/local/src/conda/python-3.11.0/Python/pythonrun.c:440
#32 0x0000555555837f03 in _PyRun_AnyFileObject (fp=0x555555b48c10, filename=0x7ffff6d53270, closeit=1, flags=0x7fffffffd888) at /usr/local/src/conda/python-3.11.0/Python/pythonrun.c:79
#33 0x000055555583206c in pymain_run_file_obj (skip_source_first_line=0, filename=0x7ffff6d53270, program_name=0x7ffff6c1f690) at /usr/local/src/conda/python-3.11.0/Modules/main.c:360
#34 pymain_run_file (config=0x555555ac6de0 <_PyRuntime+59904>) at /usr/local/src/conda/python-3.11.0/Modules/main.c:379
#35 pymain_run_python (exitcode=0x7fffffffd880) at /usr/local/src/conda/python-3.11.0/Modules/main.c:601
#36 Py_RunMain () at /usr/local/src/conda/python-3.11.0/Modules/main.c:680
#37 0x00005555557f33f9 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /usr/local/src/conda/python-3.11.0/Modules/main.c:734
#38 0x00007ffff7c29d90 in __libc_start_call_main (main=main@entry=0x5555557f3350 <main>, argc=argc@entry=2, argv=argv@entry=0x7fffffffdad8) at ../sysdeps/nptl/libc_start_call_main.h:58
#39 0x00007ffff7c29e40 in __libc_start_main_impl (main=0x5555557f3350 <main>, argc=2, argv=0x7fffffffdad8, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffdac8)
    at ../csu/libc-start.c:392
#40 0x00005555557f32a1 in _start ()

@pitrou @AlenkaF @jorisvandenbossche would appreciate if a Python maintainer can pick up this issue. It would be a bit difficult for me to find time to solve this issue.

@anjakefala I don't know if you would have time to pick this up, since you already worked on the original PR?

@pitrou Hmmmm, I don't have a lot of time, but let's see if I can get it over the finish line. It feels too bad to just close things when we were so close.


Add a quick settings tile to toggle Voice Notify

Adds a quick settings tile (TileService) that displays the current state of Voice Notify. Clicking on the tile toggles Voice Notify. Long-pressing the tile opens the main screen of Voice Notify.
Disclaimer: I'm very new to Android development and Kotlin, please let me know if there's anything I can do to improve the code.

Thanks! I hadn't even thought of that since the widget is sufficient for how I use it. It's good to offer more ways to control it.

Yeah, I tend to do things more with quick settings than widgets 😁 Thanks for merging, and thank you for creating and maintaining this project, it's been very useful!


cookies header missing in network console

when you "edit and resend" a network request any cookie info is missing from the header tab.
closing the request tab in the console also leads to weird issue trying to edit and resend again, so it feels like this part of the devTools is nobody really using?
not sure if this relates to:
https://github.com/microsoft/edge-devtools-network-console/  which looks stale..?!
AB#48956223

thank you for the update. I noticed this resend option is (afaik) not available in Chrome.

There are other ways to edit and resend requests across browsers, see https://devtoolstips.org/tips/en/edit-and-resend-network-requests/
You can also replay XHR requests: https://devtoolstips.org/tips/en/replay-xhr/
But, yes, you're right, a dedicated tool for editing (and managing) requests is only available in Edge currently.
You can, however, use Postman to achieve the same thing.
Again, the Network Console is only an experiment, which we've not been able to convert into an official feature of Edge DevTools.

thanks for the nice links. I use Fiddler for many many years but having a basic replay in the browser is so nice+conveniant.


[pipelines] add regression analysis and KPIs

@kpouget Definitely! Let me add a gating_smoke preset that's more suitable for this. I have a feeling a few things are probably missing, but I think most of the coding is done

/test rhoai-e2e pipelines gating_smoke

/test rhoai-e2e pipelines gating_smoke

hum, the test failed because of this line
                    all_workflow_start_times[f"{name}"] = datetime.datetime.strptime(
                        root_node["startedAt"], helpers_store_parsers.K8S_TIME_FMT)

because the workflow never started I guess :/

/test rhoai-light pipelines gating_smoke

looks good @drewrip , next test shoud have a regression report :)

/test rhoai-e2e pipelines gating_smoke

Test of 'rhoai test test_ci' failed after 01 hours 08 minutes 10 seconds.

Link to the test results.
Link to the reports index.

Test configuration:
# RHOAI: run pipelines test test_ci
PR_POSITIONAL_ARGS=gating_smoke
PR_POSITIONAL_ARG_0=pipelines-rhoai-e2e
PR_POSITIONAL_ARG_1=gating_smoke

[TOPSAIL auto-generated message]

hum, the test failed because of this line
                    all_workflow_start_times[f"{name}"] = datetime.datetime.strptime(
                        root_node["startedAt"], helpers_store_parsers.K8S_TIME_FMT)

because the workflow never started I guess :/

looks like the runs in OpenShift CI are failing because of this, but tests in Jenkins don't show this issue. The pods that make up the Workflows (the actual pipeline pods) are stuck pending. I need to see why

for the Pending status, you need to look at the Pod describe:
  Warning  FailedScheduling  26m (x19 over 59m)  default-scheduler  0/7 nodes are available: 1 Insufficient cpu, 1 node(s) had untolerated taint {only-test-pods: yes}, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/7 nodes are available: 1 No preemption victims found for incoming pod, 6 Preemption is not helpful for scheduling.

--> there's no CPU available
I guess that's because there's only 1 m6i.2xlarge machine
NAME                    PHASE     TYPE          REGION      ZONE         AGE
rhods-pipelines-d4czv   Running   m6i.2xlarge   us-west-2   us-west-2a   5m9s

I'm not sure anymore how this 1 machine is configured. In the notebook scale test, I had a computation that was taking into account the number of users and the requested size per notebook, and the cpu/memory of the instance type.
That was quite reliable to scale up the cluster

for the Pending status, you need to look at the Pod describe:
  Warning  FailedScheduling  26m (x19 over 59m)  default-scheduler  0/7 nodes are available: 1 Insufficient cpu, 1 node(s) had untolerated taint {only-test-pods: yes}, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/7 nodes are available: 1 No preemption victims found for incoming pod, 6 Preemption is not helpful for scheduling.

--> there's no CPU available
I guess that's because there's only 1 m6i.2xlarge machine
NAME                    PHASE     TYPE          REGION      ZONE         AGE
rhods-pipelines-d4czv   Running   m6i.2xlarge   us-west-2   us-west-2a   5m9s

I'm not sure anymore how this 1 machine is configured. In the notebook scale test, I had a computation that was taking into account the number of users and the requested size per notebook, and the cpu/memory of the instance type. That was quite reliable to scale up the cluster

Yeah this hasn't been something I've tuned yet, so whatever the previous instance requirements were are still there. We ran into this a bit during the DSP tests during the last release.  What I've just realized is that the per user resource requirements are likely much larger for DSP than notebooks. It is likely also more difficult to estimate CPU and memory requirements just based on user_count. Each user starts a pipeline that attempts to spawn an arbitrary number of pods as part of the workflow. So the number of pods, and the corresponding resources needed to accommodate them, depends on both the user_count and each specific pipeline topology.

/test rhoai-e2e pipelines gating_smoke

Test of 'rhoai test test_ci' succeeded after 00 hours 17 minutes 57 seconds.

Link to the test results.
Link to the reports index.

Test configuration:
# RHOAI: run pipelines test test_ci
PR_POSITIONAL_ARGS=gating_smoke
PR_POSITIONAL_ARG_0=pipelines-rhoai-e2e
PR_POSITIONAL_ARG_1=gating_smoke

[TOPSAIL auto-generated message]

@kpouget Looks like increasing the resource requirements associated with each user (for the SUT nodes) fixed the problem. We are getting a nice regression report now :). Good to merge from my end

thanks @drewrip , looks great!
a few minor details:
in the KPI labels,

run_count comes from pipelines_per_use. Why not using  pipelines_per_use as in the config?
runs_per_pipeline is configured, but not exposed as a label. Do you think it won't ever change in CPT?

and, very minor:
results.sleep_factor = int(results.test_config.get("tests.pipelines.sleep_factor"))

when you read from the configuration, the YAML parser already casted to the right type, so no need to cast again to int
(unless you hit an issue around that? 🤔 )

in the KPI labels,
run_count comes from pipelines_per_use. Why not using pipelines_per_use as in the config?
runs_per_pipeline is configured, but not exposed as a label. Do you think it won't ever change in CPT?

That's a good question. Originally I wanted to rename "runs_per_pipeline" to "run_count" in the labels because I thought that was more concise (and outwardly consumable) but I'm just going to keep "runs_per_pipeline" as I don't think its worth the confusion.

and, very minor:
results.sleep_factor = int(results.test_config.get("tests.pipelines.sleep_factor"))
when you read from the configuration, the YAML parser already casted to the right type, so no need to cast again to int
(unless you hit an issue around that? 🤔 )

This was just a personal preference, but I'm happy to remove the casts to attempt to follow the style of the rest of the repo. Having the cast makes it extra clear to myself and my type checker what types should be expected since we annotate types everywhere.

/test rhoai-e2e pipelines gating_smoke

Test of 'rhoai test test_ci' succeeded after 00 hours 14 minutes 43 seconds.

Link to the test results.
Link to the reports index.

Test configuration:
# RHOAI: run pipelines test test_ci
PR_POSITIONAL_ARGS=gating_smoke
PR_POSITIONAL_ARG_0=pipelines-rhoai-e2e
PR_POSITIONAL_ARG_1=gating_smoke

[TOPSAIL auto-generated message]

The labels are more complete now. Let me know what you think. Good to merge from my end

thanks @drewrip , good to merge!
/lgtm
I let you see if you want to squash or keep the commits

@kpouget squashed a few of the commits and rebased, could you give another lgtm?

thanks for the work @drewrip , looking forward to seeing true regression reports :)
/lgtm

/approve

/approve


feature: Merging the Swapping-Windows and Swap-Two-Windows features.

Did you check the docs?

[x] I have read all the docs

Is your feature request related to a problem? Please describe.
I don't see why there should be separate Swapping-Windows and Swap-Two-Windows features.
Describe the solution you would like
I would like the UI/UX to follow Swapping-Windows, and the actual change to happen with Swap-Two-Windows.
Describe alternatives you have considered
For example, I suggest that it would be better to have a function that buffers A Window, which you decide to swap, and swaps it with the buffer of the highlighted window.
In other words, it would be better if the buffers were crossed in the same way as Swapping-Windows, but the buffers were only swapped with the A window that you wanted to move to in the first place, rather than all the buffers of all the windows.
Additional context
No response

Thanks for submitting an issue 🙂
Apologies but I do not understand your issue. Could you draw some example ascii art or try to explain the issue in a different way with a concrete situation?

I don't see why there should be separate Swapping-Windows and Swap-Two-Windows features.

One use case would be if you need to swap a window across multiple tabs. If you couldn't swap two windows you would have to move the window across all tabs and then move the other window the other way assuming you had enabled the option for swapping windows across tabs. In any case, if you don't need the feature then don't use it unless it is preventing you from something?

I'm sorry if this makes you feel bad. I'm not saying it's wrong, it's just a suggestion.

You misunderstand. I don't feel bad about your suggestion, on the contrary, I appreciate you making one 👍

In the case of Swapping windows,
In “A|B|C”, if you move A to C, the end result is “B|C|A”,
I think it should be “C|B|A”.

I'm not sure I agree that end result is very intuitive but this is just my opinion of how swapping (swap mode specifically) should work and it's pretty difficult to convey via github. See below for the step-by-step changes for the different modes. I'm just using modes to explain although there's also a public api.

I'm talking about the examples in the README.

Thanks for the example, I think I understand better now. To clarify, using your example, here is what should happen using the different features of the plugin and wanting to move/swap A over to C (otherwise this might be a bug).

move mode:

Start move mode in window A (A|B|C).
Move A to the right (B|A|C).
Move A to the right (B|C|A).

swap mode:

Start swap mode in window A (A|B|C).
Swap A to the right (B|A|C).
Swap A to the right (B|C|A).

Swapping two windows:

Mark window A for swapping (A|B|C).
Navigate to window C (A|B|C).
Mark window C for swapping (C|B|A).

You cannot really tell the difference between move and swap mode in this example but the former inserts a window into the layout in a given direction (do =vim.fn.winlayout() to see the current window layout) while the former swaps position with itself and the window in a given direction.

For Swap-two-windows, are you saying that you can swap windows between tabs?

Yes. In the configuration section of the README, you can configure the at_edge behaviour of mode and swap mode. One behaviour is moving/swapping across tabs (all behaviours are listed below the lua config code).

If so, doesn't it overlap functionality with Move-between-tabs?

I don't think so since moving a window and swapping a window do different things as I explained above. Hopefully, everything makes sense. If not, please don't hesitate to respond 🙂

Thanks for explaining it well enough.
I'm actually in the middle of creating a similar project like this and find it interesting.
I will support you.