Adding pre-commit checks and fixing pre-commit issues

Adding pre-commit checks in the linting code and changes for pre-commit issues.
Fixing https://github.com/securefederatedai/openfl/issues/311
@psfoley @MasterSkepticista

Adding pre-commit checks in the linting code and changes for pre-commit issues. Fixing #311
@psfoley @MasterSkepticista

What is the mode of usage for this? Can you add instructions on how the development workflow will change (if any)?

Adding pre-commit checks in the linting code and changes for pre-commit issues. Fixing #311
@psfoley @MasterSkepticista

What is the mode of usage for this? Can you add instructions on how the development workflow will change (if any)?

Mode of Usage
Pre-commit is a framework for managing and maintaining multi-language pre-commit hooks. It automates the process of running code quality checks and formatting tasks before committing changes to the repository.
Instructions for Developers

Initial Setup:
a. After cloning the repository, run the following command to set up the pre-commit hooks:
pre-commit install
During Development:
a. Make changes to the code as usual.
b. When committing changes, the pre-commit hooks will run automatically and may prompt you to fix issues before completing the commit.
Running Hooks Manually (if needed):
a. Developers can manually run the hooks on all files using:
pre-commit run --all-files
b. This can be useful to check code quality before committing changes.

b. When committing changes, the pre-commit hooks will run automatically and may prompt you to fix issues before completing the commit.

This seems useful, but is it required to run them? In my mind, this should be optional. It would also be good to clarify in CONTRIBUTING.md what kinds of conventions users should be expected to follow.

b. When committing changes, the pre-commit hooks will run automatically and may prompt you to fix issues before completing the commit.

This seems useful, but is it required to run them? In my mind, this should be optional. It would also be good to clarify in CONTRIBUTING.md what kinds of conventions users should be expected to follow.

In my opinion, pre-commit can be included the shell/lint.sh script since it serves a similar purpose as black or flake8. As for whether the user should be required to run it, it would certainly be a good to have, to maintain a similar code-style throughout the repository.

b. When committing changes, the pre-commit hooks will run automatically and may prompt you to fix issues before completing the commit.

This seems useful, but is it required to run them? In my mind, this should be optional. It would also be good to clarify in CONTRIBUTING.md what kinds of conventions users should be expected to follow.

In my opinion, pre-commit can be included the shell/lint.sh script since it serves a similar purpose as black or flake8. As for whether the user should be required to run it, it would certainly be a good to have, to maintain a similar code-style throughout the repository.

yeah I agree I have taken care in the new commit.

Adding pre-commit checks in the linting code and changes for pre-commit issues. Fixing #311
@psfoley @MasterSkepticista

What is the mode of usage for this? Can you add instructions on how the development workflow will change (if any)?

Mode of Usage Pre-commit is a framework for managing and maintaining multi-language pre-commit hooks. It automates the process of running code quality checks and formatting tasks before committing changes to the repository.
Instructions for Developers

Initial Setup:
a. After cloning the repository, run the following command to set up the pre-commit hooks:
pre-commit install
During Development:
a. Make changes to the code as usual.
b. When committing changes, the pre-commit hooks will run automatically and may prompt you to fix issues before completing the commit.
Running Hooks Manually (if needed):
a. Developers can manually run the hooks on all files using:
pre-commit run --all-files
b. This can be useful to check code quality before committing changes.

Thanks. Request to add this to CONTRIBUTING.md with appropriate formatting and brevity.


More formal process for writing mods and adding them to OL

Something along the lines of the way UE plugins work.  Your plugin is coded to UE plugin standards and you just drop it into the plugin folder.  Then in OL maybe Material Instance you get a list of installed plugins that you can check to enable.  Once checked the particular plugin options are listed.  By having a formal plugin system this would help eliminate users from having to modify the OL material by adding the plugin to the graph.

Using a plugin won't help in this case & it's literally no point in using plugins if we don't use C++ or need additional permission. Instead, it will make it harder for people to contribute. That's a discussion for later for sure.
About adding nodes automatically. I am not 100% sure about that. Because, if you isolate things into a function it will be pretty easy to add into the custom version of the OpenLand material.
Anyway, I am planning to experiment with the Unreal Engine Scripting UI. So, we can try to add(and manage) these mods automatically to the project.


Replace textual Sierra representation with JSON serialization of the Sierra struct

Currently, when building a [lib] target we rely on sierra textual representation. This representation can then be used as an intermediate artifact, e.g. for cairo-run extension. This imposes a need to rely on the textual representation parser which is deemed to not be production grade, which may induce hard to trace errors. Instead we should serialize the sierra program to some serde format, that can serve as a "runnable" representation.

Please verify if this issue still persists after migration: https://github.com/software-mansion/scarb/issues/495


[BPHH-1982] CSRF Token Fix

Description
…in CSRF protection so that we would be safe from those attacks. This change makes the backend set a CSRF-TOKEN cookie which is then read on the requestTransform in the frontend and set so that we can maintain security
What type of PR is this? (check all applicable)

[x] 🍕 Feature
[ ] 🐛 Bug Fix
[ ] 📦 Chore (Release)
[ ] ✅ Test

Related Tickets & Documents
Implements https://hous-bssb.atlassian.net/jira/software/projects/BPHH/boards/1?issueParent=10320&selectedIssue=BPHH-1982
Steps to QA
As long as you can login as use the app (submit various forms, etc.) as usual there should be no impact on user experience. As a developer, inspect the headers to see that CSRF token is indeed passed.

https://blog.eq8.eu/article/rails-api-authentication-with-spa-csrf-tokens.html


Need "Applies to" section

I want to know what version of sqlsrv support this function.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 8471b5d7-c6c9-c158-c57e-341683a0fc36
Version Independent ID: 50e18725-91d2-21b7-864e-8b6432ad342b
Content: STRING_AGG (Transact-SQL) - SQL Server
Content Source: docs/t-sql/functions/string-agg-transact-sql.md
Product: sql
Technology: t-sql
GitHub Login: @MikeRayMSFT
Microsoft Alias: mikeray

@ahdung -- thank you for your question.
@MikeRayMSFT -- please look into this issue 2991.

@ahdung  @MikeRayMSFT Hello Ahdung and MikeRayMSFT,.....
The following links about our Versioning system, and its drop-down list above the leftNav TOC, might be helpful for questions like the question here in Issue #2991 :

https://docs.microsoft.com/esql/sql-server/sql-docs-navigation-guide?view=sql-server-2017#version-filter

https://docs.microsoft.com/sql/sql-server/versioning-system-monikers-ui-sql-server?view=sql-server-2017

To discover all the versions of SQL Server to which given article applies, the reader would have to operate the Versioning drop-down to different versions, and see whether the system places a Cautionary message across the top of the article to indicate the version you attempted to select does Not apply.
Even if the version does apply, portions of the article text or other content might have been changed without the reader immediately knowing of the change.  This opens the entire conversation about exactly what Applies To exactly means.
Basically, the APPLIES_TO should never include any version info, because contradictions will likely occur with the version info of the whatever the currently chosen Versioning moniker is.  APPLIES_TO is only for handy info about products (not versions of those products).
The only way that APPLIES_TO could safely include version info is if Microsoft hired a small army of technical writers to constantly update far more includes/ files than we already have, and swore to never use ::: in-body versioning, and to never use features like Zone Pivot.
Thanks.
. .

@ahdung - thank you for the submission. I'm going to close this issue but I've logged your comment at internally (1605663).
STRING_AGG was extended in SQL Server 2017 but it also applies to SQL Server 2016.

@ahdung by way of follow-up - STRING-AGG was introduced in SQL Server 2017. Thanks for letting me know @bluefooted

@MightyPen First of all, thank you for your description, but in your opinion this seems to be a complicated question, is it? Come on, I just want to know a function available on which version, is this difficult to know?? Why are .net documents can? eg https://docs.microsoft.com/en-us/dotnet/api/system.text.stringbuilder?view=netframework-4.8#applies-to

@ahdung We do agree that this is an issue which really should be addressed, it should be much easier to see which version the function applies to. Unfortunately fixing this across all our documentation is not as simple as it may seem. We have taken this as an action item to hopefully address in the near future.

The doc page would be clearer if this issue were open until it is resolved. I had the same question about which SQL Server version this applies to. In the absence of first-class documentation on the page, an open on the bottom page indicating that it's being worked on would have been helpful. Closed issues are less discoverable.
For issues that are unresolved and have parallel issues in another tracking system, one approach is to leave them open and use a custom label like tracked-elsewhere to help with internal querying and prioritizing issues from this repro. That way open and closed will have a more natural sense (not fixed vs fixed) to end users.

@breyed Thanks for the suggestion, about tagging with a Label instead of Closing.
The situation is more complicated than I can explain in here, but a reconsideration of our process is getting some attention.


Approvals should be visible on the dashboard both to collection reviewers and to collection managers

I submitted the item in the February Review Collection at the bottom of this screenshot for approval a little while ago and it is not showing up in the Approvals table. In fact, this table hasn't updated in quite a while (at least days) and I know I've submitted more items for approval in that time. There is an item called "Submitted for your approval" (https://sul-h2-stage.stanford.edu/works/168) in the New Review Collection that should also be showing up in this table.

I initially thought this was a failure of the table to update, but after talking with @mjgiarlo, it may be that the manager is not set to be able to see this table or these items. This may be fallout from #661 (totally my guess, not @mjgiarlo 's).

I suspect we need to tweak this query to include both collection reviewers and collection managers: https://github.com/sul-dlss/happy-heron/blob/main/app/controllers/dashboards_controller.rb#L13-L15


Store .osu files for later use

Right now we are downloading every map and removing them when done, we can better store the .osu files in a cache folder and get them if needed. This will increase our osu command times after a while if we got alot of maps.

Added this feature in commit https://github.com/CakeAndBanana/CakeBot/commit/e8dc56ed0a20283342e8f66976ba23e1bdd2d0e9


Fix parsing of Unix timestamps from results

Previously, the Unix timestamp in the results metadata was not parsed correctly by the UI. Now, upon receiving the results, the UI will replace each run's timestamp with a JS Date object. This change also includes some formatting fixes.

The time is showing down to the milliseconds which shouldn't be useful. Can we remove this on another pull request? The fix looks good so will merge. Can iterate.


Programatic Reload Current Page

We have dynamic data that we would like to periodically refresh via polling. How can we force a reload from the configured URL and replace the content?

How can we force a reload

You can set path to a function, so you can use conditional statement or whatever logic you like to reset a URL to load (i.e., reload page/2 after user hits page/5.


qs: add option to disable printing of strings with given tag

via #761 and @r0ny123
add an option to disable the printing of common/library strings to the console.
and expanding on this: any tag specified by the user.
currently #code is hidden by default, maybe provide a way to un-hide that.

it feels like this sort of interaction is best done live, such as with a quick filter panel and re-rendering the output. this way the user can play around with the output until it works for them. i'm a little worried that relying on CLI arguments is tedious to type and format. still, we should definitely have the functionality.

I agree that this would be best done interactively. Are we in good enough shape yet to start on the dynamic UI?

i would enjoy working on the interactive aspect of QS. my only hesitation right now is that it would distract from any other effort currently in progress with QS. what do we have left to explore and do we want to figure that out now?
for example, the databases need some work, especially #expert, so i think a database editor is useful, so had been working on that. this way, QS with static output is useful standalone, before the interactive view.

ok, let's not get ahead of ourselves 😄
maybe can just start collection (user-specific) configuration values like below for now
{
  "min_str_len": 6,
  "tags_to_hide": ["#code"],
  "tags_to_show": [],
  ...
}

I like @mr-tz's idea of configuration values.


Android back button isse

Hey @doonfrs
First thanks for maintaining the package,  the community does appreciate you.
But I wish you added the android back button issue as part of the fix for the updated 8.4.3 version.
:)
It will be nice if you can do something about it, I know the package was originally for desktops but a lot of people are now using it on mobile
Shalom

Hi @Jakuzeisme
Could you please include the issue link in the comment, or add more details

@doonfrs
https://github.com/doonfrs/pluto_grid_plus/issues/44

@doonfrs
Good afternoon,  are there any plans?

你好@Jakuzeisme 您可以在评论中包含问题链接，或者添加更多详细信息

https://github.com/bosskmk/pluto_grid/issues/1066
https://github.com/bosskmk/pluto_grid/pull/1070

@Comanx
Yo my man...I'm using the latest version of the plutogrid plus doesn't work.
Are we expected to fork this package merge the changes ourselves.  I'm not a big user of github  never even forked a package before or understand it in depth but I'm open to it. That being said, why can't it just be added to the plutogridplus package,  that's what u don't understand
嘿，兄弟……我在用最新版本的PlutoGrid，但PlutoGrid Plus不起作用。
我们是不是应该自己fork这个包并合并更改？我对GitHub不太熟悉，从来没有fork过包，也不太了解，但我愿意尝试。话虽如此，为什么不能直接把它加到Pluto

But why can't it just work with the package.
That's what I understand
On Thu, 21 Nov 2024, 9:44 am Jock, @.***> wrote:

@Jakuzeisme https://github.com/Jakuzeisme
说实话我并不知道在GitHub中如何将PlutoGrid的提交合并到PlutoGrid
Plus，对于没有维修或者的仓库一般我会将仓库fork或者clone到本地，修改后在pubspec.yaml文件中重新配置依赖：
To be honest, I’m not sure how to merge the PlutoGrid commits into
PlutoGrid Plus on GitHub. For repositories that are not actively
maintained, I usually fork or clone the repository locally. After making
the changes, I update the dependencies in the pubspec.yaml file as follows:
pubspec.yaml
dependencies:
pluto_grid_plus:
git:
url: 'https://your_repository/pluto_grid_plus.git'
ref: 0.0.1
or
pluto_grid_plus:
path: ../pluto_grid_plus
对于GitHub fork和pr，点击fork按钮，按说明创建仓库，添加自己的修改，然后可使用GitHub Desktop创建PR推送代码，可以试一下
—
Reply to this email directly, view it on GitHub
https://github.com/doonfrs/pluto_grid_plus/issues/89#issuecomment-2490397793,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BL3GQLLYSI4G5OL4KEYY5MD2BWMQRAVCNFSM6AAAAABQXHQVA2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOJQGM4TONZZGM
.
You are receiving this because you were mentioned.Message ID:
@.***>


Avoid SSLSocket leaks by setting sync_close by default

@tarcieri

Oh my, the build failures :cry:
They all seem unrelated at least, but...
/cc @digitalextremist

Uhhh. #134?

Yeah. These are rspec syntax failures.

Sorry, not clear but what's the problem? Does #134 block until this goes green?

There's an rspec compatibility issue. I'll try to remedy it before #134 when I'm back from lunch.

I wouldn't be opposed to landing this despite the spec failures

Yeah, I'm just going to merge this now and fiddle with tests separately.

@digitalextremist transpec helps a lot to update specs...

That's very helpful @ixti, thank you!
@e2 is this what you've been doing? :)

[ haha, now that all the tests are current :) ]

@digitalextremist - yeah, life would be hard without transpec. There are a few tricks, though (for outdated projects):

first, update to RSpec 2.99, which is highly compatible with other 2.x versions + run tests
update the spec/spec_helper config (remove stuff mostly) so tests run
transpec + commit (best to temporarily add to Gemfile - best to use with Bundler)
update to RSpec 3.2
transpec again (will be different)
remove transpec and update spec_helper with new RSpec features


When a user goes to the bastion tab, they don't see use bastion, but new setup

I've installed bastion, and when I go to the VM and click connect, I can click the bastion tab, and "Use Bastion" button and RDP to the VM just fine.
However another user does the same, and on the bastion tab they only see an option to create a new bastion, not to "Use Bastion".
When I go to the bastion instance in azure portal, and check "Access Control" tab, I can see that the user has "Virtual Machine User Login" permission to the VM. Do I need to give them other permissions so they can use this existing Bastion instance to connect to the VM?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 3d53c0bc-f5bb-0f6e-5fb7-fb775eab4f4a
Version Independent ID: b8d3a089-c160-7681-2a06-c230c19143db
Content: Connect to a Windows VM using Azure Bastion
Content Source: articles/bastion/bastion-connect-vm-rdp.md
Service: bastion
GitHub Login: @cherylmc
Microsoft Alias: cherylmc

@dazinator, Thanks for reaching out. We will investigate and update this thread.

I figured out I needed to give the user the "Reader" role on the Bastion resource. So in total the user had to have the following two roles:

"Virtual Machine User Login" role, for the VM resource.
"Reader" role for the Bastion resource.

It would be good if this could be conveyed better in the docs! thanks!

Hey,
I think this is well described as prerequisite within the MS doc:
https://docs.microsoft.com/en-us/azure/bastion/tutorial-create-host-portal#prerequisites

@kaiaschulz

Required VM roles:
Reader role on the virtual machine.
Reader role on the NIC with private IP of the virtual machine.

I agree that at the most granular level the pre-requisites are described.
What's not described is:

If when clicking on the "Bastion" tab, a user sees the option to set up a new Bastion instance, rather than the option shown in the article which is a "Use Bastion"  - this may be because the user is missing the permissions described in the pre-requisites
Users might not want to govern permissions as granular as the individual NIC of the virtual machine. What's the "Virtual Machine User Login" role for, if not to allow a user to login to a VM - perhaps that should be recommended as the typical way these permissions would be assigned rather than at individual NIC level?

Agreed. This confused me as well.
Myself and customers of us had the same problem that the behavior of trying to use the Bastion with insufficiant access rights caused a new Bastion creation dialogue:

Would like to see an error message or hint that an Bastion host exists but the prerequisites aren't fullfilled.

@dazinator, Appreciate your feedback. Here is the doc that explains clearly about all the required roles that are necessary for connecting windows vm using Bastion service.
Meanwhile we will work on updating this document accordingly.
we are closing this issue for now. If there are further questions regarding this matter, please reply and we will gladly continue the discussion.


Enhancements to job log viewer

The new Job Log tree should have the structure
job date
   object <- source
      command
      message1
      ...
      messageN
job date 2
...

The object<-source entry field will be colored red if the command failed (or at least the icon for it will be a red x or green check)
There will be a toggle action on the view to only show failed commands
There will be a filter action to only show messages with a given severity or higher

There should also be an action to view the output for individual objects


unpin pandas

Unpinning pandas, due to some recent test failures. Unfortunately, on my Mac this leads to a segfault (below). Opening this to see what CircleCI thinks of this.
::>> running: 'conda run --no-capture-output -n fastparquet python setup.py test'
running pytest
Searching for zstd
Reading https://pypi.org/simple/zstd/
Downloading https://files.pythonhosted.org/packages/92/bc/6bc155c61db55d5b3111a8afc76aaec084372bde89a82ecd4f5fd96c639a/zstd-<IP_ADDRESS>.tar.gz#sha256=b62b21eb850abd6b8c0046bfc1c5c773c873eeb94f1904ef1ff304e98b62b80e
Best match: zstd <IP_ADDRESS>
Processing zstd-<IP_ADDRESS>.tar.gz
Writing /var/folders/jk/s59wn3zx5g92kvk6t9k74fzr0000gn/T/easy_install-91sj_1pw/zstd-<IP_ADDRESS>/setup.cfg
Running zstd-<IP_ADDRESS>/setup.py -q bdist_egg --dist-dir /var/folders/jk/s59wn3zx5g92kvk6t9k74fzr0000gn/T/easy_install-91sj_1pw/zstd-<IP_ADDRESS>/egg-dist-tmp-hyghuwr2
warning: no files found matching 'README.md'
zip_safe flag not set; analyzing archive contents...
__pycache__.zstd.cpython-37: module references __file__
creating /Users/vhaenel/git/numba-integration-testing/fastparquet/.eggs/zstd-<IP_ADDRESS>-py3.7-macosx-10.9-x86_64.egg
Extracting zstd-<IP_ADDRESS>-py3.7-macosx-10.9-x86_64.egg to /Users/vhaenel/git/numba-integration-testing/fastparquet/.eggs

Installed /Users/vhaenel/git/numba-integration-testing/fastparquet/.eggs/zstd-<IP_ADDRESS>-py3.7-macosx-10.9-x86_64.egg
Searching for zstandard
Reading https://pypi.org/simple/zstandard/
Downloading https://files.pythonhosted.org/packages/04/a5/97dd40c1a2c54f6cbb41097ff0869f7c56ac12c5c4f265c982996d5dd207/zstandard-0.15.1-cp37-cp37m-macosx_10_9_x86_64.whl#sha256=8de12b37d32a9128a72b6050d4c6070a3bc944557f6b9912ecfc421f9ee97824
Best match: zstandard 0.15.1
Processing zstandard-0.15.1-cp37-cp37m-macosx_10_9_x86_64.whl
Installing zstandard-0.15.1-cp37-cp37m-macosx_10_9_x86_64.whl to /Users/vhaenel/git/numba-integration-testing/fastparquet/.eggs

Installed /Users/vhaenel/git/numba-integration-testing/fastparquet/.eggs/zstandard-0.15.1-py3.7-macosx-10.9-x86_64.egg
running egg_info
writing fastparquet.egg-info/PKG-INFO
writing dependency_links to fastparquet.egg-info/dependency_links.txt
writing requirements to fastparquet.egg-info/requires.txt
writing top-level names to fastparquet.egg-info/top_level.txt
reading manifest file 'fastparquet.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'test-data/'
no previously-included directories found matching 'docs/'
writing manifest file 'fastparquet.egg-info/SOURCES.txt'
running build_ext
copying build/lib.macosx-10.9-x86_64-3.7/fastparquet/speedups.cpython-37m-darwin.so -> fastparquet
========================================================================= test session starts =========================================================================
platform darwin -- Python 3.7.9, pytest-6.2.1, py-1.10.0, pluggy-0.13.1 -- /Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/bin/python
cachedir: .pytest_cache
rootdir: /Users/vhaenel/git/numba-integration-testing/fastparquet, configfile: setup.cfg
collected 311 items

fastparquet/test/test_api.py::test_import_without_warning PASSED                                                                                [  0%]
fastparquet/test/test_api.py::test_statistics PASSED                                                                                [  0%]
fastparquet/test/test_api.py::test_logical_types PASSED                                                                                [  0%]
fastparquet/test/test_api.py::test_empty_statistics PASSED                                                                                [  1%]
fastparquet/test/test_api.py::test_sorted_row_group_columns PASSED                                                                                [  1%]
fastparquet/test/test_api.py::test_sorted_row_group_columns_with_filters SKIPPED (could not import 'dask.dataframe': No module named 'dask')                    [  1%]
fastparquet/test/test_api.py::test_iter PASSED                                                                                [  2%]
fastparquet/test/test_api.py::test_attributes PASSED                                                                                [  2%]
fastparquet/test/test_api.py::test_open_standard PASSED                                                                                [  2%]
fastparquet/test/test_api.py::test_filelike PASSED                                                                                [  3%]
fastparquet/test/test_api.py::test_cast_index PASSED                                                                                [  3%]
fastparquet/test/test_api.py::test_zero_child_leaf PASSED                                                                                [  3%]
fastparquet/test/test_api.py::test_request_nonexistent_column PASSED                                                                                [  4%]
fastparquet/test/test_api.py::test_read_multiple_no_metadata PASSED                                                                                [  4%]
fastparquet/test/test_api.py::test_single_upper_directory PASSED                                                                                [  4%]
fastparquet/test/test_api.py::test_numerical_partition_name PASSED                                                                                [  5%]
fastparquet/test/test_api.py::test_floating_point_partition_name PASSED                                                                                [  5%]
fastparquet/test/test_api.py::test_datetime_partition_names PASSED                                                                                [  5%]
fastparquet/test/test_api.py::test_string_partition_names PASSED                                                                                [  6%]
fastparquet/test/test_api.py::test_mixed_partition_types[partitions0] PASSED                                                                                [  6%]
fastparquet/test/test_api.py::test_mixed_partition_types[partitions1] PASSED                                                                                [  6%]
fastparquet/test/test_api.py::test_filter_without_paths PASSED                                                                                [  7%]
fastparquet/test/test_api.py::test_filter_special PASSED                                                                                [  7%]
fastparquet/test/test_api.py::test_filter_dates PASSED                                                                                [  7%]
fastparquet/test/test_api.py::test_in_filter PASSED                                                                                [  8%]
fastparquet/test/test_api.py::test_in_filter_numbers PASSED                                                                                [  8%]
fastparquet/test/test_api.py::test_filter_stats PASSED                                                                                [  8%]
fastparquet/test/test_api.py::test_in_filters[vals0-None-None-False-False] PASSED                                                                               [  9%]
fastparquet/test/test_api.py::test_in_filters[vals1-3-3-False-True] PASSED                                                                                [  9%]
fastparquet/test/test_api.py::test_in_filters[vals2-2-2-True-False] PASSED                                                                                [  9%]
fastparquet/test/test_api.py::test_in_filters[vals3-None-7-False-False] PASSED                                                                                [  9%]
fastparquet/test/test_api.py::test_in_filters[vals4-None-2-True-False] PASSED                                                                                [ 10%]
fastparquet/test/test_api.py::test_in_filters[vals5-2-None-False-False] PASSED                                                                                [ 10%]
fastparquet/test/test_api.py::test_in_filters[vals6-7-None-True-False] PASSED                                                                                [ 10%]
fastparquet/test/test_api.py::test_in_filters[vals7-2-4-False-False] PASSED                                                                                [ 11%]
fastparquet/test/test_api.py::test_in_filters[vals8-5-6-False-True] PASSED                                                                                [ 11%]
fastparquet/test/test_api.py::test_in_filters[vals9-2-3-False-True] PASSED                                                                                [ 11%]
fastparquet/test/test_api.py::test_in_filters[vals10-6-7-False-True] PASSED                                                                                [ 12%]
fastparquet/test/test_api.py::test_in_filters[vals11-1-2-True-False] PASSED                                                                                [ 12%]
fastparquet/test/test_api.py::test_in_filters[vals12-7-8-True-False] PASSED                                                                                [ 12%]
fastparquet/test/test_api.py::test_in_filters[vals13-1-8-False-False] PASSED                                                                                [ 13%]
fastparquet/test/test_api.py::test_in_filters[vals14-1-8-True-False] PASSED                                                                                [ 13%]
fastparquet/test/test_api.py::test_in_filter_rowgroups PASSED                                                                                [ 13%]
fastparquet/test/test_api.py::test_index_not_in_columns PASSED                                                                                [ 14%]
fastparquet/test/test_api.py::test_no_index_name PASSED                                                                                [ 14%]
fastparquet/test/test_api.py::test_input_column_list_not_mutated PASSED                                                                                [ 14%]
fastparquet/test/test_api.py::test_drill_list PASSED                                                                                [ 15%]
fastparquet/test/test_api.py::test_multi_list PASSED                                                                                [ 15%]
fastparquet/test/test_api.py::test_hive_and_drill_list PASSED                                                                                [ 15%]
fastparquet/test/test_api.py::test_bad_file_paths PASSED                                                                                [ 16%]
fastparquet/test/test_api.py::test_compression_zstandard PASSED                                                                                [ 16%]
fastparquet/test/test_api.py::test_compression_zstd PASSED                                                                                [ 16%]
fastparquet/test/test_api.py::test_compression_lz4 PASSED                                                                                [ 17%]
fastparquet/test/test_api.py::test_compression_snappy PASSED                                                                                [ 17%]
fastparquet/test/test_api.py::test_int96_stats PASSED                                                                                [ 17%]
fastparquet/test/test_api.py::test_only_partition_columns PASSED                                                                                [ 18%]
fastparquet/test/test_api.py::test_path_containing_metadata_df PASSED                                                                                [ 18%]
fastparquet/test/test_api.py::test_empty_df PASSED                                                                                [ 18%]
fastparquet/test/test_api.py::test_unicode_cols PASSED                                                                                [ 18%]
fastparquet/test/test_api.py::test_multi_cat PASSED                                                                                [ 19%]
fastparquet/test/test_api.py::test_multi_cat_single PASSED                                                                                [ 19%]
fastparquet/test/test_api.py::test_multi_cat_fail PASSED                                                                                [ 19%]
fastparquet/test/test_api.py::test_multi PASSED                                                                                [ 20%]
fastparquet/test/test_api.py::test_simple_nested PASSED                                                                                [ 20%]
fastparquet/test/test_api.py::test_pandas_metadata_inference PASSED                                                                                [ 20%]
fastparquet/test/test_aroundtrips.py::test_map_array SKIPPED (could not import 'pyspark': No module named 'pyspark')                                            [ 21%]
fastparquet/test/test_aroundtrips.py::test_nested_list SKIPPED (could not import 'pyspark': No module named 'pyspark')                                          [ 21%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[None-row_groups0-simple] SKIPPED (could not import 'pyspark': No module named 'pyspark')           [ 21%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[None-row_groups0-hive] SKIPPED (could not import 'pyspark': No module named 'pyspark')             [ 22%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[None-row_groups0-drill] SKIPPED (could not import 'pyspark': No module named 'pyspark')            [ 22%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[None-row_groups1-simple] SKIPPED (could not import 'pyspark': No module named 'pyspark')           [ 22%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[None-row_groups1-hive] SKIPPED (could not import 'pyspark': No module named 'pyspark')             [ 23%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[None-row_groups1-drill] SKIPPED (could not import 'pyspark': No module named 'pyspark')            [ 23%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[UNCOMPRESSED-row_groups0-simple] SKIPPED (could not import 'pyspark': No module named 'pyspark')   [ 23%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[UNCOMPRESSED-row_groups0-hive] SKIPPED (could not import 'pyspark': No module named 'pyspark')     [ 24%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[UNCOMPRESSED-row_groups0-drill] SKIPPED (could not import 'pyspark': No module named 'pyspark')    [ 24%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[UNCOMPRESSED-row_groups1-simple] SKIPPED (could not import 'pyspark': No module named 'pyspark')   [ 24%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[UNCOMPRESSED-row_groups1-hive] SKIPPED (could not import 'pyspark': No module named 'pyspark')     [ 25%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[UNCOMPRESSED-row_groups1-drill] SKIPPED (could not import 'pyspark': No module named 'pyspark')    [ 25%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[GZIP-row_groups0-simple] SKIPPED (could not import 'pyspark': No module named 'pyspark')           [ 25%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[GZIP-row_groups0-hive] SKIPPED (could not import 'pyspark': No module named 'pyspark')             [ 26%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[GZIP-row_groups0-drill] SKIPPED (could not import 'pyspark': No module named 'pyspark')            [ 26%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[GZIP-row_groups1-simple] SKIPPED (could not import 'pyspark': No module named 'pyspark')           [ 26%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[GZIP-row_groups1-hive] SKIPPED (could not import 'pyspark': No module named 'pyspark')             [ 27%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[GZIP-row_groups1-drill] SKIPPED (could not import 'pyspark': No module named 'pyspark')            [ 27%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[SNAPPY-row_groups0-simple] SKIPPED (could not import 'pyspark': No module named 'pyspark')         [ 27%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[SNAPPY-row_groups0-hive] SKIPPED (could not import 'pyspark': No module named 'pyspark')           [ 27%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[SNAPPY-row_groups0-drill] SKIPPED (could not import 'pyspark': No module named 'pyspark')          [ 28%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[SNAPPY-row_groups1-simple] SKIPPED (could not import 'pyspark': No module named 'pyspark')         [ 28%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[SNAPPY-row_groups1-hive] SKIPPED (could not import 'pyspark': No module named 'pyspark')           [ 28%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[SNAPPY-row_groups1-drill] SKIPPED (could not import 'pyspark': No module named 'pyspark')          [ 29%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[LZO-row_groups0-simple] SKIPPED (could not import 'pyspark': No module named 'pyspark')            [ 29%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[LZO-row_groups0-hive] SKIPPED (could not import 'pyspark': No module named 'pyspark')              [ 29%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[LZO-row_groups0-drill] SKIPPED (could not import 'pyspark': No module named 'pyspark')             [ 30%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[LZO-row_groups1-simple] SKIPPED (could not import 'pyspark': No module named 'pyspark')            [ 30%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[LZO-row_groups1-hive] SKIPPED (could not import 'pyspark': No module named 'pyspark')              [ 30%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[LZO-row_groups1-drill] SKIPPED (could not import 'pyspark': No module named 'pyspark')             [ 31%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[BROTLI-row_groups0-simple] SKIPPED (could not import 'pyspark': No module named 'pyspark')         [ 31%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[BROTLI-row_groups0-hive] SKIPPED (could not import 'pyspark': No module named 'pyspark')           [ 31%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[BROTLI-row_groups0-drill] SKIPPED (could not import 'pyspark': No module named 'pyspark')          [ 32%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[BROTLI-row_groups1-simple] SKIPPED (could not import 'pyspark': No module named 'pyspark')         [ 32%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[BROTLI-row_groups1-hive] SKIPPED (could not import 'pyspark': No module named 'pyspark')           [ 32%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[BROTLI-row_groups1-drill] SKIPPED (could not import 'pyspark': No module named 'pyspark')          [ 33%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[LZ4-row_groups0-simple] SKIPPED (could not import 'pyspark': No module named 'pyspark')            [ 33%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[LZ4-row_groups0-hive] SKIPPED (could not import 'pyspark': No module named 'pyspark')              [ 33%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[LZ4-row_groups0-drill] SKIPPED (could not import 'pyspark': No module named 'pyspark')             [ 34%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[LZ4-row_groups1-simple] SKIPPED (could not import 'pyspark': No module named 'pyspark')            [ 34%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[LZ4-row_groups1-hive] SKIPPED (could not import 'pyspark': No module named 'pyspark')              [ 34%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[LZ4-row_groups1-drill] SKIPPED (could not import 'pyspark': No module named 'pyspark')             [ 35%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[ZSTD-row_groups0-simple] SKIPPED (could not import 'pyspark': No module named 'pyspark')           [ 35%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[ZSTD-row_groups0-hive] SKIPPED (could not import 'pyspark': No module named 'pyspark')             [ 35%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[ZSTD-row_groups0-drill] SKIPPED (could not import 'pyspark': No module named 'pyspark')            [ 36%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[ZSTD-row_groups1-simple] SKIPPED (could not import 'pyspark': No module named 'pyspark')           [ 36%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[ZSTD-row_groups1-hive] SKIPPED (could not import 'pyspark': No module named 'pyspark')             [ 36%]
fastparquet/test/test_aroundtrips.py::test_pyspark_roundtrip[ZSTD-row_groups1-drill] SKIPPED (could not import 'pyspark': No module named 'pyspark')            [ 36%]
fastparquet/test/test_aroundtrips.py::test_empty_row_groups SKIPPED (could not import 'pyspark': No module named 'pyspark')                                     [ 37%]
fastparquet/test/test_compression.py::test_compress_decompress_roundtrip[UNCOMPRESSED] PASSED                                                                   [ 37%]
fastparquet/test/test_compression.py::test_compress_decompress_roundtrip[GZIP] PASSED                                                                           [ 37%]
fastparquet/test/test_compression.py::test_compress_decompress_roundtrip[SNAPPY] PASSED                                                                         [ 38%]
fastparquet/test/test_compression.py::test_compress_decompress_roundtrip[LZO] PASSED                                                                            [ 38%]
fastparquet/test/test_compression.py::test_compress_decompress_roundtrip[BROTLI] PASSED                                                                         [ 38%]
fastparquet/test/test_compression.py::test_compress_decompress_roundtrip[LZ4] PASSED                                                                            [ 39%]
fastparquet/test/test_compression.py::test_compress_decompress_roundtrip[ZSTD] PASSED                                                                           [ 39%]
fastparquet/test/test_compression.py::test_compress_decompress_roundtrip_args_gzip PASSED                                                                       [ 39%]
fastparquet/test/test_compression.py::test_compress_decompress_roundtrip_args_lz4 PASSED                                                                        [ 40%]
fastparquet/test/test_compression.py::test_compress_decompress_roundtrip_args_zstandard PASSED                                                                  [ 40%]
fastparquet/test/test_compression.py::test_compress_decompress_roundtrip_args_zstd PASSED                                                                       [ 40%]
fastparquet/test/test_compression.py::test_errors PASSED                                                                                [ 41%]
fastparquet/test/test_compression.py::test_not_installed PASSED                                                                                [ 41%]
fastparquet/test/test_converted_types.py::test_int32 PASSED                                                                                [ 41%]
fastparquet/test/test_converted_types.py::test_date PASSED                                                                                [ 42%]
fastparquet/test/test_converted_types.py::test_time_millis PASSED                                                                                [ 42%]
fastparquet/test/test_converted_types.py::test_timestamp_millis PASSED                                                                                [ 42%]
fastparquet/test/test_converted_types.py::test_utf8 PASSED                                                                                [ 43%]
fastparquet/test/test_converted_types.py::test_json PASSED                                                                                [ 43%]
fastparquet/test/test_converted_types.py::test_bson PASSED                                                                                [ 43%]
fastparquet/test/test_converted_types.py::test_uint16 PASSED                                                                                [ 44%]
fastparquet/test/test_converted_types.py::test_uint32 PASSED                                                                                [ 44%]
fastparquet/test/test_converted_types.py::test_uint64 PASSED                                                                                [ 44%]
fastparquet/test/test_converted_types.py::test_big_decimal PASSED                                                                                [ 45%]
fastparquet/test/test_dataframe.py::test_empty PASSED                                                                                [ 45%]
fastparquet/test/test_dataframe.py::test_empty_tz_utc PASSED                                                                                [ 45%]
fastparquet/test/test_dataframe.py::test_empty_tz_nonutc PASSED                                                                                [ 45%]
fastparquet/test/test_dataframe.py::test_timestamps PASSED                                                                                [ 46%]
fastparquet/test/test_dataframe.py::test_pandas_hive_serialization PASSED                                                                                [ 46%]
fastparquet/test/test_encoding.py::test_int32 PASSED                                                                                [ 46%]
fastparquet/test/test_encoding.py::test_int64 PASSED                                                                                [ 47%]
fastparquet/test/test_encoding.py::test_int96 PASSED                                                                                [ 47%]
fastparquet/test/test_encoding.py::test_float PASSED                                                                                [ 47%]
fastparquet/test/test_encoding.py::test_double PASSED                                                                                [ 48%]
fastparquet/test/test_encoding.py::test_fixed PASSED                                                                                [ 48%]
fastparquet/test/test_encoding.py::test_boolean PASSED                                                                                [ 48%]
fastparquet/test/test_encoding.py::testFourByteValue PASSED                                                                                [ 49%]
fastparquet/test/test_encoding.py::testSingleByte PASSED                                                                                [ 49%]
fastparquet/test/test_encoding.py::testFourByte PASSED                                                                                [ 49%]
fastparquet/test/test_encoding.py::testFromExample PASSED                                                                                [ 50%]
fastparquet/test/test_encoding.py::testWidths PASSED                                                                                [ 50%]
fastparquet/test/test_output.py::test_uvarint PASSED                                                                                [ 50%]
fastparquet/test/test_output.py::test_bitpack PASSED                                                                                [ 51%]
fastparquet/test/test_output.py::test_length PASSED                                                                                [ 51%]
fastparquet/test/test_output.py::test_rle_bp PASSED                                                                                [ 51%]
fastparquet/test/test_output.py::test_roundtrip_s3 PASSED                                                                                [ 52%]
fastparquet/test/test_output.py::test_roundtrip[None-row_groups0-simple] PASSED                                                                                [ 52%]
fastparquet/test/test_output.py::test_roundtrip[None-row_groups0-hive] PASSED                                                                                [ 52%]
fastparquet/test/test_output.py::test_roundtrip[None-row_groups1-simple] PASSED                                                                                [ 53%]
fastparquet/test/test_output.py::test_roundtrip[None-row_groups1-hive] PASSED                                                                                [ 53%]
fastparquet/test/test_output.py::test_roundtrip[GZIP-row_groups0-simple] PASSED                                                                                [ 53%]
fastparquet/test/test_output.py::test_roundtrip[GZIP-row_groups0-hive] PASSED                                                                                [ 54%]
fastparquet/test/test_output.py::test_roundtrip[GZIP-row_groups1-simple] PASSED                                                                                [ 54%]
fastparquet/test/test_output.py::test_roundtrip[GZIP-row_groups1-hive] PASSED                                                                                [ 54%]
fastparquet/test/test_output.py::test_roundtrip[SNAPPY-row_groups0-simple] PASSED                                                                               [ 54%]
fastparquet/test/test_output.py::test_roundtrip[SNAPPY-row_groups0-hive] PASSED                                                                                [ 55%]
fastparquet/test/test_output.py::test_roundtrip[SNAPPY-row_groups1-simple] PASSED                                                                               [ 55%]
fastparquet/test/test_output.py::test_roundtrip[SNAPPY-row_groups1-hive] PASSED                                                                                [ 55%]
fastparquet/test/test_output.py::test_bad_coltype PASSED                                                                                [ 56%]
fastparquet/test/test_output.py::test_bad_col PASSED                                                                                [ 56%]
fastparquet/test/test_output.py::test_roundtrip_complex[simple] PASSED                                                                                [ 56%]
fastparquet/test/test_output.py::test_roundtrip_complex[hive] PASSED                                                                                [ 57%]
fastparquet/test/test_output.py::test_datetime_roundtrip[df0] PASSED                                                                                [ 57%]
fastparquet/test/test_output.py::test_datetime_roundtrip[df1] PASSED                                                                                [ 57%]
fastparquet/test/test_output.py::test_datetime_roundtrip[df2] PASSED                                                                                [ 58%]
fastparquet/test/test_output.py::test_datetime_roundtrip[df3] PASSED                                                                                [ 58%]
fastparquet/test/test_output.py::test_nulls_roundtrip PASSED                                                                                [ 58%]
fastparquet/test/test_output.py::test_decimal_roundtrip PASSED                                                                                [ 59%]
fastparquet/test/test_output.py::test_make_definitions_with_nulls PASSED                                                                                [ 59%]
fastparquet/test/test_output.py::test_make_definitions_without_nulls PASSED                                                                                [ 59%]
fastparquet/test/test_output.py::test_empty_row_group PASSED                                                                                [ 60%]
fastparquet/test/test_output.py::test_write_delta SKIPPED (unconditional skip)                                                                                [ 60%]
fastparquet/test/test_output.py::test_int_rowgroups PASSED                                                                                [ 60%]
fastparquet/test/test_output.py::test_groups_roundtrip[hive] PASSED                                                                                [ 61%]
fastparquet/test/test_output.py::test_groups_roundtrip[drill] PASSED                                                                                [ 61%]
fastparquet/test/test_output.py::test_groups_iterable PASSED                                                                                [ 61%]
fastparquet/test/test_output.py::test_empty_groupby PASSED                                                                                [ 62%]
fastparquet/test/test_output.py::test_too_many_partition_columns PASSED                                                                                [ 62%]
fastparquet/test/test_output.py::test_read_partitioned_and_write_with_empty_partions PASSED                                                                     [ 62%]
fastparquet/test/test_output.py::test_write_compression_dict[GZIP] PASSED                                                                                [ 63%]
fastparquet/test/test_output.py::test_write_compression_dict[gzip] PASSED                                                                                [ 63%]
fastparquet/test/test_output.py::test_write_compression_dict[None] PASSED                                                                                [ 63%]
fastparquet/test/test_output.py::test_write_compression_dict[compression3] PASSED                                                                               [ 63%]
fastparquet/test/test_output.py::test_write_compression_dict[compression4] PASSED                                                                               [ 64%]
fastparquet/test/test_output.py::test_write_compression_schema PASSED                                                                                [ 64%]
fastparquet/test/test_output.py::test_index PASSED                                                                                [ 64%]
fastparquet/test/test_output.py::test_duplicate_columns PASSED                                                                                [ 65%]
fastparquet/test/test_output.py::test_cmd_bytesize[None] PASSED                                                                                [ 65%]
fastparquet/test/test_output.py::test_cmd_bytesize[gzip] PASSED                                                                                [ 65%]
fastparquet/test/test_output.py::test_dotted_column PASSED                                                                                [ 66%]
fastparquet/test/test_output.py::test_naive_index PASSED                                                                                [ 66%]
fastparquet/test/test_output.py::test_text_convert PASSED                                                                                [ 66%]
fastparquet/test/test_output.py::test_null_time PASSED                                                                                [ 67%]
fastparquet/test/test_output.py::test_auto_null PASSED                                                                                [ 67%]
fastparquet/test/test_output.py::test_many_categories[10] PASSED                                                                                [ 67%]
fastparquet/test/test_output.py::test_many_categories[127] PASSED                                                                                [ 68%]
fastparquet/test/test_output.py::test_many_categories[257] PASSED                                                                                [ 68%]
fastparquet/test/test_output.py::test_many_categories[65537] PASSED                                                                                [ 68%]
fastparquet/test/test_output.py::test_write_partitioned_with_empty_categories PASSED                                                                            [ 69%]
fastparquet/test/test_output.py::test_autocat PASSED                                                                                [ 69%]
fastparquet/test/test_output.py::test_merge[dirs0-row_groups0] PASSED                                                                                [ 69%]
fastparquet/test/test_output.py::test_merge[dirs0-row_groups1] PASSED                                                                                [ 70%]
fastparquet/test/test_output.py::test_merge[dirs1-row_groups0] PASSED                                                                                [ 70%]
fastparquet/test/test_output.py::test_merge[dirs1-row_groups1] PASSED                                                                                [ 70%]
fastparquet/test/test_output.py::test_merge_s3 PASSED                                                                                [ 71%]
fastparquet/test/test_output.py::test_merge_fail PASSED                                                                                [ 71%]
fastparquet/test/test_output.py::test_append_simple PASSED                                                                                [ 71%]
fastparquet/test/test_output.py::test_append_empty[hive] PASSED                                                                                [ 72%]
fastparquet/test/test_output.py::test_append_empty[simple] PASSED                                                                                [ 72%]
fastparquet/test/test_output.py::test_append[partition0-row_groups0] PASSED                                                                                [ 72%]
fastparquet/test/test_output.py::test_append[partition0-row_groups1] PASSED                                                                                [ 72%]
fastparquet/test/test_output.py::test_append[partition1-row_groups0] PASSED                                                                                [ 73%]
fastparquet/test/test_output.py::test_append[partition1-row_groups1] PASSED                                                                                [ 73%]
fastparquet/test/test_output.py::test_append_fail PASSED                                                                                [ 73%]
fastparquet/test/test_output.py::test_append_w_partitioning PASSED                                                                                [ 74%]
fastparquet/test/test_output.py::test_empty_dataframe PASSED                                                                                [ 74%]
fastparquet/test/test_output.py::test_hasnulls_ordering PASSED                                                                                [ 74%]
fastparquet/test/test_output.py::test_cats_in_part_files PASSED                                                                                [ 75%]
fastparquet/test/test_output.py::test_cats_and_nulls PASSED                                                                                [ 75%]
fastparquet/test/test_output.py::test_consolidate_cats PASSED                                                                                [ 75%]
fastparquet/test/test_output.py::test_bad_object_encoding PASSED                                                                                [ 76%]
fastparquet/test/test_output.py::test_object_encoding_int32 PASSED                                                                                [ 76%]
fastparquet/test/test_partition_filters_specialstrings.py::test_frame_write_read_verify[input_symbols0-504-hive-2-partitions0-filters0] PASSED                  [ 76%]
fastparquet/test/test_partition_filters_specialstrings.py::test_frame_write_read_verify[input_symbols1-504-hive-2-partitions1-filters1] PASSED                  [ 77%]
fastparquet/test/test_partition_filters_specialstrings.py::test_frame_write_read_verify[input_symbols2-504-hive-2-partitions2-filters2] PASSED                  [ 77%]
fastparquet/test/test_partition_filters_specialstrings.py::test_frame_write_read_verify[input_symbols3-504-hive-2-partitions3-filters3] PASSED                  [ 77%]
fastparquet/test/test_partition_filters_specialstrings.py::test_frame_write_read_verify[input_symbols4-504-hive-2-partitions4-filters4] PASSED                  [ 78%]
fastparquet/test/test_partition_filters_specialstrings.py::test_frame_write_read_verify[input_symbols5-504-hive-2-partitions5-filters5] PASSED                  [ 78%]
fastparquet/test/test_partition_filters_specialstrings.py::test_frame_write_read_verify[input_symbols6-504-hive-2-partitions6-filters6] PASSED                  [ 78%]
fastparquet/test/test_partition_filters_specialstrings.py::test_frame_write_read_verify[input_symbols7-10-hive-2-partitions7-filters7] PASSED                   [ 79%]
fastparquet/test/test_partition_filters_specialstrings.py::test_frame_write_read_verify[input_symbols8-10-hive-2-partitions8-filters8] PASSED                   [ 79%]
fastparquet/test/test_partition_filters_specialstrings.py::test_frame_write_read_verify[input_symbols9-10-hive-2-partitions9-filters9] PASSED                   [ 79%]
fastparquet/test/test_pd_optional_types.py::test_write_nullable_columns[simple-None] PASSED                                                                     [ 80%]
fastparquet/test/test_pd_optional_types.py::test_write_nullable_columns[simple-snappy] PASSED                                                                   [ 80%]
fastparquet/test/test_pd_optional_types.py::test_write_nullable_columns[simple-gzip] PASSED                                                                     [ 80%]
fastparquet/test/test_pd_optional_types.py::test_write_nullable_columns[hive-None] PASSED                                                                       [ 81%]
fastparquet/test/test_pd_optional_types.py::test_write_nullable_columns[hive-snappy] PASSED                                                                     [ 81%]
fastparquet/test/test_pd_optional_types.py::test_write_nullable_columns[hive-gzip] PASSED                                                                       [ 81%]
fastparquet/test/test_read.py::test_header_magic_bytes PASSED                                                                                [ 81%]
fastparquet/test/test_read.py::test_read_footer_fail[1] PASSED                                                                                [ 82%]
fastparquet/test/test_read.py::test_read_footer_fail[4] PASSED                                                                                [ 82%]
fastparquet/test/test_read.py::test_read_footer_fail[12] PASSED                                                                                [ 82%]
fastparquet/test/test_read.py::test_read_footer_fail[20] PASSED                                                                                [ 83%]
fastparquet/test/test_read.py::test_read_footer PASSED                                                                                [ 83%]
fastparquet/test/test_read.py::test_read_s3 PASSED                                                                                [ 83%]
fastparquet/test/test_read.py::test_file_csv[test-data/gzip-nation.impala.parquet] PASSED                                                                       [ 84%]
fastparquet/test/test_read.py::test_file_csv[test-data/nation.dict.parquet] PASSED                                                                              [ 84%]
fastparquet/test/test_read.py::test_file_csv[test-data/nation.impala.parquet] PASSED                                                                            [ 84%]
fastparquet/test/test_read.py::test_file_csv[test-data/nation.plain.parquet] PASSED                                                                             [ 85%]
fastparquet/test/test_read.py::test_file_csv[test-data/snappy-nation.impala.parquet] PASSED                                                                     [ 85%]
fastparquet/test/test_read.py::test_null_int PASSED                                                                                [ 85%]
fastparquet/test/test_read.py::test_converted_type_null PASSED                                                                                [ 86%]
fastparquet/test/test_read.py::test_null_plain_dictionary PASSED                                                                                [ 86%]
fastparquet/test/test_read.py::test_dir_partition PASSED                                                                                [ 86%]
fastparquet/test/test_read.py::test_stat_filters PASSED                                                                                [ 87%]
fastparquet/test/test_read.py::test_cat_filters PASSED                                                                                [ 87%]
fastparquet/test/test_read.py::test_statistics PASSED                                                                                [ 87%]
fastparquet/test/test_read.py::test_grab_cats PASSED                                                                                [ 88%]
fastparquet/test/test_read.py::test_index PASSED                                                                                [ 88%]
fastparquet/test/test_read.py::test_skip_length PASSED                                                                                [ 88%]
fastparquet/test/test_read.py::test_timestamp96 PASSED                                                                                [ 89%]
fastparquet/test/test_read.py::test_bad_catsize PASSED                                                                                [ 89%]
fastparquet/test/test_read.py::test_null_sizes PASSED                                                                                [ 89%]
fastparquet/test/test_read.py::test_multi_index PASSED                                                                                [ 90%]
fastparquet/test/test_read.py::test_multi_index_category PASSED                                                                                [ 90%]
fastparquet/test/test_read.py::test_no_columns PASSED                                                                                [ 90%]
fastparquet/test/test_read.py::test_map_multipage PASSED                                                                                [ 90%]
fastparquet/test/test_read.py::test_map_last_row_split PASSED                                                                                [ 91%]
fastparquet/test/test_read.py::test_truncated_decimal PASSED                                                                                [ 91%]
fastparquet/test/test_schema.py::test_schema_eq PASSED                                                                                [ 91%]
fastparquet/test/test_schema.py::test_schema_ne_subset PASSED                                                                                [ 92%]
fastparquet/test/test_schema.py::test_schema_ne_renamed PASSED                                                                                [ 92%]
fastparquet/test/test_schema.py::test_schema_ne_converted PASSED                                                                                [ 92%]
fastparquet/test/test_schema.py::test_schema_ne_different_order PASSED                                                                                [ 93%]
fastparquet/test/test_speedups.py::test_array_encode_utf8 PASSED                                                                                [ 93%]
fastparquet/test/test_speedups.py::test_array_decode_utf8 PASSED                                                                                [ 93%]
fastparquet/test/test_speedups.py::test_pack_byte_array PASSED                                                                                [ 94%]
fastparquet/test/test_speedups.py::test_unpack_byte_array PASSED                                                                                [ 94%]
fastparquet/test/test_thrift_structures.py::test_serialize PASSED                                                                                [ 94%]
fastparquet/test/test_thrift_structures.py::test_copy PASSED                                                                                [ 95%]
fastparquet/test/test_util.py::test_analyse_paths PASSED                                                                                [ 95%]
fastparquet/test/test_util.py::test_empty PASSED                                                                                [ 95%]
fastparquet/test/test_util.py::test_parents PASSED                                                                                [ 96%]
fastparquet/test/test_util.py::test_abs_and_rel_paths PASSED                                                                                [ 96%]
fastparquet/test/test_util.py::test_file_scheme PASSED                                                                                [ 96%]
fastparquet/test/test_util.py::test_val_to_num PASSED                                                                                [ 97%]
fastparquet/test/test_util.py::test_groupby_types PASSED                                                                                [ 97%]
fastparquet/test/test_util.py::test_bad_tz PASSED                                                                                [ 97%]
fastparquet/test/test_with_n.py::test_read_bitpacked PASSED                                                                                [ 98%]
fastparquet/test/test_with_n.py::test_rle PASSED                                                                                [ 98%]
fastparquet/test/test_with_n.py::test_uvarint PASSED                                                                                [ 98%]
fastparquet/test/test_with_n.py::test_hybrid PASSED                                                                                [ 99%]
fastparquet/test/test_with_n.py::test_hybrid_extra_bytes Fatal Python error: Segmentation fault

Current thread 0x00000001056d3dc0 (most recent call first):
  File "/Users/vhaenel/git/numba-integration-testing/fastparquet/fastparquet/test/test_with_n.py", line 74 in test_hybrid_extra_bytes
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/_pytest/python.py", line 1641 in runtest
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/_pytest/runner.py", line 255 in <lambda>
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/_pytest/runner.py", line 311 in from_call
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/_pytest/runner.py", line 255 in call_runtest_hook
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/_pytest/runner.py", line 215 in call_and_report
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/_pytest/runner.py", line 126 in runtestprotocol
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/_pytest/main.py", line 323 in _main
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/manager.py", line 87 in <lambda>
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/_pytest/config/__init__.py", line 163 in main
  File "/Users/vhaenel/git/numba-integration-testing/fastparquet/.eggs/pytest_runner-5.2-py3.7.egg/ptr.py", line 220 in run_tests
  File "/Users/vhaenel/git/numba-integration-testing/fastparquet/.eggs/pytest_runner-5.2-py3.7.egg/ptr.py", line 209 in run
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/distutils/dist.py", line 985 in run_command
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/distutils/dist.py", line 966 in run_commands
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/distutils/core.py", line 148 in setup
  File "/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/lib/python3.7/site-packages/setuptools/__init__.py", line 153 in setup
  File "setup.py", line 98 in <module>
/Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet/.tmpk_4yfmx9: line 3: 94910 Segmentation fault: 11  (core dumped) python setup.py test
ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['python', 'setup.py', 'test']' command failed.  (See above for error)
::>> running: 'conda list -n fastparquet'
# packages in environment at /Users/vhaenel/git/numba-integration-testing/miniconda3/envs/fastparquet:
#
# Name                    Version                   Build  Channel
fastparquet               0.5.0                    pypi_0    pypi
llvmlite                  0.36.0dev0              py37_31    numba/label/dev
numba                     0.53.0dev0      np1.11py3.7h6440ff4_g674f77b_423    numba/label/dev
::>> The following tests failed: '['fastparquet']'
python3 switchboard.py -t fastparquet  182.12s user 48.88s system 67% cpu 5:39.88 total

Looks like the segmentation fault can not be reproduced on CircleCI.

Looks like the segmentation fault can not be reproduced on CircleCI.

The test_with_n functions are testing pure numba functionality, as it happens, so I wonder if someone here could try to debug what the fail case is? These functions have not changed in a long time.

The test_with_n functions are testing pure numba functionality, as it happens, so I wonder if someone here could try to debug what the fail case is? These functions have not changed in a long time.

@martindurant thank you for your feedback, the test is against a development version of Numba so there may very well be something we broke during development. I will look into it.

@martindurant thank you for your feedback, the test is against a development version of Numba so there may very well be something we broke during development. I will look into it.

I took another look and this seems to happen with the released 0.52.0 also. I'll have to create a standalone reproducer.

I took another look and this seems to happen with the released 0.52.0 also. I'll have to create a standalone reproducer.

I have a standalone reproducer on OSX.
It has been extracted from:
https://github.com/dask/fastparquet/blob/master/fastparquet/test/test_with_n.py#L60-L76
As:
import numpy as np
from fastparquet import encoding
count = 1000

def test_hybrid_extra_bytes():
    results = np.empty(1000000, dtype=np.int32)
    with open('hybrid') as f:
        for i, l in enumerate(f):
            if i > count // 20:
                break
            (data, width, length, res) = eval(l)
            if length is not None:
                data += b'extra bytes'
                length += len(b'extra bytes')
            else:
                continue
            i = encoding.Numpy8(np.frombuffer(memoryview(data),
                                              dtype=np.uint8))
            o = encoding.Numpy32(results)
            encoding.read_rle_bit_packed_hybrid(i, width, length, o)
            assert (res == o.so_far()[:len(res)]).all()
            assert i.loc == len(data)

test_hybrid_extra_bytes()

Where hybrid comes from: https://github.com/dask/fastparquet/blob/master/test-data/hybrid
And the conda env is:
 💣 zsh» conda list
# packages in environment at /Users/vhaenel/miniconda3/envs/fastparquet:
#
# Name                    Version                   Build  Channel
blas                      1.0                         mkl    defaults
ca-certificates           2020.12.8            hecd8cb5_0    defaults
certifi                   2020.12.5        py38hecd8cb5_0    defaults
fastparquet               0.5.0            py38he3068b8_0    defaults
flake8                    3.8.4                      py_0    defaults
importlib-metadata        2.0.0                      py_1    defaults
intel-openmp              2019.4                      233    defaults
libcxx                    10.0.0                        1    defaults
libedit                   3.1.20191231         h1de35cc_1    defaults
libffi                    3.3                  hb1e8313_2    defaults
libllvm10                 10.0.1               h76017ad_5    defaults
llvm-openmp               10.0.0               h28b9765_0    defaults
llvmlite                  0.34.0           py38h739e7dc_4    defaults
mccabe                    0.6.1                    py38_1    defaults
mkl                       2019.4                      233    defaults
mkl-service               2.3.0            py38h9ed2024_0    defaults
mkl_fft                   1.2.0            py38hc64f4ea_0    defaults
mkl_random                1.1.1            py38h959d312_0    defaults
ncurses                   6.2                  h0a44026_1    defaults
numba                     0.51.2           py38h6440ff4_1    defaults
numpy                     1.19.2           py38h456fd55_0    defaults
numpy-base                1.19.2           py38hcfb5961_0    defaults
openssl                   1.1.1i               h9ed2024_0    defaults
packaging                 20.8               pyhd3eb1b0_0    defaults
pandas                    1.2.0            py38hb2f4e1b_0    defaults
pip                       20.3.3           py38hecd8cb5_0    defaults
pycodestyle               2.6.0                      py_0    defaults
pyflakes                  2.2.0                      py_0    defaults
pyparsing                 2.4.7                      py_0    defaults
python                    3.8.5                h26836e1_1    defaults
python-dateutil           2.8.1                      py_0    defaults
pytz                      2020.5             pyhd3eb1b0_0    defaults
readline                  8.0                  h1de35cc_0    defaults
setuptools                51.1.2           py38hecd8cb5_3    defaults
six                       1.15.0           py38hecd8cb5_0    defaults
sqlite                    3.33.0               hffcf06c_0    defaults
thrift                    0.11.0           py38h0a44026_0    defaults
tk                        8.6.10               hb0a8c7a_0    defaults
tzdata                    2020d                h14c3975_0    defaults
wheel                     0.36.2             pyhd3eb1b0_0    defaults
xz                        5.2.5                h1de35cc_0    defaults
zipp                      3.4.0              pyhd3eb1b0_0    defaults
zlib                      1.2.11               h1de35cc_3    defaults

The conda environment can be created with:
conda create -n fastparquet python=3.8 numpy fastparquet
It would be good if someone else could reproduce this on their machine to confirm it's not something specific to my setup.

I have a standalone reproducer on OSX.
It has been extracted from:
https://github.com/dask/fastparquet/blob/master/fastparquet/test/test_with_n.py#L60-L76
As:
import numpy as np
from fastparquet import encoding
count = 1000

def test_hybrid_extra_bytes():
    results = np.empty(1000000, dtype=np.int32)
    with open('hybrid') as f:
        for i, l in enumerate(f):
            if i > count // 20:
                break
            (data, width, length, res) = eval(l)
            if length is not None:
                data += b'extra bytes'
                length += len(b'extra bytes')
            else:
                continue
            i = encoding.Numpy8(np.frombuffer(memoryview(data),
                                              dtype=np.uint8))
            o = encoding.Numpy32(results)
            encoding.read_rle_bit_packed_hybrid(i, width, length, o)
            assert (res == o.so_far()[:len(res)]).all()
            assert i.loc == len(data)

test_hybrid_extra_bytes()

Where hybrid comes from: https://github.com/dask/fastparquet/blob/master/test-data/hybrid
And the conda env is:
 💣 zsh» conda list
# packages in environment at /Users/vhaenel/miniconda3/envs/fastparquet:
#
# Name                    Version                   Build  Channel
blas                      1.0                         mkl    defaults
ca-certificates           2020.12.8            hecd8cb5_0    defaults
certifi                   2020.12.5        py38hecd8cb5_0    defaults
fastparquet               0.5.0            py38he3068b8_0    defaults
flake8                    3.8.4                      py_0    defaults
importlib-metadata        2.0.0                      py_1    defaults
intel-openmp              2019.4                      233    defaults
libcxx                    10.0.0                        1    defaults
libedit                   3.1.20191231         h1de35cc_1    defaults
libffi                    3.3                  hb1e8313_2    defaults
libllvm10                 10.0.1               h76017ad_5    defaults
llvm-openmp               10.0.0               h28b9765_0    defaults
llvmlite                  0.34.0           py38h739e7dc_4    defaults
mccabe                    0.6.1                    py38_1    defaults
mkl                       2019.4                      233    defaults
mkl-service               2.3.0            py38h9ed2024_0    defaults
mkl_fft                   1.2.0            py38hc64f4ea_0    defaults
mkl_random                1.1.1            py38h959d312_0    defaults
ncurses                   6.2                  h0a44026_1    defaults
numba                     0.51.2           py38h6440ff4_1    defaults
numpy                     1.19.2           py38h456fd55_0    defaults
numpy-base                1.19.2           py38hcfb5961_0    defaults
openssl                   1.1.1i               h9ed2024_0    defaults
packaging                 20.8               pyhd3eb1b0_0    defaults
pandas                    1.2.0            py38hb2f4e1b_0    defaults
pip                       20.3.3           py38hecd8cb5_0    defaults
pycodestyle               2.6.0                      py_0    defaults
pyflakes                  2.2.0                      py_0    defaults
pyparsing                 2.4.7                      py_0    defaults
python                    3.8.5                h26836e1_1    defaults
python-dateutil           2.8.1                      py_0    defaults
pytz                      2020.5             pyhd3eb1b0_0    defaults
readline                  8.0                  h1de35cc_0    defaults
setuptools                51.1.2           py38hecd8cb5_3    defaults
six                       1.15.0           py38hecd8cb5_0    defaults
sqlite                    3.33.0               hffcf06c_0    defaults
thrift                    0.11.0           py38h0a44026_0    defaults
tk                        8.6.10               hb0a8c7a_0    defaults
tzdata                    2020d                h14c3975_0    defaults
wheel                     0.36.2             pyhd3eb1b0_0    defaults
xz                        5.2.5                h1de35cc_0    defaults
zipp                      3.4.0              pyhd3eb1b0_0    defaults
zlib                      1.2.11               h1de35cc_3    defaults

The conda environment can be created with:
conda create -n fastparquet python=3.8 numpy fastparquet
It would be good if someone else could reproduce this on their machine to confirm it's not something specific to my setup.

Hm, if I repeatedly run this, I find it fails sometimes, even on my old environment. Looking now...

Hm, if I repeatedly run this, I find it fails sometimes, even on my old environment. Looking now...

Sidenote: it may be time to widen the operating systems covered by the integration testing, it was pure accident I discovered this.

Sidenote: it may be time to widen the operating systems covered by the integration testing, it was pure accident I discovered this.

https://github.com/dask/fastparquet/pull/545 fixes this locally for me. What do you suggest as a decent test matrix?

https://github.com/dask/fastparquet/pull/545 fixes this locally for me. What do you suggest as a decent test matrix?

Note that I have no idea why this would happen now and not before, nor why it would be intermittent.

Note that I have no idea why this would happen now and not before, nor why it would be intermittent.

It is a bit bizzare... Maybe something about the OS/Kernel has changed...
For the integration testing, I don't yet know what else CircleCI supports, but maybe worth activating a few more options than just Linux.

It is a bit bizzare... Maybe something about the OS/Kernel has changed...
For the integration testing, I don't yet know what else CircleCI supports, but maybe worth activating a few more options than just Linux.

@martindurant how should we proceed. Continue to investigate or move forward with the fix. And should we merge the integration test for now, because fastparquet does pass again (only on linux, but that is what is being tested right now).

@martindurant how should we proceed. Continue to investigate or move forward with the fix. And should we merge the integration test for now, because fastparquet does pass again (only on linux, but that is what is being tested right now).

Do you mean it passes because the fix is in, or that it started to pass by itself?
There was a problem on our side, I think, and my guess is that the error only happening sometimes had to do with what bytes happened to be allocated after the data - which likely depend on OS, numpy and other things.

Do you mean it passes because the fix is in, or that it started to pass by itself?
There was a problem on our side, I think, and my guess is that the error only happening sometimes had to do with what bytes happened to be allocated after the data - which likely depend on OS, numpy and other things.

Oh, I mean it passes, because the integration tests that only run on linux pass. So I would advocate to merge this PR.

Oh, I mean it passes, because the integration tests that only run on linux pass. So I would advocate to merge this PR.

OK, fine with me

OK, fine with me

@martindurant excellent, thank you! I'll call on @seibert to review and merge this PR.

@martindurant excellent, thank you! I'll call on @seibert to review and merge this PR.


STNO Hub Upgrade with Landing Zone

Hello,
IHAC who is trying to upgrade their STNO hub to version 2. They currently use STNO as a core resource in their network account. Sample Manifest below:
core_accounts:
Network account

name: network
email<EMAIL_ADDRESS>ssm_parameters:

name: /org/member/network/account_id
value: $[AccountId]
core_resources:
name: STNO-Hub
template_file: templates/core_accounts/aws-transit-network-orchestrator-hub.template
parameter_file: parameters/core_accounts/aws-transit-network-orchestrator-hub.json
deploy_method: stack_set
regions:

us-east-1
us-west-2
ca-central-1
ap-southeast-1
eu-central-1
eu-west-2
ap-southeast-2

The problem is with V2 they want to pass different prefix list IDs and custom CIDRs to the different regions, but landing zone core resources do not allow parameter overrides (like baseline_resources do). The customer is also using a single JSON parameter file.
Any guidance on how to work around or restructure ?

We recommend using the latest version of the solution (V3.2.1). Let us know if you face any issues with the upgrade.


Mrt windows provision

Description
This update will add Vagrant + Ansible + Windows.
Fixes # (issue)
Type of change

[] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[] This change requires a documentation update

All Submissions:

[X] Have you run the validate.sh command and was it successful?

This is now depricated


Rank and MAE are both increasing for SoftImpute

Starting from Iter 13 both rank and MAE are increasing.
[SoftImpute] Max Singular Value of X_init = 1345.871031
[SoftImpute] Iter 1: observed MAE=0.002085 rank=2000
[SoftImpute] Iter 2: observed MAE=0.002495 rank=2000
[SoftImpute] Iter 3: observed MAE=0.002674 rank=1994
[SoftImpute] Iter 4: observed MAE=0.002806 rank=1977
[SoftImpute] Iter 5: observed MAE=0.002879 rank=1959
[SoftImpute] Iter 6: observed MAE=0.002915 rank=1943
[SoftImpute] Iter 7: observed MAE=0.002932 rank=1924
[SoftImpute] Iter 8: observed MAE=0.002943 rank=1909
[SoftImpute] Iter 9: observed MAE=0.002951 rank=1899
[SoftImpute] Iter 10: observed MAE=0.002960 rank=1891
[SoftImpute] Iter 11: observed MAE=0.002969 rank=1887
[SoftImpute] Iter 12: observed MAE=0.002977 rank=1884
[SoftImpute] Iter 13: observed MAE=0.002984 rank=1884
[SoftImpute] Iter 14: observed MAE=0.002991 rank=1883
[SoftImpute] Iter 15: observed MAE=0.002998 rank=1884
[SoftImpute] Iter 16: observed MAE=0.003003 rank=1887
[SoftImpute] Iter 17: observed MAE=0.003008 rank=1889
[SoftImpute] Iter 18: observed MAE=0.003012 rank=1892
[SoftImpute] Iter 19: observed MAE=0.003015 rank=1896
[SoftImpute] Iter 20: observed MAE=0.003019 rank=1897
[SoftImpute] Iter 21: observed MAE=0.003022 rank=1900
[SoftImpute] Iter 22: observed MAE=0.003024 rank=1903
[SoftImpute] Iter 23: observed MAE=0.003027 rank=1906
[SoftImpute] Iter 24: observed MAE=0.003029 rank=1908
[SoftImpute] Iter 25: observed MAE=0.003031 rank=1911
[SoftImpute] Iter 26: observed MAE=0.003033 rank=1912
[SoftImpute] Iter 27: observed MAE=0.003035 rank=1915
[SoftImpute] Iter 28: observed MAE=0.003037 rank=1916
[SoftImpute] Iter 29: observed MAE=0.003038 rank=1917
[SoftImpute] Iter 30: observed MAE=0.003040 rank=1919
[SoftImpute] Iter 31: observed MAE=0.003042 rank=1922
[SoftImpute] Iter 32: observed MAE=0.003043 rank=1923
[SoftImpute] Iter 33: observed MAE=0.003045 rank=1924
[SoftImpute] Iter 34: observed MAE=0.003046 rank=1925
[SoftImpute] Iter 35: observed MAE=0.003047 rank=1926
[SoftImpute] Iter 36: observed MAE=0.003049 rank=1927
[SoftImpute] Iter 37: observed MAE=0.003050 rank=1928
[SoftImpute] Iter 38: observed MAE=0.003051 rank=1929
[SoftImpute] Iter 39: observed MAE=0.003052 rank=1930
[SoftImpute] Iter 40: observed MAE=0.003054 rank=1932
[SoftImpute] Iter 41: observed MAE=0.003055 rank=1933
[SoftImpute] Iter 42: observed MAE=0.003056 rank=1934
[SoftImpute] Iter 43: observed MAE=0.003057 rank=1934
[SoftImpute] Iter 44: observed MAE=0.003058 rank=1935
[SoftImpute] Iter 45: observed MAE=0.003059 rank=1937
[SoftImpute] Iter 46: observed MAE=0.003060 rank=1937
Used parameters:
fancyimpute.SoftImpute(shrinkage_value=2, init_fill_method="random",min_value = 0.0, max_value=200.0, max_iters=100)

@aburkov Are you able to share the data that this happens on?
@iskandr Any ideas? This seems pretty wacky.

Hard to tell what's going on without example data. However, it's not impossible for rank to go back up before coming down again, I don't think there are guarantees on the individual iterations.

Hey @aburkov, let us know if you can send us example data for this problem. Closing for now.


Added support for dynamic text/string in RegexpString() and String() …

In conjunction with this replace-response PR. Details are discussed here.

Thanks for the PR, but I'll have to decline this change for several reasons:

This is a breaking change to the public API that would require all existing users to modify their code, yet it doesn't provide any new functionality that wasn't already possible.

If a user needs to transform the strings before creating a transformer, they can simply do that at the call site:
// Instead of:
String(old, new, myReplace)

// Users can already do:
String(myReplace(old), myReplace(new))

I'm confused about the intent of the RegexpString change, why is the template string being passed to a replace function?

If you have a specific use case that isn't well-served by the current API, I'd be happy to discuss it further to find a solution that doesn't break compatibility.

Hmm, I thought the intent was to allow the strings to be generated after the creation of the transformer but before the transformations occur. Our goal is to avoid re-allocating transformers for every HTTP request (in our case).
Maybe there's a non-breaking API that could be introduced, like new methods/functions?

@icholy and @mholt I just created new PRs to both icholy/replace (https://github.com/icholy/replace/pull/8) and caddyserver/replace-response (https://github.com/caddyserver/replace-response/pull/34). I introduced two new Transformer methods to the icholy/replace to keep the already-published APIs intact. Also explained the reasons in the comment lines of the new methods. Also named the new methods with more semantics. Please review

the intent was to allow the strings to be generated after the creation of the transformer but before the transformations occur.

@mholt this PR doesn't implement anything like that though.

@icholy I see. Thanks for clarifying. I have not followed it very close.