How can i train model in unetr++ with my own dataset?

Hi! Your Idea is owesome ! And I'm interesting in how can i do in order to adjust with my own dataset. I have generated corresponding json file ,but I still don't know what I should do next.Should I create a new trainer file? or do something else?

Hi @Overflowu7 ,
Thank you for your interest in our work.
To train UNETR++ for a new dataset, you have to do the appropriate data pre-processing first. It depends on each dataset. After that, you can use, for example, the model architecture of ACDC and create new trainings files In unetr_plus_plus/tree/main/unetr_pp/training/network_training. It is similar to Trainer_acdc.py, network_trainer_acdc.py, and unetr_pp_trainer_acdc.py. Please note that the folder structure of the data should follow the same structure as in run_training_acdc.sh
I hope my answer helps you to do that.
Best regards,
Abdelrahman.

Thank you for your prompt reply! Based on your answer, I have another two questions.
1.I found that unetr_pp_trainer_synapse.py  import UNETR_PP from unetr_pp.network_architecture.synapse.unetr_pp_synapse, should I make a new file in network_architecture which is similar to synapse and acdc?
2.I found that the batch size and crop size should be adjust in the file default_configuration.py , and I also want to know if there are other files that need to be adjusted?

Yes, you have to create similar file for your dataset. Please take care that you may need to change the architecture slightly according to the size of your input data. For example, Synapse the down-sampling layers are designed for input 63x128x128. For ACDC, the down-sampling has different kernels because the input size is different.

If this is a new dataset, you just have to change in there. If you need to change it for the current datasets, you have to change the .pkl file for that.

2. 如果需要更改当前数据集的 .pkl 文件

Hi @Overflowu7 ,
Thank you for your interest in our work.
To train UNETR++ for a new dataset, you have to do the appropriate data pre-processing first. It depends on each dataset. You can follow setting-up dataset section here for pre-processing: https://github.com/282857341/nnFormer#2-setting-up-the-datasets. After that, you can use, for example, the model architecture of ACDC and create new trainings files In unetr_plus_plus/tree/main/unetr_pp/training/network_training. It is similar to Trainer_acdc.py, network_trainer_acdc.py, and unetr_pp_trainer_acdc.py. Please note that the folder structure of the data should follow the same structure as in run_training_acdc.sh
I hope my answer helps you to do that.
Best regards, Abdelrahman.

Hello, may I ask how to obtain the pkl file

2. 如果需要更改当前数据集的 .pkl 文件

Hi @Overflowu7 ,
Thank you for your interest in our work.
To train UNETR++ for a new dataset, you have to do the appropriate data pre-processing first. It depends on each dataset. You can follow setting-up dataset section here for pre-processing: https://github.com/282857341/nnFormer#2-setting-up-the-datasets. After that, you can use, for example, the model architecture of ACDC and create new trainings files In unetr_plus_plus/tree/main/unetr_pp/training/network_training. It is similar to Trainer_acdc.py, network_trainer_acdc.py, and unetr_pp_trainer_acdc.py. Please note that the folder structure of the data should follow the same structure as in run_training_acdc.sh
I hope my answer helps you to do that.
Best regards, Abdelrahman.

Hello, may I ask how to obtain the pkl file

Hi! Your Idea is owesome ! And I'm interesting in how can i do in order to adjust with my own dataset. I have generated corresponding json file ,but I still don't know what I should do next.Should I create a new trainer file? or do something else?

Hello, may I ask how to obtain the pkl file

Just change the corresponding path to the one you need.
------------------ 原始邮件 ------------------
发件人:                                                                                "Amshaker/unetr_plus_plus"                                                                                @.>;
发送时间: 2024年4月17日(星期三) 下午4:14
@.>;
抄送: "-<EMAIL_ADDRESS>主题: Re: [Amshaker/unetr_plus_plus] How can i train model in unetr++ with my own dataset? (Issue #10)
Hi! Your Idea is owesome ! And I'm interesting in how can i do in order to adjust with my own dataset. I have generated corresponding json file ,but I still don't know what I should do next.Should I create a new trainer file? or do something else?
Hello, may I ask how to obtain the pkl file
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

Just change the corresponding path to the one you need.
…
------------------ 原始邮件 ------------------ 发件人: "Amshaker/unetr_plus_plus" @.>; 发送时间: 2024年4月17日(星期三) 下午4:14 @.>; 抄送: "-<EMAIL_ADDRESS>主题: Re: [Amshaker/unetr_plus_plus] How can i train model in unetr++ with my own dataset? (Issue #10) Hi! Your Idea is owesome ! And I'm interesting in how can i do in order to adjust with my own dataset. I have generated corresponding json file ,but I still don't know what I should do next.Should I create a new trainer file? or do something else? Hello, may I ask how to obtain the pkl file — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

Thank you, could you be more specific? 1. Is this .pkl file automatically generated? 2. In which file should I change thecorresponding path

Yes, .pkl files were automatically generated in nnunetv1, but were removed in nnunetv2, now json files are used instead of pkl files! If you still use nnunetv1 ,after each dataset has been processed, you can find the corresponding path of pkl in nnUNet_processed folder like "/nnUNet/nnUNet_processed/Task11_CTPelvic1K/nnUNetPlans_plans_3D.pkl'".
------------------ 原始邮件 ------------------
发件人:                                                                                "Amshaker/unetr_plus_plus"                                                                                @.>;
发送时间: 2024年4月18日(星期四) 下午2:09
@.>;
抄送: "-<EMAIL_ADDRESS>主题: Re: [Amshaker/unetr_plus_plus] How can i train model in unetr++ with my own dataset? (Issue #10)
Just change the corresponding path to the one you need.
…
------------------ 原始邮件 ------------------ 发件人: "Amshaker/unetr_plus_plus" @.>; 发送时间: 2024年4月17日(星期三) 下午4:14 @.>; 抄送: "-<EMAIL_ADDRESS>主题: Re: [Amshaker/unetr_plus_plus] How can i train model in unetr++ with my own dataset? (Issue #10) Hi! Your Idea is owesome ! And I'm interesting in how can i do in order to adjust with my own dataset. I have generated corresponding json file ,but I still don't know what I should do next.Should I create a new trainer file? or do something else? Hello, may I ask how to obtain the pkl file — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>
Thank you, could you be more specific? 1. Is this .pkl file automatically generated? 2. In which file should I change thecorresponding path
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

Yes, .pkl files were automatically generated in nnunetv1, but were removed in nnunetv2, now json files are used instead of pkl files! If you still use nnunetv1 ,after each dataset has been processed, you can find the corresponding path of pkl in nnUNet_processed folder like "/nnUNet/nnUNet_processed/Task11_CTPelvic1K/nnUNetPlans_plans_3D.pkl'".
…
------------------ 原始邮件 ------------------ 发件人: "Amshaker/unetr_plus_plus" @.>; 发送时间: 2024年4月18日(星期四) 下午2:09 @.>; 抄送: "-<EMAIL_ADDRESS>主题: Re: [Amshaker/unetr_plus_plus] How can i train model in unetr++ with my own dataset? (Issue #10) Just change the corresponding path to the one you need. … ------------------ 原始邮件 ------------------ 发件人: "Amshaker/unetr_plus_plus" @.>; 发送时间: 2024年4月17日(星期三) 下午4:14 @.>; 抄送: "-<EMAIL_ADDRESS>主题: Re: [Amshaker/unetr_plus_plus] How can i train model in unetr++ with my own dataset? (Issue #10) Hi! Your Idea is owesome ! And I'm interesting in how can i do in order to adjust with my own dataset. I have generated corresponding json file ,but I still don't know what I should do next.Should I create a new trainer file? or do something else? Hello, may I ask how to obtain the pkl file — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.> Thank you, could you be more specific? 1. Is this .pkl file automatically generated? 2. In which file should I change thecorresponding path — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.>

Thank you, I understand. Does it mean using Json files instead of pkl files? In this paper, I have generated the Json file for my own dataset, but it shows that there is no pkl file. I found that the pkl file was called in the code. How did you handle it?

I haven't run unter++ code since the nnunetv2 update, so I can't provide any more information. I remember that the v1 version will automatically generate the pkl file, and unter++ is based on the v1 version, so I don't know why there is no when your data is generated. I suggest you try to apply the network part of unter++ to nnunetv2 with slight changes.
------------------ 原始邮件 ------------------
发件人:                                                                                "Amshaker/unetr_plus_plus"                                                                                @.>;
发送时间: 2024年4月18日(星期四) 下午4:59
@.>;
抄送: "-<EMAIL_ADDRESS>主题: Re: [Amshaker/unetr_plus_plus] How can i train model in unetr++ with my own dataset? (Issue #10)
Yes, .pkl files were automatically generated in nnunetv1, but were removed in nnunetv2, now json files are used instead of pkl files! If you still use nnunetv1 ,after each dataset has been processed, you can find the corresponding path of pkl in nnUNet_processed folder like "/nnUNet/nnUNet_processed/Task11_CTPelvic1K/nnUNetPlans_plans_3D.pkl'".
…
------------------ 原始邮件 ------------------ 发件人: "Amshaker/unetr_plus_plus" @.>; 发送时间: 2024年4月18日(星期四) 下午2:09 @.>; 抄送: "-<EMAIL_ADDRESS>主题: Re: [Amshaker/unetr_plus_plus] How can i train model in unetr++ with my own dataset? (Issue #10) Just change the corresponding path to the one you need. … ------------------ 原始邮件 ------------------ 发件人: "Amshaker/unetr_plus_plus" @.>; 发送时间: 2024年4月17日(星期三) 下午4:14 @.>; 抄送: "-<EMAIL_ADDRESS>主题: Re: [Amshaker/unetr_plus_plus] How can i train model in unetr++ with my own dataset? (Issue #10) Hi! Your Idea is owesome ! And I'm interesting in how can i do in order to adjust with my own dataset. I have generated corresponding json file ,but I still don't know what I should do next.Should I create a new trainer file? or do something else? Hello, may I ask how to obtain the pkl file — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.> Thank you, could you be more specific? 1. Is this .pkl file automatically generated? 2. In which file should I change thecorresponding path — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.>
Thank you, I understand. Does it mean using Json files instead of pkl files? In this paper, I have generated the Json file for my own dataset, but it shows that there is no pkl file. I found that the pkl file was called in the code. How did you handle it?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

I haven't run unter++ code since the nnunetv2 update, so I can't provide any more information. I remember that the v1 version will automatically generate the pkl file, and unter++ is based on the v1 version, so I don't know why there is no when your data is generated. I suggest you try to apply the network part of unter++ to nnunetv2 with slight changes.
…
------------------ 原始邮件 ------------------ 发件人: "Amshaker/unetr_plus_plus" @.>; 发送时间: 2024年4月18日(星期四) 下午4:59 @.>; 抄送: "-<EMAIL_ADDRESS>主题: Re: [Amshaker/unetr_plus_plus] How can i train model in unetr++ with my own dataset? (Issue #10) Yes, .pkl files were automatically generated in nnunetv1, but were removed in nnunetv2, now json files are used instead of pkl files! If you still use nnunetv1 ,after each dataset has been processed, you can find the corresponding path of pkl in nnUNet_processed folder like "/nnUNet/nnUNet_processed/Task11_CTPelvic1K/nnUNetPlans_plans_3D.pkl'". … ------------------ 原始邮件 ------------------ 发件人: "Amshaker/unetr_plus_plus" @.>; 发送时间: 2024年4月18日(星期四) 下午2:09 @.>; 抄送: "-<EMAIL_ADDRESS>主题: Re: [Amshaker/unetr_plus_plus] How can i train model in unetr++ with my own dataset? (Issue #10) Just change the corresponding path to the one you need. … ------------------ 原始邮件 ------------------ 发件人: "Amshaker/unetr_plus_plus" @.>; 发送时间: 2024年4月17日(星期三) 下午4:14 @.>; 抄送: "-<EMAIL_ADDRESS>主题: Re: [Amshaker/unetr_plus_plus] How can i train model in unetr++ with my own dataset? (Issue #10) Hi! Your Idea is owesome ! And I'm interesting in how can i do in order to adjust with my own dataset. I have generated corresponding json file ,but I still don't know what I should do next.Should I create a new trainer file? or do something else? Hello, may I ask how to obtain the pkl file — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.> Thank you, could you be more specific? 1. Is this .pkl file automatically generated? 2. In which file should I change thecorresponding path — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.> Thank you, I understand. Does it mean using Json files instead of pkl files? In this paper, I have generated the Json file for my own dataset, but it shows that there is no pkl file. I found that the pkl file was called in the code. How did you handle it? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

Ok, thanks for your reply


helm-release.yml: disable chart changes verification

The current scripts rely on Git tags, and it was taken from a repository that was dedicated exclusively to a helm chart. This won't work on this directory because it checks that tag v1.0.0 does not exist before releasing.
Since this is a manual action, I'm just disabling this check and let automatize/improve it for the future.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 39.43%. Comparing base (f71b3b7) to head (d039af0).

Additional details and impacted files
@@             Coverage Diff             @@
##             main     #792       +/-   ##
===========================================
- Coverage   77.58%   39.43%   -38.16%
===========================================
  Files         107      107
  Lines        9277     9277
===========================================
- Hits         7198     3658     -3540
- Misses       1699     5413     +3714
+ Partials      380      206      -174

Flag
Coverage Δ

integration-test
?

unittests
39.43% <ø> (+0.07%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Merging as everibody is on holidays. Feel free to do any post-merge review.


Feature: Remove subgraph calls and return contract based on type

Thank you for contributing to Open Format! Please follow this template to ensure a smooth pull request process. Make sure you have read and followed the guidelines in CONTRIBUTING.md before submitting your pull request.
Summary
This PR refactors the getContract function to not call the subgraph, but get the contract based on the given contractAddress and type.
Type of Change
Please check the boxes that apply to your pull request:

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Documentation update (changes to documentation only)
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes)
[ ] Other (please describe):

Checklist

[x] I have read the CONTRIBUTING.md document
[x] I have followed the project's style guide
[x] I have added or updated tests to cover my changes
[x] All new and existing tests passed
[x] My changes generate no new warnings or errors
[x] I have made corresponding changes to the documentation
[x] I have checked that my changes do not introduce any new security vulnerabilities
[x] I have provided a clear and concise description of my changes
[x] I have linked to the related issue or bounty, if applicable

Additional Information

Once again, thank you for your contribution! We appreciate your effort and dedication to improving the Open Format.

/release-pr

/release-pr

/release-pr

/release-pr

/release-pr


Deploy etcd raises error

When following instructions in https://github.com/docker/compose-on-kubernetes/blob/master/docs/deploy-etcd.md line-by-line, when running the kubectl apply -f compose-etcd.yaml command, an error is raised:
error: unable to recognize "compose-etcd.yaml": no matches for kind "EtcdCluster" in version "etcd.database.coreos.com/v1beta2"
I have looked through many related issues but have not been able to find a solution. Does compose-for-kubernetes work on AWS EKS?

The EtcdCluster CRD is installed by etcd-operator. Can you confirm that etcd-operator is running on your cluster ? Also what does kubectl get crd gives you?

I have run out of time to work on this and will just need to use Kompose instead sorry. Maybe I will get back to this eventually. But will close for now.


Add vint.version for better version handling

This will provide the correct version when using
pip install --editable.

We could also use https://pypi.org/project/setuptools_scm/ - what do you think?

Re setuptools_scm: https://github.com/Vimjas/covimerage/pull/43

Changed to use setuptools_scm.

Codecov Report

Merging #283 into master will increase coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #283      +/-   ##
==========================================
+ Coverage   70.77%   70.79%   +0.01%
==========================================
  Files         133      134       +1
  Lines        7737     7735       -2
  Branches     1252     1252
==========================================
  Hits         5476     5476
+ Misses       1932     1930       -2
  Partials      329      329


Missing parentheses in call to 'print'

Hello,
I try to install xgboost on Anaconda 3 with the command pip install xgboost.
I have the following error: SyntaxError: Missing parentheses in call to 'print'
It occurs on line 13 in setup.py, when 'print output' is called.
Thanks!

Hi,
I am still facing the same issue with the parentheses when trying to install xgboost on Anaconda3.
File "", line 20, in
File "~\pip-build-pdt_v902\xgboost\setup.py", line 18
print output
^
SyntaxError: Missing parentheses in call to 'print'
----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in
~\pip-build-pdt_v902\xgboost
Is there anything else that needs to be done? Thank you!

Please try to clone from the most recent version from github and install from there


Feat: Code Enhancements

Hi
This PR to add some code enhancements for Hotel Reservation System project.
Thanks.

I think it is now synced with original @Kushal997-das

@TawfikYasser it's says "This branch is 7 commits behind Kushal997-das:main."

I don’t know what is the problem, Before i open a PR, I have sync the repo to be sure it fetched.

Okay You can close the PR and I will open one again


LENGTH_MILLIMETERS?

home-assistant    | 2021-01-24 11:37:04 ERROR (MainThread) [homeassistant.setup] Unable to prepare setup for platform bureau_of_meteorology.sensor: Platform not found (cannot import name 'LENGTH_MILLIMETERS' from 'homeassistant.const' (/usr/src/homeassistant/homeassistant/const.py)).
home-assistant    | 2021-01-24 11:37:04 ERROR (MainThread) [homeassistant.setup] Unable to prepare setup for platform bureau_of_meteorology.weather: Platform not found (cannot import name 'LENGTH_MILLIMETERS' from 'homeassistant.const' (/usr/src/homeassistant/homeassistant/const.py)).
HA:
Home Assistant 0.114.4
Frontend version: 20200811.0 - latest
HA running on HYPRIOT in a docker container rather than using HASSIO.  I used docker exec -it  /bin/bash to jump into the container.  Indeed /usr/src/homeassistant/homeassistant/const.py does NOT contain a definition for LENGH_MILLIMETERS in the version of home-assistant I am running.  Not that keen to change versions at the mo as I have a home system up and running and the last HA upgrade I tried required rework and pain to get things up and running again.
I am otherwise running off windows on a laptop without docker, so hooking up to a remoted docker container on the RPI from windows, using vscode tools, seems off the cards for the moment.
I cannot run or install nano into the HA container as is.  I guess I am looking at a hack of the BOM sensor.py as I can get to that outside the container in the local volume?
Cheers,
A

DOH, same for PERCENTAGE.  Opted to change to UNIT_PERCENTAGE which is in the source constant file.

DOH, same for PERCENTAGE.  Opted to change to UNIT_PERCENTAGE which is in the source constant file.

You need to be running 0.116 I think. You should probably upgrade anyway after the recent security issues.

You need to be running 0.116 I think. You should probably upgrade anyway after the recent security issues.


[VL] Allow updating native hdfs credential

What changes were proposed in this pull request?
Allow update native hdfs credential if the new credential is different with before.
Related to https://github.com/oap-project/gluten/pull/4184
How was this patch tested?
Pass CI

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query
log/native_4736_time.csv
log/native_master_02_22_2024_996ff4c89_time.csv
difference
percentage

q1
35.18
32.51
-2.669
92.41%

q2
24.08
24.28
0.199
100.83%

q3
38.95
38.12
-0.836
97.85%

q4
35.79
38.44
2.647
107.40%

q5
70.65
71.06
0.406
100.57%

q6
5.51
5.36
-0.149
97.30%

q7
84.22
85.47
1.252
101.49%

q8
85.93
88.36
2.430
102.83%

q9
121.48
123.63
2.157
101.78%

q10
43.29
44.24
0.957
102.21%

q11
20.30
21.09
0.792
103.90%

q12
26.57
26.06
-0.503
98.11%

q13
45.89
46.65
0.760
101.66%

q14
21.19
15.31
-5.882
72.24%

q15
29.74
31.22
1.475
104.96%

q16
14.11
13.80
-0.311
97.80%

q17
101.87
104.44
2.572
102.52%

q18
147.98
149.48
1.505
101.02%

q19
14.03
13.72
-0.310
97.79%

q20
28.10
28.41
0.307
101.09%

q21
226.14
226.51
0.373
100.16%

q22
13.77
13.62
-0.150
98.91%

total
1234.76
1241.78
7.022
100.57%


Fix CRD registration

Signed-off-by: Frank Jogeleit<EMAIL_ADDRESS>

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

22 of 30 (73.33%)  changed or added relevant lines in 1 file are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.5%) to 92.824%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pkg/kubernetes/report_adapter.go
22
30
73.33%

Files with Coverage Reduction
New Missed Lines
%

pkg/kubernetes/report_adapter.go
2
75.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.5%

Covered Lines:
1591

Relevant Lines:
1714

💛  - Coveralls


Unread notifications without counts

Can we provide an unread notification similar to the built-in slack integration that would allow sites that either don't provide an unread count, or don't provide one on every page to at least alert?
Ideally the notification would show up like the number badges on the icon and not to the right like slack does.

The setBadge function will take text, it appears.  It would be nice to have this still put the indicator on the icon.
Franz.setBadge("·");

This should also get documented

You can use the indirectMessages parameter:
Franz.setBadge(false, true);

Cool.  Thank you.


NumberBox does not respect Background

Describe the bug
This xaml:
<StackPanel
    Orientation="Horizontal"
    Spacing="10">
    <TextBox
        Background="SkyBlue"
        Foreground="HotPink"
        Header="TextBox"
        Text="{x:Bind NumberBoxControl.Text, Mode=OneWay}" />
    <AutoSuggestBox
        Background="SkyBlue"
        Foreground="HotPink"
        Header="AutoSuggestBox"
        Text="{x:Bind NumberBoxControl.Text, Mode=OneWay}" />
    <NumberBox
        x:Name="NumberBoxControl"
        Background="SkyBlue"
        Foreground="HotPink"
        Header="NumberBox" />
</StackPanel>

renders:

Steps to reproduce the bug
Use the code above.
Expected behavior
NumberBox respect its assigned Background.
Screenshots
No response
NuGet package version
WinUI 3 - Windows App SDK 1.4.2: 1.4.231008000
Windows version
No response
Additional context
Related to:

#2998
#4469

According to NumberBox.xaml, the InputBox doesn't bring any Background.

Fixed in internal PR: 10250839


Set Voice Lowpass is missing

It seems that you forgot to call VRCPlayerApi.SetVoiceLowpass(false) on players that should be audible over longer distances. VRChat adds a low pass effect based on player distance if this is enabled, which means it is really hard to understand other players over your radio asset.
This is enabled by default, but needs to get disabled while players should be audible over radio.
See https://docs.vrchat.com/docs/player-audio#set-voice-lowpass

Thanks to you, I now understand how SetVoiceLowpass works.
I knew about the effect of the filter at long distances, however many users were positive that it was "radio-like", so I put it off.
On the other hand, I understand very well that it is hard to hear. I'll try to implement it as an switchable feature in the next update.


very slow build times

Current Behavior

Very slow build times.
running SpeedMeasurePlugin (https://github.com/stephencookdev/speed-measure-webpack-plugin) ,
including vue-loader running several times on each file, and also parsing node_modules
(despite skipLibCheck: true, and excluding node_modules in tsconfig files)
Some files are taking 12 seconds per file!
Expected Behavior

Faster build times
Steps to Reproduce

This issue may not be prioritized if details are not provided to help us reproduce the issue.
Failure Logs

Environment

Plugin name and version:

The build time is slowest when editing less files.
We're using Ant Design Vue UI, which uses less.
Interestingly:

the Ant Design Vue npm package specifies less-loader at 6.0.0 (which had significant breaking changes),
the nx-plus/vue package.json has less-loader 5.0.0 as a dependency.
the node_modules folder has less-loader at 5.0.0 (ie the same as the nx-plus/vue version).
however our package.json file has no less-loader dependencies.

haven't looked deeply enough into nx-plus/vue, but I'm wondering it it somehow installed less-loader 5.0.0, while ant design vue is expecting version 6.0.0. (as it was itself built with that version?
Does that make sense?


Unable to start the app on macOS if System settings "Always open documents in Tabs" enabled.

Describe the bug
When I started ghidra I can't press anything, check out the video.
https://user-images.githubusercontent.com/3390357/106386494-85dd7880-63d5-11eb-85fa-91b393d94fd7.mov
Steps to reproduce the behavior:

Go to macOS system Settings, General, enable "Always open documents in Tabs"
Start the ghidra

Expected behavior
Mouse clicks to work
Environment (please complete the following information):

OS: macOS latest
Java Version: tried many
Ghidra Version: latest

Duplicate of #2531

Duplicate of #2531


cleaning_mode sweeping not available on r2251a

First than you so much for this amazing repo.
Describe the bug
cleaning mode choices are only sweep/mop or mop on my dreame l10 prime which does have a lifting mop pad
To Reproduce
On install the entity is created but no sweep mode is available
Expected behavior
since my robot has a lifting mop I expect to be able to choose sweep only mode
Screenshots
If applicable, add screenshots to help explain your problem.
Additional Information (please complete the following information)

Model Name [e.g. dreame.vacuum.p2028]: r2251a dreame l10 prime
Firmware Version [e.g. 1156]: 4.3.3_1505
Home Assistant Version: 2023.10.1
Configuration Type [With or without map support]: with map support
Errors or warnings shown in the HA logs (if applicable

Confirm the same issue for r2216o

The l10 prime does have a self wash base though just FYI.
On Thu, 12 Oct 2023, 12:03 pm Tasshack, @.***> wrote:

There was only one vacuum that has mop lifting feature but does not have a
self wash base but seems like there are three now.
I have to add those specific model numbers to the project to make the
integration know that device has this feature.
—
Reply to this email directly, view it on GitHub
https://github.com/Tasshack/dreame-vacuum/issues/238#issuecomment-1758756115,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABHW5SNGUVP66RP2FXCSNYTX646WZANCNFSM6AAAAAA53U3PIM
.
You are receiving this because you authored the thread.Message ID:
@.***>

Hi,
without trying to hijack this issue. L10 Ultra is facing almost the same problem. Cleaning mode is completely unavailable. It has lifting moppads and cleaning base station.
Model number r2257o
If you need more info or testers, let me know!

If i try to call set_custom_cleaning with even the simple example yaml with dreambot L10S ultra i have:
Invocazione del servizio dreame_vacuum.vacuum_set_custom_cleaning non riuscita. Parameter count mismatch!
Using v2.0.0b6

here you are my sample... now the service has not error... but in the log of home assistant i get
Referenced devices vacuum.dreamebot_l10s_ultra are missing or not currently available
service: dreame_vacuum.vacuum_set_custom_cleaning
data:
  segment_id:
    - 7
    - 9
  suction_level:
    - 3
    - 3
  water_volume:
    - 2
    - 3
  cleaning_mode:
    - 2
    - 2
  repeats:
    - 2
    - 2
target:
  device_id: vacuum.dreamebot_l10s_ultra

Log is below... anyway my map is composed by 11 rooms, i want to make a custom button in home assistant that just clean room 7 and room 7 in a program... it is possible with that service?
2023-10-19 11:19:56.378 ERROR (MainThread) [custom_components.dreame_vacuum] Unable to call set_custom_cleaning: Parameter count mismatch! 2023-10-19 11:19:56.380 ERROR (MainThread) [homeassistant.helpers.script.websocket_api_script] websocket_api script: Error executing script. Error for call_service at pos 1: Parameter count mismatch! 2023-10-19 11:19:56.382 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [139732789998656] Error handling message: Parameter count mismatch! (unknown_error) Vanni Brutto from <IP_ADDRESS> (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/1 <IP_ADDRESS> Safari/537.36)

As i said before you cannot just set two of the segments of the whole map. You need to pass all settings for all 11 of your segments at once with this service. If you don't want set all room settings at once you need to use the segment select entities. It is not possible to disable some rooms from general cleaning, you need to use vacuum_clean_segment service for that.

Ok, but clean_segment is missing parameter "Cleaning Mode".... or a solution is a way to change first the cleaning mode and then use clean_segment....

This is a screenshot from "xiaomi app"... maybe if there is a way to change "cleaning mode" (first icon from this app) before to start clean_segment i have just solved my problem....

O thanks! select.cleaning_mode is what i need! Great!
Last question (sorry, i'm newbie), i'm trying this yaml but it give error... i thank that "Pulizia con panno e aspirazione" is wrong value, maybe i need to use english value? If it's true what are option that i need to use for call select_option service?
service: select.select_option
data:
  option: "Pulizia con panno e aspirazione"
target:
  entity_id: select.dreamebot_l10s_ultra_cleaning_mode

found looking source code! sweeping_and_mopping sweeping mopping
sorry for spam!

Could you share your solution? I’m trying to achieve the exact same thing. Documentation is pretty hard for newbies (at least for me).

found looking source code! sweeping_and_mopping sweeping mopping
sorry for spam!

Could you share your solution? I’m trying to achieve the exact same thing. Documentation is pretty hard for newbies (at least for me).

here you are a sample script that call select_option with "sweeping_and_mopping" before to select id of rooms to clean...
alias: DREAMBOT - Camere Aspira Lava 2X
sequence:
  - service: select.select_option
    data:
      option: sweeping_and_mopping
    target:
      entity_id: select.dreamebot_l10s_ultra_cleaning_mode
  - delay:
      hours: 0
      minutes: 0
      seconds: 5
      milliseconds: 0
  - service: dreame_vacuum.vacuum_clean_segment
    data:
      repeats: 2
      segments:
        - 4
        - 5
        - 6
        - 8
        - 9
        - 7
    target:
      entity_id: vacuum.dreamebot_l10s_ultra
mode: single

Any ETA on the fix? My L10s Pro is pretty much unusable from HA right now.

Hi @chrismelba,
I am sorry to post an unrelated question here but.
How how did you manage to connect your L10 prime to the Xiaomi home app?
I tried a lot of things this week, but I cannot figure out how you did it. I can't find the L10 prime in the selection and auto-detect fails. I tried selecting other bots in the L10 line up, but I get stuck at 60% wen the vacuum connect to wifi and tries to communicate with the cloud. So I left it in the dream home app but I really want it in HA
Any help would be readily appreciated


Ensure follower commitIndex is not increased beyond AppendRequest entry index

This PR fixes #265 by simply following the solution in the Raft paper. The change simply ensures that the follower's commitIndex cannot be increased beyond the index of the last entry in an AppendRequest to ensure that consistency checks occur prior to committing an entry.

Had to fix a bug in an unrelated test to get the tests passing.

Coverage increased (+0.02%) to 43.213% when pulling aa7a7b787992d8b7fff86032032a1a4580aabba1 on follower-commit-index-bug into bc0109b64bd1609ed9b19f14fa047d5bf4cd01c1 on master.


simplify embed-id test

Depends #7960
Merge after #7960

I am not getting this error locally, could someone have a look?

#7960 should fix the issue

@dido1998 can you include this code in your PR?
https://github.com/chainer/chainer/pull/7960
lets commit it all together.
Thanks!

Fix flake8
/home/travis/build/chainer/chainer/chainerx/_fallback_workarounds.py:126:1: E303 too many blank lines (3)

It is still not working. Are more changes required to the code added in #7960 ?

what is not working?
travis checks were ok

Jenkins, test this please

If I put ignore_label : 1 instead of None then I get this error module 'chainerx' has no attribute 'broadcast_arrays'

does chainerx works in your env?

yes

try import chainerx
chainerx.broadcast_arrays

@emcastillo its working now!

Jenkins CI test (for commit 38e20d2ee9a312dc8662913e88038ea155c2c5da, target branch master) succeeded!

jenkins, test this please

Jenkins CI test (for commit f475bf7ff9ee73ede812c02d5265e97351deaef4, target branch master) failed with status FAILURE.

Jenkins, test this please

Jenkins CI test (for commit f475bf7ff9ee73ede812c02d5265e97351deaef4, target branch master) failed with status FAILURE.

Jenkins, test this please

Jenkins CI test (for commit f475bf7ff9ee73ede812c02d5265e97351deaef4, target branch master) failed with status FAILURE.

I think the error is not related to this PR


Add DarkBrewery theme

This PR resolves #3034.
This PR modifies the build script to copy the content of a new folder ./themes/codeMirror/customThemes/ to the Editor Themes build folder, which already has the content of the CodeMirror NPM package's internal theme folder copied to it. The modified script then builds the content of the editorThemes.json file from the build folder, not the node_modules folder as previously.
This allows custom CSS to be created and added to the Homebrewery Editor themes, exactly as if it were one of the CodeMirror supplied themes.

While I have been able to test that the build script correctly creates editorThemes.json with the new darkbrewery-v301 file, I have not yet been able to test if the contents of that file are correctly applied in the editor when that theme is selected. As such, I have marked this PR as Draft/WIP until such time as I can access my normal dev machine.
Assuming everything works as expected, my intention is to then convert this to a full PR and flag for review by @calculuschild and @5e-Cleric.

From my local dev install:

Pinging @5e-Cleric for confirmation that this is the expected result of the DarkBrewery theme.

I cna confirm

What color do mustache blocks have right now? i think i did not get around to color those but i could be mistaken.
I'm assuming you have fixed all contrast issues, but confirming is always a good idea.
We could also benefit from nesting here. Even native CSS nesting.

Didn't think this theme as something that would get its way into the actual site, that's why it's imperfect, so yeah, let's remake everything, with nesting, proper color groups, and comments.
If you don't want to have to come up with new colors for everything, simply setting mustache blocks and injections to a lighter variant of themselves should be fine.

Color wise, i propose these two colors for block and inline mustache elements:
blocks: #ccf781
inline:  #90ff90
DL: pink
DD: #99aaff

Looks good! Thanks @G-Ambatte


KeyError: 'playwright_page'

async def errback_close_page(self, failure):
        page = failure.request.meta["playwright_page"]
        await page.close()

def start_requests(self):
        if not self.start_urls and hasattr(self, "start_url"):
            raise AttributeError(
                "Crawling could not start: 'start_urls' not found "
                "or empty (but found 'start_url' attribute instead, "
                "did you miss an 's'?)"
            )
        for url in self.start_urls:
            npo = self.npos[url]
            logging.info("### crawl: %s", url)
            yield scrapy.Request(
                url, callback=self.my_parse, dont_filter=True,meta={"playwright": True, "playwright_include_page": True, 'start_time': datetime.utcnow()}, cb_kwargs={"npo": npo},errback= self.errback_close_page

            )

can anyone please explain why am I getting this error and how can I fix this?
Traceback (most recent call last): File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/twisted/internet/defer.py", line 1065, in adapt extracted = result.result() File "/home/ec2-user/SageMaker/xx", line 50, in errback_close_page page = failure.request.meta["playwright_page"] KeyError: 'playwright_page'

It could be that you're not correctly activating the download handler. Are you seeing log lines from the scrapy-playwright logger? i.e. something like:
2024-05-25 16:06:00 [scrapy-playwright] INFO: Starting download handler
2024-05-25 16:06:05 [scrapy-playwright] INFO: Launching browser chromium
2024-05-25 16:06:05 [scrapy-playwright] INFO: Browser chromium launched

Yes ,

i have added download handlers in my settings.py
DOWNLOAD_HANDLERS = {
"http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
"https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
}

Any updates ? still not able to solve the KeyError: 'playwright_page'

There is not enough information to reproduce. Please refer to https://github.com/scrapy-plugins/scrapy-playwright?tab=readme-ov-file#reporting-issues.

More Detail
I want to use my spider to get info from https://www.manamana.net/exploreauth, I write code obey the sample, but get this error too.
And then I translate my code as this
Then I get the picture

But I want to get this

The reason is that there are several seconds between this two kind, I think there are some bug for playwright to render the page which is dynamic

The reason is that there are several seconds between this two kind, I think there are some bug for playwright to render the page which is dynamic

This is precisely why the docs suggest trying standalone Playwright for debugging. You are not getting the result you expect with standalone Playwright, so the problem is not caused by scrapy-playwright.
This is not a bug though. I'd recommend ensuring the DOM is fully loaded before returning the result by passing wait_until="domcontentloaded" to Page.goto in standalone Playwright or via the playwright_page_goto_kwargs Request meta key scrapy-playwright.

but get this error too

If by "this error" you mean the originally reported error about the a KeyError on the playwright_page meta key, there is no Scrapy code in your to work with.

The KeyError: 'playwright_page' , I solved it my declaring errback inside meta.
yield scrapy.Request(
                url, callback=self.my_parse, dont_filter=True, meta=dict(playwright=True, playwright_include_page=True, start_time=datetime.utcnow(), errback=self.errback_close_page), cb_kwargs={"npo": npo},

            )

The failing requests have a referrer header, they are not coming from your start_requests method. You haven't shared a full spider, I suspect those specific requests do not have playwright_include_page=True in their meta.
I'm closing this, if you're still having trouble and want to reopen the issue you need to include a Minimal, Reproducible Example as instructed in the Reporting issues section.

So on further analysis I was able to pin point to exact error , Forbidden by robot.txt is causing the crawler to freeze. The issue is failure requests that are forbidden by robot.txt has no playwright_page associated to them, therefore i am not able to close them using page.close which is causing my crawler to freeze where page = failure.request.meta["playwright_page"]
Note

My crawler works absolutely fine when ROBOTSTXT_OBEY is set to false, but I want to keep it true.

Here is the minimal reproduceable code
import re
import scrapy
import logging
from bs4 import BeautifulSoup
from datetime import datetime
from urllib.parse import urlparse
from scrapy.linkextractors import LinkExtractor

emails_re = re.compile(r"\b([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})\b", re.IGNORECASE)

class GrantsSpider(scrapy.Spider):
    name = "npos"
    reported_links = []
    link_extractor = LinkExtractor(unique=True)
    npos = {}

    async def errback_close_page(self, failure):
        logging.error(f"Error processing {failure.request.url}: {repr(failure)}")
        page = failure.request.meta.get("playwright_page")
        if page:
            await page.close()
            await page.context.close()
            logging.info(f"Page {page} closed on error")

    def start_requests(self):
        url = 'https://www.guidestar.org/profile/26-1700710'
        npo = {"ein": "26-1700710", "name": "Example NPO", "type": "Nonprofit"}
        logging.info(f"### crawl: {url}")
        yield scrapy.Request(
            url, callback=self.my_parse, dont_filter=True,
            meta={"playwright": True, "playwright_include_page": True, 'dont_redirect': True},
            cb_kwargs={"npo": npo},
            errback=self.errback_close_page
        )

    async def my_parse(self, response, npo):
        page = response.meta["playwright_page"]
        self.reported_links.append(response.url)
        try:
            _ = response.text
        except AttributeError as exc:
            logging.debug(f"Skipping {response.url}: {exc}")
            await page.close()
            return

        body, match = self.is_page(response, None)
        for email in emails_re.findall(body):
            yield {
                "ein": npo["ein"],
                "name": npo["name"],
                "type": npo["type"],
                "msg": "link-email",
                "match": email,
                "text": email,
                "url": response.url,
                "timestamp": datetime.utcnow(),
            }

        for link in response.xpath("//a"):
            href = link.xpath("./@href").get()

            if not href or href.startswith("javascript:") or href.startswith("#"):
                continue

            if not href.startswith("http"):
                href = response.urljoin(href)

            if href not in self.reported_links:
                yield scrapy.Request(
                    href, callback=self.my_parse,
                    meta={"playwright": True, "playwright_include_page": True, 'dont_redirect': True},
                    cb_kwargs={"npo": npo},
                    errback=self.errback_close_page
                )

        await page.close()

    def is_page(self, response, re_expression):
        sel = scrapy.Selector(response)
        sel.xpath("//head").remove()
        sel.xpath("//header").remove()
        # sel.xpath("//footer").remove()
        sel.xpath("//navbar").remove()
        sel.xpath("//a").remove()
        body = sel.get()
        bs_doc = BeautifulSoup(body, features="lxml").get_text(strip=True, separator=" ")
        if not re_expression:
            return bs_doc, None
        if re_expression.search(bs_doc):
            matches = list(set(list(re.findall(re_expression, bs_doc))[0]))
            if "" in matches:
                matches.remove("")
            return bs_doc, matches
        return None, None

Sounds reasonable, from a quick look I think this is the Scrapy exception you're getting.
I'd say that's expected behavior though, you'll just need to check in your errback if there is actually a page to close before attempting to close it. Furthermore, unless your actual spider is bigger and what you posted is a reduced example, I'd recommend you ask yourself if it's really necessary to pass playwright_include_page=True. You don't seem to be interacting with the page besides closing it, you might be better of not passing playwright_include_page and allowing the handler to close the page on its own. See also these docs.

The actually spider is much bigger and this is just a minimal code of it. I was able to solve this by changing my errback method a bit
async def errback_close_page(self, failure):
        self.logger.error(f'Error processing page: {repr(failure)}')
        if "playwright_page" in failure.request.meta:
            page = failure.request.meta["playwright_page"]
            if page:
                await page.close()
                self.logger.info(f"Closed page due to error: {page}")
        raise CloseSpider(reason='Forbidden by robots.txt')

Hi Eugenio, even after making the changes I am still facing the issue where my spider gets stuck even if its closed
{'BOT_NAME': 'test',
 'CLOSESPIDER_ITEMCOUNT': 300,
 'CLOSESPIDER_PAGECOUNT': 300,
 'CLOSESPIDER_TIMEOUT': 300,
 'CONCURRENT_REQUESTS': 8,
 'CONCURRENT_REQUESTS_PER_DOMAIN': 4,
 'CONCURRENT_REQUESTS_PER_IP': 4,
 'COOKIES_ENABLED': False,
 'DEPTH_LIMIT': 2,
 'DEPTH_PRIORITY': 1,
 'DNS_TIMEOUT': 3,
 'DOWNLOAD_DELAY': 0.3,
 'DOWNLOAD_TIMEOUT': 120,
 'FEED_EXPORT_ENCODING': 'utf-8',
 'LOG_LEVEL': 'INFO',
 'NEWSPIDER_MODULE': 'myConfig.spiders',
 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
 'RETRY_ENABLED': False,
 'ROBOTSTXT_OBEY': True,
 'SCHEDULER_DISK_QUEUE': 'scrapy.squeues.PickleFifoDiskQueue',
 'SCHEDULER_MEMORY_QUEUE': 'scrapy.squeues.FifoMemoryQueue',
 'SPIDER_MODULES': ['myConfig.spiders'],
 'TELNETCONSOLE_ENABLED': False,
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor',
 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
               '(KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36'}
2024-06-26 20:28:06 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.closespider.CloseSpider',
 'scrapy.extensions.logstats.LogStats']
2024-06-26 20:28:06 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'myConfig.middlewares.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware']
2024-06-26 20:28:06 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2024-06-26 20:28:06 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2024-06-26 20:28:06 [scrapy.core.engine] INFO: Spider opened
2024-06-26 20:28:07 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
#### Run crawlers 25040-25040
2024-06-26 20:28:07 [scrapy-playwright] INFO: Starting download handler
2024-06-26 20:28:07 [scrapy-playwright] INFO: Starting download handler
2024-06-26 20:28:12 [root] INFO: ### crawl: http://www.trinityschooloftexas.com
2024-06-26 20:28:12 [scrapy-playwright] INFO: Launching browser chromium
2024-06-26 20:28:12 [scrapy-playwright] INFO: Browser chromium launched
2024-06-26 20:28:18 [scrapy.core.scraper] ERROR: Spider error processing <GET http://www.trinityschooloftexas.com> (referer: None)
Traceback (most recent call last):
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/utils/defer.py", line 276, in aiter_errback
    yield await it.__anext__()
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/utils/python.py", line 360, in __anext__
    return await self.data.__anext__()
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/utils/python.py", line 341, in _async_chain
    async for o in as_async_generator(it):
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator
    async for r in it:
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/utils/python.py", line 360, in __anext__
    return await self.data.__anext__()
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/utils/python.py", line 341, in _async_chain
    async for o in as_async_generator(it):
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator
    async for r in it:
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/core/spidermw.py", line 118, in process_async
    async for r in iterable:
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/spidermiddlewares/offsite.py", line 31, in process_spider_output_async
    async for r in result or ():
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/core/spidermw.py", line 118, in process_async
    async for r in iterable:
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async
    async for r in result or ():
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/core/spidermw.py", line 118, in process_async
    async for r in iterable:
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async
    async for r in result or ():
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/core/spidermw.py", line 118, in process_async
    async for r in iterable:
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async
    async for r in result or ():
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/core/spidermw.py", line 118, in process_async
    async for r in iterable:
  File "/home/ec2-user/SageMaker/grant-crawler/myConfig/spiders/npos.py", line 75, in my_parse
    yield scrapy.Request(
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/http/request/__init__.py", line 93, in __init__
    self._set_url(url)
  File "/home/ec2-user/SageMaker/grant-crawler/venv/lib/python3.10/site-packages/scrapy/http/request/__init__.py", line 143, in _set_url
    raise ValueError(f"Missing scheme in request url: {self._url}")
ValueError: Missing scheme in request url<EMAIL_ADDRESS>2024-06-26 20:28:18 [scrapy-playwright] INFO: Launching browser chromium
2024-06-26 20:28:19 [scrapy-playwright] INFO: Browser chromium launched
2024-06-26 20:29:07 [scrapy.extensions.logstats] INFO: Crawled 3 pages (at 3 pages/min), scraped 0 items (at 0 items/min)
2024-06-26 20:30:07 [scrapy.extensions.logstats] INFO: Crawled 3 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
^C2024-06-26 20:30:22 [scrapy.crawler] INFO: Received SIGINT, shutting down gracefully. Send again to force
2024-06-26 20:30:22 [scrapy.core.engine] INFO: Closing spider (shutdown)
^C2024-06-26 20:30:22 [scrapy.crawler] INFO: Received SIGINT twice, forcing unclean shutdown
#### End crawlers 25040-25040
execution time:<PHONE_NUMBER>5818176
^CScript execution complete.
(venv) sh-4.2$ ./run.sh
Starting script execution...
en_core_web_sm is already downloaded.
2024-06-26 20:30:36 [scrapy.utils.log] INFO: Scrapy 2.9.0 started (bot: test)
2024-06-26 20:30:36 [scrapy.utils.log] INFO: Versions: lxml <IP_ADDRESS>, libxml2 2.9.14, cssselect 1.2.0, parsel 1.8.1, w3lib 2.1.1, Twisted 22.10.0, Python 3.10.8 | packaged by conda-forge | (main, Nov 22 2022, 08:23:14) [GCC 10.4.0], pyOpenSSL 23.2.0 (OpenSSL 3.1.1 30 May 2023), cryptography 41.0.1, Platform Linux-5.10.215-203.850.amzn2.x86_64-x86_64-with-glibc2.26
#### Create crawler for 25040: www.trinityschooloftexas.com
2024-06-26 20:30:37 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'test',
 'CLOSESPIDER_ITEMCOUNT': 300,
 'CLOSESPIDER_PAGECOUNT': 300,
 'CLOSESPIDER_TIMEOUT': 300,
 'CONCURRENT_REQUESTS': 8,
 'CONCURRENT_REQUESTS_PER_DOMAIN': 4,
 'CONCURRENT_REQUESTS_PER_IP': 4,
 'COOKIES_ENABLED': False,
 'DEPTH_LIMIT': 2,
 'DEPTH_PRIORITY': 1,
 'DNS_TIMEOUT': 3,
 'DOWNLOAD_DELAY': 0.3,
 'DOWNLOAD_TIMEOUT': 120,
 'FEED_EXPORT_ENCODING': 'utf-8',
 'LOG_LEVEL': 'INFO',
 'NEWSPIDER_MODULE': 'myConfig.spiders',
 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
 'RETRY_ENABLED': False,
 'ROBOTSTXT_OBEY': True,
 'SCHEDULER_DISK_QUEUE': 'scrapy.squeues.PickleFifoDiskQueue',
 'SCHEDULER_MEMORY_QUEUE': 'scrapy.squeues.FifoMemoryQueue',
 'SPIDER_MODULES': ['myConfig.spiders'],
 'TELNETCONSOLE_ENABLED': False,
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor',
 'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
               '(KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36'}
2024-06-26 20:30:37 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.closespider.CloseSpider',
 'scrapy.extensions.logstats.LogStats']
2024-06-26 20:30:37 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'myConfig.middlewares.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware']
2024-06-26 20:30:37 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2024-06-26 20:30:37 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2024-06-26 20:30:37 [scrapy.core.engine] INFO: Spider opened
2024-06-26 20:30:38 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
#### Run crawlers 25040-25040
2024-06-26 20:30:38 [scrapy-playwright] INFO: Starting download handler
2024-06-26 20:30:38 [scrapy-playwright] INFO: Starting download handler
2024-06-26 20:30:43 [root] INFO: ### crawl: http://www.trinityschooloftexas.com
2024-06-26 20:30:43 [scrapy-playwright] INFO: Launching browser chromium
2024-06-26 20:30:43 [scrapy-playwright] INFO: Browser chromium launched
2024-06-26 20:30:49 [scrapy.core.scraper] ERROR: Spider error processing <GET http://www.trinityschooloftexas.com> (referer: None)
2024-06-26 20:30:49 [scrapy-playwright] INFO: Launching browser chromium
2024-06-26 20:30:50 [scrapy-playwright] INFO: Browser chromium launched
2024-06-26 20:31:38 [scrapy.extensions.logstats] INFO: Crawled 3 pages (at 3 pages/min), scraped 0 items (at 0 items/min)
2024-06-26 20:32:38 [scrapy.extensions.logstats] INFO: Crawled 3 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-06-26 20:33:38 [scrapy.extensions.logstats] INFO: Crawled 3 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-06-26 20:34:38 [scrapy.extensions.logstats] INFO: Crawled 3 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-06-26 20:35:38 [scrapy.core.engine] INFO: Closing spider (closespider_timeout)
2024-06-26 20:35:38 [scrapy.extensions.logstats] INFO: Crawled 3 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-06-26 20:36:38 [scrapy.extensions.logstats] INFO: Crawled 3 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-06-26 20:37:38 [scrapy.extensions.logstats] INFO: Crawled 3 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

I am linking the GitHub repo with minimal code which will help you replicate the issue , any help would be appreciated. Thanks
https://github.com/nekender/Playwright_Issue/

Hi @elacuesta , did you get the time to have look ?

Is not clear to me what the issue is in your last report. Furthermore, what you shared is not minimal: there are multiple files, middlewares, regex processing, selector manipulation, HTML parsing with BeautifulSoup, etc, none of which are likely to be related to what you report. It is often the case that developers understand and solve issues on their own during the process of distilling a program down to the bare minimum necessary to reproduce a problem.
So far this issue doesn't contain any reproducible bug report, it has always been a support matter and there are better resources to handle those (e.g. the scrapy-playwright tag at StackOverflow).


Possible scroll issue. Bug introduced in version 6.0.2

from user @ Croge32
I'm noticing another issue that may or may not be related to your latest updates.
When I start my application I'm in one row mode at the current date. If I click my expand button I am in the same month. If I retract the calendar back to 1 row, the dates shown are the 24th - 30th of January, so almost 1 entire year back. Code above is basically the same.
Also calendarView.scrollToNextSegment() and calendarView.scrollToPreviousSegment() seem broken with 1 row. ScrollToPrevious goes back by entire months even when set to 1 row and scrollToNext doesn't scroll at all.

@patchthecode Have you discovered anything new about this?

@Croge32 Yes. I am working on it.
But just to confirm, can you paste me your configureCalendar function code for the single row calendar that is messing up?

Also, can you show me the code you use to expand and retract the calendar?
I believe I have fixed the issue where it scrolls back the entire year. But I need to see you code in order to know if my fixed works.

Sure.
Here is my configureCalendar function:
func configureCalendar(_ calendar: JTAppleCalendarView) -> ConfigurationParameters {
    let formatter = DateFormatter()
    formatter.dateFormat = "yyyy MM dd"

    let rows = calendarOut ? 6 : 1
    let inDates: InDateCellGeneration =  calendarOut ? .forAllMonths : .forFirstMonthOnly
    let outDates: OutDateCellGeneration = calendarOut ? .tillEndOfGrid : .off

    let startDate = currentCalendar.date(byAdding: .year, value: -1, to: Date())!
    let endDate = Date()
    let parameters = ConfigurationParameters(
      startDate: startDate,
      endDate: endDate,
      numberOfRows: rows,
      calendar: currentCalendar,
      generateInDates: inDates,
      generateOutDates: outDates,
      firstDayOfWeek: .sunday)

    return parameters
  }

And here is my code to expand the calendar:
func expandCalendar() {
    calendarViewHeight.constant = calendarOut ? 60 : 200
    calendarOut = !calendarOut
    gesture?.direction = calendarOut ? .up : .down
    calendarView.reloadData()

    UIView.animate(withDuration: 0.2) {
      self.view.layoutIfNeeded()
    }
  }

@Croge32 thanks. looking into it

@Croge32
Ok I had a change to look at the issue.
Concerning the calendar scrolling back to the original date, this is expected.
If you have a calendar with a certain configuration, and you change orientation, then your calendar should be visible on the same month/date. But, since you are changing the calendar layout, then the cells positions will change. Now, should the library auto scroll to the date you were previously looking at? or should it not? This is not for the library to decide. This is for you (the coder) to decide. Therefore, can do something like this?

save the date you want to scroll to
reload the calendar and then scroll to the date.

something like this:
let dateToScrollTo: Date?

let allVisibleDates = calendarView.visibleDates()
// lets pick a date
date = allVisibleDates.inDates.first!

calendarView.frame = newSize
calendarView.reload {
    self.calendarView.scrollToDate(date)
}

@Croge32 Hey, I believe I have fixed the issue now.
Do you know how to test with master branch?

Testing now.
Perhaps I didn't explain my first issue clearly enough, but let me try to clarify the sequence of events:

I open my app in 1 row mode. The calendar date selects November 23rd.
I shift to 6 row mode. The date stays selected and it is on the current month, November.
I shift back to 1 row mode. The date lands on the week of January 24-30th.
I shift back to 6 row mode. The date is still in November.

I see what you're saying about manually shifting it, but I am wondering why this doesn't occur when shifting back to 6 month mode? Also considering the range of my calendar is November 2015-November 2016, so January is a seemingly random place for it to shift to.
BTW it seems 1 or 6 row calendar scrolling is fixed!

It happens because of this (i specuate).

When you start with One-row mode, it is at the end of the UICollectionView content offset. Because according to what you have told me, Your end date is this month 2016.
So Let's assume this offset is

xOffset: CGFloat  = 2000

So since your calendar is at the end of your date boundary, your calendar is thus at the end of the content offset of the UICollectionView.
2. When you shift to 6 row, the UICollectionView content size will shrink, because 6 rows of data is on the screen (instead of being stretched out like the single row calendar). Therefore, when you shift to 6 row, the contentSize has shrunk to say 1000. Therefore on the 6 row calendar, your offset of 1000 will still be on the end of the calendar i.e. November this month.
3. You shift back to One-row. Your content size has grown to 2000 again, but your actual xOffset is on 1000 where it left from the 6-row calendar. The offset of 1000 on your single row calendar happens to land on January.
4. You shift back to 6 row. Your content Size has again shrunk back to 1000. Your xOffset is also on 10000 i.e the End of the calendar i.e This month November.
Like the library (or think my explanation was helpful)? Feel free to leave a Star rating ★ rating on Github.
I will greatly appreciate it :)

@Croge32

I'll close this issue.
If you have any new issues, then do not hesitate to let me know.
This library should be in a state of no bug.
Have a good one. 👍

Got it working for me! Awesome stuff, thanks!

Hey, my issue was actually 196 but it was tracked here. My problem is when I select one and scroll, then select another one and scroll back, it shows multiple cells as selected. This happens at 2 row view. I can create another issues if needed. I wanted to bring it up here since it is relevant.
Thank you in advance.


CMD-R does not work most of the time

Sometimes CMD-R (Reload Page) does not work.

I don't think the template is doing anything special for controlling CMD-R and this is being handled by Electron.
I'll upgrade the template to use the latest Electron. Beyond that it may be an Electron issue.
Thanks for reporting this.

I've just released a new version of the template if the latest dependencies. Hopefully this issue has been addressed in the latest electron.


Discussion: Conversion of NSException to NSError

There are several places where FBLPromise uses Objective-C exception handling to catch NSExceptions and convert them to NSErrors. I see several problems in this approach:

Exceptions in Objective-C are usually used for fatal errors or programmer errors. Recovering from most exceptions results in undefined state.
By default Objective-C is not exception-safe when using ARC. This leads to leaked memory. There is a flag to opt in to making ARC exception-safe (-fobjc-arc-exceptions) but this has to be used for all code participating in the stack where an exception is thrown which in most cases can't be guaranteed.

What was the reasoning behind recovering from exceptions?

I believe handling ObjC exceptions is indeed an error prone overkill, so let's get rid of that behavior.
Many thanks for raising the issue, Tom!

Hi @shoumikhin,
Looks like you haven't updated the docs regarding this behaviour.

@shali3 The docs don't explicitly mention exceptions at all and in my opinion this is fine. The library doesn't catch exceptions anymore which is the default and expected behavior for any library in Objective C so it's not necessary to state that exceptions are not catched.

From the docs (link in my previous comment)

From the docs (link in my previous comment):

Observing rejection
To get notified when a promise is resolved with an error (i.e. is rejected), use the catch operator.
You can reject a promise in many ways:
call reject method on a promise
call reject() in an async block or return an error from the do block
return or throw an error from the then block
Or, just create a resolved promise with an error.

From an ObjC perspective when I read error I think of NSError.


Add django-classy-tags requirement

Task linked: QF-1723 Allow permission checks in django templates to be stored as variable


Scrooge Error when defining Thrift exception

I noticed when a Thrift file defines an exception, scrooge throws an exception:
For example adding:
exception FooBar {
  1: string code
}

to test/src/main/scala/scalarules/test/twitter_scrooge/thrift/thrift2/thrift3/Thrift3.thrift results in this:
ERROR: /absolute/path/to/test/src/main/scala/scalarules/test/twitter_scrooge/thrift/thrift2/thrift3/BUILD:4:1: Couldn't build file test/src/main/scala/scalarules/test/twitter_scrooge/thrift/thrift2/thrift3/thrift3_scrooge.jar: scala //test/src/main/scala/scalarules/test/twitter_scrooge/thrift/thrift2/thrift3:thrift3 failed (Exit 1)
bazel-out/darwin-fastbuild/bin/test/src/main/scala/scalarules/test/twitter_scrooge/thrift/thrift2/thrift3/tmp4446905357582031766/scalarules/test/twitter_scrooge/thrift/thrift2/thrift3/FooBar.scala:171: error: object finagle is not a member of package com.twitter
  extends ThriftException with com.twitter.finagle.SourcedException with ThriftStruct
                                           ^
one error found
one error found
java.lang.RuntimeException: Build failed
	at io.bazel.rulesscala.scalac.ScalacProcessor.compileScalaSources(ScalacProcessor.java:244)
	at io.bazel.rulesscala.scalac.ScalacProcessor.processRequest(ScalacProcessor.java:69)
	at io.bazel.rulesscala.worker.GenericWorker.runPersistentWorker(GenericWorker.java:45)
	at io.bazel.rulesscala.worker.GenericWorker.run(GenericWorker.java:111)
	at io.bazel.rulesscala.scalac.ScalaCInvoker.main(ScalaCInvoker.java:41)

so, not everyone uses finagle (e.g. data uses of thrift which aren't about services). I'm reluctant to change the classpaths of everyone using the rules.
The alternate way is to configure which jars come in by changing the bind here:
https://github.com/bazelbuild/rules_scala/blob/master/twitter_scrooge/twitter_scrooge.bzl#L69
in your own repo, e.g. make that bind point to scala_library that exports both scrooge_core and finagle.

I'm going to try out your suggestion @johnynek gonna keep this open so I can ask you questions on how to go about it. I'll close it out once I have it working and contribute documentation for others to learn. How does that sound?


[legacy] Add finger touch events to SDL2Stage

Fix for https://github.com/openfl/lime/issues/360
Tested on a Dell XPS 13 with touchscreen on Linux with XInput2 driver.
Please note that this also requires changing SDL_Config in "project/lib/sdl" to enable XInput2, as follows:
diff --git a/include/configs/linux/SDL_config.h b/include/configs/linux/SDL_config.h
index 99b06c2..ba75343 100644
--- a/include/configs/linux/SDL_config.h
+++ b/include/configs/linux/SDL_config.h
@@ -274,14 +274,14 @@
 #define SDL_VIDEO_DRIVER_X11_DYNAMIC_XEXT "libXext.so.6"
 /* #undef SDL_VIDEO_DRIVER_X11_DYNAMIC_XCURSOR */
 /* #undef SDL_VIDEO_DRIVER_X11_DYNAMIC_XINERAMA */
-/* #undef SDL_VIDEO_DRIVER_X11_DYNAMIC_XINPUT2 */
+#define SDL_VIDEO_DRIVER_X11_DYNAMIC_XINPUT2 "libXi.so.6"
 /* #undef SDL_VIDEO_DRIVER_X11_DYNAMIC_XRANDR */
 /* #undef SDL_VIDEO_DRIVER_X11_DYNAMIC_XSS */
 /* #undef SDL_VIDEO_DRIVER_X11_DYNAMIC_XVIDMODE */
 /* #undef SDL_VIDEO_DRIVER_X11_XCURSOR */
 /* #undef SDL_VIDEO_DRIVER_X11_XINERAMA */
-/* #undef SDL_VIDEO_DRIVER_X11_XINPUT2 */
-/* #undef SDL_VIDEO_DRIVER_X11_XINPUT2_SUPPORTS_MULTITOUCH */
+#define SDL_VIDEO_DRIVER_X11_XINPUT2 1
+#define SDL_VIDEO_DRIVER_X11_XINPUT2_SUPPORTS_MULTITOUCH 1
 /* #undef SDL_VIDEO_DRIVER_X11_XRANDR */
 /* #undef SDL_VIDEO_DRIVER_X11_XSCRNSAVER */
 #define SDL_VIDEO_DRIVER_X11_XSHAPE 1

As I'm just beginning with lime/openfl I'm not sure whether this patch would have side effects on regular single pointer apps/games.
Also it would be cool if someone could try this on a Windows machine with touchscreen (might need to add more "defines" and also add if defined(HX_WINDOWS) in SDL2Scene). :smile:

I tested this using the flixel-demo "Multitouch" :smile:

Could you post a test app with a project.xml? I can test on Windows 8.1 with a touch screen.

@Type1J I have added changes for Windows, I hope it can make it work.
For a test app, install the flixel-demos package and compile the MultiTouch demo: flixel-demos/1,1,2/Input/Multitouch
(you can also find it here: https://github.com/HaxeFlixel/flixel-demos/tree/dev/Input/Multitouch)

For Linux:
After checking out the branch from this PR and also the one from https://github.com/native-toolkit/sdl/issues/3 in "project/lib/sdl", rebuild lime:
lime rebuild linux -64 -clean -release -Dlegacy (I often forgot the legacy option...)
Then compile the MultiTouch example from https://github.com/HaxeFlixel/flixel-demos/tree/dev/Input/Multitouch with:
lime test linux -64
I'm still confused about OpenFL hybrid/legacy/whatever. This PR is for lime legacy.
I wonder where I should submit another patch to make multitouch work for "regular" openfl/lime. Would be nice if someone could point me there.
Note that in non-legacy mode multitouch does not work on Linux.

Legacy is a combination of the old backend and OpenFL higher level code. The new code has Lime 2 and the OpenFL layer is written. Hybrid uses Lime 2, but has the older OpenFL level code. At this level, the C++ dispatch from Lime 2 would make it available to both hybrid or regular OpenFL

@jgranick is there a documentation or diagram that explains this ? Might be worth adding one to the README or wiki to avoid confusion for newcomers. :smile:


Proposal for accessible ButtonIcon example

Not tested, but would love to hear feedback what you think?

@shadcn can I get a :+1:?


Create vs. Upgrade: Independent Features?

To me, it feels like create-a-database-by-copying-from-assets is a logically independent feature from update-a-database-from-a-SQL-script-in-assets. IOW, some developers might want to use the update facility with a database not originally created from assets, and some developers might want to ship a database via assets but use the traditional onUpgrade() approach in Java (e.g., the upgrade cannot be expressed purely in SQL).
Are you supporting these features independently? IOW, can somebody extend SQLiteAssetHelper and elect to override onCreate() or onUpgrade() (not both) to handle that themselves? Or will doing so break SQLiteAssetHelper?
Thanks!

Good point. The answer to your question is no, I'm not supporting these features independently. Overriding those methods independently is going to break things.
The (arguably domain-specific) problem I was trying to solve in writing this was to totally avoid managing database creation and upgrade in code by using file assets generated elsewhere. I see how the some of the flexibility of the original SQLiteOpenHelper implementation has been crippled in my approach. I will endeavour to make changes to support your suggestions.
Thanks for your interest.

Overriding those methods independently is going to break things.

Actually, as I discovered after posting this issue, they're final and can't be overridden, anyway, ATM.

I will endeavour to make changes to support your suggestions.

IMHO, of the two, I suspect that more developers will be interested in packaging the database (using your logic) but handling updates separately (via their own onUpgrade()) than the reverse scenario.


As a blogger I want to be able to log into my account so that I can see all the blogs that I have created

As a blogger I want to be able to log into my account so that I can see all the blogs that I have created.

As a/an
I want to ...
So that ...

blogger
be able to log into my account
an see all the blogs that I have created

Acceptance Criteria
Notes

After logging in, the blogger should be redirected to all of their blog posts.

Fixed in #11


[WIP] Add bin packing logic for pods

Issue #, if available:
#228
Description of changes:
This is a WIP to be able to bin pack pods on instance type available. Open for feedback :)
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

It would be great to have a couple line description of the algorithm.
From our discussion it was (O(p * n))

Pack pods from largest to smallest, setting aside any pods that don't fit
pick the cheapest node that can fit the pods from step 1
If any pods didn't fit, iterate again until all pods packed.
Return nodes and their packed pods

Totally fine if you want to deviate, but it would be great to know in plain english. Also -- this needs to be explainable to customers, as well.

It would be great to have a couple line description of the algorithm (in the PR and potentially a README)
From our discussion it was (O(p * n))

Pack pods from largest to smallest, setting aside any pods that don't fit
pick the cheapest node that can fit the pods from step 1
If any pods didn't fit, iterate again until all pods packed.
Return nodes and their packed pods

Totally fine if you want to deviate, but it would be great to know in plain english. Also -- this needs to be explainable to customers, as well.

Yes, agree it will be good to have. I will take this up as a follow up PR.

Yes, agree it will be good to have. I will take this up as a follow up PR.
Do you mind writing up a couple sentences? I'm having a hard time grokking the algorithm's intent without it.


Access response body when UnprocessableEntity is raised

First of all, thanks for this amazing gem with very good documentation.
I would like to add a new feature and first I'd like to ask if it makes sense or not.
I am having a situation when I create a transmission with inline recipient and one of the recipient is invalid.
The SP returns a 422 and the gem raises an UnprocessableEntity exception. SP creates the transmission and need to threat this case. In this exception I can access the only the node errors through the method object. But I need to know the amount of accpeted_recipients, rejected_recipients, etc.
My proposal is only to add the response_body to be accessible on the exception too.
I wouldn't change any of the actual API so there is no need to change the code when upgrade.
If you say ok, I will fork and create the PR.
Best regards from Brazil.

Seems like a good idea, I never needed to deal with more than the status, so never added this originally

I think https://github.com/leadmachineapp/simple_spark/pull/19 would solve that

That's exactly what I would do, perfect. :)


Iterator::SeekToFirst happens to be very slow

What steps will reproduce the problem?
1. after opening a database, create a iterator
2. call iterator->SeekToFirst()
3.

What is the expected output? What do you see instead?

SeekToFirst() should return immediately. But it is blocked for about one
minute(60 seconds)!

What version of the product are you using? On what operating system?

1.9.0

Please provide any additional information below.

The total amount of data is 60GB, 1800 sst files.

Original issue reported on code.google.com by<EMAIL_ADDRESS>on 16 Sep 2013 at 4:54

It may have some deleted values. The DB::Open() also takes about one minute to
finish, I can see from info_log that LevelDB is doing compaction.

When mixing Seek()/SeekToFirst(), Seek() always returns immediately, but
SeekToFirst() is rather slow.

Original comment by<EMAIL_ADDRESS>on 16 Sep 2013 at 9:04

Do you have many deleted values that have not been compacted?

Original comment by<EMAIL_ADDRESS>on 16 Sep 2013 at 6:51

same question here.
total amount of data is only 512MB. SeekToFrist() blocked about 1 seconds
everytime i call it.I used a custom comparator which parse the key to uint64.
Did that causes the delay in SeekToFirst() ?

Original comment by<EMAIL_ADDRESS>on 4 Nov 2013 at 3:24


Integrate llama.cpp as an Inference Server

Overview
We need to add support for using llama.cpp as an inference server in our project. llama.cpp is known for its speed, cross-platform compatibility, and support for a wide range of embedding and LLM models.
Feature Description
Implement functionality to integrate llama.cpp as an inference server option in our project. This integration should leverage llama.cpp's advantages, including:

Fast inference speed
Cross-platform compatibility
Support for a wide variety of embedding and LLM models

Proposed Functionality
1. llama.cpp Integration

Implement a method to use llama.cpp as a backend for inference tasks
Allow users to specify llama.cpp as the inference engine of choice

2. Model Support

Enable loading and using various models supported by llama.cpp
Implement a system to manage and switch between different models

3. Inference Tasks

Support both embedding generation and text generation tasks
Implement appropriate interfaces for different types of inference tasks

4. Performance Optimization

Utilize llama.cpp's speed advantages to improve overall system performance
Implement batching and caching mechanisms if applicable

5. Cross-Platform Support

Ensure the integration works across different platforms (Windows, Linux, macOS)
Provide clear instructions for setup on various operating systems

Technical Requirements

Implement a wrapper or interface to interact with llama.cpp
Develop a model management system compatible with llama.cpp's model format
Create APIs for embedding generation and text generation tasks
Implement error handling and logging for the llama.cpp integration
Ensure proper resource management (memory, GPU utilization if applicable)

Acceptance Criteria

[ ] llama.cpp can be successfully integrated and used as an inference server
[ ] Users can specify llama.cpp as the inference engine through configuration or CLI
[ ] The system can load and use various models supported by llama.cpp
[ ] Embedding generation tasks can be performed using llama.cpp
[ ] Text generation tasks can be performed using llama.cpp
[ ] The integration demonstrates improved performance compared to current solutions
[ ] The integration works on Windows, Linux, and macOS
[ ] Comprehensive documentation is provided for setup and usage

Additional Considerations

Investigate the possibility of using llama.cpp's quantization features for model optimization
Consider implementing a benchmark system to compare performance with other inference solutions
Explore options for fine-tuning or adapting models within the llama.cpp ecosystem
Keep track of llama.cpp updates and ensure our integration can be easily updated

Resources

llama.cpp GitHub repository: https://github.com/ggerganov/llama.cpp
Documentation on supported models and setup instructions should be referenced during implementation

llama.cpp already has open ai compatible server which you can use. https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md
You can set custom api base as shown here. https://github.com/Abraxas-365/langchain-rust/blob/80f61b684e4a91b4c53d273a52db35da20325174/examples/llm_openai.rs#L12C1-L18C1
Would this work for you or are you suggesting us to embed llama.cpp? I'm not a fan of embedding llama.cpp in langchain-rust as this will increase the overhead for us due to the fact that llama.cpp updates at a very fast speed and we will be in catchup mode all the time. I follow few repos and I keep seeing lot of issues related to can it support model X and would like to avoid it.

Can you please add to you example folder this llama.cpp example. I spent a little bit time to find out the correct endpoint for llama.cpp.

Run llama.cpp: .\llama.exe --port 8181 -m models\<YOUR-MODEL-NAME>.gguf --threads 8 -ngl 100 -c 4096 --n-gpu-layers 99 --api-key <YOUR-API-KEY>
Use this code:

use langchain_rust::llm::OpenAIConfig;
use langchain_rust::{language_models::llm::LLM, llm::openai::OpenAI};

#[tokio::main]
async fn main() {
    let open_ai = OpenAI::default().with_config(
        OpenAIConfig::default()
            .with_api_base("http://localhost:8181/v1")
            .with_api_key("<YOUR-API-KEY>"), // Uncomment if you need to set your OpenAI key
    );

    let prompt = "Building a website can be done in 10 simple steps:";

    match open_ai.invoke(prompt).await {
        Ok(response) => println!("{}", response),
        Err(e) => eprintln!("Error: {:?}", e),
    }
}

Thanks for you help, @prabirshrestha!
Is there is way to also use custom API for embedding models? I would like also use llama.cpp as embedding server.

And here example of using llama.cpp as embedding server:
use langchain_rust::embedding::embedder_trait::Embedder;
use langchain_rust::embedding::openai::openai_embedder::OpenAiEmbedder;
use langchain_rust::llm::OpenAIConfig;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = OpenAIConfig::default()
        .with_api_base("http://localhost:8282/v1".to_string())
        .with_api_key("a684e734-f8b4-4967-8723-4ac393638338".to_string()); // llama.cpp doesn't require an API key

    let embedder = OpenAiEmbedder::new(config);

    let text = "Why is the sky blue?";
    let embedding = embedder.embed_query(text).await?;

    println!("Embedding for '{}': {:?}", text, embedding);

    Ok(())
}


test: adds missing unittest for leftPad

Overview
A unit test for leftPad was missing, which is added in this PR.
Checklist

[x] New and updated code has appropriate documentation
[x] New and updated code has new and/or updated testing
[ ] Required CI checks are passing
[x] Visual proof for any user facing features like CLI or documentation updates
[x] Linked issues closed with keywords

Codecov Report

Merging #2141 (b34be43) into main (b77ac63) will decrease coverage by 0.11%.
The diff coverage is 40.00%.

@@            Coverage Diff             @@
##             main    #2141      +/-   ##
==========================================
- Coverage   24.28%   24.17%   -0.11%
==========================================
  Files         125      125
  Lines       14248    14248
==========================================
- Hits         3460     3445      -15
- Misses      10432    10446      +14
- Partials      356      357       +1

Impacted Files
Coverage Δ

pkg/namespace/namespace.go
49.51% <ø> (ø)

x/blob/types/payforblob.go
69.54% <0.00%> (-4.02%)
:arrow_down:

x/mint/types/constants.go
100.00% <ø> (ø)

test/txsim/blob.go
80.00% <100.00%> (-0.65%)
:arrow_down:

x/blob/ante/ante.go
85.00% <100.00%> (-3.47%)
:arrow_down:

x/blob/keeper/keeper.go
80.00% <100.00%> (-2.76%)
:arrow_down:

... and 1 file with indirect coverage changes


Double quotes on VHDLocations value lead to an error

The value for VHDLocations registry key should not be in quotes, it leads FSLogix service to an error accessing remote share path.
That's how it worked for me:

VHDLocations
Type
Data/Value

Enabled
DWORD
1

VHDLocations
Multi-String Value
\\hostname\share

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 75645943-4897-bc27-18da-c88c35446411
Version Independent ID: 6e56fa5f-22fe-44b3-d333-1fbcdb69e808
Content: Windows Virtual Desktop FSLogix profile container share - Azure
Content Source: articles/virtual-desktop/create-host-pools-user-profile.md
Service: virtual-desktop
GitHub Login: @Heidilohr
Microsoft Alias: helohr

@cjchacon
Thanks for the feedback! I have assigned the issue to content author to check and update the document as appropriate.

@Heidilohr
Can you please check and add your comments on this doc update request as applicable.

Thanks for the feedback. This has been corrected and the change should be visible later today.
#please-close


Loading JS file working in application and not in plugin

I'm working on a multi-platform plugin for exposing Csound API to Dart.
For the web part, it is using a Javascript file generated with Emscripten (CsoundObj.js) and another one (CsoundDart.js, mostly glue upon CsoundObj.js).
The dart side is a simple wrapper using package:js.
I tested in a Flutter app, loading Javascript files from index.html <script>...</script> and it works fine.
In the plugin project, I load the Javascript files from assets with
html.document.body.append(html.ScriptElement()
   ..src = "assets/packages/csounddart/assets/CsoundObj.js"
   ..type = "text/javascript"
   ..defer = true
);

I make this manipulation for both files. Unfortunately, while the second file (CsoundDart.js) seems to load properly, the CsoundObj class is undefined, and I get the following error from CsoundDart.js : ReferenceError: CsoundObj is not defined.
I can confirm (I tested with another dummy file) that it is not a scope question (CsoundDart.js can access scope from other files dynamically loaded like showed above).
CsoundObj.js is attached below.
CsoundObj.js.zip

I suspect this is an ordering issue.  You may need to wait on Dart execution dependent on that script until the script is loaded.
Some options:

Try not setting defer - this is telling to browser it can defer loading the script.
Use ScriptElement's onLoad to signal when the script has been loaded.

Thank you for advices.
The removing of defer doesn't seem to change anything.
About the second solution, I tried the following :
html.ScriptElement elem = html.ScriptElement()
  ..src = "assets/packages/csounddart/assets/CsoundObj.js"
  ..type= "text/javascript";

elem.onLoad.lastWhere((element) {
  html.document.body.append(html.ScriptElement()
   ..src = "assets/packages/csounddart/assets/CsoundDart.js"
   ..type = "text/javascript"
   ..defer = true
);
  return true;
});

Do you think it's correct ? It returns the same CsoundObj is not defined

Anyway, I'm not sure it's an ordering issue, since I also tried to copy the content of CsoundObj.js inside CsoundDart.js, and it returns the same issue. Though, when including both JS files in the index.html of an app, it works fine.
Is there a difference in the JS engine working in plugins and apps ?

That sort of error is typically an ordering issue.  I'd suggest adding prints (e.g., console.log) in your JS files where the specific types are defined and your Dart and seeing how it differs in the various cases.
Re plugins - that's a Flutter concept, not a Dart one.  I wouldn't be surprised if there is extra async / ordering issues there, but you may want to ask on a Flutter forum.

BTW, one thing I should add - you probably need to set:
  ...async = false
  ...defer = false

explicitly is you want synchronous script loading.  This is due to browser behavior, not Dart specific.

Are you sure it could be an ordering issue even if loading both files concatenated as one file still returns the same result ?
I appended the CsoundDart.js at the end of CsoundObj.js and it's still the same. It's weird since the CsoundObj is not defined error comes from the code I appended, while CsoundObj is supposed to be defined at the begining of the file.

I tried your recommendations with same results. I will ask on the flutter side.
Thank you for your time !

If the error is coming from usage in Dart, it may be that those scripts aren't loaded before your Dart code is executed.

Closing here as I don't believe there is a Dart SDK bug or feature request here.


Rotation wrong on bottom of board for JLCPCB

The positions.csv is generated with Mid X, Mid Y, and Rotation "as viewed from top". This seems to be how KiCad works.
E.g. this is two 0603 resistors with rotation of 10 degrees.

KiCad 3D view shows clearly.

As you can see the 10 degree rotations is "as viewed from top".
The positions.csv file looks like this... showing both as 10 degrees.
Designator,Mid X,Mid Y,Rotation,Layer
R1,3.0,5.0,10.0,top
R2,7.0,5.0,10.0,bottom

However, when loaded on JLCPCB the rotation on the bottom is wrong, e.g

Clearly JLC are expecting the rotation "as viewed from above the component".
The positions.csv file needed is
Designator,Mid X,Mid Y,Rotation,Layer
R1,3.0,5.0,10.0,top
R2,7.0,5.0,350.0,bottom

This works as expected.

If a rotation offset is applied, it works as you would expect, just added to the rotation, and could be used as a -20 offset to "fix" the bottom layer in this case, but that only works if rotated 10 degrees. Rotate 20 and you need to change the JLCPCB Rotation Offset to -40 to fix it. This is why the offsets seemed not to work. It is not the offset that is wrong, it is that all rotations on the back of the board need to be inverted, so as to be "as viewed from above the component" and not "as viewed from top" in the way KiCad works.
I think if the JLCPCB Rotation Offset is applied first, when present, to correct for whatever orientation JLC have used for the part, but then all bottom side rotations are inverted, it will be consistent and avoid having to tweak the back of every board.
This was obviously not obvious as rotations 0, and 180 did not show the problem, and 90 or 270 only showed on polarised components. But even a simple 0603, with no need for any rotation offset, breaks if placed at 45 degrees, so I think this is a definite bug, and should be fixed even with the backwards compatibility issues it would cause.

Thank you for contributing!