Please I have serious issue with the CSV, Pandas, Dataframe and JSON, I need someone to assist.

`import pandas as pd
import datetime
from pandas_datareader import data, wb
import csv
import json
#declaring libraries for the analysis
out= open("testfile.csv",  "rt") # read the .csv file
data = csv.reader(out)
data = [[row[0],row[1] + "" + row[2],row[3] +"" + row[4], row[5],row[6]] for row in data] #this part is mainly to concatenate the City and country
out.close()
out=open("data.csv", "wt")
output = csv.writer(out)
for row in data:
output.writerow(row)
out.close()
#<------------ End of concatenation ------------->
#The result is written out and saved in data.csv (I don't want to override the original source)
df = pd.read_csv('data.csv') #using pandas and dataframe, I can read in data and loop it through between the Arrival Date and Departure Date
df.DateDpt = pd.to_datetime(df.DateDpt)
df.DateAr = pd.to_datetime(df.DateAr)
df = df.set_index('DateAr')
new_df = pd.DataFrame()
for i, data in df.iterrows():
data = data.to_frame().transpose()
data = data.reindex(pd.date_range(start=data.index[0], end=data.DateDpt[0])).fillna(method='ffill').reset_index().rename(columns={'index': 'DateAr'})
new_df = pd.concat([new_df, data])
new_df = new_df[['AuthorID', 'ArCity_ArCountry', 'DptCity_DptCountry', 'DateAr', 'DateDpt']]
print (new_df)
<---- End of iteration ---->
#Json file : <------ JSON ----------------->
json_dict = {}
for arrival_date, data in new_df.groupby('DateAr'):
matching_dates = data[data.DateDpt==arrival_date]
not_matching_dates = data[data.DateDpt!=arrival_date]
json_dict[arrival_date.strftime('%Y-%m-%d')] = {}
if not matching_dates.empty:
for city, flights in matching_dates.groupby('ArCity_ArCountry'):
json_dict[arrival_date.strftime('%Y-%m-%d')][city] = [str(v) for v in flights.AuthorID.to_dict().values()]
if not not_matching_dates.empty:
for city, flights in not_matching_dates.groupby('DptCity_DptCountry'):
json_dict[arrival_date.strftime('%Y-%m-%d')][city] = [str(v) for v in flights.AuthorID.to_dict().values()]
with open('json_dict.json', 'w') as f:
json.dump(json_dict, f, indent=4, sort_keys=True)
<------- End of JSON ---------------->`
CSV File (testfile.csv)
AuthorID    ArCity  ArCountry   DptCity DptCountry    DateDpt   DateAr
AAA         Paris   France      NewYork UnitedState  2008-03-10 2001-02-02
BBB         Paris   France      NewYork UnitedState  2008-03-10 2001-02-02
CCC         Paris   France      Beijing Japan        2008-03-10 2001-02-02
DDD         Paris   France      NewYork UnitedState  2008-03-10 2001-02-02
EEE         Paris   France      London  UK           2008-03-10 2001-02-02
FFF         Paris   France      NewYork UnitedState  2008-03-10 2001-02-02
GGG         Paris   France      Beijing Japan        2008-03-10 2001-02-02
Expected output:
{
    "2001-02-02": {
        "Beijing _Japan": [
            "CCC",
            "GGG"
        ],
        "London_UK": [
            "EEE"
        ],
        "NewYork_UnitedState": [
            "AAA"
            "BBB"
            "DDD"
            "FFF"
        ],
        "Paris_France": [
            "AAA"
            "BBB"
            "CCC"
            "DDD"
            "EEE"
            "FFF"
        ]

    },

    .
    .
    .
  }

But the current output is
`{
    "2001-02-02": {
        "Beijing _Japan": [
            "GGG"
        ],
        "London_UK": [
            "EEE"
        ],
        "NewYork_UnitedState": [
            "FFF"
        ]
    },
    "2001-02-03": {
        "Beijing _Japan": [
            "GGG"
        ],
        "London_UK": [
            "EEE"
        ],
        "NewYork_UnitedState": [
            "FFF"
        ]
    },
    .
    .
    .
  }`
``

you will get more help on the mailing list and/or stack overflow

this is not a library issues, rather a question how to structure your code

you will have better responses with a much shorter and copy pastable example
you are reading from a csv file which only u have

I have attached the sample csv file
AuthorID    ArCity  ArCountry   DptCity       DptCountry            DateDpt            DateAr
AAA         Paris       France      NewYork    UnitedState            2008-03-10      2001-02-02
BBB         Paris       France      NewYork    UnitedState            2008-03-10      2001-02-02
CCC         Paris      France      Beijing       Japan                      2008-03-10      2001-02-02
DDD         Paris      France      NewYork    UnitedState            2008-03-10      2001-02-02
EEE         Paris       France      London       UK                         2008-03-10      2001-02-02
FFF         Paris       France      NewYork     UnitedState           2008-03-10      2001-02-02
GGG         Paris      France      Beijing        Japan                    2008-03-10       2001-02-02
I have posted it on stackover flow, but no response and I'm right now behind schedule. Please sir! This is the sample CSV file named testfile.csv

Please assist Sir
Thank you so much for your time sir
I only seek for assistance to fix the code.
I have a csv file, which has several IDs, these fall in categories of
arrival, departure, within range of dates. I have group by date and then
find any of the IDs on particular date sharing the same place of
arrival and /or departure with any other IDs
import pandas as pd
import datetime
import numpy as np
from pandas_datareader import data, wb
import csv
import json
df = pd.read_csv('testfile.csv')
df.DepartureDate = pd.to_datetime(df.DepartureDate)
df.ArrivalDate = pd.to_datetime(df.ArrivalDate)
df = df.set_index('ArrivalDate')
new_df = pd.DataFrame()
for i, data in df.iterrows():
data = data.to_frame().transpose()

data =

data.reindex(pd.date_range(start=data.index[0],
end=data.DepartureDate[0])).fillna(method='ffill').reset_index().rename(columns={'index':
'ArrivalDate'})
new_df = pd.concat([new_df, data])

new_df = new_df[['ID', 'Arrival', 'Departure', 'ArrivalDate', 'DepartureDate']]
print (new_df)
json_dict = {}
for arrival_date, data in new_df.groupby('ArrivalDate'):
matching_dates = data[data.DepartureDate==arrival_date]

not_matching_dates = data[data.DepartureDate!=arrival_date]

json_dict[arrival_date.strftime('%Y-%m-%d')] = {}

if not matching_dates.empty:

    for city, flights in matching_dates.groupby('Arrival'):

        json_dict[arrival_date.strftime('%Y-%m-%d')][city] = [str(v) for v in flights.ID.to_dict().values()]

if not not_matching_dates.empty:

    for city, flights in not_matching_dates.groupby('Departure'):

        json_dict[arrival_date.strftime('%Y-%m-%d')][city] = [str(v) for v in flights.ID.to_dict().values()]

with open('json_dict.json', 'w') as f:
 json.dump(json_dict, f, indent=4, sort_keys=True)

The desired output:
{
"2001-02-02": {
"Japan": [
"CCC",
"GGG"
],
"UK": [
"EEE"
],
"UnitedState": [
"AAA"
"BBB"
"DDD"
"FFF"
],
"France": [
"AAA"
"BBB"
"CCC"
"DDD"
"EEE"
"FFF"
]
},

.
.
.

}
The current output
{
"2001-02-02": {
"Japan": [
"GGG"
],
"UK": [
"EEE"
],
"UnitedState": [
"FFF"
]
},
"2001-02-03": {
"Japan": [
"GGG"
],
"UK": [
"EEE"
],
"UnitedState": [
"FFF"
]
},
.
.
.
}
Date: Fri, 29 Jan 2016 18:30:01 -0800
From<EMAIL_ADDRESS>To<EMAIL_ADDRESS>CC<EMAIL_ADDRESS>Subject: Re: [pandas] Please I have serious issue with the CSV, Pandas, Dataframe and JSON, I need someone to assist. (#12183)
you will have better responses with a much shorter and copy pastable example
you are reading from a csv file which only u have
—
Reply to this email directly or view it on GitHub.

@tollycoast As Jeff already said, can you please ask this question for example at Stack Overflow (http://stackoverflow.com/) or the mailing list (https://groups.google.com/forum/?fromgroups#!forum/pydata) (I personally recommend Stack Overflow).
It is not that we don't want to help you (we can answer on those other platforms), but we like to keep this issue tracker for bugs/enhancement requests for pandas.


Fix 'WorkingSetBytes' memory metric for PA backed VMs

Added two fields (physicallyBacked and assignedMemoryInMB) to UtilityVM struct.
Stop querying the UVMs vmmem process to get the working set size for physical backed UVMs. Instead we assign the total memory assigned to the UVM to the new assignedMemoryInMB field and use this for WorkingSetBytes.

Signed-off-by: Daniel Canter<EMAIL_ADDRESS>

Making this WIP to add a new annotation akin to in here https://github.com/microsoft/hcsshim/pull/806, and a test


ubuntu x86-64 INTERL I5, last step ninja error

configure.py:
parser.add_argument("--backend", dest="backend", default="psimd",
choices=["auto", "psimd", "scalar"])
so i select psimd
result:
ninja
[0/1] CONFIGURE
Detected root directory: /home/sun/下载/NNPACK
Configuring dependency pthreadpool
Configuring dependency fxdiv
Configuring dependency googletest
Configuring dependency googlebenchmark
Configuring dependency fp16
Configuring dependency psimd
Module default: 1 libraries, 5 executables, 45 tests (11 smoke tests)
[35/150] PEACHPY bench/memread.py
FAILED: /home/sun/下载/NNPACK/build/bench/memread.py.o
python -m peachpy.x86_64 -mabi=sysv -g4 -mimage-format=elf  -I/home/sun/下载/NNPACK/include -I/home/sun/下载/NNPACK/src/x86_64-fma -I/home/sun/下载/NNPACK/src -I/home/sun/下载/NNPACK/src/ref -I/home/sun/下载/NNPACK/test -I/home/sun/下载/NNPACK/bench -I/home/sun/下载/NNPACK/include -I/home/sun/下载/NNPACK/deps/pthreadpool/include -MMD -MF '/home/sun/下载/NNPACK/build/bench/memread.py.o'.d -o '/home/sun/下载/NNPACK/build/bench/memread.py.o' '/home/sun/下载/NNPACK/bench/memread.py'
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/local/lib/python2.7/dist-packages/peachpy/x86_64/main.py", line 280, in
main()
File "/usr/local/lib/python2.7/dist-packages/peachpy/x86_64/main.py", line 267, in main
execute_script(writers, options.input[0])
File "/usr/local/lib/python2.7/dist-packages/peachpy/x86_64/main.py", line 198, in execute_script
execute_script(writers, source_filename)
File "/usr/local/lib/python2.7/dist-packages/peachpy/writer.py", line 98, in exit
self.output_file.write(self.encode())
File "/usr/local/lib/python2.7/dist-packages/peachpy/writer.py", line 125, in encode
return self.image.as_bytearray
File "/usr/local/lib/python2.7/dist-packages/peachpy/formats/elf/image.py", line 59, in as_bytearray
self.strtab.add(symbol.name)
File "/usr/local/lib/python2.7/dist-packages/peachpy/formats/elf/section.py", line 209, in add
string_bytes = codecs.encode(string, "utf-8")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe4 in position 10: ordinal not in range(128)
[40/150] CXX test/convolution-input-gradient/alexnet.cc
ninja: build stopped: subcommand failed.

ok,i find the renson.because the folder name was chinese , i change "下载" to "down",ok
thanks

[156/156] LINK bin/convolution-inference+relu-smoketest


Deployment stuck at "Retrieving deployment details"

I'm trying to deploy a simple static website from an S3 bucket, but the AWS Amplify console gets stuck every time.  I've manually uploaded my site to my S3 bucket.  I've dragged and dropped a zip file.  It never works.  Just stuck at "Retrieving deployment details..."
This is merely a single index.html file at this point.  Not very impressed with this service so far.

After much trial and error I discovered the problems. None of this is documented anywhere, but the site has to be in a zip file, and in the root of the zip file.


Invisible overflow in FluentGrid with large content which exceeds the content container

🐛 Bug Report
When using a FluentGrid and it contains so much content that the div content is being exceeded there is a nasty bug when scrolling because there are two seperate overflows created.
I've tested it with FluentStack as well and with this component it works just fine.
In my case I have a grid with 2 columns. The first column displays a list of data and the second column displays a filter area.
💻 Repro or Code Sample
I've created a small sample repo and a small video which showcases the bug.
https://github.com/MarvinKlein1508/fluent-ui-scroll-bug
https://www.youtube.com/watch?v=sqAVsGUZavs
🤔 Expected Behavior
I would expect only one scrollbar to appear.
😯 Current Behavior
You will have two scrollbars.
💁 Possible Solution
I'Ve added overflow: hidden to .content which fixes this issue for me. However this solution isn't great because it will break other things then. For example when you use a FluentSelect which displays many items.
🔦 Context
I've just updated my templates to the latest version.
🌍 Your Environment
Mozilla Firefox

For the demo you provided, I solved it by changing Home.razor line 6 to
<FluentGrid Style="overflow: hidden;">

This gives you 1 scrollbar for the whole window
I don't think this is a component issue. We provide Style and Class parameter to 'tweak' things. Also, the template is just a starter. It is not an end-all-solve-all thing. You should be able to use the components fully without or with addapted <div class="content">
Another solution is to do what we do In the demo site where we change the height of the body-content element:

(needs to be -86px for template site) This gives you a scrollbar that just scrolls the right (content)  part

You have to add bottom padding as well on things like cards or else the shadow gets cut off.
Adding overview:hidden to ~most FluentGrid uses seems like a implementation issue.
By default if someone cannot get the base template and add a Grid to the home page, it seems like this will just create more tickets/issues logged as people need to re-tweak on every use of fluentGrid (which would be a reasonably the majority of pages)

@StephenOTT My second suggestion/solution works without adding any padding anywhere. It is als obetter than the first because it makes the header 'sticky'

@vnbaaij thanks for your reply. I would still encourage you to change the template for this.
This might be really confusing for new people which try out FluentUI for the first time. Especially when you think that the template works just fine out of the box.


Support application/json-patch+json MIME type

Hi, I noticed that the application/json-patch+json MIME type is not supported, which is needed for PATCH operations. This type is defined in RFC 6902.
The problem is that the accepted MIME types are a fixed list, so if the type is not mapped there, it breaks during swag generation:
ParseComment error in file XXX.go :json-patch+json accept type can't be accepted

To solve this, a possible solution might be adding the type mapping as json-patch into the MIME list.
Other alternatives might be supporting a full Content-Type declaration, with no mapping, considering the user value as the final accept/produce values.
Thanks!

I second this, it seems like application/pdf is also not accepted.


Regarding FPGA implementation

Hey,
Thanks for the awesome work. Can you share your FPGA implementation methodology? Which tools did you use?
Thanks!
Shivam Aggarwal

For the FPGA accelerator, we design the architecture based on the systolic array, and add the support for structured dynamic sparse processing. We implement our FPGA accelerator in Verilog, synthesized with Vivado 2019.1, and evaluated on the Ultra96-v2 development board. More details of the accelerator design will be described in our later articles.


Fixing .gitattributes for binary content, removing *.class files

Signed-off-by: Andriy Redko<EMAIL_ADDRESS>Description
Specifying binary content in .gitattributes, removing accidentally added .class files by https://github.com/opensearch-project/OpenSearch/pull/1696
Issues Resolved
https://github.com/opensearch-project/OpenSearch/issues/1713
Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@saratvemulapalli could you give it a try please?

@saratvemulapalli could you give it a try please?

Haha you were faster than me. Sure will give it a shot.

(sorry for randomly commenting on this pull request uninvited - I've been reading some recent pull requests given the interest in log4j over the past few days)
Is marking these as binary file patterns (using .gitattributes) preferable to filtering them out of local unstaged changes (using .gitignore)?

thank @saratvemulapalli

Do you happen to have a windows machine to test?

Yes, I did, the clone is clean

thanks for looking @jayaddison

Is marking these as binary file patterns (using .gitattributes) preferable to filtering them out of local unstaged changes (using .gitignore)?

Those files are part of the codebase (used in tests, fe, etc.), adding them to .gitignore would prevent  them from committing.

Those files are part of the codebase (used in tests, fe, etc.), adding them to .gitignore would prevent them from committing.

Cool, that makes sense - thank you.  And *.class could also be added separately to .gitignore?

Cool, that makes sense - thank you. And *.class could also be added separately to .gitignore?

That is good point, we probably could add bin folder but I am wondering how they appear in first place, they should be in the build folder by default unless probably Eclipse is used, @saratvemulapalli is it your case? Thanks!

Cool, that makes sense - thank you. And *.class could also be added separately to .gitignore?

That is good point, we probably could add bin folder but I am wondering how they appear in first place, they should be in the build folder by default unless probably Eclipse is used, @saratvemulapalli is it your case? Thanks!

I believe so, I didnt do post mortem of who generated .class files but across the project they should live under build which is already covered.


added my anime on realmdatabase.swift

I added my favorite anime to realmdatabase.swift

@jcat1504 I love this one! Thanks for your contribution


"Browser autologin - ChatGPT Plus legacy model" and "Browser autologin - ChatGPT Plus default model" work in reverse.

hello.
"Browser autologin - ChatGPT Plus legacy model" and "Browser autologin - ChatGPT Plus default model" work in reverse.
As a result of checking "settings.json", when "Browser autologin - ChatGPT Plus legacy model" is selected, "chatgpt.gpt3.model": "text-davinci-002-render-sha" is set, and "Browser autologin - ChatGPT Plus It was confirmed that "chatgpt.gpt3.model": "text-davinci-002-render-paid" is set when "default model" is selected.
It needs to be set to "text-davinci-002-render-paid" when selecting legacy model and "text-davinci-002-render-sha" when selecting default model.
vscode-chatgpt v3.9.4
Visual Studio Code v1.76.0 (Universal)

@anjunsik It was a typo on Readme only. The settings should be correct. Paid -> Plus model. Sha -> Default model. Updated the readme. Thanks for reporting the issue


Undefined initialism in README

The README mentions ODCM numerous times but doesn't provide any definition of what it stands for.
e.g.
Vipr.Core. Provides the interfaces required for extending Vipr as well as the core Client Library generation logic. It also defines the ODCM Object Model used to describe service capabilities between Readers and Writers.
What does it stand for? Can we update the README to include this definition?

This will not be fixed currently, documentation is sufficient.

Disagree. What does ODCM stand for? It seems awfully important and awfully specific to this project. If we're going to mention it in READMEs we ought to give enough context such that people can understand what we expect them to read.


Updated the README to be more 'rubyish'

Changed method & variables from CamelCase -> underscore.
This is the preferred way in the Ruby world.

thanks for the PR. we've made quite a few changes since you sent this and i think the API is quite Ruby-ish now. sorry for leaving it open for so long!


usage backup: Raw usage/usage display to record time duration of usage

Description
This PR ensure consistency between usage details stored across different resource types. For the sake of consistency, backup usage record will show the time duration of backup usage in the raw_usage / usage_display fields.

Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[X] Bug fix (non-breaking change which fixes an issue)
[ ] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity
Bug Severity

[ ] BLOCKER
[ ] Critical
[ ] Major
[X] Minor
[ ] Trivial

Screenshots (if appropriate):
How Has This Been Tested?

@Pearl1594 does this pr fix #4982 ? is it ready for review ?
@nvazquez

Hi @Pearl1594, @weizhouapache has created PR #5259 to fix #4982, can you please sync to work on a unified solution?

Will do @nvazquez @weizhouapache - Thanks.

Hi @Pearl1594 any update on this? Can both PRs be integrated into a single one to fix the issue #4982?

Hi @nvazquez - was held up in other tasks, hence wasn't able to test #5259. However, on doing a code review, I believe, both PRs solve the same issue, and @weizhouapache's solution would be less disruptive and more apt to address the issue - so maybe we could go ahead with #5259. I'll spend some time testing it as well. Apologies for the delay.

@Pearl1594 tentatively moving to <IP_ADDRESS> milestone - please advise if it's feasible/ready for 4.15?

Since PR #5259 addresses the same issue - I'll close this PR


how to add this to android studio

I need to add this to gradle

you can use
repositories {
	// ...
	maven { url "https://jitpack.io" }
 }

Step 2. Add the dependency
dependencies {
    compile 'com.github.steelkiwi:AndroidRecording:542368cbc7'
}

+1

+1

+1

https://android-arsenal.com/details/1/744#!package
compile 'com.github.steelkiwi:AndroidRecording:master'


Releasing v0.2.0

Added

xkafka Added xkafka package with Producer and Consumer implementations that support middleware & HTTP-like handlers.

Codecov Report
Patch coverage: 100.00% and project coverage change: +0.70 :tada:

Comparison is base (70f1014) 99.29% compared to head (f849439) 100.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@             Coverage Diff             @@
##             main       #10      +/-   ##
===========================================
+ Coverage   99.29%   100.00%   +0.70%
===========================================
  Files          18         8      -10
  Lines         282        84     -198
===========================================
- Hits          280        84     -196
+ Misses          2         0       -2

Impacted Files
Coverage Δ

version.go
100.00% <100.00%> (ø)

... and 10 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

LGTM


VPL ERROR while compiling DPU-TRD

Hi,
OS: Ubuntu 16.04
DPU-TRD,Vitis,Vivado,Petalinux version: 2020.1
While running
make KERNEL=DPU DEVICE=zcu102
VPL throws the following error
INFO: [v++ 60-1454] Run Directory: /home/htic-broncho/DPU-TRD/prj/Vitis/binary_container_1/link/run_link
****** vpl v2020.1 (64-bit)
**** SW Build 2902540 on Wed May 27 19:54:35 MDT 2020
** Copyright 1986-2020 Xilinx, Inc. All Rights Reserved.
INFO: [VPL 60-839] Read in kernel information from file '/home/htic-broncho/DPU-TRD/prj/Vitis/binary_container_1/link/int/kernel_info.dat'.
 Run Status: create_bd ERROR
ERROR: [VPL 60-773] In '/home/htic-broncho/DPU-TRD/prj/Vitis/binary_container_1/link/vivado/vpl/runme.log', caught Tcl error:  ERROR: [Common 17-157] Error parsing command line options, please type 'import_files -help' for usage info.
ERROR: [VPL 60-704] Integration error, Step failed: create_bd An error stack with function names and arguments may be available in the 'vivado.log'.
ERROR: [VPL 60-1328] Vpl run 'vpl' failed
ERROR: [VPL 60-806] Failed to finish platform linker
INFO: [v++ 60-1442] [18:36:01] Run run_link: Step vpl: Failed
Time (s): cpu = 00:00:17 ; elapsed = 00:00:43 . Memory (MB): peak = 1339.039 ; gain = 0.000 ; free physical = 905 ; free virtual = 15242
ERROR: [v++ 60-661] v++ link run 'run_link' failed
ERROR: [v++ 60-626] Kernel link failed to complete
ERROR: [v++ 60-703] Failed to finish linking
INFO: [v++ 60-1653] Closing dispatch client.
Makefile:70: recipe for target 'binary_container_1/dpu.xclbin' failed
make: *** [binary_container_1/dpu.xclbin] Error 1
Attaching vivado.log for further reference
Vivado v2020.1 (64-bit)
SW Build 2902540 on Wed May 27 19:54:35 MDT 2020
IP Build 2902112 on Wed May 27 22:43:36 MDT 2020
Start of session at: Mon Nov 16 18:35:33 2020
Process ID: 29532
Current directory: /home/htic-broncho/DPU-TRD/prj/Vitis/binary_container_1/link/vivado/vpl
Command line: vivado -log vivado.log -applog -messageDb vivado.pb -mode batch -source vpl.tcl -notrace
Log file: /home/htic-broncho/DPU-TRD/prj/Vitis/binary_container_1/link/vivado/vpl/vivado.log
Journal file: /home/htic-broncho/DPU-TRD/prj/Vitis/binary_container_1/link/vivado/vpl/vivado.jou
source vpl.tcl -notrace
INFO: Dispatch client connection id - 33609
WARNING: failed to connect to dispatch server - client already initialized
[18:35:45] Run vpl: Step create_project: Started
INFO: [OCL_UTIL] current step: vpl.create_project
Creating Vivado project.
INFO: [OCL_UTIL] internal step: create_project -part xczu9eg-ffvb1156-2-e -force prj prj
INFO: [OCL_UTIL] set_property design_mode GateLvl [current_fileset]
INFO: [Vivado 12-3498] Setting project source management mode (property SOURCE_MGMT_MODE) to "None".
INFO: [OCL_UTIL] set_property PR_FLOW 1 [current_project]
[18:35:48] Run vpl: Step create_project: Completed
[18:35:48] Run vpl: Step create_bd: Started
INFO: [OCL_UTIL] current step: vpl.create_bd
INFO: [OCL_UTIL] set_property board_part_repo_paths .local/hw_platform/board [current_project]
INFO: [OCL_UTIL] set_property board_part xilinx.com:zcu102:part0:3.3 [current_project]
INFO: [OCL_UTIL] setting ip_repo_paths: /home/htic-broncho/DPU-TRD/prj/Vitis/binary_container_1/link/int/xo/ip_repo/xilinx_com_RTLKernel_DPUCZDX8G_1_0 /home/htic-broncho/Xilinx/Vitis/2020.1/data/cache/xilin$
INFO: [OCL_UTIL] internal step: update_ip_catalog
INFO: [IP_Flow 19-234] Refreshing IP repositories
INFO: [IP_Flow 19-1700] Loaded user IP repository '/home/htic-broncho/DPU-TRD/prj/Vitis/binary_container_1/link/int/xo/ip_repo/xilinx_com_RTLKernel_DPUCZDX8G_1_0'.
INFO: [IP_Flow 19-1700] Loaded user IP repository '/home/htic-broncho/Xilinx/Vitis/2020.1/data/cache/xilinx'.
INFO: [IP_Flow 19-1700] Loaded user IP repository '/home/htic-broncho/DPU-TRD/prj/Vitis/binary_container_1/link/vivado/vpl/.local/hw_platform/ipcache'.
INFO: [IP_Flow 19-1700] Loaded user IP repository '/home/htic-broncho/Xilinx/Vitis/2020.1/data/ip'.
INFO: [IP_Flow 19-2313] Loaded Vivado IP repository '/home/htic-broncho/Xilinx/Vivado/2020.1/data/ip'.
update_ip_catalog: Time (s): cpu = 00:00:08 ; elapsed = 00:00:12 . Memory (MB): peak = 2174.969 ; gain = 0.000 ; free physical = 894 ; free virtual = 15223
INFO: [OCL_UTIL] internal step: config_ip_cache -use_cache_location /home/htic-broncho/DPU-TRD/prj/Vitis/binary_container_1/ip_cache
INFO: [OCL_UTIL] internal step: import_files -norecurse .local/hw_platform/bd/hdmi_example_zcu102.bd -of_objects
ERROR: caught error: ERROR: [Common 17-157] Error parsing command line options, please type 'import_files -help' for usage info.
while executing

"import_files -norecurse $hw_platform_dr_bd {*}$rm_switch"
(procedure "ocl_util::import_dr_bd" line 40)
invoked from within
"ocl_util::import_dr_bd $hw_platform_info $config_info"
[18:36:00] Run vpl: Step create_bd: Failed
INFO: [OCL_UTIL] current step: vpl.create_bd failed. To rerun the existing project please use --from_step vpl.create_bd
Step failed: create_bd An error stack with function names and arguments may be available in the 'vivado.log'.
INFO: [Common 17-206] Exiting Vivado at Mon Nov 16 18:36:00 2020...
How to rectify this issue?
Thanks in advance

Hi @gkrislara
It seems that you choose the wrong platform source. You should use ZCU102 base platform from https://www.xilinx.com/support/download/index.html/content/xilinx/en/downloadNav/embedded-platforms/2020-1.html

Hi @gkrislara
Since we haven't received your reply for a long time, we assume you have solved this issue and I'm going to close it. If you still have any questions, please feel free to reopen it. Thank you very much.


data set

@njvisionpower
Hello! Can the data set be shared?

Hi,
The dataset has been shared in readme link, you can download with Baidu or Google drive.

@njvisionpower
Thank you for your prompt reply.


Remove playwright_cli_path ConnectOptions

I believe that this is intended to be the executable_path from LaunchOptions.
@dbrody, thoughts on this?
cc @woodward

Those are two different things, please do NOT remove playwright_cli_path.

executable_path is the path to the Chrome executable
playwright_cli_path is the path the playwright cli.js file within the npm package used to communicate with chrome

Exposing both of those options is important as this library is used in different environments and installation methods.

Ah yes, I was conflating those paths.
I do think that it makes sense to move playwright_cli_path to LaunchOptions instead of being on ConnectOptions. The ConnectionOptions configuration is meant to be applicable for cases when connecting to a running Playwright over WebSockets. No CLI is involved in those cases.
It would also make sense to have playwright_cli_path in LaunchServerOptions, once we get to implementing that functionality.

An alternative might be to include some other collection of options, for configuration that is unique to playwright-elixir, such as this (the ConnectOptions and LaunchOptions were intended to match the equivalent types in Node.js Playwright.

@coreyti Got it! Whatever you think is best, as long as the option to configure it is still available.

The "Bundle/config Playwright node CLI" commit includes the change to move :playwright_cli_path from ConnectOptions to LaunchOptions.


Stride's new library projects target uap10.0.16299 which causes build issues until manually fixed

Release Type: Official Release
Version: <IP_ADDRESS>7
Platform(s): Windows
Describe the bug
When a code library project is generated via the stride editor, the project will have uap10.0.16299 as one of its targets causing build issues.
To Reproduce
Steps to reproduce the behavior:

Open Stride
Create a code library project

Expected behavior
The generated project should only target dotnet8
Screenshots

To add more details, the template has this line:
<#@ include file="..\Common.TargetFrameworks.targets.t4" #>

Which in turns resolves to:
<TargetFrameworks><#= string.Join(";", ((IEnumerable<Stride.Core.Assets.SolutionPlatform>)Properties.Platforms).Select(x => x.TargetFramework).Distinct()) #></TargetFrameworks>

I feel like a library project (which also applies to the Game project generated by the full template) should only reference net8.0 because it is supposed to be cross-platform. If users ever need specific stuff (like for Android or iOS) they can add these additional targets manually later.
Also the Properties.Platforms collection doesn't seem to be up to date as it is still trying to add the uap` targets which don't exist anymore on our nuget packages.


Add prefix to email subject in dev/next

Get it from an environment variable, so the next emails have a prefix like [NEXT] Finish signing up for Couchers.org

How're you doing on this @boubinmj?


 is not clearing

If item is defined in  but not used in  it must be removed.
In the example below  need to be removed because it is not used anywhere:
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="128" height="128" viewBox="0 0 100 100">
<defs>
<g id="shape1">
<path
     d="m 100,47 0,-30.45 Q 100,9.7 95.15,4.85 90.3,0 83.45,0 L 16.55,0 Q 9.7,0 4.85,4.85 0,9.7 0,16.55 L 0,47 q 11.15,4.5 23.6,6.85 12.9,2.45 26.4,2.45 13.5,0 26.4,-2.45 Q 88.85,51.5 100,47 z"
     id="path3230" />
</g>
<g id="shape2">
<path
     d="m 95.15,84.1 q -4.85,4.85 -11.7,4.85 l -66.9,0 Q 9.7,88.95 4.85,84.1 0.22070313,79.470703 0,73.05 l 0,10.4 q 0,6.8 4.85,11.7 4.85,4.85 11.7,4.85 l 66.9,0 q 6.85,0 11.7,-4.85 0.4,-0.4 0.75,-0.75 1.85,-2.15 2.9,-4.55 1.2,-2.95 1.2,-6.4 l 0,-10.4 q -0.220703,6.420703 -4.85,11.05 z"
     id="path3215" style="fill:#f0a362" />
 </g>
</defs>
<use x="0" y="0" width="100" height="100" xlink:href="#shape1" />
</svg>

There is no such a plugin in svgo for now. Also, one may have an icon system with source file inlined in html and then icons used within that html code later.
Perhaps the easiest way to implement it is to extend cleanupIDs plugin by removing the whole unused element in <defs> instead of just an id attribute.

v0.5.1 with the new plugin has been just published.


[BUG] Error 500 When reordering products

When trying to reorder products, 500 SERVER ERROR
To Reproduce
Steps to reproduce the behavior:

Go to /admin/products
Click "Reorder" on some category
Reorder something
Click save
Error

Screenshots

Additional context
Paymenter version: Latest stable

I'm also getting 500 Server Error when clicking save after reordering products.

Rearranging the categories also doesn't work. It says it updates, but the categories aren't actually changing on the page. So, both ordering products and categories is busted.


Link to hack.aau.dk from frontpage of events gives error

Following the link gives a "This Connection is Not Private" error. Guess it is something with certificate of hack.aau.dk - then Per Mejdahl will be the one who can help out.

Updated with https://www.facebook.com/groups/957517617737780,
however I think, group id could be changed into text  like https://www.facebook.com/groups/aauhackers or something else which is easy to remember !
Now, the link is updated on CTFd site but will be available when we restart Haaukins or create new event.


allow configuration of test discovery paths

use-case: one of our (mono) repos has multiple sets of feature files, and we really only want to run the other tests locally
It seems like it should be possible to inspect the profile to get the paths from that, but .. barring parsing and evaluating both cucumber.yml and cucumber.mjs I'm not sure how. Open to ideas, tho!
Maybe it would be also better to have the setting be [profile]: paths?

Hey @psethwick ,
thanks again for your submission to the project.
I was reviewing your proposal and looks like a good starting point!
I was thinking about extending your proposal for complex workspaces where there are more test projects and thus more folders to check for changes.
You're checking only the first workspace folder for a configuration, that works for simpler workspaces but not for more complex ones. So the proposed change is to check for a configuration file in each workspace folder and then apply that configuration (if exist) to that specific workspace folder, otherwise default to the global configuration (or default one).
What do you think about that?

I did tilt my head a bit when I wrote workspaces[0], but I'm not super familiar with vscode workspaces.
I guess you are proposing a .cucumber-test-runner file of some sort?
Personally, I would prefer to keep the configuration in cucumberTestRunner keys in .vscode/settings.json. Reasoning being there are enough kinds of configuration files (esp in web dev) hanging around in projects and I wouldn't want to pile on.
However, configuration per workspace path makes sense, how about something like this:
  "cucumberTestRunner.workspaces": [
{
"workspacePath": ".",
"featurePaths": "features/*feature"
},
{
"workspacePath": "integration-tests",
"featurePaths": "integration-tests/features/*feature"
}
]

Then if more settings are needed per workspace in the future, the object could just be extended with more properties.
Up to you, but I would be up for implementing the vscode settings in this pr

Given that a workspace is a collection of folders that can be anywhere on disk, I was tinking of using vscode .vscode/settings.json files from each workspace folder to load the right filter for that folder.
I need to check if it is possible.
That would remove the need for the workspace folder key and so on.

ah yeah, that makes sense! That actually sounds better, I incorrectly thought you were proposing a new kind of file
happy to help, or feel free to make changes to the pr. I'm probably not going to find time today, but this week I could


fact of life? external process stops sequence

I have a task that calls an external process, then the gulp sequence stops. Is this a bug or just a fact of life? For example, this will stop the sequence:
...
import run from 'gulp-run';
...
gulp.task('well-form-2-12', done => {
  const xslSource = path.join('xsl', 'well-former-2-12.xsl');

  run(`java ${SAXON} -o:${ignoreFilePath} -s:${LOCAL_XML} -xsl:${xslSource} -x:${TAGSOUP}`).exec()
    .on('error', err => done(err))
    .on('finish', () => {
      del(ignoreFilePath);
      // done();
    });
});

Also happens with calls to apache-ant (e.g. ant some-build-task)

I don't know what you are asking here.
run-sequence just runs gulp tasks. Does the task run on it's own? Is there an error?
In your example above, you aren't ever calling the callback (// done()), which means the sequence won't ever continue.  If your tasks never complete, then the sequence never continues.

I tried with done() uncommented as well. gulp-run executes a command line java program. There is no error. I was assuming that the java process short circuits the gulp process and just ends it.

run-sequence only runs gulp tasks. If the program exits due to the task, then there's nothing here related to this library.
As long as gulp-run is forking the Java process, it's unlikely that it's killing the task.  After quickly looking at the gulp-run library, it looks like you are using it incorrectly. It's clearly designed to be returned directly as a stream, no need to use a callback at all. You should be able to just do something like:
 return run(`…your command…`).exec()
    .pipe(/* …delete the file… */);

OK, sorry for being dense (I originally was returning the stream). I was looking in the wrong place and was mistakenly focusing on the external process. It was another task running in parallel where I needed to use event-stream.merge where I was writing out multiple files. Closing...


debug run time of oc commands, line up in-script timeouts so ops-runn…

…er timeout is less likely to fire

+1
optional comments for possible improvements


Feature-based Injection

This PR is targeting on the potential user case is to inject services based on variants. For example, if the variant was an algorithm name, and a controller constructor took IHashAlgorithm as a constructor parameter. How could we make it so that the implementation was dependent upon an assigned variant?
Also, it tries to resolve #39.
This draft PR provides one APIs: FeatureManagementBuilder.AddFeaturedService<TService>
Example usage can be found here: examples/FeatureBasedInjectionPOC/Program.cs
Open to suggestion for the design of feature-based injection.

An exception should be thrown if there are multiple IVariantServiceProvider<TService> being registered.


Adding Nested & conditional fields support

Our advanced editor does not support Nested Fields
for example :
Textarea should include textfields,checkboxes,selects,radios, other  text areas etc
This should also be reflected in the JSON schema as well

https://json-schema.org/understanding-json-schema/reference/conditionals.html

Conditional Fields Support will be added for : -

Conditional field creation for text fields
Conditional field creation for checkboxes

This is example for understanding how it will work:

Development Approach :-

For each conditional field, define the triggering condition based on the value of another field.
Update the Formio object to include the conditional property for each field.
Sample code for adding conditional fields in the Formio object:

{   type: 'textfield',   key: 'firstName',   label: 'First Name',   conditional: {     show: true, // or false to hide the field     when: 'age', // the field that triggers the condition     eq: '18' // the value that triggers the condition   } }

Once the conditional fields are added to the Formio object, it is needed to convert it into JSON Schema format to handle dependent and nested fields.

For dependent fields, use the 'anyOf' keyword in JSON Schema to define the condition and required fields.

Sample code for dependent fields in JSON Schema format:
{ "type": "object", "properties": {   "firstName": {     "type": "string",     "title": "First Name"   },   // Other properties... }, "required": [   // List of required properties... ], "anyOf": [   {     "properties": {       "age": {         "const": "18"       }     },     "required": [       "firstName"     ]   },   {     "required": []   } ] }

For nested fields, the JSON Schema format is needed to define the nested structure and include the conditional logic within the nested objects.

Sample code for nested fields in JSON Schema format:
{ "type": "object", "properties": {   "person": {     "type": "object",     "properties": {       "firstName": {         "type": "string",         "title": "First Name"       },       // Other properties...     },     "required": [       // List of required properties...     ],     "anyOf": [       {         "properties": {           "age": {             "const": "18"           }         },         "required": [           "firstName"         ]       },       {         "required": []       }     ]   } } }

By following these steps and incorporating the provided code samples, we can efficiently add conditional fields to your Formio object and convert it into JSON Schema format to handle dependent and nested fields.

@Pratikshakhandagale  plzz verify the approch and give suggestions.

**How User can Implement Conditional and Nested fields : **

Identify the fields in the form that should have conditional behavior based on specific criteria

Like the form requirements and determine which fields should be conditionally shown or hidden.
Determine the triggering conditions that determine the visibility of the conditional fields.
Consider the values of other fields or user input as criteria for showing or hiding specific fields.

Use the form builder or configuration options to implement the conditional fields.

Locate the field that needs conditional behavior in the form builder.
Access the field's settings or properties to enable the conditional feature.
Define the triggering condition, such as selecting the field that triggers the condition and specifying the
required value for the condition to be met.

In this way, formio buider can be use to create Cnditional and Nested Fields :


fix: search for release commit with commit_subject

Use commit_subject template in release commit search.

@d-ryzhikov this looks like a good catch! Since we don't want to identify the current version with a commit that doesn't match the specified commit_subject, is it worth adding an explicit test that a commit in the log which doesn't match the commit_subject template but contains the current version returns None?

@bernardcooke53 I've added a test for a case when there is a commit with a message containing current version but the subject doesn't match commit_subject setting. However, in this case evaluate_version_bump shouldn't return None. Instead it should not evaluate this commit as the release commit and continue the search.

@d-ryzhikov looks good! Yes you're right about the bump, in my head I was thinking of an out-of-scope commit not triggering a bump - the way you've done it is good too :slightly_smiling_face:

I updated the release commit search logic in evaluate_version_bump and in generate_changelog. Now the release commit check is more strict: a commit message for a target version is constructed and only commits with exactly matching messages are considered to be target version commits.


[BUG] Unable to mark markdown code blocks ignored

Describe the bug
With a markdown code block that a secret is detected in. I'm unable to ignore the code block (and not affect the documentation).
To Reproduce
Steps to reproduce the behavior:

Add the following to a markdown file```bash
export ARTIFACTORY_API_KEY="<From above>"
export<EMAIL_ADDRESS>
# 👇 the `\` is to allow github to render a markdown code block inside another
# code block. It would be deleted in the test
\```

Run pre-commit
See the errorSecret Type: Secret Keyword
Location:    docs/index.md:16

There's no way to mark the line with ARTIFACTORY_API_KEY as allowed without changing how the documentation renders.

Impact
Medium. I have to ignore the whole file instead of a single line
Additional context:

Operating System, MacOs
Languages scanned etc; Markdown

I am having a similar issue, with markdown. I can not seem to find a way to put a comment in markdown.
e.g.
# Documentation

| Attribute | Where to look |
| ---| ---|
| password                 | Secret: `mas-mongo-ce-admin-admin`->`password`           |
| CA certificate           | ConfigMap: `mas-mongo-ce-cert-map`                              |
| TLS Certificate          | Secret: `mas-mongo-ce-cert-secret`->`tls.crt`                   |

To access the local Mongo DB use:

running pre-co0mmit gives
[INFO] Stashing unstaged files to /Users/matt/.cache/pre-commit/patch1670533655-25798.
Detect secrets...........................................................Failed
- hook id: detect-secrets
- exit code: 1

Potential secrets about to be committed to git repo! Please rectify.

Secret Type: Secret Keyword
Location:    MongoDB Enterprise.md:5

Secret Type: Secret Keyword
Location:    MongoDB Enterprise.md:7

Possible mitigations:

  - Mark false positives with an inline `pragma: allowlist secret`
    comment
  - Commit with `--no-verify` if this is a one-time false positive

If a secret has already been committed, visit
https://help.github.com/articles/removing-sensitive-data-from-a-
repository

[INFO] Restored changes from /Users/matt/.cache/pre-commit/patch1670533655-25798.

This should be covered via the --exclude-lines CLI argument; by doing an --exclude-lines "export ARTIFACTORY_API_KEY=\"<From above>\"" now,
cat issue7/test.md

```bash
export ARTIFACTORY_API_KEY="<From above>"
export<EMAIL_ADDRESS>
# 👇 the `\` is to allow github to render a markdown code block inside another
# code block. It would be deleted in the test
\```

detect-secrets scan --update .secrets.baseline --exclude-lines "export ARTIFACTORY_API_KEY=\"<From above>\"" --all-files issue7

detect-secrets audit .secrets.baseline
Nothing to audit!


Add 'reset' Command

Hey @eliangcs !
Awesome tool. I'm clearly going to get a lot of use out of this.  I'm sure you've thought about lots of these, but if we have them as individual Issues then we can discuss them/track changes/assign them, etc.
The first would be a simple 'reset' command. I like the current default behavior that 'name=jack' will persist, but I'd like to be able to reset it with a single command.

I'd probably name it rm * to allow users to clear all the settings they have.

rm * has been added in PR #34. Closing.


[Documentation] Update documentation with installation instructions for Vite.

Vite has specific settings that need to be set in vite.config.ts and vitest.config.ts in order for it to work with Evolu. Those are:
export default defineConfig({
	worker: {
		format: "es",
	},
	optimizeDeps: {
		exclude: ["@evolu/react", <EMAIL_ADDRESS>		include: ["react-dom"],
	},
})

It would be great if this was included as part of the installation instructions!

I will review it after the new sync and update docs. Thank you!


iOS (Unity support) marshaling delegates not supported for IL2Cpp

Currently I am trying to use Google Text-to-Speech with Nuget packages which are using for gRPC.
I see the experimental support for iOS, but when I try to use that library I am encountering "IL2CPP doesn't support marshalling delegates". How to resolve that issue?

Can anyone help me?

I have managed to find the same issue on Unity forum, so maybe this link will allow someone to understand the problem.

@ryangreen  @PawelWitak   feel free to open a pull request with the necessary patches. The potential problem is we don't want to resort to dirty hacks that would lower the quality of the C# code just because Unity happens to be very special runtime from many respects. But if we can find a clean way to work around the problem, we will be happy to accept the patch.

I am not sure this is dirty hack or workaround as I have checked that this patch works great even on real device (not only on simulator). So @ryangreen as it is your patch, maybe you will open a pull request?

I looked at @ryangreen's snippet more closely:
The idea of using [MonoPInvokeCallback(typeof(NativeMetadataInterceptor))] and a static method is the correct approach, but the solution is incomplete. Using static NativeMetadataCredentialsPlugin instance clearly doesn't allow more than one instance of NativeMetadataCredentialsPlugin to be used at a time and that's broken. So the snippet is useful as work-in-progress proof of concept, but it's definitely not acceptable into upstream as is.

I think this was fixed by #18327. The fix will be in the upcoming release v1.20.0 (or you can test using nightly build packages from packages.grpc.io, they should be available soon).
Can you confirm that this was fixed?

Confirmed, thanks :)


Configuration file uploaded into wrong place after vagrant halt && vagrant up

Trying to provision on aws using winrm communicator (using vagrant-aws, vagrant-winrm-syncedfolders plugins additionally). Provision looks like this:
config.vm.provision 'dsc' do |dsc|
    dsc.module_install = ['OctopusDSC']
    dsc.configuration_file = 'TentacleInstall.ps1'
    dsc.manifests_path = 'manifests'
    dsc.abort_on_dsc_failure = true
end

After initial vagrant up  configuration file synced correctly to folder:
tmp\vagrant-dsc-3\manifests

But after doing vagrant halt && vagrant up for some reason configuration file synced into
tmp\vagrant-dsc-3\manifests\manifests

Hm... Did some additional tests. Looks like it is issue with vagrant-winrm-syncedfolders plugin.
Cimpress-MCP/vagrant-winrm-syncedfolders#11

Interesting. I'm happy to keep this open until we get a resolution on the linked issue.


Base Variable class

Implement  a class for basic operations with variables. Arithmetics, comparison and  bitwise operators to be overridden for this class.

Variable class is already allowing us to do all kind of magic with on-the-fly C expressions generation from pure Python expressions.


ENH add missing packages to removals

Checklist

[x] Used a static YAML file for the patch if possible (instructions).
[x] Only wrote code directly into generate_patch_json.py if absolutely necessary.
[x] Ran pre-commit run -a and ensured all files pass the linting checks.
[x] Ran python show_diff.py and posted the output as part of the PR.
[x] Modifications won't affect packages built in the future.

This PR adds these missing packages to the removals so that we don't tell someone to try and download them.
xref: https://github.com/conda/infrastructure/issues/892#issuecomment-2008028131

pre-commit.ci autofix

pre-commit.ci autofix

Like the block comment idea that you used. Applied in the other cases

Thanks Matt! 🙏
Let's get this in :shipit:


⚠ ClamAV has degraded performance

In e48f6b9, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 49 ms

Resolved: ClamAV performance has improved in bd8d38a.


A100 8 GPUs extremely slow compared to a single A100

Description
TL;DR When I run a t5x script using a A100-8 GPU machine it is much slower compared to running the same script on a single A100 machine.
There are many available configurations for running language model training using the t5x library which is based on jax/flax. To reproduce this issue one can take any of the training scripts run it on a single A100 and then do the same on a A100 8 GPU machine. I also cached the tfds dataset and read it from local disk to minimize any overhead and to just compare performance when switching from 1 GPU to 8 GPU single node training. The model I use doesn't require model parallelism it only suppose to data parallelism, e.g. you can pick up any model size up to (m)T5 XL (3.7B) to reproduce.
Here are the stats I got during my training runs:
# On A100 8 GPU machine - extremely slow / slow compile warning
seqs_per_second_per_core=0.156312

# On single A100 machine / runs fine
seqs_per_second_per_core=~6

********************************
[Compiling module pjit_train_step] Very slow compile? If you want to file a bug, run with envvar XLA_FLAGS=--xla_dump_to=/tmp/foo and attach the results.
********************************

I am not able to attach the file as zst format is not supported and zip compressed folder is too large.
What jax/jaxlib version are you using?
jax==0.4.9 jaxlib==0.4.9+cuda12.cudnn88
Which accelerator(s) are you using?
GPU
Additional system info
Python 3.9.16 (main, Mar  8 2023, 14:00:05)  [GCC 11.2.0] :: Anaconda, Inc. on linux
NVIDIA GPU info
Thu May 11 09:01:07 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-SXM4-80GB           On | 00000000:05:00.0 Off |                    0 |
| N/A   30C    P0               77W / 400W|  61847MiB / 81920MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A100-SXM4-80GB           On | 00000000:06:00.0 Off |                    0 |
| N/A   29C    P0              127W / 400W|  61835MiB / 81920MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA A100-SXM4-80GB           On | 00000000:07:00.0 Off |                    0 |
| N/A   35C    P0              152W / 400W|  61835MiB / 81920MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA A100-SXM4-80GB           On | 00000000:08:00.0 Off |                    0 |
| N/A   30C    P0               79W / 400W|  61835MiB / 81920MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   4  NVIDIA A100-SXM4-80GB           On | 00000000:09:00.0 Off |                    0 |
| N/A   34C    P0               81W / 400W|  61835MiB / 81920MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   5  NVIDIA A100-SXM4-80GB           On | 00000000:0A:00.0 Off |                    0 |
| N/A   30C    P0               86W / 400W|  61835MiB / 81920MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   6  NVIDIA A100-SXM4-80GB           On | 00000000:0B:00.0 Off |                    0 |
| N/A   34C    P0               84W / 400W|  61835MiB / 81920MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   7  NVIDIA A100-SXM4-80GB           On | 00000000:0C:00.0 Off |                    0 |
| N/A   33C    P0               78W / 400W|  61835MiB / 81920MiB |    100%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A    271492      C   python3                                   61776MiB |
|    1   N/A  N/A    271492      C   python3                                   61764MiB |
|    2   N/A  N/A    271492      C   python3                                   61764MiB |
|    3   N/A  N/A    271492      C   python3                                   61764MiB |
|    4   N/A  N/A    271492      C   python3                                   61764MiB |
|    5   N/A  N/A    271492      C   python3                                   61764MiB |
|    6   N/A  N/A    271492      C   python3                                   61764MiB |
|    7   N/A  N/A    271492      C   python3                                   61764MiB |
+---------------------------------------------------------------------------------------+

@nouiz @yhtang Thanks for replying quickly. Let me give more details below by trying to reproduce the issue on 2 GPU A100 machine (I couldn't get a 8 A100 this time I will try again with it once available):
TL;DR It worked as intended in a 2 A100 GPU - more details below.
Software Stack
Running umt5 xl model.
Model implementation uses jax pjit and according to README should support TPU (most efficient-but pricey) and GPU clusters.
I am not using SLURM yet, right now for initial benchmarking I am using a script to do single node training with 8 GPUs where distribution strategy is only data parallelism. Later, I plan to try a larger model on single node and also move the training to a GPU cluster based on the price-performance compared to TPUs.
Training script:
#!/bin/bash
# set train hyperparams
unset LD_PRELOAD

# set environment variables
PROJECT_DIR=${HOME}"/<DIRECTORY_WITH_GIN_FILES>"
T5X_DIR=${HOME}"/t5x"  # directory where the t5x is cloned.
TFDS_DATA_DIR="<NOT_SO_IMPORTANT_LOCAL_CACHED_DATA_IS_USED>"
export PYTHONPATH=${PROJECT_DIR}

export XLA_PYTHON_CLIENT_PREALLOCATE=true
export XLA_PYTHON_CLIENT_MEM_FRACTION=0.9 # default is less around 61/80 GB

python3 ${T5X_DIR}/t5x/train.py \
  --gin_search_paths=${PROJECT_DIR} \
  --gin_file="xl_umt5_pretrain.gin" \
  --tfds_data_dir=${TFDS_DATA_DIR}

Content of xl_umt5_pretrain.gin:
from __gin__ import dynamic_registration
import tasks

import __main__ as train_script
from t5x import utils
from t5x import partitioning
from t5x import trainer
import seqio

include "t5x/examples/scalable_t5/umt5/pretrain_xl.gin"

MODEL_DIR = "<MODEL_DIR_TO_SAVE>"
MIXTURE_OR_TASK_NAME = "<SEQIO_MIXTURE_NAME>"
TRAIN_EVAL_MIXTURE_OR_TASK_NAME = "<SEQIO_MIXTURE_NAME>"
LOSS_NORMALIZING_FACTOR = None
# LOSS_NORMALIZING_FACTOR = "NUM_REAL_TARGET_TOKENS"
DROPOUT_RATE = 0.0
TRAIN_STEPS = 1_011_200  # 1_000_000 pretrained steps + 11_200 pre-finetuning
TASK_FEATURE_LENGTHS = {"inputs": 512, "targets": 512}
RANDOM_SEED = 42
# BATCH_SIZE = 1024
# BATCH_SIZE = 16
BATCH_SIZE = 128 # 16 x 8 = 128 with 8 A100 GPUs
SHUFFLE_TRAIN_EXAMPLES = False # shuffles files for tfds data source
USE_CACHED_TASKS = True

# ------------------- Training specification overrides --------------------------
train_script.train:
  eval_period = 2000
  stats_period = 20
  eval_steps = 20
  random_seed = %RANDOM_SEED # (used for dropout) but just to be sure in general
  train_dataset_cfg = @train/utils.DatasetConfig()
  # train_eval_dataset_cfg = @train_eval/utils.DatasetConfig()
  train_eval_dataset_cfg = None # gradient accumulation not working?
  infer_eval_dataset_cfg = None
  run_eval_before_training = False # OOM with microbatches when True
  partitioner = @partitioning.PjitPartitioner()

partitioning.PjitPartitioner:
  model_parallel_submesh = (1,1,1,1)
  num_partitions = None

trainer.Trainer:
  num_microbatches = 0 # 1024 / 32 = 32 (bs 32 tpu v3-8)

utils.create_learning_rate_scheduler:
  factors = 'rsqrt_decay'
  step_offset = 960_000 # step = 1_000_000 - 960_000, 1 / sqrt(40_000) = 5e-3

utils.DatasetConfig:
  seed = 42 # read_config.shuffle_seed in the case of TfdsDataSource

utils.SaveCheckpointConfig:
  period = 2000
  save_dataset = True
  # keep = 5 # default is None which keeps all checkpoints

utils.RestoreCheckpointConfig:
  mode = 'specific'
  dtype= 'bfloat16'
  path = 'gs://t5-data/pretrained_models/t5x/umt5_xl/checkpoint_1000000'
  assignment_map = None
  strict = True
  dtype = None

Machine Specs
1000 GB RAM | 128 vCPUs | 4000GB SSD
lscpu.info.txt
cpu.info.txt
Base Ubuntu 20.04
Dependency Installation Steps:

CUDA:

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb
sudo apt-get update
sudo apt-get -y install cuda

Miniconda:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
...# install miniconda
conda create -n myenv python=3.9

Verify jax with GPUs:

python -c "import jax; print(jax.device_put(jax.numpy.ones(1), device=jax.devices('gpu')[0]).device())"
python -c "import jax; print(jax.devices())"

gpu:0
[StreamExecutorGpuDevice(id=0, process_index=0, slice_index=0), StreamExecutorGpuDevice(id=1, process_index=0, slice_index=0)]

T5X:

git clone https://github.com/google-research/t5x.git
cd t5x
pip install -e .
# also install dependencies
pip install t5

Download cached dataset to machine.

Run single node training scripts

Running with 1 device visible with CUDA_VISIBLE_DEVICES=0
PROJECT_DIR=
T5X_DIR=
TFDS_DATA_DIR=
export PYTHONPATH=${PROJECT_DIR}

# Setting XLA flags
export XLA_FLAGS="--xla_gpu_simplify_all_fp_conversions \
--xla_gpu_all_reduce_combine_threshold_bytes=136314880 ${XLA_FLAGS}\
--xla_dump_to=xla_run_dump"

export CUDA_VISIBLE_DEVICES=0
export XLA_PYTHON_CLIENT_PREALLOCATE=true
export XLA_PYTHON_CLIENT_MEM_FRACTION=0.9

python3 ${T5X_DIR}/t5x/train.py \
  --gin_search_paths=${PROJECT_DIR} \
  --gin_file="xl_umt5_pretrain.gin" \
  --tfds_data_dir=${TFDS_DATA_DIR}

single_gpu_train.log
Note: Interestingly I got OOM with a
single_gpu_train_bs8_oom.log
batch size = 8 - in a single machine I was able to train with a batch size of 16.
Adding training logs:
single_gpu_train.log
Running with all 2 devices (assuming data parallelism via JAX pjit)
Adding training logs:
two_gpus_train.log
Performance per core / gpu is similar but need to try again with 8 GPUs.
1 gpu training: timing/seqs_per_second_per_core=5.6435
2 gpu training: timing/seqs_per_second_per_core=4.85872

Is this still a problem? We've done lots of work on LLM training in JAX, and we expect it to be screamingly fast these days on A100 GPUs.


libwebsocket init failed for relay

Mentioned this in the forum, though pretty sure it's a bug so reporting here.
The new relay feature merged into master recently doesn't seem to be working on the Pi.
I have configured:
{
  "listen": ["<IP_ADDRESS>", 7890],
  "relay": ["<IP_ADDRESS>", 7891],
  "verbose": true
  ..
}

with another server listening from <IP_ADDRESS>:7891.
Running fcserver with the above config, however, results in:
ERR: ERROR on binding to port 7891 (-1 99)
ERR: libwebsocket init failed for relay

Same error for "listen" when I put an IP address other than null or the localhost (it works with <IP_ADDRESS> or its actual LAN IP address) on a Pi Zero W (Raspbian Stretch Lite)
ERR: ERROR on binding to port 7890 (-1 99)
ERR: libwebsocket init failed

with config
{
    "listen": ["<IP_ADDRESS>", 7890],
    "relay":  ["<IP_ADDRESS>", 7891],
    "verbose": true,
    ..
}

When I use null and connect a relay client, the fcserver on RPI crashes with a Segmentation fault.  Where running the same client examples connected to fcserver running on Mac OSX everything works as expected.

Same thing. Even with the default config on a Rpi. Config:
    "listen": ["<IP_ADDRESS>", 7890],
    "relay":  [null, 7891],
    "verbose": true,

Error:
$ ./fcserver-rpi config.json
[1534618035:8677] ERR: ERROR on binding to port 7890 (-1 98)
[1534618035:8679] ERR: libwebsocket init failed

And with no config file:
$ ./fcserver-rpi
No system config file found, using default
[1534618109:8172] ERR: ERROR on binding to port 7890 (-1 98)
[1534618109:8173] ERR: libwebsocket init failed

Same results as root and as pi user.

Mentioned this in the forum, though pretty sure it's a bug so reporting here.
The new relay feature merged into master recently doesn't seem to be working on the Pi.
I have configured:
{
  "listen": ["<IP_ADDRESS>", 7890],
  "relay": ["<IP_ADDRESS>", 7891],
  "verbose": true
  ..
}

with another server listening from <IP_ADDRESS>:7891.
Running fcserver with the above config, however, results in:
ERR: ERROR on binding to port 7891 (-1 99)
ERR: libwebsocket init failed for relay

I believe it is actually functioning as intended. "Relay" creates a second server on the host running fcserver, it does not create a client that connects to a server on another host. If you look at the error you are receiving, the (-1 99) indicates bind failed because "A nonexistent interface was requested or the requested address was not local." and based on your config and description, that is not a local address. The other error mentioned in this thread (-1 98) indicates the port is already in use so another instance of fcserver is probably already running.
As for the other problems mentioned in this thread, the only valid values you should be able to pass to either "listen" or "relay" would be null, "<IP_ADDRESS>", or the ip of one of the interfaces on that host. That is because both create a server that waits for someone to connect to it, it is not creating a client that connects to another host.
Unfortunately that means the relay feature cannot be used to connect multiple instances of fcserver across hosts, which is what I was hoping it did.


Add blockpage fingerprint for CY

A blockpage fingerprint for Cyprus.
Example fingerprint:
http://www.safeinternet.cytanet.com.cy:8080/webadmin/deny
Tag: netsweeper

Do you have an example OONI Explorer measurement for this?

I do not recall the measurement in question, I will have a deeper look in the measurements.
Here is another fingerprint from Fortinet:
https://explorer.ooni.org/measurement/20190601T094329Z_AS6866_Ic09ftdU4F6shVRRfQ2HmndCrdyOuaZzOIqWWKTlhtqDzXmVmA?input=http://www.http-tunnel.com/
BTW it will be very useful to tag all SSL/TLS verification errors such as the one below, that is actually on the same network where the Fortinet middlebox is located:
https://explorer.ooni.org/measurement/20190601T094329Z_AS6866_Ic09ftdU4F6shVRRfQ2HmndCrdyOuaZzOIqWWKTlhtqDzXmVmA?input=https://ooni.torproject.org/
Feel free to open a new issue if you prefer.


⚠ Turnos 24x7 has degraded performance

In 702af3e, Turnos 24x7 (https://reservas-dev.alternativasinteligentes.com/) experienced degraded performance:

HTTP code: 200
Response time: 809 ms

Resolved: Turnos 24x7 performance has improved in c548659.


GLMakie not plotting lines correctly

When I plot lines with GLMakie things don't appear to plot correctly. For example, this snippet from the basic tutorial:
using GLMakie

f = Figure()
ax = Axis(f[1, 1])
x = range(0, 10, length=100)
y = sin.(x)
lines!(ax, x, y)
f

produces this:

Any ideas of what's happening?
I'm using:
  [e9467ef8] GLMakie v0.5.5
  [ee78f7c6] Makie v0.16.6

on Julia 1.6.7.
PS: I tried using CairoMakie to check if this happened too but I'm using it from the REPL and I haven't figured out how to make cairo plots show up yet

Another case of #1460?

Ah, yeah, good catch. I'm also using AMD on Linux so it's gotta be it...
I'm closing this


Roadmap

Thank you for your library! Great work! 👍
Do you have a roadmap for future development?

@harg Thanks for you issue, I have created a roadmap on Wiki, welcome to discuss a new feature or change.


Pass resolved color instead of resource id

Change to getResources().getColor() method to avoid the lint message: "Should pass resolved color instead of resource id"

Ali: Note that you'll have to manually merge this into the internal repo. Using GitHub's automatic merge will cause the changes to get lost with the next automerger run.


3.0~b36 no longer works on iOS 11

Unc0ver 3.0~b36 no longer works on iOS 11.3.1 (iPhone 6 Plus). The phone just reboots on 38/38, even with "Load Tweaks" and "Reload System Daemons" unchecked. Persists even after a rootFS restore.

Have same Phone on IOS 11.4.1 was JB with B26 and went to B36 last night! No Issues what so ever here.
I Icleaned 1st
rebooted Phone
Uninstalled B26
Cydia Impacted in B36
Turned on Airplane Mode
Jail Broke

Same issue on b37 on iOS 11.4.1 (iPad Air). After multiple tries and kernel panics it gets to 38/38 then briefly shows a message about jailbreak completed but then it panics and reboots.

Oh. I know what’s wrong!! - This happens because the v1ntex exploit doesn’t properly clean up after stealing the kernel credentials. - There’s code in Apple’s sandbox that causes a kernel panic if these stolen credentials get detected. - Older builds used to patch data in Sandbox to break the code. - This build unpatches the data as soon as it gets done with it. - So I will have to change it to not unpatch it.

Oh, I know what’s wrong. - This happens because the v1ntex exploit doesn’t properly clean up after stealing the kernel credentials. - There’s code in Apple’s sandbox that causes a kernel panic if these stolen credentials get detected. - Older builds used to patch data in Sandbox to break the code. - This build unpatches the data as soon as it gets done with it. - So I will have to change it to not unpatch it.

Great! Waiting for the next release then.
PS. Is that considered normal that the exploit fails at step 2 a lot? Before updating to b37 I had to try at least 5 times and 5 reboots before the exploit worked and completed successfully.


ValueError: Can't bulk create a multi-table inherited model

Problem
I have a super Notification(models.Model) class which I use to create many notification subclasses such as PostNotification(Notification), CommentNotification(Notification), etc., and when trying to run CommentNotification.objects.bulk_create(my_list_of_notification_objects), i get the following traceback:
File "/home/me/.virtualenvs/project/local/lib/python2.7/site-packages/django/db/models/query.py", line 429, in bulk_create
    raise ValueError("Can't bulk create a multi-table inherited model")
ValueError: Can't bulk create a multi-table inherited model

and upon inspecting the query.py file, we get this causes the error:
for parent in self.model._meta.get_parent_list():
      if parent._meta.concrete_model is not self.model._meta.concrete_model:
           raise ValueError("Can't bulk create a multi-table inherited model")

Environment

Django Model Utils version: 3.1.1
Django version: 1.11.7
Python version: 2.7.3

Example
PostNotification.objects.bulk_create(
   [PostNotification(related_user=user, post=instance) for user in users]
)

throws the above exception

I FOUND THE SOLUTION MY DUDES
Simply run BaseClass.objects.bulk_create(list_of_SubClass_objects) instead of SubClass.objects.bulk_create(list_of_SubClass_objects)! :)

CORRECTION: IT WAS NOT ACTUALLY FIXED!
running Notification.objects.bulk_create([SubClassNotification(),SubClassNotification()] only creates base class Notification objects and subsequently running SubClassNotification.objects.all() returns an empty queryset.

Hi @aldenjenkins, some progress trying doing that?

Any update on this issue?

I believe the issue is impossible to fix. My solution months ago was to redo the wntire notification backend and just use a single Notification() model with a type char selection field to differentiate the type of notification that it was

I found a hacky solution. Check this out.
https://stackoverflow.com/questions/49826482/django-valueerror-cant-bulk-create-a-multi-table-inherited-model/62899646#62899646

Thanks @sajid1189. I am no longer working on the project which raised this issue so its not trivial for me to test this, but i will close this issue now that it seems you have created a potential solution.


When hovering & scrolling, the editor should consume available scroll space before parent can scroll

Current behavior:
When the mouse cursor is over the editor, the editor has scrollable content, and the page is being scrolled by the mouse wheel, the page and editor scroll at the same time. It's not possible to use the mouse scroll wheel to scroll to the bottom of the editor, because the editor moves at the same time while its inner content is scrolling.

The above example uses a SingleChildScrollView and a Column. I also tried to share a ScrollController between the Editor and SingleChildScrollView, but there was no difference in behavior.
Reproducible sample

import 'package:flutter/material.dart';
import 'package:super_editor/super_editor.dart';

void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: MyHomePage(title: 'Flutter Demo Home Page'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  MyHomePage({Key key, this.title}) : super(key: key);
  final String title;

  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  ScrollController _scrollController;
  Document _doc;
  DocumentEditor _docEditor;

  @override
  void initState() {
    super.initState();
    _scrollController = ScrollController();
    _doc = _createInitialDocument();
    _docEditor = DocumentEditor(document: _doc);
  }

  @override
  void dispose() {
    _doc.dispose();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      backgroundColor: Colors.blue,
      body: Scrollbar(
        child: SingleChildScrollView(
          controller: _scrollController,
          child: Column(
            children: [
              Center(
                child: ConstrainedBox(
                  constraints: BoxConstraints(maxWidth: 800),
                  child: Row(
                    children: [
                      Expanded(
                        child: Text(
                          'A supercharged rich text editor for Flutter',
                          style: TextStyle(fontSize: 40),
                        ),
                      ),
                      Expanded(
                        child: Container(
                          color: Colors.red,
                          height: 320,
                          child: Editor.standard(
                            scrollController: _scrollController,
                            editor: _docEditor,
                            padding: const EdgeInsets.symmetric(
                                vertical: 56, horizontal: 24),
                          ),
                        ),
                      ),
                    ],
                  ),
                ),
              ),
              const SizedBox(height: 16),
              for (var i = 0; i < 200; i++) ...[
                const SizedBox(height: 16),
                Text('blah blah blah blah blah blah blah blah blah blah blah')
              ]
            ],
          ),
        ),
      ),
    );
  }
}

Document _createInitialDocument() {
  return MutableDocument(
    nodes: [
      ParagraphNode(
        id: DocumentEditor.createNodeId(),
        text: AttributedText(
          text: 'Try Me! I’m live!!!!',
        ),
        metadata: {
          'blockType': 'header1',
        },
      ),
      ParagraphNode(
        id: DocumentEditor.createNodeId(),
        text: AttributedText(
          text: 'Try typing here and editing this text.',
        ),
      ),
    ],
  );
}

Expected behavior:
When hovering the editor and scrolling, the page does not scroll if the editor has "unconsumed" scrollable content.

Possibly related to #106

@roughike do you know how this is typically handled in a Flutter UI? I imagine that the parent scroll view has to do something to make this possible, even if we also need changes to the editor UI. Any thoughts?

@rutvik110 would you like to give this ticket a try? If so, let me know, and I'll assign it to you.

yah, I'll check on this issue.

@roughike This issue doesn't seem to be present any longer. Testing this on master, and the scrolling behaviour's working as you wished it should when the editor is present within an parent scrollable.
Here's a latest demo where you can see the editor's scrollable content if any is consumed if it's in focus/we're hovering over it before parent scrollable is scrolled. Also tested this with mouse scroll wheel, works fine.
https://github.com/superlistapp/super_editor/assets/65209850/93260d80-9413-45ab-bb29-b5b82fd5f9bf

@matthew-carroll The mentioned issue is no longer present within super_editor. Likely resolved when #106 got fixed.
Though right now we seem to be lacking tests for this scrolling behaviour. If you would like I can go ahead and add tests relevant to it to super_editor.

@rutvik110 I'm curious how this could have been solved by #106 when this issue is #120?
In any event, yes, let's add tests for this.

@rutvik110 I'm curious how this could have been solved by #106 when this issue is #120?

I assumed it may've been part of #106 as it was mentioned earlier in this issue. That may've been totally another thing though.

@matthew-carroll I think we can close down this issue as the behaviour is as expected in super_editor as @roughike mentioned.
Beside this, I'm working on some tests for locking down this behaviour in #1733 PR but that could go on separately.

If #1733 is about locking down behavior for this issue ticket then we can consider that to be a PR to close this issue. Let's get that work finished and merged and have it close this issue.

We spoke offline. Apparently the web scrolling behavior for this ticket works as expected, but the desktop (Mac, Windows, Linux) scrolling doesn't work as expected. The goal right now is for @rutvik110 to find out why there's a difference. Once we know exactly why things work on one platform and not another platform, we'll decide what to do in terms of changes/fixes/tests.

@matthew-carroll Rutvik's sample seem to work both on desktop and web. This sample works because the editor is wrapped with a ListView. Therefore it's just like the editor was inside a CustomScrollView, where the editor scrolls the ancestor scrollable.
If wrapping the editor with a ListView is acceptable, I think we should just close this ticket.

Ok. We can close this. If others have related issues, please file a new ticket with clear reproduction steps and desired behaviors.


feat: phoenixd ln backend

Phoenixd doesnt seem to let you set bolt11 expiry so quote should be set to what the invoice is


Incorrect usage of datetime.now(UTC)

I am encountering an issue while running the migration script to create a database while setting up the project.
This piece of code imports UTC from datetime, yet UTC is not a standalone object in datetime:
from datetime import UTC, datetime, timedelta
datetime.now(UTC)
I believe the correct way to use UTC is throgh the piece of code below:
from datetime import timezone
datetime.now(timezone.utc)
This approach correctly creates a datetime object representing the current time in UTC.

We are using python 3.12 for this project, and datetime.UTC is an alias for timezone.UTC (added in Python 3.11).

As mentioned in docs


[BUG]

ElasticDump version  - 6.8.2
Elasticsearch version - 7.2.0
Node.js version - 12.10
Full command you are having issue with (with inputs and outputs)

node ./node_modules/.bin/elasticdump --input=http://localhost:9301 --output=/data/myindex/myindex.json --fileSize=10mb
After downloading for 3 hrs it stopped with the following error.
Fri, 11 Oct 2019 14:48:14 GMT | Error Emitted => {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [16347185302/15.2gb], which is larger than the limit of [16249153126/15.1gb], real usage: [16347168248/15.2gb], new bytes reserved: [17054/16.6kb]","bytes_wanted":16347185302,"bytes_limit":16249153126,"durability":"PERMANENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [16347185302/15.2gb], which is larger than the limit of [16249153126/15.1gb], real usage: [16347168248/15.2gb], new bytes reserved: [17054/16.6kb]","bytes_wanted":16347185302,"bytes_limit":16249153126,"durability":"PERMANENT"},"status":429}
Fri, 11 Oct 2019 14:48:14 GMT | Total Writes: 12443900
Fri, 11 Oct 2019 14:48:14 GMT | dump ended with error (get phase) => Error: {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [16347185302/15.2gb], which is larger than the limit of [16249153126/15.1gb], real usage: [16347168248/15.2gb], new bytes reserved: [17054/16.6kb]","bytes_wanted":16347185302,"bytes_limit":16249153126,"durability":"PERMANENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [16347185302/15.2gb], which is larger than the limit of [16249153126/15.1gb], real usage: [16347168248/15.2gb], new bytes reserved: [17054/16.6kb]","bytes_wanted":16347185302,"bytes_limit":16249153126,"durability":"PERMANENT"},"status":429}

Then, when attempting to restart with offset it failed due to from/scroll incompatibility.
node ./node_modules/.bin/elasticdump --input=http://localhost:9301 --output=/data/myindex/myindex.json --fileSize=10mb --offset 12443800
Mon, 14 Oct 2019 14:27:40 GMT | starting dump
Mon, 14 Oct 2019 14:27:40 GMT | Warning: offsetting 12443800 rows.
Mon, 14 Oct 2019 14:27:40 GMT |   * Using an offset doesn't guarantee that the offset rows have already been written, please refer to the HELP text.
Mon, 14 Oct 2019 14:27:40 GMT | Error Emitted => {"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: using [from] is not allowed in a scroll context;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: using [from] is not allowed in a scroll context;"},"status":400}
Mon, 14 Oct 2019 14:27:40 GMT | Total Writes: 0
Mon, 14 Oct 2019 14:27:40 GMT | dump ended with error (get phase) => Error: {"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: using [from] is not allowed in a scroll context;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: using [from] is not allowed in a scroll context;"},"status":400}

This is an Elasticsearch issue.

Scroll no longer supports offset that's a breaking change introduced in ES v6.
circuit_breaking_exception is a JVM heap issue. Read here: https://www.elastic.co/guide/en/elasticsearch/reference/current/circuit-breaker.html. Either raise the limit or get more memory

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
 {
     "transient" : {
         "indices.breaker.total.limit" : "80%"
     }
 }

Thanks,
Does (1) mean there is no longer a way to handle continuing from failed dump?

Not really, unless u designed the index with an auto-incremented offset. That u can then filter on and continue. Sorry for the inconvenience.

I've added a change that outputs the scrollId, once the scrollTime is set to a high value. You can now input the scrollId and pick up from where you let off (providing that the scroll request hasn't expired)

@ferronrsmith

I've added a change that outputs the scrollId of each get request; once the scrollTime is set to a high value. You can now input the scrollId and pick up from where you let off (providing that the scroll request hasn't expired)

Hello! I don't see scrollId output here. Would you kindly to clarify, am I doing something wrong?
set NODE_OPTIONS=--max-old-space-size=4096; elasticdump --input=http://localhost:19201/$indexName --output=http://localhost:19202/$indexName --type=data --limit=500 --scrollTime=24h searchBody='{"query": {"match_all": {}}, "sort": ["_id": "asc"]}' --support-big-int
Wed, 06 Nov 2024 18:12:44 GMT | starting dump
(node:36508) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Wed, 06 Nov 2024 18:12:48 GMT | got 500 objects from source elasticsearch (offset: 0)
Wed, 06 Nov 2024 18:13:01 GMT | sent 500 objects to destination elasticsearch, wrote 500
Wed, 06 Nov 2024 18:13:04 GMT | got 500 objects from source elasticsearch (offset: 500)
Wed, 06 Nov 2024 18:13:12 GMT | sent 500 objects to destination elasticsearch, wrote 500
Wed, 06 Nov 2024 18:13:15 GMT | got 500 objects from source elasticsearch (offset: 1000)

--debug to output console data


[FEATURE] (#679) i18n for editor plugins

[DESC.] i18n for editor plugins

plugins/webida.editor.text-editor
plugins/webida.editor.code-editor
plugins/webida.editor.code-editor.content-assist.tern

Unintended white spaces are removed.


removed pytest

3.0.3 is available upstream and there is no need to duplicate it here.

Looks like the failure was the PyQT4 issue again. I retriggered the build, but we can merge after that even if the PyQT4 failure occurs again since this seems to be unrelated.


New properties page for file handles

I've implemented a new page for file handles in their properties. It shows the type of the handle (Pipe / File / Directory / Other), the mode, current position of the file pointer, and the size of the file. And it also allows you to flush the file buffer.

Looks good. Thanks!


Fix issues with gulp-sass

I just read the article about css namespacing on css wizardry and i love the idea of bem constructor but unfortunately it doesn't work with gulp-sass, it throws some errors like this one:
{ status: 1,
file: '../bower_components/bem-constructor/dist/_bem-constructor.scss',
line: 476,
column: 13,
message: 'error reading values after ',
code: 1 }
Even if i comment out line 476 it throws like 6-7 more errors. I think this isn't an issue with your library but it's probably node sass and gulp sass are still not updated to use some features that ruby sass has. Can this be fixed somehow?

JFYI: if you are using gulp-ruby-sass, the gem must be installed on your environment otherwise you will get this error: execvp(): No such file or directory.


Issue downloading blobs from Asia regions

or one at least. I ran the basic demo for an Oak-D which required checking out https://github.com/luxonis/depthai.git and running depthai_demo.py after installing dependencies including blobconverter.  This failed downloading the model with the error
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://blobconverter.s3.asia-southeast1.amazonaws.com/0a63afdfadf54974f7c890295c9c2b64f493980804133abbe2c0d29227ba1320.blob"

The host blobconverter.s3.asia-southeast1.amazonaws.com does not exist. I figured that it's very likely that the S3 buckets haven't been replicated or it's some region/multi-region thing as boto3 tries to automagically determine the closest region to download from.
The fix is to simply specify the region when creating the boto3 resource in cli/blobconverter/init.py
from
bucket = boto3.resource('s3', config=botocore.client.Config(signature_version=botocore.UNSIGNED))\
    .Bucket('blobconverter')

to
# region must be pinned to prevent boto3 specifying a bucket/region that doesn't exist
bucket = boto3.resource('s3',
    region_name = 'us-east-1',
    config=botocore.client.Config(signature_version=botocore.UNSIGNED))\
    .Bucket('blobconverter')

Hi @gmerrall , thanks for reporting this issue!
I introduced a quick fix here, but since I cannot reproduce this issue on my end, could you give it a try and see if it helps? If yes, I'll release this version as 0.0.12

Unfortunately it doesn't work. That's because the exception is raised against the bucket.download() method. That puts you in a tricky spot as you need to set the region when initialising the bucket object, but you don't know it's wrong until you actually. go to try and download the file.
I brutally tested this by initialising the bucket object as per defaults then modifying __download_from_s3_bucket with a try/except block.
i.e. (pruned)
def __download_from_s3_bucket()
  global bucket # required because of scope
  try:
    progress = __S3ProgressPercentage(bucket, key)
    bucket.download_file(key, str(fpath.absolute()), Callback=progress)
 except botocore.exceptions.EndpointConnectionError:
    bucket = boto3.resource('s3', region='us-east-1')
    progress = __S3ProgressPercentage(bucket, key)
    bucket.download_file(key, str(fpath.absolute()), Callback=progress)

That's pretty awful. :)
I think you may have to refactor things to create a method that inits bucket, and does a quick 0 byte or HEAD request for the object and if it fails, reinit bucket with the region added.
That might mean having the blob/key available earlier but from looking at the code, there's no reason you couldn't move initialising bucket into __download_from_s3_bucket() as it's the only place it's used.
The other option is to fix whatever is going on in S3. It's a bit odd IMHO for boto3 to redirect you to a region where the host doesn't exist.
I'm duty bound to say this doesn't happen on Google Cloud :)
If you'd prefer and don't mind waiting until the weekend I can refactor as described and submit a pr.

@gmerrall thank you for testing it! I did a quick google search and found that the head_bucket method can be the best way to check if the bucket exists in the region - and it worked! I was even able to test the bucket initialization error by fixing the region name of the first bucket initialization call to the one you're assigned by default

try:
    s3 = boto3.resource('s3', config=botocore.client.Config(signature_version=botocore.UNSIGNED, region_name='asia-southeast1'))
    bucket = s3.Bucket('blobconverter')
    s3.meta.client.head_bucket(Bucket=bucket.name)
except botocore.exceptions.EndpointConnectionError:
    # region must be pinned to prevent boto3 specifying a bucket/region that doesn't exist
    s3 = boto3.resource('s3', config=botocore.client.Config(signature_version=botocore.UNSIGNED), region_name='us-east-1')
    bucket = s3.Bucket('blobconverter')
    s3.meta.client.head_bucket(Bucket=bucket.name)

I updated the linked branch and if that resolves the issue on your end I think we're free to merge and release a new version

@gmerrall I merged mentioned PR with the hope that it will resolve your issue. If not, please feel free to reopen