Secondary install brocken and adguardhome-sync stuck

Hi,
I try to set up new installs of AGH on seperated hardware (PIs)
My main is a 3B+, AGH in docker, with 100-ish lists in the filter. That's also where I have AGH-sync installed in docker.
My secondary is a pi zero W, first with AGH in docher and it breaks once.
Then I installed AGH with apt directly and it broke again.
when I say brocken, everything is actually frozen, and I can ssh into the pi anymore.
At this moment, i have the logs below in AGH-sync and AGH-sync is displayed as running but the logs are frozen too.
Can you help please?
To Reproduce
Steps to reproduce the behavior:

What version of AdGuardHome sync used?
docker, fresh install
What version of AdGuardHome us used?
v0.106.3
How does the configuration look?
looks ok :-)
What is the error message?

n"}

2021-05-22T12:40:37.721Z	INFO	client	client/client.go:228	Add filter	{"host": "<IP_ADDRESS>", "url": "https://v.firebog.net/hosts/static/w3kbl.txt", "whitelist": false, "enabled": true}

2021-05-22T12:40:37.721Z	DEBUG	client	client/client.go:133	do post	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas"}

2021-05-22T12:40:39.362Z	DEBUG	client	client/client.go:138	got response	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas", "status": 200, "body": "OK 909 rules\n"}

2021-05-22T12:40:39.362Z	INFO	client	client/client.go:228	Add filter	{"host": "<IP_ADDRESS>", "url": "http://sysctl.org/cameleon/hosts", "whitelist": false, "enabled": true}

2021-05-22T12:40:39.362Z	DEBUG	client	client/client.go:133	do post	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas"}

2021-05-22T12:40:40.886Z	DEBUG	client	client/client.go:138	got response	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas", "status": 400, "body": "Couldn't fetch filter from url http://sysctl.org/cameleon/hosts: Get \"http://sysctl.org/cameleon/hosts\": couldn't do a GET request to 'https://dns10.quad9.net:443/dns-query', cause: Get \"https://dns10.quad9.net:443/dns-query?dns=Zw0BAAABAAAAAAAABnN5c2N0bANvcmcAAAEAAQ\": read tcp <IP_ADDRESS>:40118-><IP_ADDRESS>:443: read: connection reset by peer\n"}

2021-05-22T12:40:40.887Z	ERROR	sync	sync/sync.go:190	Error syncing filters	{"from": "<IP_ADDRESS>", "to": "<IP_ADDRESS>", "error": "400 Bad Request"}

github.com/bakito/adguardhome-sync/pkg/sync.(*worker).syncTo

	/go/src/app/pkg/sync/sync.go:190

github.com/bakito/adguardhome-sync/pkg/sync.(*worker).sync

	/go/src/app/pkg/sync/sync.go:146

github.com/bakito/adguardhome-sync/pkg/sync.Sync.func2

	/go/src/app/pkg/sync/sync.go:42

github.com/robfig/cron/v3.FuncJob.Run

	/go/pkg/mod/github.com/robfig/cron/v3@v3.0.1/cron.go:136

github.com/robfig/cron/v3.(*Cron).startJob.func1

	/go/pkg/mod/github.com/robfig/cron/v3@v3.0.1/cron.go:312

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add log files or json responses from AdGuardHome to help explain your problem.
Additional context
Add any other context about the problem here.

I just tried another hardware, Pi 3B+ and I have a similar issue. it's brocken..

2021-05-22T20:21:18.561Z	DEBUG	client	client/client.go:133	do post	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas"}

2021-05-22T20:21:19.260Z	DEBUG	client	client/client.go:138	got response	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas", "status": 200, "body": "OK 2057 rules\n"}

2021-05-22T20:21:19.260Z	INFO	client	client/client.go:228	Add filter	{"host": "<IP_ADDRESS>", "url": "https://raw.githubusercontent.com/VeleSila/yhosts/master/hosts", "whitelist": false, "enabled": true}

2021-05-22T20:21:19.261Z	DEBUG	client	client/client.go:133	do post	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas"}

2021-05-22T20:21:19.906Z	DEBUG	client	client/client.go:138	got response	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas", "status": 200, "body": "OK 9469 rules\n"}

2021-05-22T20:21:19.906Z	INFO	client	client/client.go:228	Add filter	{"host": "<IP_ADDRESS>", "url": "https://winhelp2002.mvps.org/hosts.txt", "whitelist": false, "enabled": true}

2021-05-22T20:21:19.906Z	DEBUG	client	client/client.go:133	do post	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas"}

2021-05-22T20:21:23.377Z	DEBUG	client	client/client.go:138	got response	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas", "status": 200, "body": "OK 8732 rules\n"}

2021-05-22T20:21:23.377Z	INFO	client	client/client.go:228	Add filter	{"host": "<IP_ADDRESS>", "url": "https://v.firebog.net/hosts/neohostsbasic.txt", "whitelist": false, "enabled": true}

2021-05-22T20:21:23.378Z	DEBUG	client	client/client.go:133	do post	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas"}

2021-05-22T20:21:23.851Z	DEBUG	client	client/client.go:138	got response	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas", "status": 200, "body": "OK 2411 rules\n"}

2021-05-22T20:21:23.852Z	INFO	client	client/client.go:228	Add filter	{"host": "<IP_ADDRESS>", "url": "https://raw.githubusercontent.com/RooneyMcNibNug/pihole-stuff/master/SNAFU.txt", "whitelist": false, "enabled": true}

2021-05-22T20:21:23.853Z	DEBUG	client	client/client.go:133	do post	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas"}

2021-05-22T20:21:24.584Z	DEBUG	client	client/client.go:138	got response	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas", "status": 200, "body": "OK 41339 rules\n"}

2021-05-22T20:21:24.584Z	INFO	client	client/client.go:228	Add filter	{"host": "<IP_ADDRESS>", "url": "https://raw.githubusercontent.com/Kees1958/W3C_annual_most_used_survey_blocklist/master/TOP_EU_US_Ads_Trackers_HOST", "whitelist": false, "enabled": true}

2021-05-22T20:21:24.585Z	DEBUG	client	client/client.go:133	do post	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas"}

2021-05-22T20:21:25.226Z	DEBUG	client	client/client.go:138	got response	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas", "status": 200, "body": "OK 3615 rules\n"}

2021-05-22T20:21:25.226Z	INFO	client	client/client.go:228	Add filter	{"host": "<IP_ADDRESS>", "url": "https://raw.githubusercontent.com/HorusTeknoloji/TR-PhishingList/master/url-lists.txt", "whitelist": false, "enabled": true}

2021-05-22T20:21:25.227Z	DEBUG	client	client/client.go:133	do post	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas"}

2021-05-22T20:21:31.493Z	DEBUG	client	client/client.go:138	got response	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas", "status": 200, "body": "OK 798438 rules\n"}

2021-05-22T20:21:31.493Z	INFO	client	client/client.go:228	Add filter	{"host": "<IP_ADDRESS>", "url": "https://raw.githubusercontent.com/chadmayfield/my-pihole-blocklists/master/lists/pi_blocklist_porn_all.list", "whitelist": false, "enabled": true}

2021-05-22T20:21:31.494Z	DEBUG	client	client/client.go:133	do post	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas"}

2021-05-22T20:21:38.375Z	DEBUG	client	client/client.go:138	got response	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas", "status": 200, "body": "OK 1906594 rules\n"}

2021-05-22T20:21:38.375Z	INFO	client	client/client.go:228	Add filter	{"host": "<IP_ADDRESS>", "url": "https://raw.githubusercontent.com/chadmayfield/my-pihole-blocklists/master/lists/pi_blocklist_porn_top1m.list", "whitelist": false, "enabled": true}

2021-05-22T20:21:38.375Z	DEBUG	client	client/client.go:133	do post	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas"}

2021-05-22T20:21:38.952Z	DEBUG	client	client/client.go:138	got response	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas", "status": 200, "body": "OK 11868 rules\n"}

2021-05-22T20:21:38.952Z	INFO	client	client/client.go:228	Add filter	{"host": "<IP_ADDRESS>", "url": "https://raw.githubusercontent.com/anudeepND/blacklist/master/facebook.txt", "whitelist": false, "enabled": true}

2021-05-22T20:21:38.952Z	DEBUG	client	client/client.go:133	do post	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas"}

2021-05-22T20:21:39.562Z	DEBUG	client	client/client.go:138	got response	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/add_url", "username": "Nicolas", "status": 200, "body": "OK 4000 rules\n"}

2021-05-22T20:21:39.562Z	INFO	client	client/client.go:252	Update filter	{"host": "<IP_ADDRESS>", "url": "https://adaway.org/hosts.txt", "whitelist": false, "enabled": true}

2021-05-22T20:21:39.562Z	DEBUG	client	client/client.go:133	do post	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/set_url", "username": "Nicolas"}

2021-05-22T20:21:40.003Z	DEBUG	client	client/client.go:138	got response	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/set_url", "username": "Nicolas", "status": 200, "body": ""}

2021-05-22T20:21:40.004Z	INFO	client	client/client.go:252	Update filter	{"host": "<IP_ADDRESS>", "url": "https://www.malwaredomainlist.com/hostslist/hosts.txt", "whitelist": false, "enabled": true}

2021-05-22T20:21:40.005Z	DEBUG	client	client/client.go:133	do post	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/set_url", "username": "Nicolas"}

2021-05-22T20:21:40.433Z	DEBUG	client	client/client.go:138	got response	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/set_url", "username": "Nicolas", "status": 200, "body": ""}

2021-05-22T20:21:40.434Z	INFO	client	client/client.go:263	Refresh filter	{"host": "<IP_ADDRESS>", "whitelist": false}

2021-05-22T20:21:40.435Z	DEBUG	client	client/client.go:133	do post	{"host": "<IP_ADDRESS>", "method": "POST", "path": "/filtering/refresh", "username": "Nicolas"}

2021-05-22T20:23:29.429Z	ERROR	sync	sync/sync.go:190	Error syncing filters	{"from": "<IP_ADDRESS>", "to": "<IP_ADDRESS>", "error": "Post \"http://<IP_ADDRESS>/control/filtering/refresh\": EOF"}

github.com/bakito/adguardhome-sync/pkg/sync.(*worker).syncTo

	/go/src/app/pkg/sync/sync.go:190

github.com/bakito/adguardhome-sync/pkg/sync.(*worker).sync

	/go/src/app/pkg/sync/sync.go:146

github.com/bakito/adguardhome-sync/pkg/sync.Sync.func2

	/go/src/app/pkg/sync/sync.go:42

github.com/robfig/cron/v3.FuncJob.Run

	/go/pkg/mod/github.com/robfig/cron/v3@v3.0.1/cron.go:136

github.com/robfig/cron/v3.(*Cron).startJob.func1

	/go/pkg/mod/github.com/robfig/cron/v3@v3.0.1/cron.go:312

It seems, that the target AGH is failing after the filter urls are refreshed. This might be a memory issue since you have that many filter lists.
Is AGH running if you set it up manually?

Hi,
I did set it up manually before. it worked fine until I want to use AGH sync.
Uploading my massive list to any instance using this python script works well though.
https://www.reddit.com/r/Adguard/comments/isfjz3/adguard_home_batch_block_list_made_easy/
I think I will upload the list to both instances before starting AGH sync.
The other option would be to copypaste the main AGH config file to the second instance before initiallizing.
But this doesn't make me try your soft.
I have high expectations from it :-D
good point is that its dev is around to adress issues 👍
Thanks for passing by and trying to help

I am progressing... not the way I expected but I am anyway
My 3B+ primary instance broke today while updating the filters and pull updates from all the lists.
It shows the same symptomes as when AGH-sync was breaking it too.
I might have to focus on a few lists to avoid hardware limitations.
I doesn't make much sense to have so many of them anyway.
It only saves me the pain of selecting a few, which will be hard actually.
Thank you for your software, I carry on my investigation and hope to use it in production soon

Hi,
the status of the lists (enabled or disabled) are not synced.
Is it an expected behaviour?
Thanks

enabled / disabled of lists are synched

thank you and sorry to pester


autotests updated 8.2.0 cuisine on ays & docker

dependencies

cuisine clean
ays clean
docker working well in ays

Very good!
I suggest we don't do it with a test method, but rather with unit/integration testing. Will add links to how we can do that cleanly and easily so we can later have a testsuite.

We have ays_build tests for that. Need to be verified on 8.2.0

@abdulrahmantkhalifa Closely coupled with https://github.com/Jumpscale/jumpscale_core8/issues/659. After fixing builds, please validate the tests all pass successfully.


Add staging table prefix as optional parameter to scd1 template

What is the problem that needs to be solved?
Currently, if a user has same table names across different schemas there will be a need of separate staging schema for each DWH.
This feature request is to add staging table prefix as optional parameter to scd1 template. The change will allow users to have single staging schema for all their DWHs.

Can you please provide an example.
E.g what is the case
job_input.execute_template(target_schema="vmc_schema", target_table="esx")

and another job
job_input.execute_template(target_schema="vsphere_schema", target_table="esx")

And both produce staging_schema.vdk_check_esx table which conflicts
?
Is that the problem .

Why not prefixing the staging table with the schema name? I think this is unique combination ? And would provide the needed uniqueness.
In any case you need to be able that The maximum number of characters allowed in Impala for a table name in Impala is 128 characters (https://impala.apache.org/docs/build/html/topics/impala_identifiers.html) and the solution need to take care of this.

Yes, the example you provided describes perfectly our case.
Do you mean providing the target schema as a prefix for the staging table?
If I reuse your example that would mean "staging_schema.vmc_schema_esx".
This case seems fine to me, except a user hasn't provided a staging schema and the target schema is used instead.
This would mean a table like this will be created "vmc_schema.vmc_schema_esx" which could be a little bit confusing.
I'm thinking of sticking to the current "vdk_check_" prefix, but also adding the schema to the table name. Something like "vmc_schema.vdk_check_vmc_schema_esx".
What do you think?

Yes - that's what I meant - vmc_schema.vdk_check_vmc_schema_esx . I think all tables used by vdk check should be prefixed the same way (wiht vdk_check_) to make them easer to find, separate, reduce chance of conflicts in case of same table name

Yes, next question is. How long it can become. Identifies must not be longer than 128 in Impala. Puting vdk_check_(which his 10 symbols + schema name which can arbitrary long . You should check inside your tables and use-cases which is the longer and likely you may find some that table + schema + prefix > 128)
I was think that if it's longer than 128 ... what can we do
A) We can hash it .eg. table name = vdk_check_ + hash("{target_schema}{target_table} ") - that would work but make the table hard to decifer
B) Or we can truncate it "vdk_check{target_schema}_{target_t" (thought there's risk of conflict (schema.sales with schema.sales_currency table :) )
C) hash only the schema name . That might also work.

Currently, the longest length based on our production data would be 66 symbols with prefix+schema+table. The table is called vcf_dw.map_of_release_bom_components_and_bundle_elements.
I assume that we won't hit a problem with our tables for now.
My thoughts on the suggested approaches:
A) Seems the only suggestion that we could take for avoiding to hit the 128 limit and unique name conflicts at the same time. It would bring some more complexity in extracting the table name as well as additional complexity in implementation.
B) Truncating the names over 128 to fit the limit looks like measured risk as I don't think we will face many such cases. Moreover, if the tables are with same names after truncating this won't be a problem since the current logic covers recreating the table if any difference is found. I assume that the only risk here for really edge case scenarios is to increase a little bit the DDL statements that will be executed while processing.
C) If we are hashing anything I would prefer hashing schema + table as this way we are still not insured that won't hit the limit if the table name is lets say 120 chars. Moreover, there should be an additional action to find out what is the exact table checks are being ran on in case of duplicate name tables across different schemas.
D) If people are following the best practices for sticking to shorter self-explanatory names the limit will be hardly reached. However I don't think it is right to restrict them to have short schema/table names in order to be able to use this functionality.
If I had to choose one of these it would be B.

I'd prefer sticking to raising an error. Schema+table name of 120 characters is unwisely long.


Few changes to encrypt-decrypt- page

I would like to add some features to the encrypt-decrypt- webpage

/assign

Go for it @z3dc0ps! Do share the screenshot of the changes when you make pull request


Crop timers?

Hello, I see that Accountant has a setting for crop timers, but I don't see my crop beds show up in there. I have crop timers enabled, and the plots added to the Plot Names list. Am I doing something wrong?
Thank you.

You need to interact with crops for them to show up (tending/planting). If you have done that and they did not appear, what's your client language?

Yes I've done that and they don't show up.
I suppose it may be related to the language yeah... I play in English and Japanese simultaneously (thanks to an XIVAlexander feature). So crop names show up in both English and Japanese...

Both in the original game interface? If not, which one is in the actual dialogue boxes? I can't read Japanese so the matching and parsing functions may be off for japanese dialogue.
 :D

Yeah... they're displayed like this:

I may be able to put an exception into the script that converts their names. I've had to do that before anyway.

Think this is something you have to solve on your side, I'll close this for now.


Passing custom payload when handling message / text to retrieve in action

Rasa Core version: >=0.11.12
Python version: 3.6.6
Operating system (windows, osx, ...): Windows
Issue: Is there any way of passing custom payload when handling / parsing a message that I can retrieve from for example the tracker during the custom action?
I have some additional information which I need when actions are happening.
Basically I would like to:
Use handle_message / text, pass any kind of container as payload as well as the message.
in the custom action retrieve the payload object and utilize the information.
Content of domain file (if used & relevant):

Just to clarify, do you want to pass this custom payload with the bot message or with the user message?

With the user message.
To be more clear, something like:
agent.handle_message(UserMessage(text="Hello World", sender_id="Testperson1", payload={"additional_information:" additional_information_object}))
or
agent.handle_text("Hello World", payload={"additional_information:" additional_information_object})
So I can retrieve that additional_information_object from the tracker of the custom action for example.

i don't think this is something we'll add, how would you know which payload to send with which user message?

In my case I don't get the messages directly. They're coming in asynchronous with information about channel / user / permissions / respond where / etc.. I can work around it and build a temporary storage then map the payload with the message + sender_id to identify it, the hassle will be with the lifetime of that temporary stored object as only some messages trigger actions, so I figured I'd ask if there's a way that would save me some headache :)

Yeah I think that's something you need to handle in your custom implementation then. As of now this is not something we'll be adding


Any way to add in the ability to move blocks after they've been placed?

Per title, it would be really good to be able to move blocks after they've been placed. Seems like it would probably be pretty difficult, but I haven't dug into how TM/OP plugins are built, so not sure if it's possible.

Do you mean that you would like to nudge a block slightly after placing it in free mode?
There isnt really an existing API in the game for that, but there is a plugin that does something similar https://openplanet.nl/files/109

Yes, that was what I wanted. Thanks!


Fix incorporation date error

Fixed Issues
$ https://github.com/Expensify/App/issues/13261
Tests
Same as QA

[ ] Verify that no errors appear in the JS console

Offline tests
QA Steps

Go to Profile > Workspace > Connect a bank account > Connect manually
Enter a valid routing number and account number (Tip: See https://stackoverflowteams.com/c/expensify/questions/342)
Click save & continue
On company information page, click on the incorporation date field
Enter a future date manually and click Save & Continue
Verify that an error appears indicating the date is not in range

[ ] Verify that no errors appear in the JS console

PR Author Checklist

[x] I linked the correct issue in the ### Fixed Issues section above
[x] I wrote clear testing steps that cover the changes made in this PR

[x] I added steps for local testing in the Tests section
[x] I added steps for the expected offline behavior in the Offline steps section
[x] I added steps for Staging and/or Production testing in the QA steps section
[x] I added steps to cover failure scenarios (i.e. verify an input displays the correct error message if the entered data is not correct)
[x] I turned off my network connection and tested it while offline to ensure it matches the expected behavior (i.e. verify the default avatar icon is displayed if app is offline)
[x] I tested this PR with a High Traffic account against the staging or production API to ensure there are no regressions (e.g. long loading states that impact usability).

[x] I included screenshots or videos for tests on all platforms
[x] I ran the tests on all platforms & verified they passed on:

[x] iOS / native
[x] Android / native
[x] iOS / Safari
[x] Android / Chrome
[x] MacOS / Chrome
[x] MacOS / Desktop

[x] I verified there are no console errors (if there's a console error not related to the PR, report it or open an issue for it to be fixed)
[x] I followed proper code patterns (see Reviewing the code)

[x] I verified that any callback methods that were added or modified are named for what the method does and never what callback they handle (i.e. toggleReport and not onIconClick)
[x] I verified that comments were added to code that is not self explanatory
[x] I verified that any new or modified comments were clear, correct English, and explained "why" the code was doing something instead of only explaining "what" the code was doing.
[x] I verified any copy / text shown in the product was added in all src/languages/* files
[x] I verified any copy / text that was added to the app is correct English and approved by marketing by adding the Waiting for Copy label for a copy review on the original GH to get the correct copy.
[x] I verified proper file naming conventions were followed for any new files or renamed files. All non-platform specific files are named after what they export and are not named "index.js". All platform-specific files are named for the platform the code supports as outlined in the README.
[x] I verified the JSDocs style guidelines (in STYLE.md) were followed

[x] If a new code pattern is added I verified it was agreed to be used by multiple Expensify engineers
[x] I followed the guidelines as stated in the Review Guidelines
[x] I tested other components that can be impacted by my changes (i.e. if the PR modifies a shared library or component like Avatar, I verified the components using Avatar are working as expected)
[x] I verified all code is DRY (the PR doesn't include any logic written more than once, with the exception of tests)
[x] I verified any variables that can be defined as constants (ie. in CONST.js or at the top of the file that uses the constant) are defined as such
[x] I verified that if a function's arguments changed that all usages have also been updated correctly
[x] If a new component is created I verified that:

[x] A similar component doesn't exist in the codebase
[x] All props are defined accurately and each prop has a /** comment above it */
[x] The file is named correctly
[x] The component has a clear name that is non-ambiguous and the purpose of the component can be inferred from the name alone
[x] The only data being stored in the state is data necessary for rendering and nothing else
[x] For Class Components, any internal methods passed to components event handlers are bound to this properly so there are no scoping issues (i.e. for onClick={this.submit} the method this.submit should be bound to this in the constructor)
[x] Any internal methods bound to this are necessary to be bound (i.e. avoid this.submit = this.submit.bind(this); if this.submit is never passed to a component event handler like onClick)
[x] All JSX used for rendering exists in the render method
[x] The component has the minimum amount of code necessary for its purpose, and it is broken down into smaller components in order to separate concerns and functions

[x] If a new CSS style is added I verified that:

[x] A similar style doesn't already exist
[x] The style can't be created with an existing StyleUtils function (i.e. StyleUtils.getBackgroundAndBorderStyle(themeColors.componentBG)

[x] If the PR modifies a generic component, I tested and verified that those changes do not break usages of that component in the rest of the App (i.e. if a shared library or component like Avatar is modified, I verified that Avatar is working as expected in all cases)
[x] If the PR modifies a component related to any of the existing Storybook stories, I tested and verified all stories for that component are still working as expected.
[x] I have checked off every checkbox in the PR author checklist, including those that don't apply to this PR.

Screenshots/Videos

Web

Mobile Web - Chrome
Same as Android native - can't test due to inability to enter a future date

Mobile Web - Safari

Desktop

iOS
Tests are not relevant on iOS since the input prevents you from entering anything in the future but leaving proof that I tried 🤔

Android
Not really testable since future dates are not allowed.

@marcaaron, this PR will cause a regression. After implementing this the user will see Incorporation date cannot be in the future error even for invalid dates

Please check my proposal It is already approved by @thesahindia

Thanks, I prefer the else if approach as it is less likely that someone will change the order of the validate methods.

Updated

@marcaaron, I believe the logic at isValidDate also needs to be changed
It should be like this -
function isValidDate(date) {
    if (!date) {
        return false;
    }

    const pastDate = moment().subtract(1000, 'years');
    const testDate = moment(date);

    return testDate.isValid() && testDate.isAfter(pastDate);
}

It needs to be changed because

the specific error that we have for future dates doesn't show up unless the future date is only between current day and 1000 years in future becauseAt isValidDate we check if the date is between 1000 years back and 1000 years in future and when it's more than 1000 years in future then isValidDate function returns false and we see Please enter a valid date
So, at isValidDate function we should only check that the date is not more than 1000 years back in the past and we won't return false for future dates. isValidPastDate function can handle the future dates if user uses a future date it will return false and they will see Incorporation date cannot be in the future

Less than 1000 year in future

+1000 years in future -

Seems like a separate issue to me. However, one that I'd argue we don't need to fix because it is very unlikely for someone to enter this year into the date picker.

all yours @danieldoglas

:hand: This PR was not deployed to staging yet because QA is ongoing. It will be automatically deployed to staging after the next production release.

🚀 Deployed to production by @chiragsalian in version: 1.2.38-6 🚀

platform
result

🤖 android 🤖
success ✅

🖥 desktop 🖥
success ✅

🍎 iOS 🍎
success ✅

🕸 web 🕸
success ✅


1129 - Consistency Checker - English Tutorial (Trie Tree Problem)

I will soon provide the english tutorial for the problem on trie "Consistency Checker" problem tag 1129

Thanks for your contribution @sayeed1999 . Waiting for more from you. 👍


Instance of a new connected client and subscribed Topics

Hi!
I 'm implementing a Router with custom Auth then setting a RatchetTransportProvider and then I've added a new custom InternalClient to my Router.
My custom Client acts like a Pusher, I'm using ZMQ to handle the communication between this Wamp Server and the other parts of my application and Publish intelligently according to the ZMQ request.
I'ts working great!
So I 'm facing a problem that I cannot solve:
Every Client(Browser with AutobahnJS) subscribe to a different topic and I need to access this information when the Client subscribe... Something like onSubscribe.
I'm planning register inside my Pusher(custom InternalClient) a list of subscribed clients, pushing into list when the Client subscribe and pull off when the Client unsubscribe? I'ts possible? How I handle this?
Where is the best way to deal with single connected Client?(and get their subscribed topics, attributes, etc.)

You can create a module that keeps track of the subscribed events.  Something like this: https://github.com/voryx/Thruway/issues/149#issuecomment-153184958
If you want to access the subscription information from another client, have the module extend Thruway\Module\RouterModuleClient and then you can register RPCs in onSessionStart.  You'll need to register it as a router module instead of a realm module:
$router->registerModule(new SubscribeModule('realm1'));
You can take a look at https://github.com/voryx/Thruway/blob/master/src/Thruway/Authentication/AuthenticationManager.php to see how that type of module can be implemented.

@davidwdan Nice! I understand the concept of a RouterModule, probably this will be the final implementation!
So, now I'm facing another issue, you're showing to me an implementation of dev-master version of Thruway, right?
I was using 0.3.10, but when I change to dev-master my Pusher implementation crashes, I cannot get the messages from ZMQ..
Resuming my custom internal client:

use Thruway\Peer\Client;
use React\ZMQ\Context;
use ZMQ;

class Pusher extends Client {
    protected $sessions = [];
    public function onSessionStart( $session,  $transport)
    {
        $context = new Context($this->getLoop());
        $pull    = $context->getSocket(ZMQ::SOCKET_PULL);
        $pull->bind('tcp://<IP_ADDRESS>:5555');
        $pull->on('message', [$this, 'onPullMessage']);
    }

    public function onPullMessage($data)
    {
        $entryData = json_decode($data, true);
        if (!isset($entryData['channel'])) {
            return;
        }
        if (isset($entryData['session_id'])) {
            return;
        }
        $this->getSession()->publish($entryData['channel'], [$entryData]);
    }
}

Could be something with the Loop?

@douglasgalico how are you adding the Pusher client to the router?

I've added by the router, using addInternalClient:
 $router->addInternalClient(new Pusher('services', $router->getLoop())
Now I'm trying like this comment too: https://github.com/voryx/Thruway/issues/96#issuecomment-98416776
Occurs in both ways...
What makes it happen is when I change the version to dev-master, from 0.3.x(10) to dev-master.

Hi @douglasgalico ,
I have setup a project with the Pusher class you have and a simple router:
<?php

require __DIR__ . "/vendor/autoload.php";
require __DIR__ . "/Pusher.php";

use Thruway\Peer\Router;
use Thruway\Transport\RatchetTransportProvider;

$router = new Router();

$transportProvider = new RatchetTransportProvider("<IP_ADDRESS>", 9090);

$router->addTransportProvider($transportProvider);

$router->addInternalClient(new Pusher('services', $router->getLoop()));

$router->start();

This setup works for me in both 0.3.10 and dev-master.
If the only place you reference the loop is in the Context constructor, that should not be an issue as it will be the same as the loop from the router.
Is it possible that it is something else that is causing the issue? Either something more to the pusher or some other part of the router that is killing it?
If you remove $router->addInternalClient(new Pusher('services', $router->getLoop()) from your router, do the other parts still work fine?

Hi @mbonneau could be something right after ZQM emits the 'message' event.
Well, if I remove $router->addInternalClient(new Pusher('services', $router->getLoop()) the other parts work without problem...  Actually the problem seems to be when we try to Publish after the event.
I'm using this resumed code:
        $loop   = \React\EventLoop\Factory::create();
        $pusher = new \Thruway\Peer\Client("services", $loop);

        $pusher->on('open', function ($session) use ($loop) {
            Logger::info(null,'Opened!');
            $context = new \React\ZMQ\Context($loop);
            $pull    = $context->getSocket(\ZMQ::SOCKET_PULL);
            $pull->bind('tcp://<IP_ADDRESS>:5555');

            $pull->on('message', function ($entry) use ($session) {

                Logger::info(null,'MessageSent!');

                $entryData = json_decode($entry, true);
                $session->publish('main', [$entryData]);

            });
        });

        $router = new \Thruway\Peer\Router($loop);
        $router->addInternalClient($pusher);
        $router->addTransportProvider(new \Thruway\Transport\RatchetTransportProvider("<IP_ADDRESS>", 7474));
        $router->start();

With the 0.3.x everithing works fine and when I push new messages to ZMQ I got my output like this:
2015-12-07T11:24:23.8067360 info       MessageSent!
2015-12-07T11:24:24.0490690 debug      [Thruway\Role\Broker 2537] Broker onMessage for {"type":"internalClient","transportAddress":"internal"}: [16,3843112381797210,{},"main",[{"channel":"main","type":"main","data":["sdsgf"]}]]

But with dev-master everything runs 'fine' until the Publish step, output after push to zmq:
2015-12-07T11:40:49.8067360 info       MessageSent!

ZMQ event bing is ok, but I can't publish to $session.
Really strange, this occur when I change the version.

Are you getting some sort of error or are you going by the log?  The debug info for "Broker on Message" was removed in 0.4.

@davidwdan "Broken on Message"  is a custom message that i made.
So.. The version problem was my mistake, now I'ts all fixed and working.
I'm finishing here and I'll post here the resume of the implementation for anyone who needs for the same purpose(RouteModule, Oauth2 Auth, Subscribed topics, ...).
@davidwdan and @mbonneau  thanks for the help and the tips, this library is awsome!
Let's consider closed and then when I finish here I comment with the code.

@douglasgalico I am closing this now but please still post your implementation for others if you get a chance.


Reconnect Issue Showing up in Streaming

https://github.com/nats-io/java-nats-streaming/issues/107

likely related to #191


Forced back into game if not tabbed into.

Description
When you are tabbed out and get an ahelp you will be forced back into the game. This can be annoying if you are trying to write an issue and get forced back into the game mid-
Reproduction
Get an Ahelp while game is in fullscreen while tabbed into youtube or something

I do want to add, this is only an issue for fullscreen


Find user from mongo DB using phone number

How are you?
I hope you are doing great.
I need a function to find a user from the database using a phone number.
But in the database, the user's phone number has many formats
+1 (1234) 123-1234
1234567890
(1234) 124 1234
+1234567890
+1<PHONE_NUMBER>
really many formats...
.....
In this case, How can I find a user with this format "1234567890" phone number?
Thank you

in this situation, regex is your friend.

Could you please make a sample?

If your format is like "1324654897" that. You can simply use this regex :  .*\S\d

In the database, a user's phone number is "+1<PHONE_NUMBER>"
How can i find the user with "13245647890"?


Tally top 80 results exactly and fix "monday" timestamp calculation

Fixes

Staci and I noticed two issues while we were deploying the API today:

If the page_size changes (for example to 200), then if we tally the first 4 pages, we'd be tallying the top 800 results, which does not really reflect the intention of the tallying (for seeing the provider density of "top results"
On weeks that include a month boundary, the "monday" date stamp calculation was incorrect. We only found this because we were working on this literally as 00:00 UTC 1 Feb 2023 rolled over from January!

Description

Applies fixes for the above described issues. There are (hopefully) sufficient comments to hopefully describe the approach and intentions as the logic is a bit convoluted.
Testing Instructions

Check out the much expanded unit tests. You can test locally following the process Staci describes here: https://github.com/WordPress/openverse-api/pull/1088#pullrequestreview-1266702601. However, because local results for any given query do not really have 80 results, you'll need to test without a query or with a * query.
Checklist

[x] My pull request has a descriptive title (not a vague title like
Update index.md).
[x] My pull request targets the default branch of the repository (main) or
a parent feature branch.
[x] My commit messages follow best practices.
[x] My code follows the established code style of the repository.
[x] I added or updated tests for the changes I made (if applicable).
[x] I added or updated documentation (if applicable).
[x] I tried running the project locally and verified that there are no visible
errors.

Developer Certificate of Origin

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.

Thanks for the reviews. The failing test was due to the fact that I'd accidentally removed the falsey check on results, so None was getting passed to the tallies function. It is fixed now (thank goodness for tests :sweat_smile:)


Beta draft

This includes the results for the full run of the data (seeing how the trained FSMs perform is fantastic). A couple of points:

The training of the FSMs and the handshake is a big narrative. I have written a % TODO where we will insert the images from #28
I think we should include some pseudo code for the data collection (% TODO where that goes). #29
I've removed "we" and "our" throughout to go with no pronouns (I had a ridiculous review once where this became a big issue so since then I've preferred to avoid them, I hope that's fine).

Over to you now @marcharper :) I suggest we merge this as is, as a starting point, then all the modifications you make (and please feel free) can either be discussed or PR'd to master.

LGTM. I'll make a pass over the text soon.

LGTM. I'll make a pass over the text soon.

:+1:
I've just added the pseudo code and fsm diagrams @Nikoleta-v3 drew (which look great to me!), I also fixed my confusion with TF1 and TF3 (I mentioned that to you a little while ago). I think they're all the correct way around now but this is worth checking throughout the text.


Why does travis require graphicsmagick?

If node-resemble-js is used for image comparision why does travis.yml  require graphicsmagick?
Just curious.

Graphicsmagick is a dependency of wdio-screenshot, which is used for capturing screenshots.
But there is a PR for replacing graphicsmagick with jimp (image processing library written in JS).

I see, thanks!


Homebrew tap

We should add a new command: devicefarm update. It will download the latest binary and replace itself.
EDIT: See comment below

Actually, for the short term, we should set up a homebrew tap. It's really simple so it'll be a much faster way to provide easy install/update for end-users.
Changing title of this issue. Not going to open another issue for devicefarm update unless we start to feel there's a need.

Done


MAT-4877 use component for error, warning, success messages for Test …

…Case List page
MADiE PR
Jira Ticket: MAT-4877
(Optional) Related Tickets:
Summary
All Submissions

[x] This PR has the JIRA linked.
[ ] Required tests are included
[x] No extemporaneous files are included (i.e Complied files or testing results)
[x] This PR is into the correct branch.
[ ] All Documentation as needed for this PR is Complete (or noted in a TODO or other Ticket)
[ ] Any breaking changes or failing automation are noted by placing a comment on this PR.

DevSecOps
If there is a question if this PR has a security or infrastructure impact, please contact the Security or DevOps engineer assigned to this project to discuss it further.

[x] This PR has NO significant security impact (i.e Changing auth methods, Adding a new user type, Adding a required but vulnerable package)
[ ] All CDN/Web dependencies are hosted internally (i.e MADiE-Root Repo)

Reviewers
By Approving this PR you are attesting to the following:

Code is maintainable and reusable, reuses existing code and infrastructure where appropriate, and accomplishes the task’s purpose
The tests appropriately test the new code, including edge cases
If you have any concerns they are brought up either to the developer assigned, security engineer, or leads

https://www.figma.com/file/1RPYLI39hfMczAL81ARArX/MADiE-Application?node-id=7153%3A68992&t=1Wxli9Cpa2KIn941-4

I think this is how it's expected to look. There's a MadieAlert component that can be used for this from design system

@mcmcphillips I'm using the same component to replace the different Alerts, so it's consistent with other apps. For Test Case List page, there is no listed errors like what Measure updates would have with CQL errors.

The alert component does not only take lists.
It takes a prop called content that can be html strings.
The alert component also does not appear to be in the same place. You can make a single alert component the highest level of the component heirarchy, and pass props to the other components that allow them to update the state. You can blank the alert state whenever you'd want


Error in retrofit generated file -- constructor Response.fromJson not defined

I'm using retrofit in my flutter application and I'm using json_serialzable as well to generate model classes (I followed the steps in the Readme section). When I run flutter pub run build_runner build --> it successfully generates the part file for both retrofit service (by retrofit generator) and model that contains fromJson and toJson (I'm using json_annotation lib with json_serializable to generate fromJson and toJson as stated in the readme as well in the Task example). The problem is that the class generted by retrofit has an error that Response doesn't have the method fromJson. Here is a snippet from the generated file of a user service

and here are the dependencies in my pubspec.yaml:
dependencies:
  flutter:
    sdk: flutter

  cupertino_icons: ^1.0.2
  get: ^3.24.0
  flutter_custom_clippers: ^1.1.2
  font_awesome_flutter: ^8.11.0
  firebase_core: ^0.7.0
  firebase_auth: ^0.20.1
  cloud_firestore: ^0.16.0+1
  google_sign_in: ^4.5.9
  chips_choice: ^2.0.1
  logging: ^0.11.4
  path_provider: ^1.6.27
  hive: ^1.4.4+1
  hive_flutter: ^0.3.1
  retrofit: ^1.3.4+1
  dio: ^3.0.10
  json_annotation: ^3.1.0

dev_dependencies:
  flutter_test:
    sdk: flutter
  hive_generator: ^0.8.2
  retrofit_generator: ^1.3.7+5
  build_runner: ^1.10.0
  json_serializable: ^3.5.0

Flutter 2.0.1 • channel stable • https://github.com/flutter/flutter.git
Framework • revision c5a4b4029c (12 days ago) • 2021-03-04 09:47:48 -0800
Engine • revision 40441def69
Tools • Dart 2.12.0
Note: I've also tried to use 'any' instead of a certain version (even though no problem with pub get) but still the same result. Your help is much appreciated and thank you for the awesome library and great efforts in advance.

Hi @mohammad-melhem, after a few hours of research, I found retrofit_genrator to add a new method isGenericArgumentFactories to check class is generic. So, just add:
@JsonSerializable(genericArgumentFactories: true)
to resolve this error.

Hi @rinlv , firstly, thank you very much for your effort and sorry if it took a while till I respond. I tried the suggested solution but it didn't fix the issue and I got the following in the logs:
The class User is annotated with JsonSerializable field genericArgumentFactories: true. genericArgumentFactories: true only affects classes with type parameters. For classes without type parameters, the option is ignored.
and as it is stated my user class don't have any type parameters, here is the user model I'm using:
import 'package:json_annotation/json_annotation.dart';

part 'user.g.dart';

@JsonSerializable(includeIfNull: false, genericArgumentFactories: true)
class User {
  int id;
  int userType;
  String firstName;
  String lastName;
  String email;
  String profilePicture;
  bool isOpenForRemote;

  User(
      {this.id,
      this.userType,
      this.firstName,
      this.lastName,
      this.email,
      this.profilePicture,
      this.isOpenForRemote});

  factory User.fromJson(Map<String, dynamic> json) => _$UserFromJson(json);
  Map<String, dynamic> toJson() => _$UserToJson(this);
}

Hi @mohammad-melhem,
In this case,
@JsonSerializable(genericArgumentFactories: true)
only work when you added on the Response class, not the User class.

Hi @rinlv,
yes you are right as Response is the generic class. Thank you very much for your help.

Hi, How did you manage to get response object ?


fixed some clippy warnings

Sorry I left this open for so long. Closing, but will happily accept new formatting PRs!

Actually, this is still valid so merging :smile:. Thanks again @llogiq and sorry for the extreme delay.


perf(linter): use CompactStr in prefer-lowercase-title

[!WARNING]
This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

#6405  👈
#6403
#6402
#6401
#6400
#6399
#6398
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @camchenry and the rest of your teammates on  Graphite


Add NgModule To Provide CookieService

This PR creates a CookieModule in which the CookieService
is provided. Users are encouraged to add CookieModule to the
imports section of an NgModule as opposed to adding CookieService
to the providers section. However, adding CookieService as a
provider will still work, so existing users of the library won't be
required to make any changes.

Hello,
first of all, thanks for your contribution. Maybe I am a bit out of the loop here, but why should we encourage all users to use a CookieModule instead of a CookieService? After all, the only thing it does is to add a nested module that adds our service as provider.
Cheers from Berlin

@CunningFatalist thank you for taking a look at my PR. I agree that all the PR does is add a nested module that adds the service as a provider. However, it has been my experience in using several angular third party libraries that the most common pattern is to give users access to an NgModule they can import when they want to use the library, rather than ask them to add a service as a provider in their app. Imagine that you wanted to add several additional @Injectable() classes to ngx-cookie-service. With the current architecture users would need to add each additional class as a provider within their application. However, if all classes were provided in a CookieModule then users would not be required to make any changes to their providers array.

Hey @swseverance ,
I can see where you are coming from, but I do not see much value in adding the component wrapper at the moment. However, we discussed this, and if we are ever going to extend the service in a way that would require several providers, we will remember your PR and go with your changes (which look really good, by the way!).
Again, thank you very much for your contribution. I will close the PR for now, but I will definitely keep it in the back of my mind.
Cheers


Gloves distortion

Sup, gloves after agr importing so glitching

Please provide more information:

Is the glove actually used by any player in the AGR?
Which version of Blender Source Tools are you using?
Which version of afx-blender-scripts are you using?
Which version of Crowbar are you using?
When did you last update the models from Crowbar?
What are you AGR / other recording settings?
Consider providing a demo / the AGR file.

We can not solve this issue without further information requested above.


Add support for MacOSX

smart.go has a partial support for MacOSX. https://github.com/anatol/smart.go/blob/main/nvme_darwin.cpp provides an example how nvme SMART information can be obtained at macosx. Other drives (ATA and SCSI) should be done similar way.


[Bug]: Prevent creating managed resources from Crossplane without an existing providerconfig

Is there an existing issue for this?

[X] I have searched the existing issues

Affected Resource(s)
dns.gcp.upbound.io/v1beta1 - RecordSet
Resource MRs required to reproduce the bug
---
apiVersion: gcp.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: toosl
spec:
  credentials:
    impersonateServiceAccount:
      name: ""
    source: InjectedIdentity
  projectID: toosl
---
apiVersion: dns.gcp.upbound.io/v1beta1
kind: RecordSet
metadata:
  name: podinfo-private
spec:
  deletionPolicy: Delete
  forProvider:
    managedZone: some-zone
    name: podinfo-some-zone.
    project: tools
    rrdatas:
      - gw.some-other-zone.
    ttl: 300
    type: CNAME
  managementPolicies:
    - '*'
  providerConfigRef:
    name: tools

Steps to Reproduce
I shared previously the manifests yaml before with the providerconfig and the recourse recorset with a typo error in the provideconfig referenced

Create a provider config using gcp upbound family called "toosl"
Create some recordset using the providerconfig bad referenced "tools"

What happened?
By introducing a typo in the name of a GCP providerconfig, we created resources that referenced a non-existent providerconfig. This generated a conversion error in the Kubernetes API, preventing the creation of the resources.
During a recent operation, a typo was introduced into the name of a GCP provider configuration. Instead of naming it "tools", we mistakenly named it "toosl".
We believe it would be beneficial to implement a validation that prevents the creation of resources that reference non-existent providerconfigs. Additionally, we suggest improving error messages to make them more descriptive and facilitate the identification of the problem.
Subsequently, when creating "recordset" objects, this incorrect configuration was referenced. As a result, ArgoCD and the Kubernetes API returned the following error: Error from server: conversion webhook for dns.gcp.upbound.io/v1beta1, Kind=RecordSet failed: Post "https://provider-gcp-dns.crossplane-system.svc:9443/convert?timeout=30s": net/http: request canceled while waiting for the connection (Client.Timeout exceeded while awaiting headers)"
To temporarily resolve this issue, we had to delete the "dns.gcp.upbound.io" CRD. However, we would like to know if deleting the "providerconfigusage" objects where the "resource-kind" was "RecordSet" would have been sufficient. Would this action have prevented ArgoCD or the Kubernetes API from returning the aforementioned error?"
In this particular case, we had to delete the corresponding CRD to resolve the error. We would like to know if deleting the objects of type providerconfigusage where the resource-kind is RecordSet would have been sufficient to prevent the error.
Relevant Error Output Snippet
Argocd output
 to load initial state of resource recordset.dns.gcp.upbound.io: conversion webhook for dns.gcp.upbound.io/v1beta1, kind=recordset failed: post "https://provider-gcp-dns.crossplane-system.svc:9443/convert?timeout=30s

Crossplane Version
v1.18.0
Provider Version
v1.10.0
Kubernetes Version
v1.30.5-gke.1443001
Kubernetes Distribution
GKE
Additional Info
By introducing a typo in the name of a GCP providerconfig, we created resources that referenced a non-existent providerconfig. This generated a conversion error in the Kubernetes API, preventing the creation of the resources.
We believe it would be beneficial to implement a validation that prevents the creation of resources that reference non-existent providerconfigs. Additionally, we suggest improving error messages to make them more descriptive and facilitate the identification of the problem.
In this particular case, we had to delete the corresponding CRD to resolve the error. We would like to know if deleting the objects of type providerconfigusage where the resource-kind is RecordSet would have been sufficient to prevent the error.

Hi @marccortinas,
Thank you for the detailed issue report. I tried to reproduce the issue in a kind cluster but I couldn't. When I try to create a resource with the wrong ProviderConfig I get the expected error below:
  conditions:
  - lastTransitionTime: "2024-12-09T17:28:59Z"
    message: 'connect failed: cannot initialize the Terraform plugin SDK async external
      client: cannot get terraform setup: cannot get referenced ProviderConfig: ProviderConfig.gcp.upbound.io
      "tools" not found'
    reason: ReconcileError
    status: "False"
    type: Synced

The problem here seems to be that the conversion webhook mechanism is not working properly. Does this error occur consistently in your environment?

We're using :crossplane: in multiple gke cluster and this issue only appears in one of them. Let me add I uninstalled/installed crossplane three times (basically via helm and after that "kubectl deletes") in this cluster and it seems there are not working fine. Let me explain....
Now, all is working fine as expected, crossplane, providerconfig,
it's strange that the recordset is created but the get command fails with a timeout.
I checked and I can see fine:
➜ kubectl get providers.pkg.crossplane.io
NAME                          INSTALLED   HEALTHY   PACKAGE                                                      AGE
provider-gcp-dns              True        True      xpkg.upbound.io/upbound/provider-gcp-dns:v1.10.0             3h5m
provider-gcp-secretmanager    True        True      xpkg.upbound.io/upbound/provider-gcp-secretmanager:v1.10.0   3h5m
upbound-provider-family-gcp   True        True      xpkg.upbound.io/upbound/provider-family-gcp:v1.11.1          3h5m

 ➜ kubectl get providerrevisions.pkg.crossplane.io upbound-provider-family-gcp-f4e89e32a1cf
NAME                                       HEALTHY   REVISION   IMAGE                                                 STATE    DEP-FOUND   DEP-INSTALLED   AGE
upbound-provider-family-gcp-f4e89e32a1cf   True      1          xpkg.upbound.io/upbound/provider-family-gcp:v1.11.1   Active                               3h1m

 ➜ kubectl get providerrevisions.pkg.crossplane.io provider-gcp-dns-d621c8ebcfeb
NAME                            HEALTHY   REVISION   IMAGE                                              STATE    DEP-FOUND   DEP-INSTALLED   AGE
provider-gcp-dns-d621c8ebcfeb   True      1          xpkg.upbound.io/upbound/provider-gcp-dns:v1.10.0   Active   1           1               3h2m

➜ kubectl get providerconfigs.gcp.upbound.io -o yaml|kubectl neat
apiVersion: v1
items:
- apiVersion: gcp.upbound.io/v1beta1
  kind: ProviderConfig
  metadata:
    annotations:
      argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
    labels:
      argocd.argoproj.io/instance: gcp-providerconfig-edge-some-gcp-project-name-gke1
    name: gcp-some-gcp-project-name
  spec:
    credentials:
      impersonateServiceAccount:
        name: ""
      source: InjectedIdentity
    projectID: some-gcp-project-name
- apiVersion: gcp.upbound.io/v1beta1
  kind: ProviderConfig
  metadata:
    annotations:
      argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
    labels:
      argocd.argoproj.io/instance: gcp-providerconfig-edge-some-gcp-project-name-gke1
    name: gcp-some-gcp-project-name-resources
  spec:
    credentials:
      impersonateServiceAccount:
        name: ""
      source: InjectedIdentity
    projectID: edo-prod-resources
kind: List
metadata: {}

The issue appears always when I try to get the recordset.
➜ kubectl get RecordSet abollado
Error from server: conversion webhook for dns.gcp.upbound.io/v1beta1, Kind=RecordSet failed: Post "https://provider-gcp-dns.crossplane-system.svc:9443/convert?timeout=30s": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

➜ kubectl getRecordSet
Error from server: conversion webhook for dns.gcp.upbound.io/v1beta1, Kind=RecordSet failed: Post "https://provider-gcp-dns.crossplane-system.svc:9443/convert?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

But we can get with file specifying....
❯ kubectl get -f brais-dns-prod.yaml
NAME       SYNCED   READY   EXTERNAL-NAME   AGE
abollado   True     True    projects/ofuscated/managedZones/ofuscated-edo-world/rrsets/abollado.ofuscated.ofuscated./CNAME   166m

I increase logs to debug level and I can see the following:
➜ kubectl logs provider-gcp-dns-d621c8ebcfeb-cc8997ffd-k5n9l
2024-12-19T14:57:23Z	DEBUG	provider-gcp	Starting	{"sync-interval": "1h0m0s", "poll-interval": "10m0s", "poll-jitter": "30s", "max-reconcile-rate": 100}
2024-12-19T14:57:26Z	INFO	provider-gcp	Beta feature enabled	{"flag": "EnableBetaManagementPolicies"}
2024-12-19T14:57:26Z	INFO	provider-gcp	Alpha feature enabled	{"flag": "EnableAlphaExternalSecretStores"}
2024-12-19T14:57:26Z	INFO	provider-gcp	ESS TLS certificates path is set. Loading mTLS configuration.
2024-12-19T14:57:26Z	DEBUG	provider-gcp	Calling the inner handler for Create event.	{"gvk": "dns.gcp.upbound.io/v1beta1, Kind=RecordSet", "name": "abollado", "queueLength": 0}
2024-12-19T14:57:26Z	DEBUG	provider-gcp	Reconciling	{"controller": "managed/dns.gcp.upbound.io/v1beta1, kind=recordset", "request": {"name":"abollado"}}
2024-12-19T14:57:26Z	DEBUG	provider-gcp	Connecting to the service provider	{"uid": "ef6806d4-1781-4e7b-a7c1-a5b453798096", "name": "abollado", "gvk": "dns.gcp.upbound.io/v1beta1, Kind=RecordSet"}
2024-12-19T14:57:26Z	DEBUG	provider-gcp	Instance state not found in cache, reconstructing...	{"uid": "ef6806d4-1781-4e7b-a7c1-a5b453798096", "name": "abollado", "gvk": "dns.gcp.upbound.io/v1beta1, Kind=RecordSet"}
2024-12-19T14:57:26Z	DEBUG	provider-gcp	Observing the external resource	{"uid": "ef6806d4-1781-4e7b-a7c1-a5b453798096", "name": "abollado", "gvk": "dns.gcp.upbound.io/v1beta1, Kind=RecordSet"}
2024-12-19T14:57:27Z	DEBUG	provider-gcp	External resource is up to date	{"controller": "managed/dns.gcp.upbound.io/v1beta1, kind=recordset", "request": {"name":"abollado"}, "uid": "ef6806d4-1781-4e7b-a7c1-a5b453798096", "version": "1935994642", "external-name": "projects/ofuscated/managedZones/ofuscated-edo-world/rrsets/abollado.ofuscated.edo.world./CNAME", "requeue-after": "2024-12-19T15:06:58Z"}
2024-12-19T15:06:58Z	DEBUG	provider-gcp	Reconciling	{"controller": "managed/dns.gcp.upbound.io/v1beta1, kind=recordset", "request": {"name":"abollado"}}
2024-12-19T15:06:58Z	DEBUG	provider-gcp	Connecting to the service provider	{"uid": "ef6806d4-1781-4e7b-a7c1-a5b453798096", "name": "abollado", "gvk": "dns.gcp.upbound.io/v1beta1, Kind=RecordSet"}
2024-12-19T15:06:58Z	DEBUG	provider-gcp	Observing the external resource	{"uid": "ef6806d4-1781-4e7b-a7c1-a5b453798096", "name": "abollado", "gvk": "dns.gcp.upbound.io/v1beta1, Kind=RecordSet"}
2024-12-19T15:06:58Z	DEBUG	provider-gcp	External resource is up to date	{"controller": "managed/dns.gcp.upbound.io/v1beta1, kind=recordset", "request": {"name":"abollado"}, "uid": "ef6806d4-1781-4e7b-a7c1-a5b453798096", "version": "1935994642", "external-name": "projects/ofuscated/managedZones/ofuscated-edo-world/rrsets/abollado.ofuscated.edo.world./CNAME", "requeue-after": "2024-12-19T15:17:26Z"}
2024-12-19T15:17:26Z	DEBUG	provider-gcp	Reconciling	{"controller": "managed/dns.gcp.upbound.io/v1beta1, kind=recordset", "request": {"name":"abollado"}}
2024-12-19T15:17:26Z	DEBUG	provider-gcp	Connecting to the service provider	{"uid": "ef6806d4-1781-4e7b-a7c1-a5b453798096", "name": "abollado", "gvk": "dns.gcp.upbound.io/v1beta1, Kind=RecordSet"}
2024-12-19T15:17:26Z	DEBUG	provider-gcp	Observing the external resource	{"uid": "ef6806d4-1781-4e7b-a7c1-a5b453798096", "name": "abollado", "gvk": "dns.gcp.upbound.io/v1beta1, Kind=RecordSet"}
2024-12-19T15:17:26Z	DEBUG	provider-gcp	External resource is up to date	{"controller": "managed/dns.gcp.upbound.io/v1beta1, kind=recordset", "request": {"name":"abollado"}, "uid": "ef6806d4-1781-4e7b-a7c1-a5b453798096", "version": "1935994642", "external-name": "projects/ofuscated/managedZones/ofuscated-edo-world/rrsets/abollado.ofuscated.edo.world./CNAME", "requeue-after": "2024-12-19T15:27:33Z"}```

Hi @turkenf
I'm @marccortinas team mate and I'm working in the same issue.
We've been further investigating and we can extend the information about this issue
We've tried upgrading crossplane operator and provider to the latest version, but we obtain the same behaviour.
Then we tried downgrading the provider and we found that the last working version is 1.1.0
Checking changelog for version 1.2.0 https://github.com/crossplane-contrib/provider-upjet-gcp/compare/v1.1.0...v1.2.0 we found that a new CRD version is introduced: v1beta2
First thought was issue comes with this version and we are right:
Applying this manifest is accepted:
kind: RecordSet
metadata:
  name: abollado
spec:
  deletionPolicy: Delete
  forProvider:
    managedZone: <managed-zone>
    name: abollado.<domain>.
    project: <project>
    rrdatas:
      - www.google.es.
    ttl: 300
    type: CNAME
  managementPolicies:
    - '*'
  providerConfigRef:
    name: <project>```
but applying this manifest is not accepted:
```apiVersion: dns.gcp.upbound.io/v1beta2
kind: RecordSet
metadata:
  name: abollado
spec:
  deletionPolicy: Delete
  forProvider:
    managedZone: <managed-zone>
    name: abollado.<domain>.
    project: <project>
    rrdatas:
      - www.google.es.
    ttl: 300
    type: CNAME
  managementPolicies:
    - '*'
  providerConfigRef:
    name: <project>```

  We get the following error:

```Error from server: conversion webhook for dns.gcp.upbound.io/v1beta1, Kind=RecordSet failed: Post "https://provider-gcp-dns.crossplane-system.svc:9443/convert?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)```

We have tried deleting providers and crossplane itself from this cluster, but every time we reinstall it, we find the same issue. We haven't been able to reproduce the issue in other clusters, so this doesn't occur consistently.

As we want to keep this cluster and avoid migrating all its applications to a new one, could you help us in some way? Do you have any clue? Which things do you recommend us to check? provider logs aren't useful as @marccortinas  mentioned before... our guess is that something remains in the cluster although we uninstall everything carefully. Do you thing we are missing something?

Thanks in advance!


EOF error when chaining endpoints

Hi,
I defined two RESTful endpoints under the same Remote Service and tried to chain them together, but I ended up with EOF error although each endpoints worked fine and can even be nested as long as I don't use parameters to chain them.
My endpoints configuration is
Remote Service Name: ctapi
Endpoint 1 Name: events, Path: events
Endpoint 2 Name: attendance, Path: attendance?eventId={args.id}

Run the events endpoint alone
query {
  events @ctapi {
    id
  }
}

, and I got
{
  "data": {
    "events": [
      {
        "id": 98
      },
      {
        "id": 103
      }
    ]
  }
}

Run the attendance endpoint alone
query {
  attendance(id:103) @ctapi {
    source
  }
}

, and I got
{
  "data": {
    "attendance": [
      {
        "source": "R"
      },
      {
        "source": null
      },
      {
        "source": "M"
      }
    ]
  }
}

Running both endpoints nested but not chained
query {
  events @ctapi {
    attendance(id:103) @ctapi {
      source
    }
  }
}

, and I got
{
  "data": {
    "events": [
      {
        "attendance": [
          {
            "source": "R"
          },
          {
            "source": null
          },
          {
            "source": "M"
          }
        ]
      },
      {
        "attendance": [
          {
            "source": "R"
          },
          {
            "source": null
          },
          {
            "source": "M"
          }
        ]
      }
    ]
  }
}

But when I run chained
query {
  events @ctapi {
    attendance(id:"events.id") @ctapi {
      source
    }
  }
}

, I ended up with
{
  "errors": [
    {
      "message": "EOF"
    }
  ]
}

What's wrong please?

Hey @waynesi Can you confirm that the attendance endpoint doesn't throw an error for the other event.id (98)?
Seems like there might be a problem with the attendance endpoint.
To reproduce what you said, I made the following service:
Remote Service Name : myservice Remote Service URL : https://httpbin.org
Endpoint 1 Name : post path: /post
Endpoint 2 Name: post2 path: /post?arg1={args.k1}

Note: https://httpbin.org is an online service that just sends the details of HTTP request (like args, body, headers, etc) back in the response

Then I made the following chained request (k1 is taken from post.json.k1):
{
  post(foo: "bar", k1: "somekey") @myservice {
    post2(k1: "post.json.k1") @myservice {
      args
      json
    }
  }
}

And I got the following response:
{
  "data": {
    "post": {
      "post2": {
        "args": {
          "arg1": "somekey"
        },
        "json": {
          "k1": "somekey"
        }
      }
    }
  }
}

Also can you see if there's anything helpful in the logs of space-cloud-gateway?

Hi @HeyItsJs,
Thanks a lot for your very timely reply. Much appreciated. You're right, the attendance endpoint doesn't always have data for all the possible event ids. When I tried with such event id on the nested but but not chained query, I ended up with the same EOF error.
However, our attendance endpoint does return proper 204 No Content status code which according to HTTP protocol is a succeeded response rather than an error. Is there any chance the space-cloud gateway can be improve to handle such situation more gracefully?
What we're trying to do is to build a GraphQL layer in front of our Web API layer, in order to avoid very verbose and chatty Web API calls from the client perspective. So basically we want the GraphQL layer to "join" the results of several Web API calls. But if the whole GraphQL query will simply fail because one nested Web API calls returns nothing, I'm afraid the practicality of your great feature is significantly limited. In my case, if the nested Web API call fails or returns nothing, I would like to see a null value instead. Maybe an advanced option can be provided for either fail-fast (the current behaviour) or use null.
Once again, thanks a lot for your help. Your fast turnaround and the level of help are very impressive. I'm sure your business will flourish.

Hey @waynesi, Thanks for the kind words! Words like this keep us motivated!
You have pointed a very valid point for the status code 204. If the status code is 204, then we can send null instead of trying to pare the response from the remote endpoint.
This is an easy fix. you can expect it to be fixed it in a patch version soon. Closing this issue. Have created a separate specific issue for the same - #1077 to track this problem.
Thanks for bringing such use cases! It would be great if you can join the community discord channel of Space Cloud as well 😄 - https://discord.com/invite/RkGjW93

Re: Discord. Thanks for the invite. Now downloading. Man, you have no idea how many comm apps I've installed since I joined Github 😂


snapshot create from task in CLI and SDK interface

CLI interface
datmo snapshot --task-id <task_id>

SDK interface
import datmo
task_dict = {"command":"python script.py"}
task_id = datmo.task.run(task_dict)
datmo.snapshot.create(task_id=task_id)

this interface makes sense.
Currently in the SDK there is already a create_from_task function that is used separately (https://github.com/datmo/datmo/blob/master/datmo/snapshot.py#L172-L206)
If we would like to merge these functions into 1 then, we will need to make sure there is validation and checks for mutually exclusive arguments

@asampat3090 https://github.com/datmo/datmo/blob/master/datmo/core/controller/snapshot.py#L226-L232
this has after snapshot id being given back rather than before snapshot id (for scripts)
return self.dal.snapshot.update({
            "id": task_obj.after_snapshot_id,
            "message": message,
            "stats": task_obj.results,
            "visible": True
        })

From the last comment, actually, we do not allow non-completed tasks to be snapshots for the time being which means we would never have after_snapshot_id be None. if we want to change that we can reopen a new issue regarding that.
For now as per PR #97, we are pulling from the after_snapshot_id, updating the values, and then making it visible to the end user.

merged PR #97


[dart:js_interop] Add JSArray.from static method.

Array.from (MDN).

It depends. :)
Creating an array using Array.from will be faster than iterating the original object and placing in each value in a Dart list. toDart should also be quite fast since it's either a cast or a wrap operation.
On the other hand, accesses on the toDart'd array may be slower than accesses on the Dart list you created. When compiling to JS, they will likely be the same performance, but when compiling to Wasm, you'll need to use interop for every access, so it'll be slower.


8muses  " Non-retriable status code 400 while downloading"

Ripme version: ripme-1.7.96-1-c7af5c50
Java version: 1.8.0_281
Operating system: Windows 7 & 10

Exact URL you were trying to rip when the problem occurred: https://comics.8muses.com/comics/album/Various-Authors/Chessire88
Please include any additional information about how to reproduce the problem:

Expected Behavior
Download images from 8muses
Actual Behavior
Folder are created but no files are downloaded with error  " Non-retriable status code 400 while downloading"

ripme.log
Including Log from ripme-1.7.96-3-431d382c compiled 13 April 2021 from this repository.

if one does "view page source" and "inspect" in firefox, the page content differs. the image is inserted into the

lets close this, until somebody opens it for a current version or provides a pull request.


error domain=kAFAssistantErrorDomain Code=1101"(null)"

Hi Peter,
I am using your plugin in cordova app,it give me error on simulator(error domain=kAFAssitantErrorDomainCode=1101"(null)")
and says missing permission in isListening function.
Also in device it give me error(error domain=kAFAssitantErrorDomainCode=216"(null)").
Kindly help.
Thanks in advance

You have to run on a device.
The iOS simulator does not have access to your Mac’s microphone.

when i run on device it give me this error(error domain=kAFAssitantErrorDomainCode=216"(null)")

Did you request permission?
What is your environment? cordova, xcode, ios version?

yes i have already given these permission (NSMicrophoneUsageDescription permission &
NSSpeechRecognitionUsageDescription permission) with its string value in info.plist.
My cordova version:6.3.1
xcode:Version 8.0 (8A218a)
IOS:OS X EI Capitan..10.11.6 (15G1004)

I've already implemented requestPermission() method as shown in below code-
window.plugins.speechRecognition.requestPermission(function(requestpermissionResult){
console.log("Stop message: " + requestpermissionResult);
alert("requestPermission::"+requestpermissionResult);
}, function(requestpermissionErrorMessage){
console.log("Error message: " + requestpermissionErrorMessage);
alert("Error message :requestPermissionError::"+requestpermissionErrorMessage);
});
Is this enough or should I have to implement in other way ??

You should call it before using startListening().
You should see a popup window of permission request.

We are calling it before startListening(). We are getting Popup asking for permission also. After clicking on ok, it  goes to startListening method and we get the error domain=kAFAssitantErrorDomainCode=216"(null)").

Where do you see this error?
Does it come with the error callback of startListening() ?

You have to run it on a device.
The iOS simulator does not have access to your Mac’s microphone
On 24 Oct 2016 15:54, "Rabindramohapatra2016"<EMAIL_ADDRESS>wrote:

Hi Peter,
I am using your plugin in cordova app,it give me error on simulator(error
domain=kAFAssitantErrorDomainCode=1101"(null)")
and says missing permission in isListening function.
Also in device it give me error(error domain=kAFAssitantErrorDomainCode=
216"(null)").
Kindly help.
Thanks in advance
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pbakondy/cordova-plugin-speechrecognition/issues/4,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAej2aPevYci-3ZRqYk0NJjb16l3qDEiks5q3LiXgaJpZM4Ke0Th
.

Yes i am running on actual device with ios version 10.0.2(14A456)and that error comes in error callback alert of startListening() method.

I could not found any solution yet.
I don't know what this error code means.
The Apple docs does not list this error.
This forum topic is about the same issue without resolve:
https://forums.developer.apple.com/thread/51938
Please try reinstall cordova and Xcode.

We tried reinstalling as well but did not work.

Please try with latest XCode 8.1 and iOS 10.1

Have updated to ios version:10.1 (14B72) and xcode version:Version 8.1 (8B62) still same issue persist.

I have same problem, someone have a solution?
Functions
Avaliable - works
HasPermition - works
RequestPermition - works
But startListening - not work
I am with:
Cordova CLI: 6.5.0
Ionic Framework Version: 2.0.1
Ionic CLI Version: 2.1.18
Ionic App Lib Version: 2.1.9
Ionic App Scripts Version: 1.0.0
ios-deploy version: 1.9.1
ios-sim version: 5.0.13
OS: macOS Sierra
Node Version: v6.9.2
Xcode version: Xcode 8.2.1 Build version 8C1002
I need help.

you can send me your project that works on your computer to iOS?
I can test here...

Please check last release

@pbakondy does the latest release solve the problem? I have the same problem on the device itself. everything is using the latest version (device, itunes, xcode)

Yes it does

I have the versión cordova-plugin-speechrecognition 1.1.2 "Speech Recognition"
and I have this problem
cordova 7.0.0
Iphone 6s IOS 10.3.1
Ipad pro IOS 10.0.2
Xcode Version 8.3.2 (8E2002)

I start seeing the problem when I call te startListening method very often like I push the button 3 times in a short period of time ...

I solved the problem i describe above, the problem is caused because I call the method startListening before the last called returns, so, I used a flag to prevent to call startListening if the last call didn't returned yet. However there still a problem, when I call multiple times the startListening the plugin returns
"Error Domain=kAFAssistantErrorDomain Code=203 \"Corrupt\" UserInfo={NSLocalizedDescription=Corrupt, NSUnderlyingError=0x1704506e0 {Error Domain=SiriSpeechErrorDomain Code=102 \"(null)\"}}"

This error doesn't cause a plugin crash like the previous one, after this error occur the plugin is able to listen again

code=216 -> in my case it was because I initialized the SFSpeechRecognizer without locale. So, the code that worked is:
 private SFSpeechRecognizer SpeechRecognizer = new SFSpeechRecognizer(new NSLocale("en_US"));


feature - User's comment textfield

Github Issue URL/Tags (required)
What's this PR do? Please give a technical summary (required)

[x] add comment textfield

Any references (stackoverflow …) (optional)

:+1: There are no linter violations.


use identifier and num option in same time

Currently, we cannot use num and identifier options in same time. But it will be enable if set process_name as delayed_job.#{@options[:identifier]}#{worker_index}.
The pool option is ok, but we cannot detect which worker is running for which queue.
$ bundle exec bin/delayed_job --pool=mailers:2 --pool=default:2 start
delayed_job.0: process with pid 12992 started.
delayed_job.1: process with pid 12994 started.
delayed_job.2: process with pid 12996 started.
delayed_job.3: process with pid 12998 started.

$ ps aux | grep [d]elay
kenjiszk        12998  22.3  2.1  2679988 179992   ??  S    11:19PM   0:02.52 delayed_job.3
kenjiszk        12996   9.9  2.1  2676916 176792   ??  S    11:19PM   0:02.69 delayed_job.2
kenjiszk        12994   6.0  2.1  2670772 173492   ??  S    11:19PM   0:02.75 delayed_job.1
kenjiszk        12992   1.9  2.1  2671796 174548   ??  S    11:19PM   0:02.70 delayed_job.0

This PR samples are followings.
$ bundle exec bin/delayed_job -n 2 -i mailers --queue=mailers start
delayed_job.mailers0: process with pid 12918 started.
delayed_job.mailers1: process with pid 12920 started.

$ bundle exec bin/delayed_job -n 2 -i default --queue=default start
delayed_job.default0: process with pid 12943 started.
delayed_job.default1: process with pid 12945 started.

$ ps aux | grep [d]elay
kenjiszk        12920  11.4  1.9  2666676 162916   ??  U    11:17PM   0:04.56 delayed_job.mailers1
kenjiszk        12918  10.2  2.0  2665652 170820   ??  R    11:17PM   0:04.81 delayed_job.mailers0
kenjiszk        12945   0.0  2.1  2675892 178828   ??  S    11:17PM   0:02.35 delayed_job.default1
kenjiszk        12943   0.0  2.2  2683060 184472   ??  S    11:17PM   0:02.47 delayed_job.default0

Coverage decreased (-0.3%) to 90.867% when pulling c0c55491c872e1c1b435dced290b405e0456839e on kenjiszk:use-number-and-identifier-options-in-same-time into e3772d4f0c8470d0fcba00c86ca3bc4f5e876830 on collectiveidea:master.


DEV transform

Is there any example on how to create a visitor that would achieve the same thing that is done in https://github.com/facebook/react/blob/master/vendor/constants.js ?
I'm trying to port that transform code to a transform, but didn't find much documentation yet (for ex: use of this.parent.node does not seem to work inside a visitor function)
This seems quite a handy transform (I'm trying to integrate with reactify, using a custom visitor for example). Any hints?

Thank you for reporting this issue and appreciate your patience. We've  notified the core team for an update on this issue. We're looking for a response within the next 30 days or the issue may be closed.


[RFE] Investigate creating a library to leverage Apache Kafka single message transforms

The Kafka connect API contains among other things an interface for doing transformations on messages as they flow from sources to sinks.  It'd be interesting to see if we could leverage these interfaces to allow users to define these transformations in AtlasMap.

Glueing data shapes for this usecase is something we have to consider well it seems like. There could be 2 options:
Serialized JSON/XML/CSV as a single key/value
In this case kafka schema is not involved and instead the schema has to be fed into AtlasMap UI individually. For example, using FileStreamSource connector, a record is sent for each line in the input file, having the text data as a value with null key. Although this case leverages AtlasMap's cross format data mapping feature, it doesn't look like a common usage for kafka.
kafka structured data as a key and a value
The kafka connect can set up a converter to serialize/deserialize from/to kafka internal data structure. We'll need to create a kafka AtlasMap module and let it handle the mapping between kafka data structure and AtlasMap one. It would be simpler/smaller than others like JSON/XML since kafka's is very similar to what AtlasMap do with AtlasMap Field. e.g. The kafka Struct is an equivalent to FieldType.COMPLEX in AtlasMap.
How the schema is passed into AtlasMap is a question, especially in AtlasMap UI. It is recommended to use Confluent schema-registry to manage kafka schema. Putting integration with the schema-registry aside, for the 1st attempt, we would assume the user would manually load the schema into AtlasMap UI just like we expect in standalone usage today. But even so, we'd have to support Avro schema and protobuf schema in AtlasMap.
Also, instead of deliverying kafka record as a single AtlasMap Document, key and value are better to be handled as a separate Document as they have dedicated kafka runtime schema maintained for each, attached to the record. We'd need to investigate how we can attach those kafka runtime schema as a generic runtime Document metadata into AtlasMap context.
Or kafka schema might not really required for AtlasMap runtime? in theory all the schemas are already fed into AtlasMap UI at design time, so at runtime there shouldn't be a need for updating schema but the design time schemas could be used as-is for both of source and target.
In this usecase, AtlasMap has to internally use KafkaField that handles kafka data structure like Struct. AtlasMap never knows the real data format, e.g. JSON, CSV, XML - seralize/desrialize is done by kafka connect. That means plain Avro support in AtlasMap if there is, would be a different implementation than kafka Avro support.
Fact
We would assume that, whichever the schemas are kafka aware or only for AtlasMap, they are loaded into AtlasMap UI. And it has to distinguish if it's kafka aware or serialized into kafka key or value.


coqPackages.mathcomp: 1.7.0 -> 1.8.0

coqPackages.mathcomp-finmap: 1.1.0 -> 1.2.0
coqPackages.mathcomp-analysis: 0.1.0 -> 0.2.0
Motivation for this change
Major update.
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Assured whether relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

@GrahamcOfBorg build coqPackages.mathcomp-analysis


Proposal for the minimal design change for the sample app

At the moment we need to make two changes to the current sample app design

Add support for showing the complete KendoUI charts family
Add support for showing the complete set of KendoUI controls.

Note: the change 2. naturally includes change 1. above - but since we want to create an Alpha release of the KendoUI plugin as soon as it is feasible, I am proposing such staged approach, which starts with item 1. above

Add support for showing the complete KendoUI charts family
Concept 1

Note: one could say that we could simply replace the menubar "Area Chart" item with the DropDownList, I believe that this approach is more balanced than having the DropDownList "hang" on the menubar.
Issue 1: I would not like to invest more time in the current (Bootstrap based) menubar, which would be required to do it I want to show the DropDownList at the current "Area Chart" location.In that case it would be better to switch to Kendo menubar, without "showing it as a demo".
Issue 2. Even if we "resurrect" the existing KendoUI menu code, I do not know how to properly integrate it with Aurelia router so that the main app would work unchanged.
Concept 2

isn't this the same issue as this one? The question is: how will we allow users to navigate through all the samples, right?

Correct - and again I would like to merge them having in mind that my most current proposal would be to have the most popular controls

"listed" on the main manubar - so the user can access them directly, and then have the "control chooser" main menubar item that when clicked shows this popup

superseded by https://github.com/aurelia-ui-toolkits/aurelia-kendoui-plugin/issues/206


pyinstaller interaction with torchvision

Setting: torchvision 0.5.0
When compile with torchvision 0.5.0 it has an error comming with
File "site-packages \ torchvision \ ops \ misc.py", line 135, in

File "site-packages \ torchvision \ ops \ misc.py", line 148, in FrozenBatchNorm2d

File "site-packages \ torch \ jit \ __ init__.py ", line 1204, in script_method

File" site-packages \ torch \ jit \ frontend.py ", line 156, in get_jit_def
File" inspect.py ", line 955, in getsourcelines

File" inspect .py ", line 786, in findsource
OSError: could not get source code
but most of the library requires version newer than 0.2.2 which is not basic solution @ # #

this is the exact error that pops up

I have the same error in ubuntu 18.04. :'(

@anhoangphuc uploading your code status will also help solving this problem

Got the same error in ubuntu 16.04, do you have any update, thoughts or solution on this ?

@Sstrap Some suggestion on online offers to downgrade torchvision to 0.2.2 however my current version is 0.5.0 and need some utils in order to run my python script so at this moment I couldn't find any solution for this.

Had the same problem on Windows 10.
I didn't manage to solve it, but I can confirm that it works when downgrading torchvision to 0.2.2.

@SangbumChoi since the 0.5.0 release of torchvision, we don't use @torch.jit.script in FrozenBatchNorm anymore, but other parts of the code still use it.
I don't have any experience with pyinstaller, so I would expect that it might be better to open an issue in pyinstaller reporting the issue. There might be a flag that you can pass to pyinstaller to make it work with torchvision, but I'm not aware of it.

FYI, I opened an issue on PyInstaller (https://github.com/pyinstaller/pyinstaller/issues/4740#issue-578771079).
The only workaround I found to make things work was to downgrade to torchvision 0.2.2 and manually add the code I needed. This is clearly a dirty solution, hopefully I will find something better in the future.

Here are the workarounds I found to make things work.
Solution 1: patch torchvision 0.2.2 (dirty and painful)
Pyinstaller works well with torchvision 0.2.2, thus if you do not use too many functionnalities of more recent version of torchvision, you can "patch" old torchvision 0.2.2 by adding the needed new tools. Note that this may need you first use compiled file of torchvision >= 3.0 (_C.so).
Solution 2: Keep recent torchvision & patch torch.jit in your entry point
I found some monkey patch on stackoverflow and dev blogs that simply overwrite 2 methods of torch.jit.
Simply add this path at the beginning of your entry point:
def script_method(fn, _rcb=None):
    return fn
def script(obj, optimize=True, _frames_up=0, _rcb=None):
    return obj
import torch.jit
torch.jit.script_method = script_method
torch.jit.script = script

I still think there should be a more direct way to do that with pyinstaller, but this works for me.
I hope it will help others too.

@Sstrap I will try second one and tell if it is work

Having this same problem.  Not 100% sure where to put that patch, but not having any success yet.

Here are the workarounds I found to make things work.
Solution 1: patch torchvision 0.2.2 (dirty and painful)
Pyinstaller works well with torchvision 0.2.2, thus if you do not use too many functionnalities of more recent version of torchvision, you can "patch" old torchvision 0.2.2 by adding the needed new tools. Note that this may need you first use compiled file of torchvision >= 3.0 (_C.so).
Solution 2: Keep recent torchvision & patch torch.jit in your entry point
I found some monkey patch on stackoverflow and dev blogs that simply overwrite 2 methods of torch.jit.
Simply add this path at the beginning of your entry point:
def script_method(fn, _rcb=None):
    return fn
def script(obj, optimize=True, _frames_up=0, _rcb=None):
    return obj
import torch.jit
torch.jit.script_method = script_method
torch.jit.script = script

I still think there should be a more direct way to do that with pyinstaller, but this works for me.
I hope it will help others too.

Solution 2 worked for me. Thanks.
A more detailed solution could be as:

Adding above code to the beginning of your main program
Adding the files/packages you need in a.datas in .spec file, as well as hidden_import (just for sure :)), for example:
datas=[(path.join(site_packages,"av"), "av"),
(path.join(site_packages,"fvcore"), "fvcore"),
(path.join(site_packages,"torch"), "torch"),
(path.join(site_packages,"torchvision"), "torchvision"),
('D:/detectron2_repo', "detectron2")],
hiddenimports=['av', 'fvcore', 'torch', 'torchvision', 'detectron2'],

Hi,
We have now hopefully fix this issue by replacing instances of @torch.jit.script with @torch.jit._script_if_tracing, and this fix will be present in the next release of torchvision, which will be out in a few weeks.
As such, I'm closing this issue as I believe this should be fixed with the 0.7.0 release, but please let us know if this isn't fixed when the 0.7.0 release comes out

Here are the workarounds I found to make things work.
Solution 1: patch torchvision 0.2.2 (dirty and painful)
Pyinstaller works well with torchvision 0.2.2, thus if you do not use too many functionnalities of more recent version of torchvision, you can "patch" old torchvision 0.2.2 by adding the needed new tools. Note that this may need you first use compiled file of torchvision >= 3.0 (_C.so).
Solution 2: Keep recent torchvision & patch torch.jit in your entry point
I found some monkey patch on stackoverflow and dev blogs that simply overwrite 2 methods of torch.jit.
Simply add this path at the beginning of your entry point:
def script_method(fn, _rcb=None):
    return fn
def script(obj, optimize=True, _frames_up=0, _rcb=None):
    return obj
import torch.jit
torch.jit.script_method = script_method
torch.jit.script = script

I still think there should be a more direct way to do that with pyinstaller, but this works for me.
I hope it will help others too.

Solution 2 worked for me. Thank you very much <3

next

Here are the workarounds I found to make things work.
Solution 1: patch torchvision 0.2.2 (dirty and painful)
Pyinstaller works well with torchvision 0.2.2, thus if you do not use too many functionnalities of more recent version of torchvision, you can "patch" old torchvision 0.2.2 by adding the needed new tools. Note that this may need you first use compiled file of torchvision >= 3.0 (_C.so).
Solution 2: Keep recent torchvision & patch torch.jit in your entry point
I found some monkey patch on stackoverflow and dev blogs that simply overwrite 2 methods of torch.jit. Simply add this path at the beginning of your entry point:
def script_method(fn, _rcb=None):
    return fn
def script(obj, optimize=True, _frames_up=0, _rcb=None):
    return obj
import torch.jit
torch.jit.script_method = script_method
torch.jit.script = script

I still think there should be a more direct way to do that with pyinstaller, but this works for me. I hope it will help others too.

then next,I encounter “ File "torch_ops.py", line 61, in getattr
op = torch._C._jit_get_operation(qualified_op_name)
RuntimeError: No such operator torchvision::_cuda_version”
can have any idear？

This would've worked perfeclty if not for the fact that it completely eliminates the ability to use CUDA and forces CPU.

Here are the workarounds I found to make things work.
Solution 1: patch torchvision 0.2.2 (dirty and painful)
Pyinstaller works well with torchvision 0.2.2, thus if you do not use too many functionnalities of more recent version of torchvision, you can "patch" old torchvision 0.2.2 by adding the needed new tools. Note that this may need you first use compiled file of torchvision >= 3.0 (_C.so).
Solution 2: Keep recent torchvision & patch torch.jit in your entry point
I found some monkey patch on stackoverflow and dev blogs that simply overwrite 2 methods of torch.jit. Simply add this path at the beginning of your entry point:
def script_method(fn, _rcb=None):
    return fn
def script(obj, optimize=True, _frames_up=0, _rcb=None):
    return obj
import torch.jit
torch.jit.script_method = script_method
torch.jit.script = script

I still think there should be a more direct way to do that with pyinstaller, but this works for me. I hope it will help others too.

can you share the full spec file i am not sure where to add it

@Sstrap  How can i patch like in second example couldn't understand it. Can you please eleborate ? Like to which file torch.jit._script ? of which line

Here are the workarounds I found to make things work.
Solution 1: patch torchvision 0.2.2 (dirty and painful)
Pyinstaller works well with torchvision 0.2.2, thus if you do not use too many functionnalities of more recent version of torchvision, you can "patch" old torchvision 0.2.2 by adding the needed new tools. Note that this may need you first use compiled file of torchvision >= 3.0 (_C.so).
Solution 2: Keep recent torchvision & patch torch.jit in your entry point
I found some monkey patch on stackoverflow and dev blogs that simply overwrite 2 methods of torch.jit. Simply add this path at the beginning of your entry point:
def script_method(fn, _rcb=None):
    return fn
def script(obj, optimize=True, _frames_up=0, _rcb=None):
    return obj
import torch.jit
torch.jit.script_method = script_method
torch.jit.script = script

I still think there should be a more direct way to do that with pyinstaller, but this works for me. I hope it will help others too.

Could you elaborate on what you mean by "entry point"?


Dealing with long text

Hi, may I ask a question about dealing with tong texts?
I try replicating the "Load one of the pre-trained models" in README.md to correct the long OCR text. My code is below:
text = 'here is the long text ... ' text = correct_punctuation_error(text) new_source = [list(text)] X_new = source_index.text2tensor(new_source) predictions, log_probabilities = seq2seq.beam_search(     model,      X_new,     progress_bar = 0) just_beam = target_index.tensor2text(predictions[:, 0, :])[0] just_beam = re.sub(r"<START>|<PAD>|<UNK>|<END>.*", "", just_beam)
However, here shows an error:

How to fix it?
Thank you so much!

The error you are getting is because you are trying to process a sequence that is longer that what the model can process. Remember that Transformers require positional embeddings to train, so they always have a limit of the sequence length they can accept.
The Transformer models trained here can only process sequences of 100 characters (if I remember correctly), so you will not be able to process sequences longer than that with just the methods from Pytorch Beam Search (Greedy Search, Sampling and Beam Search). However, what you can do is to use the methods from this library, (correction.disjoint and correction.n_grams in the example code) which were designed exactly to do that.

Thank you so much, it works!


Fix typo causing broken link

Summary
Fixing a typo.
Reason
Issue #2919.
Testing

Turns out it was not only a typo, but the source link was wrong entirely. Fixing that too.


Use is-promise

is-promise
Less redundancy. When we need to write our own middleware to catch errors not caught by this library, we can simply import the same dependency.

I'd prefer not to add a dependency to the project.

Why? Dependencies are good. They prevent unnecessary tests.


Provide better Guidance on how to get the project working from clean install

The guidance on getting the solution working from a clean install is causing confusion and needs to be tidied up.
Fixes: #373
This issue is for a: (mark with an x)
- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce
Follow the instructions from the main readme.md

I second this - I need to be able to use already provisioned resources given the limited regional availability of OpenAI. I'm working on modifying the main.bicepparam file but having an variables file that could be used in place of the environment variables configured during deploy would make this much easier to work with.

@Lawndemon - can I just understand what your requirements are here so I can see if we can address them?

Do you want to deploy some resources in different regions than others? Can you give an example?
You want to reuse existing resources? You can do something like azd env set MY_ENV_KEY value and that will write it to the default .env file. We could then read it in the bicepparams. Is that what you mean?