[FEA] dynamic sizing of input data

This is something that might be interesting to push back to Spark too. When we are trying to figure out the splits for reading, can we look at the read schema and the file schema to get an idea of how much data is going to be thrown out when we do a read so we can better size the splits. I can see a few options for this.
One option is where we look at a small amount of data (1 or 2 files at most) to try and determine the file schema.  The other option is to launch an actual job that all it does is read the file schema to get better knowledge about exactly what is happening.  Option 1 is nice because it can be fast and probably can give us a decent estimate on what things will look like. Option 2 fits more with things like delta lake that will read and cache metadata information before running a much larger query.  We might even be able to switch between the two based off of the number of files, or the number of directories involved in a single query.

we look at a small amount of data (1 or 2 files at most) to try and determine the file schema.

The file schema is already known, fetched by Spark as part of planning and validating the query, and available via the dataSchema field of the HadoopFsRelation.   Do you mean something more sophisticated like parsing the footer of the files and looking at relative column data sizes or something more simplistic like a blind guess as to the relative data sizes of the columns based on their known Spark data types?

I would start off with the simplest possible approach and see how far that gets us.  I would only make it more complicated if we run into real world situations where it is too far off for us to get the benefit we want.
The simplest I can think of is to keep targeting maxPartitionBytes, but we look at the read schema vs the file schema and SWAG how much of each batch we are going to be able to skip and read the block size accordingly. Alternatively we could try and target the GPU target batch size instead. But then we have to try and understand the compression ratio of the columns that we are going to read in. That feels harder to do.

I have been working on this for a while and there is no simple way to do what we want with only the information we have ahead of time. I tried a few heuristics based off of information that we currently have access to through Spark when creating the splits. I mainly looked at the number and types of columns to try and increase the maximum batch size config per-input. That way the batches would get larger and the sub-linear scaling of the GPU works better. This worked well for a few extreme cases, NDS queries 9, 7, and 88 but not as well in the general case.
There are a number of problems with the initial approach that I took.

Predicate push down, compression ratios, and row groups. I was able to get the size of the batches to increase in the general case, but it was far from consistent. This means that the maximum batch that was processed grew about 4 fold, to the point that it was larger than the "maximum" batch size. It also spread the sizes out a lot more, which made getting predictable computation much more difficult.
gpu decoding is volatile. For smaller batch sizes (up to about 200 MiB) there is a clear trend that more data is better.  After that point I don't think I have enough data to really come to any real conclusion. There is a lot of volatility from one bucket to another. There is huge volatility within buckets.  The size has a clear impact on computation time, but predicting what that computation time will be is not as clear.
Increased host memory pressure.  Before we send the data to the GPU we buffer it on the CPU.  If we increase the size of the batches, we now have more data held on the CPU at any point in time and that increases memory pressure, especially in relation to pinned memory.
Buffer time.  The time to decode parquet is not all about the GPU decoding time. We also have to read that data in and cache it in host memory. In general we want to overlap buffering data and computation on the GPU. But we still have to pay the initial cost of downloading the first batch of data before we can put anything on the GPU. If we increase the average size of a batch we are also increasing the amount of time it takes to download the data before we can put it on the GPU.
Side impacts.  I also saw a number of other impacts caused by changing the number of tasks and size of the batches. We saw increased contention for the GPU, which slowed down other processing too. We saw in many cases fewer tasks later in the processing too. I am not totally sure why AQE was changing things like this, but generally if there were fewer upstream tasks it resulted in fewer downstream tasks, even if the total amount of data stayed the same.  We also saw some impact to shuffle and compression on the GPU.

Despite all of this I have hope that this can be useful and we should look into this more. Even this imperfect code saved over 3,400 seconds of GPU compute time from decoding parquet. That is about 27.8% of the total compute time used to parse the parquet data on the GPU, and about 1.7% of the total run time of all of the queries, assuming that the computation could be spread evenly among all of the tasks.  So there is potential here to make a decent improvement. But it has not worked out perfectly.
We are looking at ways to improve the decoding performance of the parquet kernels themselves. In addition to this we might want to look at more of a control system approach instead. We lack information up front and it looks to be expensive to try and get that information early on. It might be better to try and dynamically adjust the sizes of the batches we buffer/send to the GPU based off of throughput rates that we are able to achieve while buffering and guesses about how much data the GPU can processes efficiently. But we first need to do a fair amount of benchmarking to understand what really impacts the performance of the buffering and the performance of decode. We also would need to look how AQE will impact downstream processing.


Ac admin scenarios

Add new admin scenarios for CCS Category and CCS Sourcing.
Refactored some steps and did some tidying up

I found it really hard to follow what each commit was addressing. They each seemed to include a few different things. I think in future it would be helpful to split commits so they each address one concept and provide more detailed comments.
Commits from the data api may be good examples.
Also the GDS git styleguide provides some good guidelines.
That said, the changes seem good. :+1:


fix(option): fix small issues

Suggessions by @coderabbitai:

Name Value() method to something else
use helpers in tests to validate Some/None values

Summary by CodeRabbit

New Features

Renamed method for retrieving values in the option package from Value() to GetValue() for improved clarity.
Introduced helper functions assertSome and assertNone to enhance test readability and maintainability.

Bug Fixes

No functional changes, but improved method naming clarifies expected behavior.

Tests

Streamlined test suite for the option package to improve verification of Some and None values.

@coderabbitai review

@coderabbitai resolve


Top Safe Area Inset of the presented VC does not match the post-transition value.

Clarification:

When it is fully expanded, there is an additional margin added, which doesn't exist during the interactive transition


feat: Add MergeRight derive macro.

Summary:
Briefly describe the changes made in this PR.
Issue Reference(s):
Fixes #... (Replace "..." with the issue number)
Close #1700
/claim #1700
Build & Testing:

[ ] I ran cargo test successfully.
[ ] I have run ./lint.sh --mode=fix to fix all linting issues raised by ./lint.sh --mode=check.

Checklist:

[ ] I have added relevant unit & integration tests.
[ ] I have updated the documentation accordingly.
[ ] I have performed a self-review of my code.
[ ] PR follows the naming convention of <type>(<optional scope>): <title>

We could keep a few if they are very custom.

@Shylock-Hg Thanks! Just tiny feedback here —

[ ] Move the macro logic into tailcall-macros/src/merge_right.rs so that we can make space of other macros in this workspace, and keep lib.rs light and simple exports as much as possible.
[ ] Resolve conflicts

Overall it looks like is ready to merge.

/tip 50$


[BUG] : Sign in page isn't responding

Description
sign in page isn't responding while Login the credentials.
Screenshots

Any additional information?
No response
What browser are you seeing the problem on?
No response
Checklist

[X] I have checked the existing issues
[X] I have read the Contributing Guidelines
[X] I'm a GSSoC'24-Extd contributor
[X] I'm a Hacktoberfest'24 contributor
[X] I am willing to work on this issue (optional)

Hello, I tried to fix the issue
This is what I did:
I have updated the auth.html file to fix the login functionality. The main changes include adding an event listener to the login form and implementing a basic login validation function. Here's a summary of the changes:

Added an event listener to the sign-in form:

const signInForm = document.querySelector('.sign-in-form');
signInForm.addEventListener('submit', handleSignIn);

Implemented a handleSignIn function to validate the login:

function handleSignIn(e) {
    e.preventDefault();
    const username = document.querySelector('.sign-in-form input[type="text"]').value;
    const password = document.querySelector('.sign-in-form input[type="password"]').value;

    if (username && password) {
        // Here you would typically send a request to your server to validate the credentials
        console.log('Login attempt:', username, password);
        alert('Login successful!'); // Replace this with actual login logic
    } else {
        alert('Please enter both username and password');
    }
}

These changes should make the login form functional and responsive to user input.
I also created pull request: https://github.com/vishanurag/Canvas-Editor/pull/1147

[!CAUTION]
Disclaimer: This fix was created by Latta AI and you should never merge before you check the correctness of generated code!
The fix provided by Latta AI might not be complete and it can serve as an inspiration.

This bug was fixed for free by Latta AI - https://latta.ai/ourmission
If you no longer want Latta AI to attempt fixing issues on your repository, you can block this account.


What's the status on the iOS port?

With the risk of being really annoying I would just like to ask: What's the status for the iOS port of this project?

For those who are following this, experimental support of iOS is now working.  I am still in the process of writing documentation, but if you want to play with it, you can either open the LiquidCoreiOS directory in XCode (there is a test app which runs a node console), or you can build the framework on the command line with Carthage (carthage build --no-skip-current).

iOS support is now relatively complete in 0.5.0-experimental-6.  I will publish it on the front page with the full release 0.5.0, but it is good to go now.


The "Hmm... it's not here" page is displayed when attempting to view a profile after splitting the bill.

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Action Performed:

Open the app and click on the plus sign, then choose "split bill".
Enter the desired amount and click on  "Next".
Choose who you want to split with
Click on "split the bill" and click on the Profile. The detail section will appear on the right.
Click on the profile, but instead of displaying the profile details, the "Hmm... it's not here" page appears.

Expected Result:
Clicking on a profile after splitting the bill should display the profile details or images.
Actual Result:
Clicking on a profile after splitting the bill displays the "Hmm... it's not here" page instead of the profile details.
Workaround:
Can the user still use Expensify without this being fixed? Have you informed them of the workaround?
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android / native
[ ] Android / Chrome
[ ] iOS / native
[ ] iOS / Safari
[x] MacOS / Chrome / Safari
[ ] MacOS / Desktop

Version Number: 1.29-0
Reproducible in staging?: y
Reproducible in production?: n
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation
screen-capture (15).webm
https://github.com/Expensify/App/assets/43996225/0a01ce7e-0a84-4b5f-99f8-998618b50490
Expensify/Expensify Issue URL:
Issue reported by: @ayazhussain79
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1686899440198109
View all open jobs on GitHub

:wave: Friendly reminder that deploy blockers are time-sensitive ⏱ issues! Check out the open StagingDeployCash deploy checklist to see the list of PRs included in this release, then work quickly to do one of the following:

Identify the pull request that introduced this issue and revert it.
Find someone who can quickly fix the issue.
Fix the issue yourself.

Proposal
Please re-state the problem that we are trying to solve in this issue.
The "Hmm... it's not here" page is displayed when attempting to view a profile after splitting the bill
What is the root cause of that problem?
We are using login in MoneyRequestConfirmationList.
https://github.com/Expensify/App/blob/83ab0160a3b5c385fd26fffe20bdbccabf9249c0/src/components/MoneyRequestConfirmationList.js#L236-L241
What changes do you think we should make in order to solve the problem?
We shall update the login to accountID as per new implementations.
What alternative solutions did you explore? (Optional)

Proposal
Please re-state the problem that we are trying to solve in this issue.
The "Hmm... it's not here" page is displayed when attempting to view a profile after splitting the bill.
What is the root cause of that problem?
Empty string gets passed for ReportUtils.getAccountIDForLogin(option.login) here as that function call won't work bcoz we are passing loginId here and inside function getAccountIDForLogin it is fetching from personalDetailsList onyx key which has the mapping with accountID's.
https://github.com/Expensify/App/blob/bee8fd3e1634fa11761435ef405a3d3efd654b07/src/components/MoneyRequestConfirmationList.js#L236-L241
What changes do you think we should make in order to solve the problem?
We have the accountID here for the option object which we can use directly.
https://github.com/Expensify/App/blob/bee8fd3e1634fa11761435ef405a3d3efd654b07/src/components/MoneyRequestConfirmationList.js#LL240C2-L240C2
Navigation.navigate(ROUTES.getProfileRoute(option.accountID));
Also this condition can be updated with accountID.
https://github.com/Expensify/App/blob/bee8fd3e1634fa11761435ef405a3d3efd654b07/src/components/MoneyRequestConfirmationList.js#L237
!option.accountID

@0xmiroslav sorry it looks like you linked the PR twice, did you mean to link an issue?

Aahhh this is the "Also, there was another new issue happening on main" issue, yeah? Great, i'll make this not a deploy blocker 😬
@luacmartins whenever you see this, feel free to close if it's not reproducible (since it shouldn't be 🙏 )

I can't reproduce this, but I also see no emails/display names in the list

@0xmiroslav same as the OP.

FYI...

I also see no emails/display names in the list

This was fixed earlier in PR: https://github.com/Expensify/App/pull/21113

This specific issue has been fixed. We had other issues for the missing display names, etc. Closing.


#0: Add device perf CI to PR template

Ticket
Problem description
Developers were missing device perf CI
What's changed
Hint at it explicitly on PR template
Checklist

[ ] Post commit CI passes
[ ] Blackhole Post commit (if applicable)
[ ] Model regression CI testing passes (if applicable)
[ ] Device performance regression CI testing passes (if applicable)
[ ] New/Existing tests provide coverage for changes

https://github.com/tenstorrent/tt-metal/pull/12518#event-14219222530
let this PR link stand as proof of who merged first

TOXIC PERMABAN @ttmchiou
look at this guyyyyy @TT-billteng aiyaaaaaa


get rid of deprecated charcters api usage and migrate to swift 4

hi there, .charcters method of String is deprecated in swift 4

@dynzhdjl The validator is up to date with Swift 4.2 now


feat: integrate vue-docgen-api in preview

Using this new integration we can now leverage docgen on component pages.
I was thinking of making an extra component on the right side (collapsible) that would display all this info and allow to interract with the previews.
It has been my experience that people love to interact and modify preview examples.

extra component on the right side
That's a nice idea, go ahead.

(collapsible)
We should make left-side collapsible too.

Here is a screenshot of the status today:

Added the two collapse buttons (no animation, no design)
Started rendering the documentation on the right panel

LGTM.

I will finish rendering the docs this morning. Then I will merge.

I have not gotten .__docgen__ urls to refresh using HMR. I will try to do this in a next PR.


Integration and easy replication into utsa.club

Would like to make instances easy to spin up for other organizations, including a subdomain on utsa.club. Needs planning.

First time I've really heard of utsa.club; is there a Slack discussion I'm missing lol? Any serious planning on what it was supposed to be?


[FEATURE REQUEST] Access logs

The problem is I can't find the access logs when running RR as a frontend web server for PHP Laravel app.
When running the app as Nginx+PHP-FPM you got the access logs from nginx plus slow logs from FPM. Both of them can be routed to stderr inside a container and caught for analysis.
I see the two options to get the logs using RR:

run RR in debug mode. You got a lot more logging than the access logs. It can affect the performance.
create the logs with Laravel logger. It should affect the performance plus increase the TTFB (time to first byte) lowering the server responsiveness. Plus it cannot log the static files.

Is it possible to output the access logs by RR in production env?

@cidious Feature will be available by the end of the week in the v2.5.0-beta.2 with the following fields:
method - http method.
remote_addr - request remote address.
bytes_sent - content-length,
http_host - host.
request - request Query.
time_local - local time in Common Log Format.
request_length - request body with headers size (content-len + size of all headers) in bytes. Max allowed headers size for the RR is 1MB.
request_time - request processing time in seconds with a milliseconds resolution.
status - http response status.
http_user_agent - http user agent
http_referer - http referer


Feature youtube connector

The Youtube Connector relies on yt-dlp, a tool that enables the downloading of YouTube captions. To use the Youtube Connector, you must install yt-dlp and Python in your environment.
To install yt-dlp, run this command on terminal:
pip install yt-dlp

The Youtube Connector relies on yt-dlp, a tool that enables the downloading of YouTube captions. To use the Youtube Connector, you must install yt-dlp and Python in your environment.
To install yt-dlp, run this command on terminal: pip install yt-dlp

I'd love to see no dependency on other languages.

I will close the PR for now.


How to Use Proxy with DataFlowKit?

How to use proxy to prevent blocking IPs?
Plz give some example/guide tutorial with docker or standalone

"Chrome fetcher"
https://github.com/slotix/dataflowkit/blob/afc0d3fe495fa97406d0034a4f386e4f88985d2c/docker-compose.yml#L9
Uncomment the string starting from --proxy-server and enter here real IP address of proxy server.
"Base fetcher"
https://github.com/slotix/dataflowkit/blob/d423a7b11b2849df5f4443b4be6c232dc3cf4167/docker-compose.yml#L40
add - --PROXY=http://username:password@proxy-host:port
parameter after this string. This parameter is responsible for specifying proxy server for base fetcher requests.

Run "fetch.d" standalone server with --PROXY parameter as specified above to pass requests through a proxy server.

@slotix
Thanks for the answer !
how to randomize proxy ip list in the ip pool or check and failover depending on whether its alive or not

You can use something like "super" proxy or proxy manager service which will rotate proxies and validate them before forwarding requests. So the whole scheme looks like dfk fetcher -> proxy manager -> pool of proxies.
Chrome browser cannot rotate proxies in run time itself. The only possible way is to set up --proxy-server parameter at start. So you can point chrome to this proxy manager and it will do that work for you.


bump pipelines-as-code to v0.17.3

Fix for https://github.com/redhat-appstudio/infra-deployments/pull/1725

/test build-service-e2e

/retest

/test build-service-e2e


errors(full: true), for nested input, could be better

Describe the bug
result = contract.call(params)

# errors(full: false) works fine ..
result.errors.to_h.to_json
# "{\"items\":{\"0\":{\"weight\":{\"unit\":[\"must be one of: g, kg, lb, oz\"]}}}}"

# errors(full: true), for nested input, could be better
result.errors(full: true).to_h.values.join(', ')
# "{0=\u003e{:weight=\u003e{:unit=\u003e[\"unit must be one of: g, kg, lb, oz\"]}}}"

I'm happy to attempt a PR, if requested.
To Reproduce
Call a Contract that validates nested input, as shown above. If the above is unclear, I'm happy to provide more detail.
Expected behavior
It'd be great to have better output. For example:
result.errors(full: true).to_h.values.join(', ')
# "items 0 weight unit must be one of: g, kg, lb, oz"

My environment

Affects my production application: No, because I guess I won't be using errors(full: true) in production .. :)
Ruby version: 2.7
OS: N/A

Could you please post sample code for reproducing it?


Add support for aerosol optical depth climatology metrics to the AutoAssess replacement

Description
Support will be added within the Met Office's AutoAssess replacement, the Climate Model Evaluation Workflow (CMEW) for aerosol optical depth (AOD) climatology metrics. This work adds a recipe and diagnostic to ESMValTool to compare AOD at 440 nm from modelled multi-annual seasonal means (DJF, MAM, JJA, SON) with a climatology derived from ground based AeroNET observations.
The ESMValTool diagnostic is based on the existing AutoAssess metric for comparing modelled and observed AOD at 440 nm, but here an updated observational climatology is used. Please note that the climatology will need to be used 'offline' because further work will be required to CMORize the station based observational data.

Closes #2775
Link to documentation:

Summary of testing:
UKESM model output for AOD at 440nm (AOD440) was evaluated against climatologies generated from AeroNET observations using the original AutoAssess methodology and the new recipe and diagnostic developed in this work. Model output from a UKESM1 historical run (available on the ESGF) and an UKESM1 AMIP run were tested. Statistics for both model outputs are shown in Table 1. Only output from the UKESM1 historical run is shown in Figures 1-3.
The original AutoAssess climatology made use of observations from 67 AeroNET sites. From inspection of the original climatologies, it appears that a minimum of three months were used to generate a multi-annual monthly mean from which the multi-annual seasonal means were calculated. Here we tested thresholds of three and seven for the minimum number of months used to generate a multi-annual monthly mean. This resulted in a total of 77 and 219 available sites respectively.
Table 1 shows that the seasonal root mean square error (RMSE) calculated for the AutoAssess and ESMValTool methodologies is similar in DJF, MAM and SON, but higher in JJA. In general, a threshold of three months per multi-annual monthly mean gives a higher RMSE than when a threshold of seven months is set, but these differences are small. The higher RMSE values calculated in JJA are due to the additional sites used for the new climatologies. Figures 1 – 3 show that many of the new sites are located in Europe and North America. The model overpredicts AOD in these regions during JJA and the greater coverage in these regions, particularly in the polluted Eastern USA, central Europe and southern Europe means that overall, model bias is larger.
Table 1: Summary of the seasonal root mean square error for the evaluation of modelled AOD at AeroNET stations using the AutoAssess and proto-Climate Assessment Workflow methodologies. ‘Min months’ is the minimum number of months per multiannual monthly mean.

UKESM1 historical
UKESM1 historical
UKESM1 historical
UKESM1.1 AMIP
UKESM1.1 AMIP
UKESM1.1 AMIP

Original climatology, AutoAssess
New climatology, ESMValTool
New climatology, ESMValTool
Original climatology, AutoAssess
New climatology, ESMValTool
New climatology, ESMValTool

Min months
3 (?)
7
3
3 (?)
7
3

Number of AeroNET stations
67
77
219
67
77
219

Obs. Time period
1993-2011
1993-2014
1993-2014
1993-2011
1993-2014
1993-2014

DJF
0.091
0.091
0.103
0.068
0.077
0.102

MAM
0.128
0.107
0.121
0.100
0.091
0.108

JJA
0.122
0.150
0.145
0.144
0.162
0.164

SON
0.108
0.102
0.113
0.100
0.101
0.110

Figure 1: AutoAssess evaluation of AOD at 440 nm between UKESM1 historical ensemble member r1i1p1f2 (u-bc179) and the original AeroNET climatology. Multi-annual seasonal means are shown for DJF (top left), MAM (top right), JJA (bottom left) and SON (bottom right). In each plot model output is overlaid with observational data.

Figure 2: ESMvalTool evaluation of AOD at 440 nm between UKESM1 historical ensemble member r1i1p1f2 (u-bc179) and the new AeroNET climatology generated using a threshold value of seven monthly means per multi-annual monthly mean. Multi-annual seasonal means are shown for DJF (top left), MAM (top right), JJA (bottom left) and SON (bottom right). In each plot model output is overlaid with observational data.

Figure 3: ESMvalTool evaluation of AOD at 440 nm between UKESM1 historical ensemble member r1i1p1f2 (u-bc179) and the new AeroNET climatology generated using a threshold value of three monthly means per multi-annual monthly mean. Multi-annual seasonal means are shown for DJF (top left), MAM (top right), JJA (bottom left) and SON (bottom right). In each plot model output is overlaid with observational data.

Before you get started

[ ] ☝ Create an issue to discuss what you are going to do

Checklist
It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.

[ ] 🛠 This pull request has a descriptive title
[ ] 🛠 Code is written according to the code quality guidelines
[ ] 🛠 Documentation is available
[ ] 🛠 Tests run successfully
[ ] 🛠 The list of authors is up to date
[ ] 🛠 Any changed dependencies have been added or removed correctly
[ ] 🛠 All checks below this pull request were successful

New or updated recipe/diagnostic

[ ] 🧪 Recipe runs successfully
[ ] 🧪 Recipe is well documented
[ ] 🧪 Figure(s) and data look as expected from literature
[ ] 🛠 Provenance information has been added

New or updated data reformatting script

[ ] 🛠 Documentation is available
[ ] 🛠 The dataset has been added to the CMOR check recipe
[ ] 🧪 Numbers and units of the data look physically meaningful

To help with the number of pull requests:

🙏 We kindly ask you to review two other open pull requests in this repository

awesome, thanks a lot @catherinehardacre - I am the "historical" AutoAssess contributor, so I think we can can let @zklaus off the review hook, if that's ok with you - unless you are looking at specific stuff from Klaus :+1: I'll have a look at the code in a few days, we are under a bit of v2.8 release snow atm

Hi V,
The reason I’d asked Klauz/Remi to review was for their interest in observational data in ESMValTool. I’ve had some communication with Remi about this Issue – he’d like the AeroNET obs to be CMORized and available as Tier 3 data before this PR. This is so the recipe and diagnostic can be tested properly on the ESMValTool systems. Currently the AeroNET obs are only on internal Met Office systems… I’m going to have a chat with Remi this week and open an issue linked to #2775 to document the obs CMORization. I hope the CMORIzation should not be too difficult as the NetCDF files I’ve generated are CF-compliant (i.e. they load via Iris…).
So there is not any urgency for #2775 (yet… I’m hoping for vn2.9) I’ll make sure you’re on the Issue and let you know how we get on with the CMORization. You and Klaus can then argue over who gets to review 😉
Thanks,
Catherine
From: Valeriu Predoi @.>
Sent: 27 February 2023 14:32
To: ESMValGroup/ESMValTool @.>
Cc: Hardacre, Catherine @.>; Mention @.>
Subject: Re: [ESMValGroup/ESMValTool] Add support for aerosol optical depth climatology metrics to the AutoAssess replacement (PR #3048)
This email was received from an external source.   Always check sender details, links & attachments.
awesome, thanks a lot @catherinehardacrehttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcatherinehardacre&data=05|01|catherine.hardacre%40metoffice.gov.uk|2bf4c766256246e1784c08db18cf6ca1|17f1816120d7474687fd50fe3e3b6619|0|0|638131051377060002|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=%2BJcM6Kcfenb0F5HKVmXXEPR8n5K6fqmj6JUlbt4xDXg%3D&reserved=0 - I am the "historical" AutoAssess contributor, so I think we can can let @zklaushttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fzklaus&data=05|01|catherine.hardacre%40metoffice.gov.uk|2bf4c766256246e1784c08db18cf6ca1|17f1816120d7474687fd50fe3e3b6619|0|0|638131051377060002|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=4m987r4lgpeQt%2FUHvio1Voj3WDhJQJrrMS4Iwnyi%2BYY%3D&reserved=0 off the review hook, if that's ok with you - unless you are looking at specific stuff from Klaus 👍 I'll have a look at the code in a few days, we are under a bit of v2.8 release snow atm
—
Reply to this email directly, view it on GitHubhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FESMValGroup%2FESMValTool%2Fpull%2F3048%23issuecomment-1446430197&data=05|01|catherine.hardacre%40metoffice.gov.uk|2bf4c766256246e1784c08db18cf6ca1|17f1816120d7474687fd50fe3e3b6619|0|0|638131051377060002|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=ueoMzi%2BF930JCxRjEw7wqHU%2Flf%2FfLpqqxl5YgsNNlpc%3D&reserved=0, or unsubscribehttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAUCKZARUFMOV66PWEVBJQCTWZS3GZANCNFSM6AAAAAAVDEAOKA&data=05|01|catherine.hardacre%40metoffice.gov.uk|2bf4c766256246e1784c08db18cf6ca1|17f1816120d7474687fd50fe3e3b6619|0|0|638131051377060002|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=y7WIkfXV%2FaT9K2kES1GMFNspJB2%2BH6WqiGRJkNXRg3k%3D&reserved=0.
You are receiving this because you were mentioned.Message ID: @.***>

sounds like a good plan, @catherinehardacre :beer: Pls ping me once technical review can commence here - I can do that, and will let Remi/Klaus handle the fancy sciencey stuff :grin:

Thanks V! Will keep you posted Not so much ‘fancy science’ in this one, but you have to start somewhere…
From: Valeriu Predoi @.>
Sent: 27 February 2023 14:50
To: ESMValGroup/ESMValTool @.>
Cc: Hardacre, Catherine @.>; Mention @.>
Subject: Re: [ESMValGroup/ESMValTool] Add support for aerosol optical depth climatology metrics to the AutoAssess replacement (PR #3048)
This email was received from an external source.   Always check sender details, links & attachments.
sounds like a good plan, @catherinehardacrehttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fcatherinehardacre&data=05|01|catherine.hardacre%40metoffice.gov.uk|3f4db6eeee194bb37eac08db18d1f08c|17f1816120d7474687fd50fe3e3b6619|0|0|638131062186603868|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=4UbjR4OqTDUP3eoEgJvy%2F4CGLIuzFuJsU8rDdbvWuJ0%3D&reserved=0 🍺 Pls ping me once technical review can commence here - I can do that, and will let Remi/Klaus handle the fancy sciencey stuff 😁
—
Reply to this email directly, view it on GitHubhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FESMValGroup%2FESMValTool%2Fpull%2F3048%23issuecomment-1446468261&data=05|01|catherine.hardacre%40metoffice.gov.uk|3f4db6eeee194bb37eac08db18d1f08c|17f1816120d7474687fd50fe3e3b6619|0|0|638131062186603868|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=uS3uXY2pntnQWRdI2MobSwoElhQ1wHqSRkEZfY90qsg%3D&reserved=0, or unsubscribehttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAUCKZAWVF53I4Q7JSML3MIDWZS5KNANCNFSM6AAAAAAVDEAOKA&data=05|01|catherine.hardacre%40metoffice.gov.uk|3f4db6eeee194bb37eac08db18d1f08c|17f1816120d7474687fd50fe3e3b6619|0|0|638131062186603868|Unknown|TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D|3000|||&sdata=A1aKqu%2FEt6Wx%2BUtuQl8AT4zy85fqLe0polX148G2vYc%3D&reserved=0.
You are receiving this because you were mentioned.Message ID: @.***>

The deadline for including new work in v2.9 has passed, so I'm removing this pull request from the milestone. Please add it to the next milestone if you plan to include this in the next release.

@valeriupredoi or @zklaus: Is anybody of you looking at this from the technical point of view?
@alistairsellar: do you want to do the scientific review, or should I do it?
It would be great if we could get this in the release of v2.10.0!

@valeriupredoi or @zklaus: Is anybody of you looking at this from the technical point of view? @alistairsellar: do you want to do the scientific review, or should I do it? It would be great if we could get this in the release of v2.10.0!

Hi @hb326, I had a quick chat with @catherinehardacre on this before in July. @zklaus has been her main collaborator, and I think this PR was waiting on small remaining effort from one of them.
I have already science-reviewed the methodology, and happy with it overall, Am just awaiting finalisation of code before approving.
(P.S. Hi @catherinehardacre, hope the move to NZ went smoothly. I forget when you start your new job?)

@valeriupredoi or @zklaus: Is anybody of you looking at this from the technical point of view? @alistairsellar: do you want to do the scientific review, or should I do it? It would be great if we could get this in the release of v2.10.0!

Hi @hb326, I had a quick chat with @catherinehardacre on this before in July. @zklaus has been her main collaborator, and I think this PR was waiting on small remaining effort from one of them.
I have already science-reviewed the methodology, and happy with it overall, Am just awaiting finalisation of code before approving.
(P.S. Hi @catherinehardacre, hope the move to NZ went smoothly. I forget when you start your new job?)

Great and I too hope @catherinehardacre 's move went OK and we are already missing her in UKESM! Let me know if I can help in any way - I reckon @zklaus is prob best person to operate significant code changes, but I can review :+1:

@alistairsellar Many thanks for your review! I will get on with tidying up the aero_utils.py and aod_aeronet_assess.py scripts as suggested in your comments above :)
Could you please point me to what the failing tests are? are they the Codacy Static Code Analysis? or am I missing something else - if it's not just the code styling, any pointers to sort it out would be great :)
@alistairsellar did you mean PR #3227 - Klaus's Aeronet cmorizer? I'll double check that is up to date too..
@valeriupredoi - If you are able to the code review, that would be ace!
Thanks everyone

Could you please point me to what the failing tests are? are they the Codacy Static Code Analysis? or am I missing something else - if it's not just the code styling, any pointers to sort it out would be great :)

Yes, I was referring to the the Codacy Static Code Analysis, and it does indeed look to be all code style guidance. There is some guidance here on code standards: https://docs.esmvaltool.org/en/v2.3.0/community/code_documentation.html#python
Best place to start is probably the 3 commands at the bottom of that section (isort, yapf and docformatter). Please shout if you get any problems running prospector on the command line (I have seen it throw confusing errors in the past), or if any of the style comments are not understandable.

@alistairsellar did you mean PR #3227 - Klaus's Aeronet cmorizer? I'll double check that is up to date too..

Oops, yes I meant #3227. Now corrected in my comment above. It is in draft, but since it's output has been successfully tested by this diagnostic, perhaps it "just" needs tidying and submitting for review?

hi @catherinehardacre sure thing, will review today, cheers :beer:

OK am done with me review - very nice work, and plots, Catherine - really cool sciencey stuff! Code looks good too, but it'd benefit from a few adjustments here and there - @alistairsellar makes an excellent point of maybe trying to use as much as possible esmvalcore's preprocessors (import esmvalcore as a  package, then you can use its preprocs as modules). It'd also be nice to have a look through the Codacy warnings - a lot of it is stuff we normally don't enforce in ESMValTool (a sopposed to ESMValCore) but it'd be cool if you could fix some of the low hanging fruit there :+1:

@valeriupredoi Thank you for the review and the coding comments - I'm always trying to write nicer code ;) I'll pull these together with Alistair's comments and (hopefully) get it tidied up this week...

Changes so far are looking good @catherinehardacre! Would you like a re-review yet, or is it still a work in progress?

@alistairsellar @valeriupredoi I've tidied up the code as per your suggestions. I believe I've just understood V's points about kwargs, so I'll sort that tomorrow. I've reduced the Codacy warnings, but not eliminated them entirely, so let me know if more needs do there... I'll have this ready for re-review tomorrow @alistairsellar - apologies for the delay, I always need a clear run to shift into ESMValtool gear...

I'm happy to do it, but can't we just edit CITATION.cff within this branch?

yeah we could, but my GitHub OCD tells me we should do PRs with their own scopes 😁

It feels like this is in scope - the current PR is Catherine's first contribution, so seems natural to me to include contributor file updates in your "maiden" PR. Similar to adding name to config-references.yml?

If Catherine plops her name, affiliation, and ORCID in CITATION.cff, I won't complain though 😆
Great, thanks for pragmatism. I've gone ahead and done that myself, to save Catherine tracking down where these files are (feels like there's enough other stuff for her to be getting on with in our reviews). @catherinehardacre hope you don't mind me throwing these commits into your branch. Do you happen to have an ORCID?

@alistairsellar @valeriupredoi I've tidied up the code as per your suggestions. I believe I've just understood V's points about kwargs, so I'll sort that tomorrow. I've reduced the Codacy warnings, but not eliminated them entirely, so let me know if more needs do there... I'll have this ready for re-review tomorrow @alistairsellar - apologies for the delay, I always need a clear run to shift into ESMValtool gear...

No need to apologise, and PR is looking in great shape. Just three tiny outstanding things left from my review (unresolved comments above):

Include yourself as author on recipe
Decide whether to list your name as a maintainer (optional)
Add a comment about why preprocessors are not suitable for doing what you need

=> Done!
=> Done!
=> Done
Fantastic - thanks @catherinehardacre! I've approved my review. Many thanks for your work on this.

Let me know if I've missed anything or anything needs doing for [(https://github.com/Add AERONET cmorizer #3227)].
Thanks - I'll head over to look at that now.


Chapter 3 - Dependency Service is not working

I attempt to deploy the TripLog app to the Android simulator and I get a NullReferenceException at
var navService = DependencyService.Get() as XamarinFormsNavService;
I have no other errors in Shared Lib or Android project. I get the same error running the code from github as well.
I updated all Nugets to latest. The ONLY time I had success was based on the guthub code BEFORE I updated all the nugets. (Xamarin.Forms == <IP_ADDRESS>047)
What changes to the DependencyService need to be made to make this work? Very frustrating.

My own fault... typo... :/ the danger of having Property and local vars named the same MainPage vs mainPage...


Maximum balance when adding liquidity to pool

When adding liquidity to a pool clicking on MAX will enter your maximum available balance -1.
I think this should not be the case, as it is possible to enter the 'real' maximum available balance by hand.
If I'm not mistaken the line responsible for this is the following: webapp/src/containers/SwapPage/index.tsx+226

it was added by cd6361f though i am not sure as to why

Hi @thunderstorm99 , thanks for your feedback, please give us time to investigate this issue.

Hi @thunderstorm99 , thank you for waiting. After some investigation, this seems like a visual bug. The team is in the midst of fixing this. In the mean time, you may try the temporary solution as stated, thank you.
In input type 1 > click on MAX

Hi @thunderstorm99 , thank you for waiting. After some investigation, this seems like a visual bug. The team is in the midst of fixing this. In the mean time, you may try the temporary solution as stated, thank you.
In input type 1 > click on MAX

Hi @thunderstorm99 , update v2.1.6 is out https://defichain.com/downloads/.
I've notified team of this visual bug. Proceeding to close this issue for now, you may reopen this issue if there's new information (include screenshots) and your app logs. Thank you.

Hi @thunderstorm99 , update v2.1.6 is out https://defichain.com/downloads/.
I've notified team of this visual bug. Proceeding to close this issue for now, you may reopen this issue if there's new information (include screenshots) and your app logs. Thank you.


restore-state-cb and image-file-reactive can crash annoyingly

This fixes a variety of small issues related to selecting reprexes, or source-image-directories

but I am sure I haven't fixed them completely

But I am sure I haven't fixed it completely, given that this issue is so annoyingly hard to diagnose.


Use R kernel with vscode-jupyter get error with output

Applies To

[x] Notebooks (.ipynb files)
[ ] Interactive Window and/or Cell Scripts (.py files with #%% markers)

What happened?
ParseError: KaTeX parse error: No such environment: tabular at position 7

VS Code Version
1.77.0
Jupyter Extension Version
v2023.3.1000892223
Jupyter logs
ParseError: KaTeX parse error: No such environment: tabular at position 7: \begin{̲t̲a̲b̲u̲l̲a̲r̲}̲{lll} 1 & 2…

Coding Language and Runtime Version
R 4.2.2
Language Extension Version (if applicable)
R extension v2.7.2
Anaconda Version (if applicable)
No response
Running Jupyter locally or remotely?
Local

I have recreated a workspace and solved this problem!

Recreated a workspace and solved this problem.

As long as I put the .ipynb file in the workspace and run, I will get this error, but this is not the case when .ipynb file is executed alone

oh I found it, changing the presentation to text/latex gives the error.
With the text/html renderer, the matrix is displayed:

oh I found it, changing the presentation to text/latex gives the error. With the text/html renderer, the matrix is displayed:

Ops... it works! Thank you! I thought it was a bug after the update, sorry.


docs: new format for the "Usage" section in the README.md files (proposal #2)

Description
As described in #8613 are linking all README.md to the archived docs. We should change this because the docs are archived and it makes the impression that the packages are not maintained anymore, when you are linking to archived docs.
This is one of the two proposals for fixing this issue (other proposal: https://github.com/firebase/flutterfire/pull/8813). In this proposal, we just link to the first article of the docs.
Advantages:

Fit into the existing style of the README.md files
No maintaining (in comparison to #8813)

Disadvantages:

The links are sometimes a bit random because the format doesn't fit anymore. For the old docs we had in most cases a "Usage". But now we have a lot of small pages.

This PR is a proposal and should show a proof of concept for the new format. If this is a good solution, I will continue adding the links for the other packages.
Related Issues
Closes #8613
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]).
This will ensure a smooth and quick review process. Updating the pubspec.yaml and changelogs is not required.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See Contributor Guide).
[x] All existing and new tests are passing.
[x] I updated/added relevant documentation (doc comments with ///).
[x] The analyzer (melos run analyze) does not report any problems on my PR.
[x] I read and followed the Flutter Style Guide.
[x]  I signed the CLA.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change.
[x] No, this is not a breaking change.

@kevinthecheung @russellwheatley Any thoughts on this proposal?


How to update settings.json in the managed vm

Hi,
Not sure whether it should be raised here or bosh-agent.
We are trying to migrate our director compiled_package_cache s3-compatible bucket (not only the name but also the endpoint etc) from current one to a new one.
After bosh director updated successfully, we found that below settings in the managed vms are still the old ones - which results in issue when there is a need to do the s3 upload (for instances a new release with new compiled packages). We did another test by spinning up a new vm without any other changes, the result shows that the new vm has the updated settings.

/var/vcap/bosh/settings.json
/var/vcap/bosh/etc/blobstore-s3.json

I want to confirm whether this is the expected behaviour and what's the practice the update these setting after a compiled_package_cache settings change (seems restart agent can regenerate the blobstore-s3.json if settings.json has been updated in advance)?
Thanks!

@aad you ll have to recreate vms.

Thanks!


[misc] Improve manual tests to auto using ATIP for sampleapp

Impacted TCs num(approved): New 0, Update 9, Delete 0
Unit test Platform: Android
Android test result summary: Pass 9, Fail 0, Block 0

LGTM


when run at server throw Cannot find compilation library location for package error

i create the project by asp.net core 2.1 ,Microsoft.AspNetCore.App version is 2.1.3,,my server is windows 2012 64x,.net core runtime is 2.1.3,,i use the publish the website ,configuation is
 ,,,but when i request the site,it throw the error like this

but i never references this package....how can i solve this ,,,please help me

this application run  in vs2017 mode is no problem

@dougbu / @pranavkm / @mkArtakMSFT - any ideas on this?

Try setting <CopyRefAssembliesToPublishDirectory>true</CopyRefAssembliesToPublishDirectory> in your project file.. https://docs.microsoft.com/en-us/aspnet/core/razor-pages/sdk?view=aspnetcore-2.1 has some documentation about the setting.
That said, I'd recommend not trying to compile views as part of a published application. The most common case for this is using an EmbeddedFileProvider as a source for views. Have a look at Razor Class Libraries - https://docs.microsoft.com/en-us/aspnet/core/razor-pages/ui-class?view=aspnetcore-2.1&tabs=visual-studio - instead.


[Request] Upload it to F-droid and FOSSdroid

maybe the app will became more popular and more people will contribute to this

duplicate of https://github.com/gazlaws-dev/codeboard/issues/74

fossdroid doesnt need additional publishing:  https://fossdroid.com/blog/about


AnimStateGraph constructor is documented as having zero arguments

https://github.com/playcanvas/engine/blob/dev/src/anim/state-graph/anim-state-graph.js#L1-L7
If we intend for this to be public API, we should document the argument that it takes.

Note that there is already an open PR by @kungfooman that tries to address this here: https://github.com/playcanvas/engine/pull/2986
However, for now, I'm leaning more towards just specifying data as {object} which should then appear in the TS definitions as any. This should at least avoid TS compiler errors.


Unable to perform helm upgrade due to resource conflict

Output of helm version: v3.0.0-rc.1
Output of kubectl version: Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T12:36:28Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.10-eks-5ac0f1", GitCommit:"5ac0f1d9ab2c254ea2b0ce3534fd72932094c6e1", GitTreeState:"clean", BuildDate:"2019-08-20T22:39:46Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}
Cloud Provider/Platform (AKS, GKE, Minikube etc.): AWS
Seem to be experiencing a weird bug when doing helm upgrade. The error states "Error: UPGRADE FAILED: rendered manifests contain a new resource that already exists. Unable to continue with update: existing resource conflict: kind: ServiceMonitor, namespace: dcd, name: bid-management".
We've tested on the following helm versions:
Helm Version:"v3.0.0-beta.2", "v3.0.0-beta.3"
We get the following error - "Error: UPGRADE FAILED: no ServiceMonitor with the name "bid-management" found". Though I can confirm it exists.
Helm Version:"v3.0.0-rc.1", "3.0.0-beta.4", "3.0.0-beta.5"
We get the error above "Error: UPGRADE FAILED: rendered manifests contain a new resource that already exists. Unable to continue with update: existing resource conflict: kind: ServiceMonitor, namespace: dcd, name: bid-management"

Can you provide a set of steps to reproduce the issue?

@bacongobbler Apologies for the delay. Realised it's harder to reproduce locally with minikube since we've got everything setup for/with AWS EKS. At the moment I can confirm the apiVersion of the serviceMonitor doesn't change so it doesn't seem to be a relation to #6583.
When I run helm template the first time:
`---
Source: k8s-service/templates/servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: bid-management
namespace: dcd
labels:
chart: k8s-service-v0.0.11
app: bid-management
heritage: "Helm"
release: prometheus
spec:
endpoints:
- honorLabels: true
interval: 10s
path: /prometheus
port: metrics
scheme: http
scrapeTimeout: 10s
selector:
matchLabels:
app.kubernetes.io/name: bid-management`
After upgrading and once the resource gets created successfully, I run helm template again and get back the following below:
`---
Source: k8s-service/templates/servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: bid-management
namespace: dcd
labels:
chart: k8s-service-v0.0.11
app: bid-management
heritage: "Helm"
release: prometheus
spec:
endpoints:
- honorLabels: true
interval: 10s
path: /prometheus
port: metrics
scheme: http
scrapeTimeout: 10s
selector:
matchLabels:
app.kubernetes.io/name: bid-management`
After running helm upgrade a second time, I get back the error mentioned above
Error: UPGRADE FAILED: rendered manifests contain a new resource that already exists. Unable to continue with update: existing resource conflict: kind: ServiceMonitor, namespace: dcd, name: bid-management

@bacongobbler Going to still try and re-produce the steps locally with minikube but might take longer than expected.

Facing the same issue here. @bacongobbler , @efernandes-ANDigital
kubectl version: 1.14.7
kubernetes version 1.15 (PKS)
First time happened to me using Helm v3.0.0-rc.1, after updating to Helm v3.0.0-rc.2 this is still happening.
Made a rollback successfully to the previous state with rc.2 and upgraded again but it didn't solved it: after upgrading successfully trying a new upgrade gave the same error message.
helm diff shows no issues (it detects the resources correctly), and even if I look inside the secret related to the revision, it shows the resources there, so it shouldn't try to re-deploy them.
Not a complex chart, just using range to iterate a list (~100 elements) to generate some resources (namespaces, configmaps, etc.)
Cannot repro it (tried on GKE)

@aespejel What kinds of resources did it fail on?

Namespaces, which sounds logic having in mind the order helm is trying to apply manifests, right?

Yep, Namespaces go first, but I was just checking if this was happening with specific kinds of resources or with an assortment

Just to add we noticed something else, upon disabling the service monitor. When running helm upgrade, it returns back a success message "Release "bid-mangement" has been upgraded. Happy Helming! etc. However upon checking the api-resources servicemonitors we still see the servicemonitor that was created.

What we've just noticed is for same charts with other services it works just fine and we don't have the issue. The services use the same exact charts with just a few configuration changes for the services....very weird

The problem also happen while trying to install a chart (e.g. prometheus-operator), if the install fails and you try to install it again, helm complains about resource conflict, and if you try to remove the chart, complains saying it never been deployed.

@vakaobr I doubt it's the same issue. When the first install fails (and only with the first install), as you noticed, helm doesn't make a release. Hence helm wont have any information about a release to compare with already deployed resources and will try to install them showing that message because actually some of the resources were installed. You can probably solve this by using --atomic with the installation or using helm upgrade --install --force  being careful with the --force since it will delete and re-create resources.
Here we are facing an issue that happens with already successfully installed charts.

Update: still happening after updating to helm v3.0.0 (stable). @efernandes-ANDigital , @bacongobbler , @thomastaylor312
If I use helm diff, it shows NO differences, if I use helm upgrade --install --dry-run, it fails with the following error: "Error: UPGRADE FAILED: rendered manifests contain a new resource that already exists."
Using helm get manifest (of the last release), shows these resources.
Using helm template shows theses resources too.
Maybe this is related with how helm compares resources with the manifest vs the templates?
I'm creating resources iterating a list with range, could it be related with this?

Workaround:
Running the upgrade --install with --dry-run --debug  and -v 10 option showed us that helm was somehow using some really old revisions. We deleted all revisions but the last 2 and it started to work again.

Did you manage to save the release ledger before deleting them? Would've been helpful to reproduce the issue if we had our hands on a solid reproducible case.

I get this error when trying to change api version of deployment to apps/v1 from deprecated extensions/v1beta1. Helm refuses to deploy without me manually removing old deployment.

@sheerun did you see my answer in regards to apiVersion changes in this comment: https://github.com/helm/helm/issues/6646#issuecomment-547650430?
The tl;dr is that you have to manually remove the old object in order to "upgrade". The two schemas are incompatible with each other and therefore cannot be upgraded from one to the next in a clean fashion. Are you aware of any tooling that handles this case?

This doesn't really help because I still need to manually remove old resources. I'd expect that flag for helm update command like --force would automatically remove and add resources that have api-incompatibility, but it's not the case. This makes upgrades of apps in kubernetes clusters very cumbersome. If --force is not responsible for this then other flag would be useful.
It's very important issue right now because kubernetes 1.16 just drops support for old apis so we need to upgrade.

I see your point... We could potentially support a new flag for that.
If you have suggestions, we'd love a PR with tests, docs, etc. It's certainly cropping up more and more especially with the 1.16 release, so we'd be happy to look at proposals to handle that case.

any updates on this?

#7082 should handle this case if someone wants to start working on that feature.

If you are having to use these workarounds: https://github.com/helm/helm/issues/6646#issuecomment-546603596, you can use the following script I crated to automate that process: https://gist.github.com/techmexdev/5183be77abb26679e3f5d7ff99171731

Fixed by this PR

reproduce step
I have install 10 revisions, when I install new sub charts in revision 11, deloyed OK.
then upgrade again with same charts, helm complains rendered manifests contain a new resource that already exists.
reason
helm compare the current revision with the first deployed revision which has no resources we installed in the last revision just now
fix
use the last deployed revision as currentRelease instead of fist revision

work around
delete old secrets

@jason-liew This issue is about different thing that is not related to number of releases. You're fixing other bug with similar error. This bug is related to change of resource's api version.

@sheerun sorry,  i have delete the reference in the commit message.

I have the same issue. Production env is blocked and not possible to fix.
$ helm version
version.BuildInfo{Version:"v3.0.2", GitCommit:"19e47ee3283ae98139d98460de796c1be1e3975f", GitTreeState:"clean", GoVersion:"go1.13.5"}
Error: UPGRADE FAILED: rendered manifests contain a new resource that already exists. Unable to continue with update: existing resource conflict: kind: Service, namespace: mynamespace, name: my-service
Please add "Warning! There is risk to have brick instead of chart after update" to https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/
Amazing work guys. I'm proud!

Also bumped into this problem.
After migration to v3, I'm unable to update charts, using the helm-v2-to-helm-v3/ plugin:
Error: UPGRADE FAILED: rendered manifests contain a new resource that already exists. Unable to continue with update: existing resource conflict: kind: Deployment, namespace: default, name: grafana-main
This would be totally understandable if would have happened in a beta version, but I'm running v3.0.2 after following instructions on the official blog, reaching a issue/ bug reported during the beta version. It doesn't feel nice.
Is there any non-destructive work-arround for the time being? What this comment proposes feels quite destructive https://github.com/helm/helm/issues/6646#issuecomment-546603596

Also having the same problem when upgrading a resource to a new api version using helm3.
For the time being, we're in need of a clean workaround as well. Deleting the old resource isn't really an option for our production workloads.

Helm 3.0.2. Can't deploy or even rollback when the previous deploy changed the number of deployments (removed or added a Deployment). Fails with error:
Error: no Deployment with the name "server2" found
Extremely frustrating.

I agree. It's a frustrating point of view from the Kubernetes API point of view.
If anyone is aware of a a solution within Kubernetes that allows one to upgrade apiVersions without having to re-create the object, we'd love to hear about it. We are currently unaware of an API endpoint available for third party tools like Helm to "convert" an object from one apiVersion to the next.
The only option we're aware of from Kubernetes' API is to delete and create the object, which is how this error appears.
It was mentioned in https://github.com/helm/helm/issues/7219 that upgrading from Kubernetes 1.15 to 1.16 migrated the objects from a deprecated apiVersion (like extensions/v1beta1) to apps/v1 on the backend. If someone could confirm that behaviour as well as gain a better understanding about how Kubernetes achieves this, that could be a possible solution to this problem.

I tried to perform the redeploy from another machine (not the one that had previously deployed the modified number of deployments), and it went smoothly. Could possibly be a local caching issue?

@youurayy Do you use Stateful sets ? Or Deployments ? Stateful sets only support seamless resources upgrade in K8S. Deployments still have issue with "Upgrade error, Not possible to upgrade" service for example.

I faced the same issue here. For your information, I migrated from v2 to v3. At first it didn't work since it was complaining that some of the APIs were not valid. I upgrade the APIs (Deployments and Statefulsets), and then I had this issue

Same issue here. Doesn't look like a PR has been opened yet. Anyone working on this?

nope, feel free to investigate further. Have a look at my previous comment for more information on where to get started.

This is kind of a big deal no?  If I'm not mistaken you have no choice with helm3 to change your api revisions from v1beta1 to v1, so doesn't that mean everyone that uses helm pretty much that stays current has to delete deployments prior to upgrading a chart?

@bacongobbler I agree that from k8s point of view, it's the broken change between API versions. However, in k8s there IS design to handle such a case to migrate one object from one version to another.
For example, in one 1.14 cluster, if a deployment created in version 'apps/v1', it's also available in the version 'apps/v1bet1', 'apps/v1bet2', 'extensions/v1beta1'. See https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.14/#deployment-v1-apps.
So I think the gvk design of the helm3 is ok, but the implementation should be more complicated. The old release object should not only be retrieved from helm release storage, but also from the current running environment.
Thanks

I agree. It's frustrating that the Kubernetes API treats updating the apiVersion as a breaking change.
If anyone is aware of a a solution within Kubernetes that allows one to upgrade apiVersions without having to re-create the object, we'd love to hear about it. We are currently unaware of an API endpoint available for third party tools like Helm to "convert" an object from one apiVersion to the next.
The only option we're aware of from Kubernetes' API is to delete and create the object. It is certainly not ideal, but it's the only option we are aware of at this time.
It was mentioned in #7219 that upgrading from Kubernetes 1.15 to 1.16 migrated the objects from a deprecated apiVersion (like extensions/v1beta1) to apps/v1 on the backend. If someone could confirm that behaviour as well as gain a better understanding about how Kubernetes achieves this, that could give us a possible solution to this problem.

A k8s single object may convert from one version to another version if they are compatible. See https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/apps/v1/conversion.go as an example.

I've ran into an issue that's related to this.
In my case, I've enabled NamespaceAutoProvision admission controller to avoid failures due to non-existent namespaces. That mostly helped, but here's where it created a new problem: in our charts, we created few namespaces explicitly in order to set some labels on them, which are used for network policies. This was done as a separate step before installing charts that use those namespaces. With helm 3 and the NamespaceAutoProvision controller, chart installation now fails with
Error: rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: kind: Namespace, namespace: , name: monitoring

which makes sense, but blocks provisioning.
I don't know if this has been floated before, but maybe there could be a way to tell Helm to "adopt" resources if they already exist – i.e., in case of an existing namespace it would be patched with user-supplied manifest and understood to be managed by Helm from that point.

A k8s single object may convert from one version to another version if they are compatible. See https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/apps/v1/conversion.go as an example.

That is a conversion from apps/v1 to an internal representation. It does not convert from v1beta1 to v1. Look at the code more closely.

For example, in one 1.14 cluster, if a deployment created in version 'apps/v1', it's also available in the version 'apps/v1bet1', 'apps/v1bet2', 'extensions/v1beta1'

Kubernetes clusters support multiple API versions, but they are treated as separate iscrete objects. The internal schemas are completely different. There is no "convert from v1beta1 to v1" API we're aware of at this time.

I don't know if this has been floated before, but maybe there could be a way to tell Helm to "adopt" resources if they already exist – i.e., in case of an existing namespace it would be patched with user-supplied manifest and understood to be managed by Helm from that point.

See #2730

@bacongobbler thanks for your answers and help here. I have same issue with api version, but in cluster itself our deployment have - apiVersion: apps/v1
New version of chart also have: apiVersion: apps/v1
But in Helm 3 metadata/release we have this:
apiVersion: extensions/v1beta1
kind: Deployment
Its really not convenient that you need to reinstall production workload just to fix Helm metadata, since real deployment have correct API version. Any suggestions here? I am thinking to tweak metadata manually.

Any suggestions here?

https://github.com/helm/helm/issues/7219#issuecomment-568175917

@bacongobbler
I will show how kubernetes hanlde multiple API versions

Desing doc
https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md#operational-overview.
There is an example in  the doc on how to create a resource in v7beta1 and retrieve in v5. So from kubernetes design perspective, it allows a single object conversion among multiple versions.

The conversion process is logically a "star" with the internal form at the center. Every versioned API can be converted to the internal form (and vice-versa)

Kubernetes source code
As I mentioned above, there are such conversions
https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/apps/v1/conversion.go Convert_v1_DeploymentSpec_To_apps_DeploymentSpec and Convert_apps_DeploymentSpec_To_v1_DeploymentSpec
Also, https://github.com/kubernetes/kubernetes/blob/master/pkg/apis/apps/v1beta2/conversion.go Convert_v1beta2_DeploymentSpec_To_apps_DeploymentSpec and Convert_apps_DeploymentSpec_To_v1beta2_DeploymentSpec. The code use the interal data structure as a hub in different versions.

Runtime behavior
I use a 1.14 cluster and kubectl.
kubectl get -n kube-system deployment coredns -o yaml -v 8
Force to use another api version
kubectl get -n kube-system deployment.v1.apps coredns -o yaml -v 8
You can see a single object (by its uuid) can be retrieve by multiple API versions.

Hey,
I am having the same issue. My issue is regarding a pre existing stateful set. Any advice would be much appreciated.
Thanks,
Richard

Excited to see #7082 get released!

Let's open up discussions:
https://github.com/helm/helm/pull/7575

Hello,
I am facing same issue the only think i have done i upgraded from helm 2.14.1 to latest, we are getting the error as mentioned above : **rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: kind: Service, namespace: *, name: *** . All stuff mentioned up about deleting that wont work for us as this is production and the API is critical with 0 down time required. kindly assist ...
Thanks
Hany

Here's a dirty hack that we use whenever a resource, such as a PV or PVC, already exists and we don't want to delete it. This typically happens whenever we do a helm upgrade whatever.
kubectl get deployments -n monitoring
kubectl get -n monitoring deployment/prometheus-operator-grafana -o jsonpath="{.spec.replicas}"

# Set spec.replicas to 0
kubectl edit -n monitoring deployments/prometheus-operator-grafana
watch -n1 -d "kubectl get pods -n monitoring"

# Once all pods have terminated, set the spec.replicas back to 1 or whatever value you want
kubectl edit -n monitoring deployments/prometheus-operator-grafana
watch -n1 -d "kubectl get pods -n monitoring"

# At this point you'll have an updated pod/deployment/whatever

I got this error just following the basic tutorial

Hitting this install kubernetes dashboard chart
Error: UPGRADE FAILED: rendered manifests contain a new resource that already exists. Unable to continue with update: existing resource conflict: namespace: , name: kubernetes-dashboard, existing_kind: rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding, new_kind: rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding

Same issue when installing chart in two namespaces. My chart depends on prometheus-operator chart which will create ClusterRole .
Error: rendered manifests contain a resource that already exists. Unable to continue with install: existing resource conflict: kind: ClusterRole, namespace: , name: p8s-operator

Same here. I migrated the helm2 to a helm 3 deployment and afterwards its no longer upgradeable because of the same error
./helm upgrade public-nginx-ingress stable/nginx-ingress --install --namespace mbsys-ingress --wait -f public-nginx-ingress.yaml coalesce.go:199: warning: destination for extraContainers is a table. Ignoring non-table value [] coalesce.go:199: warning: destination for extraVolumes is a table. Ignoring non-table value [] coalesce.go:199: warning: destination for extraVolumeMounts is a table. Ignoring non-table value [] Error: UPGRADE FAILED: rendered manifests contain a new resource that already exists. Unable to continue with update: existing resource conflict: namespace: , name: public-nginx-ingress, existing_kind: rbac.authorization.k8s.io/v1, Kind=ClusterRole, new_kind: rbac.authorization.k8s.io/v1, Kind=ClusterRole

Could someone clarify what the solution is here? I see that this got reopened then closed again 39 minutes later but I didn't see an obvious solution in this thread.

Could someone clarify what the solution is here? I see that this got reopened then closed again 39 minutes later but I didn't see an obvious solution in this thread.

There is no solution yet but this one is promising and almost ready to implement:
https://github.com/helm/helm/pull/7649

#7649 was merged this morning.

#7649 was merged this morning.

Ohh, missed that ;) well, then answer to @micseydel question is in the first post of #7649 in Release Notes section


Seperate SCSS files,

Hey guys, just a quick question if anyone could point me in the correct direction that would be appreciated. Currently we have SCSS files that we want processed but would like to create them as separate css files, so no bundling for these. Is this possible with webpack? I have used extract-text-webpack-plugin but this seems to only create bundled css files.

What makes you think it is only for css files? I think you can use it for anything you don't want to include in the bundle.
https://stackoverflow.com/questions/29210325/webpack-sass-where-is-the-css-file

Thanks Adam,  appreciate the help. I tried this bid let's say I have two completely separate session files e.g. desktop.scss and mobile.scss how would we go about extracting a mobile.css file and desktop.css file?

It's impossible now, if you want entry SCSS file and output only CSS file
import ExtractTextPlugin from 'extract-text-webpack-plugin';

const config = {
  entry: {
    desktop: './styles/desktop.scss',
    mobile: './styles/mobile.scss',
  },
  moduleExtensions: ["-loader"],
  output: {
    filename: '[name]'
  },
  module: {
    loaders: [
      {
        test: /.\scss$/, loader: ExtractTextPlugin.extract({
          notExtractLoader: 'style',
          loader: 'scss'
        })
      }
    ],
    plugins: [
      new ExtractTextPlugin({
        filename: "build/css/[name].css",
        disable: false,
        allChunks: true
      }),
    ]
  }
}
export default [config]

output will be:
desktop.js, mobile.js, desktop.css, mobile.css,

Oleg's answer is correct. Closing.
Please use Stack Overflow for questions like this.

@OlegLustenko notExtractLoader should be fallbackLoader as of extract-text-webpack-plugin beta.3.

Hey @OlegLustenko, I was trying out your example, and I kept getting an error, even though I had the sass-loader installed -
   ERROR in ./main.scss
   Module parse failed: /Users/arunmichaeldsouza/Desktop/wb/main.scss Unexpected token (1:5)
   You may need an appropriate loader to handle this file type.

I am using your above mentioned config. Any idea what I could be doing wrong ?
Thanks.


Add in the GpuArrayFilter command

this fixes #10760
I still need to do some performance comparisons/testing, but functionality wise it looks good.  Will update it shortly.

build

For performance numbers I ran the following query on Spark 3.4.2
spark.time(spark.range(0,<PHONE_NUMBER>0L, 1, 512).selectExpr("array(id, id + 1, id - 100, id + 3, id + 2) as ar").selectExpr("filter(ar, f -> f % 3 == 0) as f").selectExpr("SUM(size(f)) as r").show())

On an a6000 running with 4 concurrent It took 36 seconds in the worst case, I ran it 3 times and the best was about 33 seconds.
On a 32-core/64-thread Threadripper PRO 5975WX, using all 64 cores it took 688 seconds (and I didn't really want to run it again).
That is a speedup of about 19x.

build

@jlowe @abellina please take another look

build

The link check is failing because of https://github.com/NVIDIA/spark-rapids/issues/10768


Issue When Compiled on Linux

try's to find a non exsistient so file

what does it say on your terminal?

that should work

thanks

np bud

welp guest this isn't a problem any more

do you want to close it?

yeah

it didn't work

https://user-images.githubusercontent.com/91945956/161276030-fbea57fa-3f44-4d23-b9b1-217ee150d2e3.png
IT STILL DIDN'T WORK

ohh
put if desktop instead of if windows

so it can use the linc_luajit

opps nvm

i fixed it myself

how did you fixed it?

put the so file it was tring to find in the lib dictroy

the lib file was in linc_luajit


Add files via upload

Added code to print statistical difference between cosecutive models with n+1 subtypes

Had a quick look on my phone. Is this a t test between BIC? Are you sure that this is valid?

Yes, this is the t test between BIC. I've checked it on the simulated data which supports ground truth of two subtypes and not three subtypes. See "SuStaIn tutorial using simulated data updated stats between models" file in notebooks

I think Neil's comment is more about how meaningful a t-test between BIC is. That justification is key, and I think it'd be prudent to have that rather than integrate this based on empirical results in the simulated data.
From a code perspective, if we were to do this, you should be looping over all pairs of subtypes, rather than the for loop currently implemented. After this, there should also be the option for multiple comparison correction.

Thanks for clarifying the issue @sea-shunned. I guess increasing the sample size for the t-test i.e number of cross-validations wouldn't work either as we'd end up with similar BIC values for each of the cross-validations. I think a simplier method is to compare the difference of BICs for each of the models trained on the main dataset. I'll update the code in another notebook.


I cannot start the bot

Hey there! Whenever I start the bot, this error happens: https://imgur.com/a/ePoiGXC
Please find attached the config file (which I had to change to a txt file because it doesn't allow me to upload .json files)..
config.txt
Thanks!

Dupe.


Forcemerge should report index size, segment count, and heap usage

Running force-merge changes index size, segment count, and heap usage.  It would be nice if esrally reported the new values after force-merge runs in addition to the original values from the original index operation.

That is the point of force-merge: We run it as an intermediate step in our benchmarks to get better reproducible results for queries. The metrics you describe are also not gathered during the benchmark but rather at the very end of it (index size is even determined only after the cluster has been shutdown).
If you are interested in metrics without running force-merge, I think the best option is to create two challenges:

run only an index operation
run indexing, then force-merge

However, you should account for run-to-run variation by repeating the benchmark several times.


请问header_comment怎么设置标题不要替换描述

比如我设置了一个header_comment

我要怎么样设置让描述那里，不要每次格式化的时候被格式化

@elliot @sebastianbergmann

Hi @233cy,
could you maybe following the reporting template and translate this into English as I don't think we can be of much help otherwise sadly.


unit testAdd unit test case for auto analyze.

Add unit test case for auto analyze.

Further comments
If this is a relatively large or complex change, kick off the discussion at<EMAIL_ADDRESS>by explaining why you chose the solution you did and what alternatives you considered, etc...

run buildall

run buildall


feat: generate_library.sh with postprocessing

Current progress demoed in https://github.com/googleapis/google-cloud-java/pull/9792
This modifies generate_library.sh so it calls the owlbot postprocessor and some adapted postprocessing scripts originally meant to help new-client.py
** notes **

The owlbot SHA will be inferred from the monorepo's HEAD, if provided
repo-metadata.json will not be inferred as it's equivalent to an eventual "monorepo folder" input variable

Pending items are mainly explained by the demo diff PR:

[x] libraries bom version in readme

NOTE: this version is brought by owlbot updates instead of region tags. Maybe this needs a region tag

[x] bom's parent pom
[x] gapic library description - fixed by using an owlbot sha image that matches the one used by the commit that initially brought the library (i.e. this error happened because of owlbot image mismatch)
[x] dependency order
[x] indentation
[x] newline at EOF

@diegomarquezp could you reformat the code in accordance with https://google.github.io/styleguide/shellguide.html?
The obvious ones are quoting and case statement

Sure thing, I missed this as a preparation step for review.

Let's try to merge https://github.com/googleapis/sdk-platform-java/pull/1935 first before merging this. Since the showcase module generation is changed to use the script, the changes in this PR would affect showcase as well, we would probably need more changes to accommodate showcase module.

@diegomarquezp could you reformat the code in accordance with https://google.github.io/styleguide/shellguide.html?
The obvious ones are quoting and case statement

@JoeWang1127 I did most of necessary formatting (i.e. quoting variables with braces and constants) but couldn't quite get the problem with case - it looks as suggested by the formatting guide.
(not intended to start a re-review)

I found out that since the scripts replace the poms, the generation of alloydb disregards the existence of other apis (e.g. generating alloydb/v1 will remove v1beta1 from the poms with the current setup).

Could you modify library_generation/README.md to reflect the change as well?

https://github.com/googleapis/google-cloud-java/commit/4c9115650a51b522432cfa198d6922311a2568e7 causes different output in poms (artifact will be apigeeconnect instead of apigee-connect)

https://github.com/googleapis/google-cloud-java/commit/5c5078db41a7c7bcfe107815088df16bdd5a07df#diff-f475a3a9d1237c0f96b9e69f817bf8a42cf682a743f1427463410c6f8f2a66a8 added accesscontextmanager which is not currently handled by the script

@blakeli0
I agree with Joe as well, we should try to only use one path. I still don't quite get the need of an additional repo path, can you please explain more about it?

I ended up using repository_path and infer destination_path from googleapis-gen + proto_path

Looks good overall. Just some minor comments. Also, can you share the performance of the generation process?

@meltsufin I added a small time-tracking step and the results were the following
+ echo 'generation times in seconds:'
+ cat /home/runner/work/sdk-platform-java/sdk-platform-java/output/generation_times
google-cloud-java/java-apigee-connect 107
google-cloud-java/java-asset 11
google-cloud-java/java-asset 10
google-cloud-java/java-asset 10
google-cloud-java/java-asset 10
google-cloud-java/java-asset 23
google-cloud-java/java-compute 177
google-cloud-java/java-kms 22
google-cloud-java/java-redis 15
google-cloud-java/java-redis 14
google-cloud-java/java-containeranalysis 10
google-cloud-java/java-iam 6
google-cloud-java/java-iamcredentials 10

This is only considering the call of generate_library.sh, without checking out reposiories for comparison


fix: Upload conda package to both rapidsai & rapidsai-nightly orgs

This PR updates the release scripts to ensure that DFG is uploaded to both the rapidsai and rapidsai-nightly Anaconda.org organizations.

technically this isn't a fix but I classified it as such to ensure that a package is published to the rapidsai-nightly org

:tada: This PR is included in version 1.13.4 :tada:
The release is available on:

GitHub release
PyPI release
Conda release - rapidsai
Conda release - rapidsai-nightly

Your semantic-release bot :package::rocket:


use tailwindcss text-[#dfdfdf] in @nuxt/content's markdown file, not work

the hex color string becomes an id

Might be related to MDC, cc @farnabaz

@klren0312 could you please share the source notation in your markdown (.md) file for this please?

add this to markdown file
::p{.text-lg .leading-8 .text-[#dddddd]}
testtesttest testtesttest
::

Unfortunately this wouldn't be an issue with Tailwind or this module (or content module imo), it would just become invalid syntax. As a workaround, you can do
::p{class="text-lg leading-8 text-[#dddddd]"}
::


fix: broken search url

Description
This PR fixes an issue where an invalid URL was generated due to passing an enum itself instead of its value to a template. Below are the URLs before and after the change:

Before: /api/datasets/dataset__07_03_2024_15_06_50/TaskType.text_classification:search
After: /api/datasets/dataset__07_03_2024_15_06_50/TextClassification:search

Type of change
(Please delete options that are not relevant. Remember to title the PR according to the type of change)

[x] Bug fix (non-breaking change which fixes an issue)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

How Has This Been Tested
(Please describe the tests that you ran to verify your changes. And ideally, reference tests)

[ ] Test A
[ ] Test B

Checklist

[x] I followed the style guidelines of this project
[x] I did a self-review of my code
[x] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] I filled out the contributor form (see text above)
[ ] I have added relevant notes to the CHANGELOG.md file (See https://keepachangelog.com/)

It seems to be due to changes in the Enum specification in 3.11. I found that it works correctly in 3.10.

https://github.com/Eladlev/AutoPrompt/issues/47
https://github.com/python/cpython/issues/100458

Thanks for contributing,  @katsuya
We may have other places where this kind of error arises. I would love to know how did you find it.

@frascuchon  I discovered this issue while running autoprompt. Specifically, an error occurs at https://github.com/Eladlev/AutoPrompt/blob/7f373f219aa360cd2de38c6aa700c1dff282d7de/estimator/estimator_argilla.py#L106-L111. This section of the code checks if all records have been annotated.

I checked the source code to see if this issue still occurs, and it seems to have been fixed in https://github.com/argilla-io/argilla/pull/4672. Are you still experiencing bugs even after applying this patch?

Thanks for all this info @katsuya.

I checked the source code to see if this issue still occurs, and it seems to have been fixed in #4672. Are you still experiencing bugs even after applying this patch?

Not really, I refreshed your PR with the latest changes from develop branch and it looks is already fixed. I will close this PR


Anticaptcha consumes too much balance

Summary.
Expected Result
Hello, have tested to bypass hcaptcha using anti-captcha, all work fine, just a problem, use to much $ using multi requests/threads, in 2 mins have used 1$ balance without proxies, proxyless
I don't know if it is a bug or is nomal, but i know with cloudscraper (nodejs) just used 0.02$
Also, tested 2captcha, but can't bypass hcaptcha, dont have any error, just try and waiting, but nothing
tested url : https://exitus.xyz
Actual Result
What happened instead.
Reproduction Steps
import cloudscraper

scraper = cloudscraper.create_scraper(
  interpreter='nodejs',
  recaptcha={
    'provider': 'anticaptcha',
    'api_key': 'my-key'
  }
)

scraper.get(url)

System Information
$ python -m cloudscraper.help

{
    "platform": {
        "system": "Linux",
        "release": "4.9.0-12-amd64"
    },
    "interpreter": {
        "version": "3.5.3",
        "name": "CPython"
    },
    "cloudscraper": "1.2.40",
    "requests": "2.12.4",
    "urllib3": "1.24.3",
    "OpenSSL": {
        "version": "OpenSSL 1.1.1  11 Sep 2018",
        "ciphers": "get_ciphers() is unsupported"
    }
}

Debug Log

Add debug=True to create_scraper()

<paste_here>

This command is only available on cloudscraper v1.2.33 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system).

re threads / sessions... are you using a new create_scraper each time? are you caching your cookies / headers ? are you using multiple proxies? you do realize that Cloudflare cookies are only valid for the ip / subnet you solve on.
This is not a bug, I am not going to do that for you, that code logic is entirely up to you if you are switching proxies etc, it is up to you to manage your headers / cookies in a caching manner.

Im not using proxies, just multi threads, and no just one time create_scraper
I will check my script again to see

Paste example code if you want some help

Paste example code if you want some help

``python
import cloudscraper
def bypass():
scraper = cloudscraper.create_scraper(
interpreter='nodejs',
recaptcha={
'provider': 'anticaptcha',
'api_key': 'key'
}
)
time.sleep(5)
while True:
	while oo:
		try:
			scraper.get(uri)
			print(Fore.GREEN + "Conexion Succesed")
			try:
				for g in range(per):
					scraper.get(uri)
					print(Fore.GREEN + "Conexion Succesed")
				scraper.close()
			except:
				s.close()
		except:
			scraper.close()
			print(Fore.RED + "Conexion Failed")

if name == "main":
main()


Restore backwards compatibility to older node

Coverage decreased (-0.3%) to 99.659% when pulling 92f1780618e86470ed1f693d33a9ecd4db207ff9 on restore-backwards-compatibility into 0d8f41990cc8c15e3cdad8ecd3379d0b73b90215 on master.

Coverage decreased (-0.3%) to 99.659% when pulling 92f1780618e86470ed1f693d33a9ecd4db207ff9 on restore-backwards-compatibility into 0d8f41990cc8c15e3cdad8ecd3379d0b73b90215 on master.

Coverage decreased (-0.3%) to 99.659% when pulling 92f1780618e86470ed1f693d33a9ecd4db207ff9 on restore-backwards-compatibility into 0d8f41990cc8c15e3cdad8ecd3379d0b73b90215 on master.

Coverage remained the same at 100.0% when pulling c1971a264e1fb833d1c1b6286fcbe7105aad6e23 on restore-backwards-compatibility into 0d8f41990cc8c15e3cdad8ecd3379d0b73b90215 on master.


handle specifiable limits (twitter dev account got suspended)

Describe the bug
calling collect_results of API directly calls the API v2 without considering limits.
To Reproduce
for time_period in time_priod_list:
for query in query_list:
collect_results()
Expected behavior
To handle specifiable limits, search-tweets-python needs to handle limits as per my queries and pagination for each query.
Environment

OS:Ubuntu 22
Python version: 3.8

Additional context
if this issue is not in the scope, please tell me where is pagination handling code so that I can handle it with user queries

I'm not sure what you mean - rate limits are already handled, in https://github.com/twitterdev/search-tweets-python/blob/v2/searchtweets/result_stream.py#L55 decorator at request level, however - there is an edge case where in loops, calling the Academic access endpoint that has an extra 1 call per second limit, will cause a premature rate limit error, but in this case the solution is to add an extra second sleep to whatever loop you're running, so adding it to for query in query_list: loop should fix it.

Thanks time.sleep() fixed the 300 request per 15 min case.


CI for Node

Created a new workflow using Node.js template

Failing tests
Initializes with two players
Starts the game with a random player
Sets the current player to be whoever it is not
Contains the compiled JavaScript


adding the possibility for extracting www-based URLs as well

Have you had some time to look at the updated pull request?

Sorry, thanks for pinging :). I don't think we should also accept WWW.example.org, that doesn't seem to be something someone would type that often. And if they really want that, they can add http:// in front. Would you mind removing it? The rest looks good now!

I have removed the check for uppercase W.
I also have a suggestion to rename UrlScanner to HttpScanner and introduce an extra link type, so we have a LinkType.HTTP, LinkType.WWW and LinkType.URL, whereas the latter includes to two former.
This will not exactly break backwards compatibility, but just add extra functionality to the same methods.
What do you say?

HttpScanner wouldn't be accurate because it also scans https or ssh or other schemes. I think leaving it as now is good.
What would be gained by adding another LinkType? I can see it lead to bugs because a user could get confused when checking the type of an extracted LinkSpan. And what happens when you tell it to extract type URL and HTTP, would be redundant.

My thinking was to rename UrlScanner -> HttpScanner
and WwwUrlScanner -> WwwScanner
I think this would more appropriately tell what the classes do.
The addition to LinkType just needed to reflect this and thus have an additional type. The LinkType.URL would then be used when you wanted to extract ALL links (http and www) but not email.
However, I do not mind, if you find this confusing to the user. It was just a thought.
I could create a new pull request, so you could see what I meant

The thing that might confuse users is that www.example.org is not a URL at all, see here: https://en.wikipedia.org/wiki/Uniform_Resource_Identifier#Syntax
You can already extract URL and WWW links but not emails by configuring the builder like this:
LinkExtractor linkExtractor = LinkExtractor.builder()
        .linkTypes(EnumSet.of(LinkType.URL, LinkType.WWW))
        .build();

Given that, I'm not sure what having an additional LinkType that is a combination of other link types would improve.

Merged. Thanks a lot for submitting the pull request and seeing it through :)!
I noticed that the README also needs a section for the new functionality. Would you mind submitting another PR for that?

Released in 0.6.0 now :tada:: https://github.com/robinst/autolink-java/releases/tag/autolink-0.6.0