Cancellation not detected

Hi,
I'm revisiting an issue that I thought had seemed to be resolved, specifically, cancelling auto-renewable subscriptions using either XCode->Storekit->Manage Transactions or AppStore.showManageSubscriptions() from within an app. I've tested this with the StoreHelperDemo app and with my own code, and both consistently fail for me.
I'm on iOS 16.2 and the latest version of StoreHelper. While I can correctly monitor subscription status using: Product.SubscriptionInfo.Status.updates, the various transactions in StoreHelper don't seem to pick this up consistently.
What I believe needs to happen is that Status updates need to be monitored and the StoreHelper caches updated appropriately... Something along the lines of:
`for await status in Product.SubscriptionInfo.Status.updates {
            let result: VerificationResult<Product.SubscriptionInfo.RenewalInfo> = status.renewalInfo
            switch result {
                case .unverified(let unverifiedTransaction, let error):
                    print("unverified")

                case .verified(let verifiedTransaction):
                    let productId = verifiedTransaction.currentProductID
                    print("\(pid) verified")

                   if status.state == .subscribed || status.state == .inGracePeriod {
                      print("status subscribed or in grace")
                  } else if status.state == .revoked || status.state == .expired {
                     print("status cancelled")
                     updatePurchasedIdentifiers(productId, purchased: false)
                 }
            }

}`
Thanks.

Sorry for the delay in replying - been on a short vacation!
Re your issue, I'm certain that StoreHelper does (or did) pick up subscription cancellations correctly. It's possible I've done something to break it, but I'll take a look and let you know.

Yes, you're right. Cancellations are not being correctly picked up. I'll put out a fix asap!

Hi Russel, thanks for picking it up. Sorry to bother you post-holiday :).

I've added a few things to help with the situation where subscriptions are renewed or expire when the app's not running.
See AppStoreHelper.paymentQueue(_:updatedTransactions:) and StoreHelper.handleStoreKit1Transactions(productId:status:transaction:).
From the research I've done, it looks like when using Xcode StoreKit Testing and Sandbox Testing subscription renewal transactions that happen when the app's not running are NEVER picked up by StoreKit2! That is, the transactions don't appear in StoreKit.Transaction.all or Transaction.currentEntitlement(for:). This seems to have been a known issue since the release of StoreKit2. So, you can have situations where a user has paid to renew their subscription but StoreKit2 has no knowledge of it. However, this only seems to affect Xcode StoreKit testing and sandbox testing. Production builds using the live App Store DO NOT appear to suffer from this issue.
That's great, but you really want to be convinced during testing that your app works correctly! So, as a workaround, StoreHelper now maintains a transactionUpdateCache that keeps track of subscription transactions that happen when the app's not running. This cache is used in isPurchased(productId:) when a call to Transaction.currentEntitlement(for: productId) returns nil.

Thanks Russel. Can confirm that's working nicely now. FWIW, I'd found a decent workaround by introspecting currentEntitlements. That seemed to work in the Debugger, etc... Eventually, entitlements would be updated, but likely not in the background as you've mentioned, so you do have to be more generous in checking subscription status. That is, not simply relying on the state at the very beginning of app execution. The following code was working consistently via an extension on StoreHelper...
public func purchasedSubscription() async -> Product? {
    if self.hasStarted == false {
        await self.start()
    }

    let result : Set<ProductId> = await self.currentEntitlements()
    guard let productId = result.first else {
        return nil
    }

    guard let products : [Product] = await self.requestProductsFromAppStore(productIds: [productId]) else {
        return nil
    }
    guard let product = products.first else {
        return nil
    }

   return product
}


test: fix samples for reverse scans

Change-Id: Id03b5105afe609cae61ae84e484b53b1860daf92
Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

[ ] Make sure to open an issue as a bug/issue before writing your code!  That way we can discuss the change, evaluate designs, and agree on the general idea
[ ] Ensure the tests and linter pass
[ ] Code coverage does not decrease (if any source code was changed)
[ ] Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> ☕
If you write sample code, please follow the samples format.

Please note that the samples tests will fail until the next libraries-bom release


Bump JNA from 5.5 to 5.13

Description
JNA version of 5.5 is still used causing some issues running integTest on Mac M1/2.  JNA 5.7 introduced support for Mac aarch64.  I see that we are already using JNA 5.13 in 'server'.
Related Issues
Resolves #[Issue number to be closed when this PR is merged]

Check List

[ ] New functionality includes testing.

[ ] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[ ] Commits are signed per the DCO using --signoff
[ ] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@austintlee could you please fix DCO check? thank you

@reta it's a Github settings issue.  I think it's because my primary email does not match what's in my signature.
Commit sha: [1b6d88d](https://github.com/opensearch-project/OpenSearch/pull/9963/commits/1b6d88df30968a65be09eafe16c62670bf02ec3d), Author: Austin Lee, Committer: GitHub; Can not find "Austin Lee [austin.t.lee@gmail.com](mailto:austin.t.lee@gmail.com)", in ["Austin Lee [austin@aryn.ai](mailto:austin@aryn.ai)", "Austin Lee [austin@aryn.ai](mailto:austin@aryn.ai)", "Austin Lee [austin@aryn.ai](mailto:austin@aryn.ai)", "Austin Lee [austin@aryn.ai](mailto:austin@aryn.ai)"].
I switched my primary email, but I think we can override this check since I do have a proper DCO in every commit?  Or can you force it to re-run the check?  I don't know for certain exactly how Github is picking up that old email (I did use it for my first few contributions).

I switched my primary email, but I think we can override this check since I do have a proper DCO in every commit? Or can you force it to re-run the check?

We could override DCO check when the only option, but you could amend the last commit and do force push to fix the issue.

The problem is that that would require me to use an email that I don't want to put in the DCO.  Since I corrected my email in my Github settings, do you think we can rerun the DCO check?

@austintlee sadly needs manual backport to 2.x

I'm on it!


snappy-<IP_ADDRESS> fails to compile tests

Preprocessing test suite 'tests' for snappy-<IP_ADDRESS>..
Building test suite 'tests' for snappy-<IP_ADDRESS>..
[1 of 1] Compiling Main             ( tests/Properties.hs, dist/build/tests/tests-tmp/Main.o )

tests/Properties.hs:5:1: error:
    Could not find module ‘Functions’
    Use -v to see a list of the files searched for.
  |
5 | import Functions (rechunk)
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^

Skipping them for now.

See https://github.com/bos/snappy/issues/1.

Closing; this is reported on the snappy repo as linked above.


Node label alpha thing not working

In Preferences, selecting "alpha" for the node labels should only display alphabetically labelled nodes, but it is currently showing the numerically labelled nodes as well.

Fixed!


Wrong predictions after loading a saved model

Hi,
I have a trained a model which is a combination of keras and theano. I have the cifar10 convolution and pooling layers and then defined three dense layers in theano. Each dense layer gives a 10 dimension one-hot vector. To obtain the predictions, I use the theano function as below.
f_test = theano.function([model.layers[0].input],[dense.get_output(train=False) for dense in dense_layer])

I saved the weights of the dense layer and model separately. I have saved in two ways. I used both pickling as well as model.save_weights() for the convolution and pooling layers. For the dense layers, I used pickling and dense_layer.get_weights().
When I load the weights and run my predictions, I get a different values from when the predictions were obtained while training. My predictions are obtained as given below.
predictions = f_test(test_images)

The error is similar to issue #884. Can someone please tell me how to correct this?

@hemanth2090, did you figure this out? I'm having the same issue and am pretty stumped...


feat: create survey banner

closes #624

Still needs work :

on trigger condition
on cohabitation with existing NPS Banner
on UI


apply rustfmt (cargo fmt)

Note that the config is outdated:
Warning: Unknown configuration option `array_layout`
Warning: Unknown configuration option `fn_args_layout`
Warning: Unknown configuration option `fn_call_style`
Warning: Unknown configuration option `generics_indent`
Warning: Unknown configuration option `where_style`

I'm happy to amend the config to match your current stylistic preferences; one thing I love about rustfmt is that we can encode those preferences and then not have worry about them as much during review :)

I'll close this as we are currently working on redoing more or less the whole codebase, see the refactor branch and #46 for more information. I'll also update the README soon


Implement persistence

Released in v2.1.0

Quick question about this, how does it behave if several instances of gatus are using the same file?
Thanks for your work!

@rissson Unfortunately, that won't do anything constructive.
The only interactions with the file are:

At application start, read the file
Every 5 minutes, auto save
On application stop (i.e. On SIGKILL), try to save one more time before exiting

Every save operations overwrites the previous data persisted, thus, multiple applications pointing to the same file would just independently persist their data over the other applications. Since Gatus only reads from the file once on application start, each application wouldn't be influenced by any changes made by other applications on the file.
TL;DR: Persistence is really just to ensure that the data can be read on the next application start, once the data is read, the entire dataset is in memory and any changes made to the file is inconsequential, thus, the current persistence implementation cannot be used for synchronization between multiple Gatus instances, which I assume is why you're asking.

Actually, my question wasn't exactly about syncing multiple Gatus instances. My main problem is that my Gatus pods restart quite frequently:
devoups-gatus-6d6c579b67-4hmv4   1/1     Running   798        5d4h
devoups-gatus-6d6c579b67-82xw6   1/1     Running   797        5d4h
devoups-gatus-6d6c579b67-84r8s   1/1     Running   276        2d16h
devoups-gatus-6d6c579b67-n5ptf   1/1     Running   418        2d16h
devoups-gatus-6d6c579b67-w94wv   1/1     Running   418        2d16h

According to my logs, they get OOM killed by the kernel. So, the only thing I'm really looking for is whether they'll get some state after they're back up. Thus, it doesn't really matter if they don't sync their state. The only question left is what happens if two instances try to write to the file at the same time.

@rissson If you care about persistence, you may want to use a StatefulSet instead, but before even worrying about that, I'd suggest that you use only a single pod for Gatus (unless you're using it for stress test purposes).
Furthermore, you should give the deployment a bit more memory if it's getting OOMKilled that often.
Would you mind showing me the output of kubectl get deploy devoups-gatus -o yaml as well as the output of kubectl top pods | grep devoups-gatus?
For reference, I use a single pod for a Gatus instance that has 50+ services to monitor and it hasn't restarted a single time in the past 26 days.

Here is kubectl get deploy devoups-gatus -o yaml: http://ix.io/2Rza. kubectl top pods is not working on my cluster somehow. Looking for the process manually on the nodes, they all use about 40M of RAM.

@rissson Since Gatus keeps all of its data in memory, if you have enough monitored services to cause the instance to use more than 30M of memory, I suggest that you increase your memory request to 40M and the limit to 100M.
i.e. Update the following:
        resources:
          limits:
            cpu: 200m
            memory: 50M
          requests:
            cpu: 50m
            memory: 20M

to
        resources:
          limits:
            cpu: 200m
            memory: 100M
          requests:
            cpu: 50m
            memory: 40M

As I've mentioned, I also suggest you decrease the number of replicas (spec.replicas) from 5 to 1, as Gatus is meant to be run on a single instance, since it doesn't benefit from using multiple instances because they don't share their data with other instances (unless you're using Gatus for stress testing purposes, in which case using multiple instances is reasonable).
P.S. For kubectl top pods to work, you must have metrics-server running in your cluster.

I increased the limits as you suggested, no restart since. Again, thanks a lot for your wonderful work!

@rissson Glad to hear it! 🥳


Put extensions to release, add artifacts

Aha, it is done, here is a test release: https://github.com/aeternity/superhero-wallet/releases/tag/test


JavaScript errors when refreshing the page

I confirm (by marking "x" in the [ ] below):

[X] This is not a troubleshooting question. Troubleshooting questions go here: http://www.mattermost.org/troubleshoot/.
[X] This doesn't reproduce on web browsers (such as in Chrome). If it does, issue reports go to the Mattermost Server repository.
[X] I have read contributing guidelines.

Summary
JavaScript errors when refreshing the page
Steps to reproduce
Windows 10; Mattermost Desktop v3.6; Mattermost Server v3.6

Go to View > Toggle Developer Tools
On the Developer console, choose "Console"
Refresh the page

Expected behavior
No JavaScript errors
Observed behavior

I'll have a look right away. I don't remember those messages to be honest but maybe I introduced them with one of the later commits of my PR and simply didn't check again.


Activate amplifier & held gpio buttons

Hi,
hopefully I have done everything right now with branching and pulling. ;-) . It is late and I had many fun.

Added an service for activating my amplifier. I have a PAM8403 and with a little hardware hack on the PCB I was able to switch the amplifier on via GPIO and default is it off.
I need to write a little bit of documentation for that hw hack but now it's to late. The HW modification is on this picutre.

I've implemented a function when holding the e.g. Volume Up button for more then 0.3 seconds it will raise in 0.3 second intervals the volume based on configured volume steps.

Kind regards
Marco

Hi @marcohorstmann
this is the right way to develop and create pull requests.
What's missing: please add the information you added here in the comments as a text file (or better even: markdown) do the docs folder. So others can benefit from the work.

Hi needs so days to add this. In the Moment i play with Home Assistant and used my dev Pi for it.

@MiczFlor I see you added a wiki page. will it better that I write down the GPIO part and this hardware hack into the wiki than the folder?

Hi @marcohorstmann
if you could give this a try, that would be great. I am not sure if you have access to edit. But I set it to allow all github registered users to edit the wiki.
Not sure if the wiki is the right place for the documentation, actually. We will see. Please give me some feedback.

@MiczFlor I added the info to the wiki. The GPIO file in docs can be deleted now because infos are in Wiki.
Only thing I had not get to work was to add pictures to my new wiki page for amplifier. I tried to add this files here via Github desktop client and commited to wiki master two files but I cannot see them.

@MiczFlor checked and I can confirm that wifi toogle works as expected after merge.

@marcohorstmann
I committed the image to the readme in the wiki, but I could only find one image. Please post the other one here and tell me which one I need to replace. The first or the second?
https://github.com/MiczFlor/RPi-Jukebox-RFID/wiki/Hardware-Hack-PAM8403-Poweroff

You can stop the service and start it with a little delay before resume. But you need to e.g. add this to gpio buttons.py. With a little sleep and finding out if mpd is actually playing.
But I have no time actually to test it. i‘m very busy at my work


License command - Create a stdout table with [module name | license] fields

We already have the module name + license content.
It would be good to show that content using cli-table3 instead of console.log

Closing that for now related to https://github.com/nodeshift/oratrix/issues/50


[ui] Improve font displays in all GUIs

Font display looks dithered and generally bad using the Swing defaults.  For instance, the file chooser dialog window font for file paths does not look polished at all.
Identify a few better font choices and present them to the team for a concensus, then implement that change in the GUI for all areas.

Consider using a look/feel to package with the application to alleviate problems with cross platform font support


More output data to match capy

There are following differences with capy's output:

status field: to indicate how parsing went, e.g. "file not found", "missing file" (fgosccnt detects missing screenshots by checking current vs previous scroll bar state), etc.
image_path field: the path of the parsed image,
for items (most extra fields are due to them already being present in fgosccnt anyway):

id field: uses DW's item ids as suggested by Cereal
name field: name of the item. The name of the image file used to match
dropPriority field: apparently something used to sort stuff in the UI by DW
bonus field: the bonus gained for the dropped item
category field: classifies item more or less. Can be "Quest Reward" for fixed QP, "Craft Essence", "Exp. UP" for embers (I assume, not tested), and "Item" for any other item.
no score field as it's not present in fgosccnt afaik

fgosccnt reports drops in a different way. If it detects the screenshot to not be the 1st page, aka it is scrolled down, it sometimes ignores the top row. This is to prevent duplication of items, as that row is already supposed to be counted in on the 1st page.

fgosccnt (left) not counting top row, while capy (right) does.
fgosccnt also includes QP drops, while capy doesn't. QP could be ignored as well if we use the category and allow that field to detect QP drops.

fgosccnt (left) reports 9 drops, while capy (right) only reports 4.
fgosccnt was unable to parse da_vinci.png and Screenshot_20190329-114404_Fate_GO.jpg from capy's examples. Still looking into this.

I think it also needs json file output. If there is currently an output location flag for the csv file, we can probably use it.

I think it also needs json file output. If there is currently an output location flag for the csv file, we can probably use it.

Original fgosccent simply prints a csv format. I've completely removed the csv, and replaced it with json output. Do we still need csv format?
Also, I think redirecting stdout should be easy enough. Or should a cli arg be added to specify an output file?

capy in production mode writes the json file so I would add it.
@sadisticsolution I assume you would still need the output jsons?

Yes to adding a cli flag to write output files.
That fgosccnt's behavior should be removed then. We can handle matching screenshots downstream.

Added hard coded values for the QP's location in images. This is in case fgosccnt fails to detect them due to obfuscation of the area, e.g. for capy's example Screenshot_20190329-114404_Fate_GO.jpg.
All other issues known to me have been resolved. Ready to merge after peer confirmation.


Idea: server offers to generate a config salt if the user didn't set one up

Just thought of this.
Why not, if the server detects someone didn't change the configuration file salts that it either a) throws an ascii art of Naka-chan and quits or b) asks the user if they want to let it make a random 16 - 32 character string?
I know this is promoting lazy "eh just download and lol oss kc server".

I'll probably make it just print a "you are shit" error at somepoint.


Update workflows to admb 13.1

Satisfies issue #392 in the stock synthesis repository.

Also, nice to have the "squash and merge" option so that all the exploratory commits in this branch can be squashed down to just one. I'll let you take that step @e-gugliotti-NOAA.


[SPARK-31179] Fast fail the connection while last connection failed in fast fail time window

What changes were proposed in this pull request?
For TransportFactory, the requests sent to the same address share a clientPool.
Specially, when the io.numConnectionPerPeer is 1, these requests would share a same client.
When this address is unreachable, the createClient operation would be still timeout.
And these requests would block each other during createClient, because there is a lock for this shared client.
It would cost connectionNum * connectionTimeOut * maxRetry to retry, and then fail the task.
It fact, it is expected that this task could fail in connectionTimeOut * maxRetry.
In this PR, I set a fastFail time window for the clientPool, if the last connection failed in this time window, the new connection would fast fail.
Why are the changes needed?
It can save time for some cases.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Existing UT.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

cc @cloud-fan

are you saying we should fail fast for bad clients? When/How a client can go "bad"?

Thanks for the reply.
We meet this issue when ESS(node manager) is busy for full gc, and the task costs long times(connectionTimeoutmaxRetry(number of requests to the same ESS)) and then became fetch failed.
We expect the task could fast fail instead of wasting too much times to wait the lock of client.

I think it may happen for these case below:

nm GC
nm crash
temporary network issue

makes sense to me to treat conf.ioRetryWaitTimeMs as per client not per task, as multiple tasks may share the same client.
cc @squito @vanzin  @jiangxb1987 @Ngone51

OK to test

ok to test

retest this please

Test build #119986 has started for PR 27943 at commit 73d2a63.

Test build #119986 has finished for PR 27943 at commit 73d2a63.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119986/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24707/
Test PASSed.

Test build #119989 has started for PR 27943 at commit 5d40e8d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24710/
Test PASSed.

Test build #119989 has finished for PR 27943 at commit 5d40e8d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119989/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24718/
Test PASSed.

Test build #119998 has started for PR 27943 at commit fa266f2.

Test build #119998 has finished for PR 27943 at commit fa266f2.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/119998/
Test FAILed.

IIUC throw an IOException here would eventually lead to task failure with FetchFailedException? If that's really the case then I would be really conservative here and wait instead of fail fast.

How about that, if the last connection failed in the last retry io wait, the new connection would be break but its retry count would not increase.

How about that, if the last connection failed in the last retry io wait, the new connection would be break but its retry count would not increase.

So here, you don't want to reduce the connection chances for the request/task in case of it may could establish connection successfully without giving up connecting within ioRetryWait?
If so,  I think it can be a valid concern but really make things complicate and I'm wondering if it's worth to do so.
And at least, as per @tgravescs comment, I think we should record the failed time for the whole ClientPool instead of every client from the same ClientPool.

I think the problem can be described as: the client pool has no more healthy clients, and we need to run a task, what shall we do?
Currently we just run and timeout 3 times, and this PR proposes to fail fast.

Currently we just run and timeout 3 times, and this PR proposes to fail fast.

We should not be failing without retrying.  Is that really what this does?  I'd have to take a closer look but I thought the RetryingBlockFetcher caught this and did its normal retries within it, but that was my question yesterday to confirm?

Just attach the example mentioned in the description.

For example: there are two request connection, rc1 and rc2.
Especially, the io.numConnectionsPerPeer is 1 and connection timeout is 2 minutes.
1: rc1 hold the client lock, it timeout after 2 minutes.
2: rc2 hold the client lock, it timeout after 2 minutes.
3: rc1 start the second retry, hold lock and timeout after 2 minutes.
4: rc2 start the second retry, hold lock and timeout after 2 minutes.
5: rc1 start the third retry, hold lock and timeout after 2 minutes.
6: rc2 start the third retry, hold lock and timeout after 2 minutes.
It wastes lots of time.

The concern is that, for some case, these request connections block each other.
If the rc1 connect timeout, then we fast break the first retry of rc2 but don't increase the retry count of rc2.
Then rc1 will wait a io retry wait, and then start the second retry and connect timeout.
Then we fast break the rc2 and don't increase its retry count.
Then rc1 will wait a io retry wait, and then start the third retry and connect timeout, then rc1 throw fetch failed exception.
I think it is better than the request connections block each other.

Currently we just run and timeout 3 times, and this PR proposes to fail fast.

We should not be failing without retrying. Is that really what this does? I'd have to take a closer look but I thought the RetryingBlockFetcher caught this and did its normal retries within it, but that was my question yesterday to confirm?

@cloud-fan @tgravescs  IIUC, this PR only fail fast in a single one connection try but will still retry if it's a RetryingBlockFetcher.

Currently we just run and timeout 3 times, and this PR proposes to fail fast.

We should not be failing without retrying. Is that really what this does? I'd have to take a closer look but I thought the RetryingBlockFetcher caught this and did its normal retries within it, but that was my question yesterday to confirm?

@cloud-fan @tgravescs IIUC, this PR only fail fast in a single one connection try but will still retry if it's a RetryingBlockFetcher.

In fact, the current implementation in this patch would fast fail all connections.
I just propose a compromise solution that just fast fail a single one connection in the comments.
I prefer to fast fail all connections related with the unreachable ESS.

So I'm really not clear on what the desired behavior is and I think we are using different terminology here.  I would like you to clearly define what you want the desired behavior to be compared to now.
@turboFei  you say "In fact, the current implementation in this patch would fast fail all connections."
What is all connections?  If a single task fails to fetch from an address (host/port) is the intention for you to fail that task and all other subsequent tasks trying to fetch from that address immediately and not retry?  If so I'm against this. The retry is there for a reason, many times we see temporary issues with shuffle servers on node managers that on retry it works fine. People can configure the number of retries and wait based on how they want their job to act.
If that is not what you are proposing please describe in detail what you want the first task that fails to do, subsequent tasks fetching from same address, and then what happens in the retrying fetcher with retries.

Thanks for the reply @tgravescs
Sorry for the unclear description.
All connections I mentioned above is that the sent request connections to the same unreachable address.
It is my mistake that does not recognize  that there maybe several clients for the same address, may be we need keep a lastConnectionFailedTime variable for one clientPool.
The problem is that, for a task, there maybe several request connections to the same address.
Specially, for a shuffle read task and there is only one client in the client pool and it would always been picked by the connections, which want to connect the same ESS.
If this address is unreachable, these connections would block each other(during createClient).
These connections owned to a same task and want to connect the same ESS,  if this ESS was still unreachable.
It  would cost connectionNum * connectionTimeOut * maxRetry to retry, and then fail the task.
It is ideally that this task could fail in connectionTimeOut * maxRetry.

ok thanks for the description. So are you only trying to fail faster when only fetching blocks for the same task?  Or Also across other tasks fetching from same shuffle server?
numConnectionsPerPeer = 1 by default so by default you get a client pool of size 1 but its configurable so it needs to be able to handle the case its bigger.  Yeah I think if you just have something per clientPool it would handle that.
I also wonder if it shouldn't be like 95% of conf.ioRetryWaitTimeMs() or something to make sure there isn't race condition in when the first one actually tries again. I would think there is enough delay in retrying to not hit that exactly but might be better to be safe.
The only other question I have is connections not going through the retryingblockfetcher, this could potentially fail them much faster, if its a one time fetch is that what we want. I would need to look a bit more at the usages there.

Agree the fail fast time window length should be a little less than conf.ioRetryWaitTimeMs() so you will not fail the retried connection immediately. Also I'm curious could this lead to the scenario that, when you have two tasks and only one client, the connection request from the second task may fail fast every time it tries to connect (because the connection from the first task always fail beforehand)?

Test build #120266 has started for PR 27943 at commit 62eedae.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24977/
Test PASSed.

Agree the fail fast time window length should be a little less than conf.ioRetryWaitTimeMs().

The only other question I have is connections not going through the retryingblockfetcher, this could potentially fail them much faster, if its a one time fetch is that what we want. I would need to look a bit more at the usages there.

I think we can only fast fail the connection if conf.maxIORetries >0.

Also I'm curious could this lead to the scenario that, when you have two tasks and only one client, the connection request from the second task may fail fast every time it tries to connect (because the connection from the first task always fail beforehand)?

I think if the  service address is still unreachable after maxIORetries retries, it does not matter to fail the another task.

Test build #120269 has started for PR 27943 at commit 02537b3.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/24980/
Test PASSed.

Test build #120266 has finished for PR 27943 at commit 62eedae.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120266/
Test PASSed.

Test build #120269 has finished for PR 27943 at commit 02537b3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120269/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25010/
Test PASSed.

Test build #120298 has started for PR 27943 at commit 9ff5ca2.

Test build #120298 has finished for PR 27943 at commit 9ff5ca2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120298/
Test PASSed.

Test build #120312 has started for PR 27943 at commit dd126cf.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25022/
Test PASSed.

LGTM

Test build #120312 has finished for PR 27943 at commit dd126cf.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120312/
Test FAILed.

UT had passed before, the latest test is killed manually.
cc @cloud-fan  @Ngone51

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25044/
Test PASSed.

Test build #120333 has started for PR 27943 at commit dd126cf.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25045/
Test PASSed.

Test build #120333 has finished for PR 27943 at commit dd126cf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120333/
Test PASSed.

I'll take a closer look this afternoon, but can you update the description to just mention that retries still take affect since we had discussion about that.

thanks, have updated the description.

have addressed the comments and added ut

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25098/
Test PASSed.

Test build #120389 has started for PR 27943 at commit 3ef1612.

Test build #120389 has finished for PR 27943 at commit 3ef1612.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120389/
Test FAILed.

Test build #120402 has started for PR 27943 at commit 8784906.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25111/
Test PASSed.

Test build #120402 has finished for PR 27943 at commit 8784906.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120402/
Test FAILed.

retest this please

retest this, please

Test build #120412 has started for PR 27943 at commit 8784906.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25120/
Test PASSed.

Test build #120412 has finished for PR 27943 at commit 8784906.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120412/
Test PASSed.

thanks @HeartSaVioR
test passed cc @Ngone51 @tgravescs @jiangxb1987  @cloud-fan

Test build #120423 has started for PR 27943 at commit d7a1ad5.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25131/
Test PASSed.

Test build #120426 has started for PR 27943 at commit 3246606.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25134/
Test PASSed.

Test build #120427 has started for PR 27943 at commit 8e354da.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25135/
Test PASSed.

Test build #120427 has finished for PR 27943 at commit 8e354da.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120427/
Test FAILed.

Merged build finished. Test FAILed.

Test build #120426 has finished for PR 27943 at commit 3246606.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120426/
Test FAILed.

Test build #120431 has started for PR 27943 at commit 7af60c8.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25139/
Test PASSed.

Test build #120423 has finished for PR 27943 at commit d7a1ad5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120423/
Test PASSed.

Test build #120431 has finished for PR 27943 at commit 7af60c8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120431/
Test FAILed.

Test build #120448 has started for PR 27943 at commit 66adc09.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25156/
Test PASSed.

Test build #120448 has finished for PR 27943 at commit 66adc09.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120448/
Test PASSed.

Thank you for your patient review, thanks a lot @tgravescs

Test build #120498 has started for PR 27943 at commit 0b5dfe1.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25204/
Test PASSed.

Test build #120502 has started for PR 27943 at commit 8942ae5.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25208/
Test PASSed.

Test build #120502 has finished for PR 27943 at commit 8942ae5.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120502/
Test FAILed.

Test build #120505 has started for PR 27943 at commit 49c8e13.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25211/
Test PASSed.

Test build #120498 has finished for PR 27943 at commit 0b5dfe1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120498/
Test PASSed.

Test build #120505 has finished for PR 27943 at commit 49c8e13.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120505/
Test PASSed.

cc @travishegner

gentle ping @tgravescs

Test build #120661 has started for PR 27943 at commit fe8c08c.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25360/
Test PASSed.

Test build #120661 has finished for PR 27943 at commit fe8c08c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120661/
Test PASSed.

sorry it looks like a previous comment I typed originally got put here, removed that. Just minor doc updates and then the parameter setting when maxIORetries is 0

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25381/
Test PASSed.

Test build #120682 has started for PR 27943 at commit 9739b8a.

Test build #120682 has finished for PR 27943 at commit 9739b8a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120682/
Test PASSed.

Test build #120718 has started for PR 27943 at commit f9be82e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25416/
Test PASSed.

Test build #120718 has finished for PR 27943 at commit f9be82e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120718/
Test PASSed.

merging to master

I have applied this patch to our in-house spark distribution and run for more than one week. Sometimes, making things worse, especially, when the shuffle server JUST in GC STW periods but not eventually become unreachable. Tasks become more sensitive to fail and stage retries occur which may be unnecessary at all.  Also, the upper limit of spark.stage.maxConsecutiveAttempts is much easier to hit and fail the job rather than to get more chances to recover.
org.apache.spark.shuffle.FetchFailedException:
 Connecting to xxxx failed in the last 5700 ms, fail this connection directly

thanks for the feedback. In some ways that is what I would expect, it is to fail faster. You should be able to adjust
spark.shuffle.io.maxRetries and spark.shuffle.io.retryWait.  I think its hard to balance people that want it to fail fast vs ones who want to run no matter what as well as nodes just GCing vs really have issues.  Are you using blacklisting?

Thanks, @tgravescs for the quick reply.
Yes, I have considered adjusting spark.shuffle.io.maxRetries and spark.shuffle.io.retryWait and hope that would help. I have one more question here, does the first task that failed to create the client and set the lastConnectionFailed  will hit the fast fail logic when it retries later?
Blacklisting is enabled.

do you see the nodes being blacklisted or are there not enough failures to cause blacklisting?  In general though it sounds like its GC or some other slowness and the node manager comes back fairly quickly?  Can I ask what your settings are now and perhaps how long of GC pauses you are seeing?

I have one more question here, does the first task that failed to create the client and set the lastConnectionFailed will hit the fast fail logic when it retries later?

No, its supposed to fail any attempts that happen after the first task failed up til 95% of the spark.shuffle.io.retryWait timeout.  At that point it should let any create calls go through to try again. That means the first task that failed should be trying to connect again.  Now I guess there is the possibility that if another task create call came in at that 95% retryWait time that task may try again rather then the first task that failed, but some task should try again at that point.

do you see the nodes being blacklisted or are there not enough failures to cause blacklisting?

The driver log was too large for me to do analysis, I didn't pay much attention to blacklisting. The heavy load is from the NodeManager(shuffle server), IIUC, we do not have a blacklist rule for External Shuffle Server?

Can I ask what your settings are now and perhaps how long of GC pauses you are seeing?

The workload that this spark job take is very heavy, it has upper limited 3000 executors with 4 cores and 20g(memory+overhead) each,  retryWait(6s) and max retries(5), the largest stage has 60k tasks, the biggest input table has 16~20TB data and the one single largest shuffle read can up to 50TB compressed data. The pause that causes the client timeout is NM being in high IOWAIT status. The reason why the IOWAIT is extremely high is due to the high workload itself and dynamic allocation sometimes makes shuffle write data distributed unevenly. In the 2 joint shuffle write/read stages, during the write stage, the earlier start executors get more task to run and produce more shuffle blocks to its local Shuffle server(NM). These NMs are more likely to suffer from high iowait offering more data during the read stage because the guava cache hit for shuffle index files can be greatly reduced.
Before applying this patch, the large connectionTimeout(default 120s) and those unnecessary retries seems to give the tasks more chance to recover. After applying this patch, the shuffle server-status become a sort of immediate factor for clients to do retries. So the tasks become much more sensitive to fail. I have to admit that the retryWait(6s) and max retries(5) is too smaller for that particular job after fastfail. But now I am wondering that whether the default retryWait and maxRetries are still suitable for general cases with fastfail.

No, its supposed to fail any attempts that happen after the first task failed up til 95% of the spark.shuffle.io.retryWait timeout. At that point it should let any create calls go through to try again. That means the first task that failed should be trying to connect again.

Ah, I see, that task will pass that window time and retry creating client again if there is still none exists.

See config:
spark.blacklist.application.fetchFailure.enabled (http://spark.apache.org/docs/latest/configuration.html)
But here you need to be careful with this as well if its just an intermittent type failure where it would recover shortly from.
ok, yeah your node manager are probably very busy and possibly disks are pegged with all the shuffle and different requests.   Some of that can be helped with the shuffle configs like spark.reducer.maxBlocksInFlightPerAddress but those aren't guaranteed to solve your problems, basically throttling heuristics.
Yeah the issue was that the wait time before wasn't really what it was waiting, it could be waiting much longer.  I think it makes sense for you to increase the retryWait time and give it a try.  I know on certain jobs we run where it isn't necessarily all about speed but we really want it to finish no matter what we make those pretty high.

Thanks, @tgravescs,
The spark.blacklist.application.fetchFailure.enabled looks too risky for me.
The spark.reducer.maxBlocksInFlightPerAddress looks interesting and very useful as it is documented, but I think I have to do more testing before production use.
Increasing maxRetries and retryWait is the way I am going to try first. But I think increasing maxRetries is much better, as most retries will fail directly and fast on the client-side with no requests to servers, and short retryWaits can give clients more opportunity to get connected with servers.

BTW, this is a good improvement. But we should pay more attention to imperceptible change on
maxRetries and retryWait although they just come back to what they exactly mean.
The reason why I urgently patched this to our in-house distribution is that this same spark job I mentioned above was delayed for hours just because of one single NodeManager crashed during the shuffle read phase.
The 2 stages were aborted and resubmitted but most of the executors were not freed. One thing is that we don't kill the tasks from the aborted stages but let them fail by themselves, the other is these tasks which were doomed to fail are in the issue was that the wait time before wasn't really what it was waiting. However, the tasks in the new attempt could not get enough resources to run.
When the hosts for client and server are in the same network segment, NoRouteToHostException will immediately fail the connecting attempt and the task eventually when the maxReties exhausted, but when they are in the different network segments, it will wait for connectionTimeOut  util get SocketException of the connection timeout and retry... In general, the hosts are more likely to be in the different network segments for large clusters, that's one reason why the fast fail here matters.
As this is merged into master only, I hope the story I share will help others when they are looking for something just like I did.

I was running with spark.blacklist.application.fetchFailure.enabled in production at Yahoo, but our default timeouts and retries were much higher. It worked well for us.  We had also configured the various configs for spark.reducer.maxBlocksInFlightPerAddres and trying to throttle the large jobs from overwhelming the node managers though too. The scenario you mention I think could benefit from that because the node manager is dead and isn't coming back. You can tie this with  spark.blacklist.killBlacklistedExecutors. But agree with you, I definitely suggest testing and rolling out rather then throwing into production.

Thanks for your sharing @tgravescs,
These configs you offer give me more ways to optimize such kinds of jobs with heavy shuffle loads.


Botomongo - S3 spans throwing an "validation error: span: context: destination: service: 'name' required"

Flask Python application doesn't record the SPAN with S3 call:
To Reproduce

Flask Application doing S3 call
In console you will see the exception Failed to submit message: 'HTTP 400: {"accepted":3,"errors":[{"message":"validation error: span: context: destination: service: \'name\' required",...

Environment (please complete the following information)

OS: Linux
Python version: 3.6
Framework and version: Flask 1.1.2
APM Server version: v7.12.0
Agent version: 6.1.0

From the APM version 7.12 name field is required
Problem is located here:
elasticapm/instrumentation/packages/botocore.py
context["destination"]["service"] = {"type": span_type}
for destination.service there is no destination.service.name element
IMHO: destination.service.name should be set as in the elasticapm/instrumentation/packages/elasticsearch.py

@tonyman19 thanks for the heads up! I opened a PR with a fix in #1090. Will do a release ASAP!

Reviewed the PR - looks good!

@tonyman19 thanks for the review! The fix is released in version 6.1.1 of the agent. Thanks again for the report!


refactor: update deps + upgrade example structure to use a clean approach

Codecov Report

Merging #67 (c505b09) into master (55a6635) will decrease coverage by 6.98%.
The diff coverage is 87.12%.

@@            Coverage Diff             @@
##           master      #67      +/-   ##
==========================================
- Coverage   93.06%   86.08%   -6.99%
==========================================
  Files          13       25      +12
  Lines         101      194      +93
  Branches       13       27      +14
==========================================
+ Hits           94      167      +73
- Misses          7       27      +20

Impacted Files
Coverage Δ

src/shared/components/Header/index.tsx
100.00% <ø> (ø)

src/shared/locales/translations.ts
100.00% <ø> (ø)

src/core/Counter/counterReducers.ts
42.85% <42.85%> (ø)

src/core/Counter/CounterViewModel.ts
50.00% <50.00%> (ø)

src/core/Counter/counterActions.ts
60.00% <60.00%> (ø)

src/core/Counter/CountUseCase.ts
81.25% <81.25%> (ø)

src/app.tsx
100.00% <100.00%> (ø)

src/config/index.ts
100.00% <100.00%> (ø)

src/core/Counter/CounterActionTypes.ts
100.00% <100.00%> (ø)

src/core/Counter/CounterView.tsx
100.00% <100.00%> (ø)

... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 55a6635...c505b09. Read the comment docs.


Added FFG-LMD vote consistency checks

Added consistency check for FFG & LMD vote in validate_on_atttestation(), fixes #1636, fixes #1456, fixes #1408

@adiasg can you rebase this to v012x branch to queue it up for the v0.12 release?

@adiasg Looks good! added a test if you want to check it out before merge


move config and add ToConfig

Thank you for contributing to Coraza WAF, your effort is greatly appreciated
Before submitting check if what you want to add to coraza list meets quality standards before sending pull request. Thanks!

Make sure that you've checked the boxes below before you submit PR:

[x] My code includes positive and negative tests.
[x] I have an appropriate description with correct grammar.
[x] I have read Contribution guidelines, maintainers note and Quality standard.
[x] My code is properly linted and passes pre-commit tests.

Thanks for your contribution :heart:

Codecov Report
Base: 78.08% // Head: 76.98% // Decreases project coverage by -1.10% :warning:

Coverage data is based on head (c162b12) compared to base (63a49c8).
Patch coverage: 5.66% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##           v3/dev     #521      +/-   ##
==========================================
- Coverage   78.08%   76.98%   -1.11%
==========================================
  Files         145      146       +1
  Lines        6876     6981     +105
==========================================
+ Hits         5369     5374       +5
- Misses       1270     1370     +100
  Partials      237      237

Flag
Coverage Δ

default
72.30% <5.66%> (-1.05%)
:arrow_down:

examples
27.22% <66.66%> (+0.02%)
:arrow_up:

ftw
55.44% <66.66%> (+<0.01%)
:arrow_up:

tinygo
70.21% <5.66%> (-1.09%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

config/config.go
0.00% <0.00%> (ø)

waf.go
50.00% <100.00%> (+4.23%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

Is this still needed @jptosso ?

Might be required when we come back to nginx

LEt's close it and open back if needed.


Fill in all target configurations

fixes https://github.com/AngeloAvv/flutter_flavorizr/issues/184

It weirdly nuking all macos configs but works great on ios. @AngeloAvv Can you take a look?

Hey @JasCodes , what if you do it starting from native_target instead? Do macOS configs get erased too?
native_target = project.native_targets

I don't remember PR 😂, So what you may with it.


try and fix the notary issues

Uses new secrets and forces code signing.
Also updates to new version of relocatable-python that has some additional fixes/features.

🥳

finally
./recommended/payload/Library/ManagedFrameworks/Python/Python3.framework/Versions/3.9/Python: rejected
source=Unnotarized Developer ID
origin=Developer ID Application: Clever DevOps Co. (9GQZ7KUFR6)
./recommended/payload/Library/ManagedFrameworks/Python/Python3.framework/Versions/3.9/lib/libssl.1.1.dylib: rejected
source=Unnotarized Developer ID
origin=Developer ID Application: Clever DevOps Co. (9GQZ7KUFR6)

Current status: Accepted.................Processing complete
  id: ad883496-392c-4047-b130-eb13d2cf82d0
  status: Accepted


PR: Rename donejs-* repos that are not a donejs generator!

i found out that many repos here are sharing name space with the reserved donejs-* name space thats used for donejs-* generators
my suggestion rename them if they are not generators to done-

can you give examples?

https://github.com/donejs/donejs-streaming-test-comparison
https://github.com/donejs/donejs-streaming-product-page

We're not going to restrict the names people can use for donejs repositories, nor do github repository names interfere with generators.

@phillipskevin but you remember that the donejs binary will auto install and download packages called donejs-* ?
but ok its fine for me as i don't rellay on that tooling feel free


Crawl ThaiSubtitle Website

http://www.thaisubtitle.com/forums/index.php

From 27446 Episodes that translate completed only 12046 episodes.
Dataset : nakcnx/thai_subtitle (Huggingface)


3D editor transform gizmo is no longer visible by default

Tested versions
master
System information
win10 21H2,Vulkan API 1.2.0
Issue description
https://github.com/user-attachments/assets/2752e0eb-e33e-4884-8d70-20947f21d628
Steps to reproduce
look video
Minimal reproduction project (MRP)
none

Could you check if "View Transform Gizmo" option is turned on?

I tested master (v4.4.dev.custom_build [6c05ec3d6]) rn and indeed this version is starting new projects with View Transform Gizmo being false for some reason.
@clayjohn I think this issue is not OS related, but if is the author OS is windows, but you put as android 😄

Bisected to #96922, @Nodragem

I will have a look tomorrow at how to solve the issue. For context, my PR intends to solve some input conflicts between Gridmap editor and 3d editor. When a Gridmap is selected, there are several input modes (e.g. transform, selection, paint, erase, picker), and I make the gizmo only visible and usable when the user selects the Transform mode.
The intended behaviour is that the gizmo is made invisible when a Gridmap is selected and it is made visible again if the transform mode is selected or if the Gridmap is deselected.

I checked a bit no this, and running a gdb session on a empty project.godot I managed to pinpoint the trace at the end.
TLDR : while le plugin is loaded, calling set_pressed on select_mode_button (grid_map_editor_plugin.cpp:1354) executes connected call _on_tool_mode_changed which set default viewport View Transform Gizmo parameter to false,
as created scenes viewport options seems to carry form the last one, it carries on.
The relevant stack trace :
Thread 1 "godot.dev" hit Breakpoint 1, GridMapEditor::_show_viewports_transform_gizmo (this=0x555574c12e70, p_value=false)
    at modules/gridmap/editor/grid_map_editor_plugin.cpp:635
635             new_state["transform_gizmo"] = p_value;
(gdb) where
#0  GridMapEditor::_show_viewports_transform_gizmo (this=0x555574c12e70, p_value=false)
    at modules/gridmap/editor/grid_map_editor_plugin.cpp:635
#1  0x00005555574c3677 in GridMapEditor::_on_tool_mode_changed (this=0x555574c12e70)
    at modules/gridmap/editor/grid_map_editor_plugin.cpp:1255
#2  0x00005555574cb355 in call_with_variant_args_helper<GridMapEditor>(GridMapEditor*, void (GridMapEditor::*)(), Variant const**,
 Callable::CallError&, IndexSequence<>) (p_instance=0x555574c12e70,
    p_method=(void (GridMapEditor::*)(GridMapEditor * const)) 0x5555574c362c <GridMapEditor::_on_tool_mode_changed()>,
    p_args=0x7fffffffc2e0, r_error=...) at ./core/variant/binder_common.h:304
#3  0x00005555574cac2b in call_with_variant_args<GridMapEditor> (p_instance=0x555574c12e70,
    p_method=(void (GridMapEditor::*)(GridMapEditor * const)) 0x5555574c362c <GridMapEditor::_on_tool_mode_changed()>,
    p_args=0x7fffffffc2e0, p_argcount=0, r_error=...) at ./core/variant/binder_common.h:418
#4  0x00005555574ca3fa in CallableCustomMethodPointer<GridMapEditor, void>::call (this=0x555574c6a180,
    p_arguments=0x7fffffffc2e0, p_argcount=0, r_return_value=..., r_call_error=...)
    at ./core/object/callable_method_pointer.h:107
#5  0x000055555aecb156 in Callable::callp (this=0x555574c6a1f0, p_arguments=0x7fffffffc2e0, p_argcount=0, r_return_value=...,
    r_call_error=...) at core/variant/callable.cpp:57
#6  0x000055555aecf81d in CallableCustomUnbind::call (this=0x555574c6a1e0, p_arguments=0x7fffffffc2e0, p_argcount=1,
    r_return_value=..., r_call_error=...) at core/variant/callable_bind.cpp:260
#7  0x000055555aecb156 in Callable::callp (this=0x7fffffffc120, p_arguments=0x7fffffffc2e0, p_argcount=1, r_return_value=...,
    r_call_error=...) at core/variant/callable.cpp:57
#8  0x000055555b29873e in Object::emit_signalp (this=0x555574c66420, p_name=..., p_args=0x7fffffffc2e0, p_argcount=1)
    at core/object/object.cpp:1200
#9  0x0000555558e8f869 in Node::emit_signalp (this=0x555574c66420, p_name=..., p_args=0x7fffffffc2e0, p_argcount=1)
    at scene/main/node.cpp:3986
#10 0x00005555576a47ed in Object::emit_signal<bool> (this=0x555574c66420, p_name=...) at ./core/object/object.h:920
#11 0x0000555558fa0fc3 in BaseButton::_toggled (this=0x555574c66420, p_pressed=true) at scene/gui/base_button.cpp:144
#12 0x0000555558fa1577 in BaseButton::set_pressed (this=0x555574c66420, p_pressed=true) at scene/gui/base_button.cpp:229
--Type <RET> for more, q to quit, c to continue without paging--
#13 0x000055555635de3e in GridMapEditor::GridMapEditor (this=<optimized out>, this=<optimized out>)
    at modules/gridmap/editor/grid_map_editor_plugin.cpp:1354
#14 0x00005555563633a4 in GridMapEditorPlugin::GridMapEditorPlugin (this=<optimized out>, this=<optimized out>)
    at modules/gridmap/editor/grid_map_editor_plugin.cpp:1742
#15 0x00005555574b6e53 in EditorPlugins::creator<GridMapEditorPlugin> () at ./editor/plugins/editor_plugin.h:279
#16 0x0000555557ec751e in EditorPlugins::create (p_idx=62) at ./editor/plugins/editor_plugin.h:286
#17 0x00005555565156af in EditorNode::EditorNode (this=<optimized out>, this=<optimized out>) at editor/editor_node.cpp:7769
#18 0x00005555569de4fd in Main::start () at main/main.cpp:4043
#19 0x00005555568ff674 in main (argc=4, argv=0x7fffffffdea8) at platform/linuxbsd/godot_linuxbsd.cpp:84

Oh right, I did not realise that the GridmapEditor initialisation function would be called at Godot's start-up. I assumed it was called on the creation of a Gridmap node.
The idea is that when a Gridmap Editor node is created, the default mode is the selection mode. In selection mode the gizmo is hidden.
There are two possible fixes:

move the code that selects the default mode to another event such as when the Gridmap Editor is shown/clicked in the bottom tabs
change the default tool mode to Transform mode (which makes the Gizmo visible)

The first solution is probably the best. I will test it out now.

It should work now; sorry for the inconvinience!

I've tested #99354 on my project. It seems like opening some scene, gizmo is always ok, other is not.
The way to make the bad scene gizmo back, is to add a gridmap and then delete it. After that, everytime I open this scene, it will be with gizmo.

Note that if your project had already the gizmo hidden (because of the bug), this fix won't restore its visibility.
This fix prevents the gridmap to hide the gizmo in the first place; it does not restore the gizmo visibility.
To restore the gizmo visibility, you can simply go in the viewport menu and check the option 'View Transform Gizmo', as shown in a screenshot above this post.
Let me know if that explains what you've experienced.


Upstream analysis module - add run qc function

What are the scientific goals of the analysis?
This is to add the function-run-QC.R in the upstream-analysis module.

Closed here https://github.com/stjude-dnb-binfcore/sc-rna-seq-snap/pull/21


Fixed bug css, js about auth views (css, js not loaded)

When you start the command
php artisan make:auth
We have all our files related to authentication. the problem is that If we use the routes managed by the command, files css/app.css and js/app.js are not loaded.
Exemple : your-website.com/public/register

The problem only comes from '/' before each name folder. If we remove it, we have our views with their css and js.

Please send to 5.4 since this is breaking.


Distinguish between datanotavailableinthis-state errors and other errors

In a time series, its possible that some plotted objects don't exist in all states. As a user moves through states, if a plot is associated with an object that doesn't exist in that state, the plot is sent into an error state and an error message is generated. There are many things that can cause plots to go into error states. The case of the data associated with the plot not being present in the current state is kind of a special one that I think requires some special handling. First, its quite common. Second, its quite annoying for users to deal with. An error popup always occurs and there is no way to disable it. That said, in other respects, VisIt seems able to continue moving through time and even recover an error'd plot when the data for it because available again in a later state. So, maybe all that is required here is to change how we handle this one special kind of situation from an Error to a Warning (which can be disabled but nonetheless, a user can still go to the output window to see whats happened if the ever need to). As well, maybe instead of turning the plot red in the plot list (en error) turn it to some other color, maybe orange or blue or something to indicate it isn't so much in Error as it is there isn't data for it in the current state.
-----------------------REDMINE MIGRATION-----------------------
This ticket was migrated from Redmine. As such, not all
information was able to be captured in the transition. Below is
a complete record of the original redmine ticket.
Ticket number: 2467
Status: Pending
Project: VisIt
Tracker: Feature
Priority: Normal
Subject: Distinguish between datanotavailableinthis-state errors and other errors
Assigned to: -
Category: -
Target version: -
Author: Mark Miller
Start: 12/03/2015
Due date:
% Done: 0%
Estimated time:
Created: 12/03/2015 09:34 pm
Updated: 12/15/2015 06:32 pm
Likelihood:
Severity:
Found in version: 2.12.3
Impact: 3 - Medium
Expected Use: 3 - Occasional
OS: All
Support Group: Any
Description:
In a time series, its possible that some plotted objects don't exist in all states. As a user moves through states, if a plot is associated with an object that doesn't exist in that state, the plot is sent into an error state and an error message is generated. There are many things that can cause plots to go into error states. The case of the data associated with the plot not being present in the current state is kind of a special one that I think requires some special handling. First, its quite common. Second, its quite annoying for users to deal with. An error popup always occurs and there is no way to disable it. That said, in other respects, VisIt seems able to continue moving through time and even recover an error'd plot when the data for it because available again in a later state. So, maybe all that is required here is to change how we handle this one special kind of situation from an Error to a Warning (which can be disabled but nonetheless, a user can still go to the output window to see whats happened if the ever need to). As well, maybe instead of turning the plot red in the plot list (en error) turn it to some other color, maybe orange or blue or something to indicate it isn't so much in Error as it is there isn't data for it in the current state.
Comments:

Move back into the new state. Engineering/Diablo needs a fix for this. Would like to discuss with group.

Looks like creating empty objects (in Silo databases anyways) addresses this problem (for Silo). Should we develop a more general solution?


Move prompb to its own module

Proposal
Use case. Why is this important?
For our project, I need to implement Prometheus Remote Write to collect Prometheus metrics sent from Prometheus servers. It only requires the github.com/prometheus/prometheus/prompb package, but I have to add the entire github.com/prometheus/prometheus module to go.mod. This pulls in a bunch of irrelevant go modules and some of the dependency modules expose security vulnerabilities per our code scan.
Moreover, as reported in https://github.com/prometheus/prometheus/issues/8417 and https://github.com/prometheus/prometheus/issues/8852, we have to use github.com/prometheus/prometheus v1.8.2-0.20210331101223-3cafc58827d1 in go.mod, which raises a security vulnerability in code scan.
Prometheus Remote Write is a nice feature and easy to implement. However, it becomes complicated because we have to include entire prometheus server module. Moving github.com/prometheus/prometheus/prompb to its own module will solve the problem.

Could you try https://buf.build/prometheus/prometheus/assets ?
go get go.buf.build/protocolbuffers/go/prometheus/prometheus

Thank you for the response. I was able to run go get go.buf.build/protocolbuffers/go/prometheus/prometheus and added it to go.mod. However, when I ran go mod tidy, it automatically added github.com/prometheus/prometheus v1.8.2-0.20210331101223-3cafc58827d1 back to go.mod and go.sum. Therefore, it doesn't solve the problem.
Is there a way to add the prometheus/prompb to go.mod without pulling in entire github.com/prometheus/prometheus module?

When using go.buf.build/protocolbuffers/go/prometheus/prometheus v1.1.1 module, I got panic when trying to marshal and unmarshall a WriteRequest object. Is there a newer version?
panic: protobuf tag not enough fields in WriteRequest.state:

goroutine 1 [running]:
github.com/gogo/protobuf/proto.(*unmarshalInfo).computeUnmarshalInfo(0xc0000c2320)
	/Users/wenjian@us.ibm.com/gopath/pkg/mod/github.com/gogo/protobuf@v1.3.2/proto/table_unmarshal.go:341 +0x138a
github.com/gogo/protobuf/proto.(*unmarshalInfo).unmarshal(0xc0000c2320, {0x123f720}, {0x147d548, 0x0, 0x0})
	/Users/wenjian@us.ibm.com/gopath/pkg/mod/github.com/gogo/protobuf@v1.3.2/proto/table_unmarshal.go:138 +0x67
github.com/gogo/protobuf/proto.(*InternalMessageInfo).Unmarshal(0xc0000cc0a0, {0x12a6d80, 0xc00008c300}, {0x147d548, 0x0, 0x0})
	/Users/wenjian@us.ibm.com/gopath/pkg/mod/github.com/gogo/protobuf@v1.3.2/proto/table_unmarshal.go:63 +0xd0
github.com/gogo/protobuf/proto.(*Buffer).Unmarshal(0xc0000b3ef0, {0x12a6d80, 0xc00008c300})
	/Users/wenjian@us.ibm.com/gopath/pkg/mod/github.com/gogo/protobuf@v1.3.2/proto/decode.go:424 +0x153
github.com/gogo/protobuf/proto.Unmarshal({0x147d548, 0x0, 0x0}, {0x12a6d80, 0xc00008c300})
	/Users/wenjian@us.ibm.com/gopath/pkg/mod/github.com/gogo/protobuf@v1.3.2/proto/decode.go:342 +0xef
main.main()
	/Users/wenjian@us.ibm.com/gopath/src/github.ibm.com/Observability/main.go:17 +0xe5

My code can be found at https://go.dev/play/p/zsTA3N25ss2 -  it simply marshals a WriteRequest and then unmarshals it.

Please use golang/protobuf instead: https://go.dev/play/p/yz7yH5NQnCG

Thank you. Changing to github.com/golang/protobuf/proto works, but the package is deprecated. The google.golang.org/protobuf package also works.
$ go get github.com/golang/protobuf
go: module github.com/golang/protobuf is deprecated: Use the "google.golang.org/protobuf" module instead.

I have one more question. As I mentioned earlier in this comment https://github.com/prometheus/prometheus/issues/10371#issuecomment-1055468283, github.com/prometheus/prometheus/prompb and go.buf.build/protocolbuffers/go/prometheus/prometheus have different definitions for Timeseries struct. Is it on purpose or just different versions?
type TimeSeries struct {
	Labels               []Label  `protobuf:"bytes,1,rep,name=labels,proto3" json:"labels"`
	Samples              []Sample `protobuf:"bytes,2,rep,name=samples,proto3" json:"samples"`
        ......
}

vs.
type TimeSeries struct {
	Labels    []*Label    `protobuf:"bytes,1,rep,name=labels,proto3" json:"labels,omitempty"`
	Samples   []*Sample   `protobuf:"bytes,2,rep,name=samples,proto3" json:"samples,omitempty"`
	......
}

go.buf.build/protocolbuffers/go/prometheus/prometheus is autogenerated, this is something we have introduced recently. it's very much possible that there are minor changes, but they should both work on the wire.

The go.buf.build/protocolbuffers/go/prometheus/prometheus module solved the dependency problem. We include the following module in go.mod.
go.buf.build/protocolbuffers/go/prometheus/prometheus

To unmarshal the protobuf message, you need to import google.golang.org/protobuf/proto or github.com/golang/protobuf/proto (already deprecated).

The go.buf.build/protocolbuffers/go/prometheus/prometheus module solved the dependency problem. We include the following module in go.mod.
go.buf.build/protocolbuffers/go/prometheus/prometheus

To unmarshal the protobuf messages, you need to import google.golang.org/protobuf/proto or github.com/golang/protobuf/proto (already deprecated).

I confirm that it works but I could not find where this is documented.
Where should I look?


[DO NOT REVIEW/MERGE] Swagger for Lease Container API from Oct17 to latest

Latest improvements:
MSFT employees can try out our new experience at OpenAPI Hub  - one location for using our validation tools and finding your workflow.

Contribution checklist:

[ ] I have reviewed the documentation for the workflow.
[ ] Validation tools were run on swagger spec(s) and have all been fixed in this PR.
[ ] The OpenAPI Hub was used for checking validation status and next steps.

ARM API Review Checklist

[ ] Service team MUST add the "WaitForARMFeedback" label if the management plane API changes fall into one of the below categories.
adding/removing APIs.
adding/removing properties.
adding/removing API-version.
adding a new service in Azure.

Failure to comply may result in delays for manifest application. Note this does not apply to data plane APIs.

[ ] If you are blocked on ARM review and want to get the PR merged urgently, please get the ARM oncall for reviews (RP Manifest Approvers team under Azure Resource Manager service) from IcM and reach out to them.
Please follow the link to find more details on API review process.

Automation for azure-sdk-for-ruby

Automation for azure-sdk-for-python

Automation for azure-sdk-for-go

Automation for azure-sdk-for-js

Can one of the admins verify this patch?

Automation for azure-sdk-for-java

Automation for azure-sdk-for-net
A PR has been created for you:
https://github.com/azure/azure-sdk-for-net/pull/-1
.NET SDK Commits:
https://github.com/adxsdknet/azure-sdk-for-net/pull/-1/commits/18485f183248c6ffb1a3b61a5dc507a114081c59

Automation for azure-sdk-for-net
A PR has been created for you:
https://github.com/azure/azure-sdk-for-net/pull/-1
.NET SDK Commits:
https://github.com/adxsdknet/azure-sdk-for-net/pull/-1/commits/18485f183248c6ffb1a3b61a5dc507a114081c59

Automation for azure-sdk-for-net
A PR has been created for you:
https://github.com/azure/azure-sdk-for-net/pull/-1
.NET SDK Commits:
https://github.com/adxsdknet/azure-sdk-for-net/pull/-1/commits/3229c268a755934bb834edbe65f8403621849a93

startbuild

Context from @huizlAzure – KPI team has discovered that the SRP lease container API doesn’t have Swagger coverage – they detected low usage on the API from Azure Portal, which was only for testing purpose, and the system alerted because there was no Swagger. As per KPI team it must be added for all SRP API versions where lease container is supported.
@blueww it seems BlobContainers_Lease API is missing from latest api-versions as well. Could you please sync with @huizlAzure and share thoughts?

startbuild

startbuild

Automation for azure-sdk-for-net
A PR has been created for you:
https://github.com/azure/azure-sdk-for-net/pull/5612
.NET SDK Commits:
https://github.com/adxsdknet/azure-sdk-for-net/pull/5612/commits/e382b0883b0f1c44f99840207ba7510df217a9aa


Can't use vimdeck

Everytime i start vimdeck i get this error message and i have no idea why:
/usr/lib/ruby/2.2.0/rubygems/dependency.rb:315:in to_specs': Could not find 'artii' (~> 2.0.4) - did find: [artii-2.1.1] (Gem::LoadError) Checked in 'GEM_PATH=/home/benedikt/.gem/ruby/2.2.0:/usr/lib/ruby/gems/2.2.0', execute gem envfor more information 	from /usr/lib/ruby/2.2.0/rubygems/specification.rb:1311:inblock in activate_dependencies'
from /usr/lib/ruby/2.2.0/rubygems/specification.rb:1300:in each' 	from /usr/lib/ruby/2.2.0/rubygems/specification.rb:1300:in activate_dependencies'
from /usr/lib/ruby/2.2.0/rubygems/specification.rb:1282:in activate' 	from /usr/lib/ruby/2.2.0/rubygems/core_ext/kernel_gem.rb:67:in block in gem'
from /usr/lib/ruby/2.2.0/rubygems/core_ext/kernel_gem.rb:66:in synchronize' 	from /usr/lib/ruby/2.2.0/rubygems/core_ext/kernel_gem.rb:66:in gem'
from /usr/lib/ruby/gems/2.2.0/bin/vimdeck:22:in `'

Looks like artii, one of vimdeck's dependencies didn't install correctly.
Try (might need to put sudo in front of these):
gem uninstall vimdeck
gem install vimdeck

Let me know what output you see when you do that.

Closing for now. Will re-open if you're still seeing the issue.


FHRS Re: 'Food Hygiene Ratings Scheme (FHRS)'

Problem with 'Food Hygiene Ratings Scheme (FHRS)' (https://www.api.gov.uk/fsa/food-hygiene-ratings-scheme-fhrs/)

Could you please describe the problem.

I am afraid that with a description of the problem that we are unable to help. The API contact is<EMAIL_ADDRESS>I suggest you contact them if there are problems with the operation of that API.


Request for platform support for leetcode

@raj454raj Please provide the platform for leetcode as it also well-known competitive programming platform

Hey @maverickLez, we have already reached out to Leetcode for this and awaiting a response. Current issue is that they don't have a public page or API from which submissions can be fetched and stored into StopStalk's DB given a user's handle. Closing this for now.

@raj454raj No Problem, Thank you for considering


Single policy rule for Mutation and Validation

In current design there are two separate calls for validation and mutation. This is not ideal as

in some cases the API server will not have proximity with the web hooks (for instance on a public network),
validation will fail if the object is not mutated
Policy owner and user have to define two separate ruled for validation and mutation

Can the policies be defined in a manner that they more desired state instead of separate mutation and validation policies and mutation can be indirectly inferred ?

MUTATION has additively more information than VALIDATION.  MUTATION checks not only if there is a problem but says how to fix it.  So to unify MUTATION and VALIDATION, we make it so the admission controller always runs the same query but where a MUTATION includes additional information in each decision that OPA returns.  Let's start with some sample decisions:
VALIDATION decision example.  This decision says the admission controller should reject an ingress because it has an invalid host.  The admission controller should send the user the specified error message.
# Validation
{
    "id": "ingress-host-fqdn",  # identifies type of violation
    "resource": {
        "kind": "ingresses",    # identifies kind of resource
        "namespace": "retail",  # identifies namespace of resource
        "name": "frontdoor"     # identifies name of resource
    },
    "resolution": {             # defines how to resolve problem
        "message": "invalid ingress host fqdn front.hooli.com"
    }
}

MUTATION decision example.  This decision says the admission controller should patch a pod so the 2nd container has the imagePullPolicy set to Always.  The format is exactly the same as the VALIDATION example, but where the resolution field includes a patch field.
# Mutation
{
    "id": "imagepullpolicy-always",  # identifies type of violation
    "resource": {
        "kind": "pod",               # identifies kind of resource
        "namespace": "retail",       # identifies namespace of resource
        "name": "frontdoor"          # identifies name of resource
    },
    "resolution": {                  # defines how to resolve problem
        "patch": [{"op": "replace",
                   "path": "spec/containers/2/imagePullPolicy",
                   "value": "Always"}],
        "message": "Change imagePullPolicy to Always"  # Optional
    }
}

Note that the patch field is an array of JSON patch operations so that if multiple operations are necessary to express the required change, the user has the full power of JSON patch at their disposal.
As usual, these decisions are generated by OPA policies and the rules they contain.  Here are examples of OPA rules generating the decisions shown above.  Notice that both of them contribute to the deny set.
# Reject any ingress with a bad FQDN
deny[x] {
    ingress := data.kubernetes["ingresses"][namespace][name]
    host := ingress.spec.rules[_].host
    endswith(host, "hooli.com")     # In reality a more comprehensive test
    msg := sprintf("invalid ingress host fqdn %q", [host])
    x := {
        "id": "ingress-host-fqdn",  # identifies type of violation
        "resource": {
            "kind": "ingresses",    # identifies kind of resource
            "namespace": namespace, # identifies namespace of resource
            "name": name            # identifies name of resource
        },
        "resolution": {
            "message": msg          # human-readable message to display
        }
    }
}

# Patch any pod so that imagePullPolicy is Always
deny[x] {
	pod := data.kubernetes["pod"][namespace][name]
	pod.spec.containers[i].imagePullPolicy != "Always"
	path := sprintf("spec/containers/%v/imagePullPolicy", [i])
    x := {
        "id": "image-pull-policy",  # identifies type of violation
        "resource": {
            "kind": "pod",          # identifies kind of resource
            "namespace": namespace, # identifies namespace of resource
            "name": name            # identifies name of resource
        },
        "resolution": {
            "patch": [{"op": "replace",
                       "path": path,
                       "value": "Always"}],
            "message": "Change imagePullPolicy to Always"  # Optional
        }
    }
}

The policy author now has a unified way of authoring both validation and mutation decisions.  The Validating and Mutating admission controllers have a unified query interface as well.  Below are several examples of relevant queries.
# all types of violations on all resources with all resolutions
deny[{"id": id,
      "resource": resource,
      "resolution": resolution}]

# all "image-pull-policy" violations
deny[{"id": "image-pull-policy",
      "resource": resource,
      "resolution": resolution}]

# all violations on namespace "retail"
deny[{"id": id,
      "resource": {"kind": kind,
                   "namespace": "retail",
                   "name": name},
      "resolution": z}]

# all violations for ingresses in namespace "retail" named "frontdoor"
deny[{"id": id,
      "resource": {"kind": "ingresses",
                   "namespace": "retail",
                   "name": "frontdoor"},
      "resolution": z}]

The results of any of these queries is a collection of the decisions shown earlier, some of which should be rejected with a message sent to the user, and some of which have a patch that should be applied to fix the problem.

The validating admission controller will reject any resources that DO NOT INCLUDE a resolution/patch field.

The mutating admission controller will apply patches for the resources that DO INCLUDE a resolution/patch field.

Going one step further, we need to identify mutation conflicts (e.g. where 2 different decisions on the same resource have conflicting patches) and resolve them.  A reasonable choice is to reject any resource that has conflicting patches and synthesize an appropriate error message from the multiple resolution/message fields.

MUTATION has additively more information than VALIDATION.  MUTATION checks not only if there is a problem but says how to fix it.  So to unify MUTATION and VALIDATION, we make it so the admission controller always runs the same query but where a MUTATION includes additional information in each decision that OPA returns.  Let's start with some sample decisions:
VALIDATION decision example.  This decision says the admission controller should reject an ingress because it has an invalid host.  The admission controller should send the user the specified error message.
# Validation
{
    "id": "ingress-host-fqdn",  # identifies type of violation
    "resource": {
        "kind": "ingresses",    # identifies kind of resource
        "namespace": "retail",  # identifies namespace of resource
        "name": "frontdoor"     # identifies name of resource
    },
    "resolution": {             # defines how to resolve problem
        "message": "invalid ingress host fqdn front.hooli.com"
    }
}

MUTATION decision example.  This decision says the admission controller should patch a pod so the 2nd container has the imagePullPolicy set to Always.  The format is exactly the same as the VALIDATION example, but where the resolution field includes a patch field.
# Mutation
{
    "id": "imagepullpolicy-always",  # identifies type of violation
    "resource": {
        "kind": "pod",               # identifies kind of resource
        "namespace": "retail",       # identifies namespace of resource
        "name": "frontdoor"          # identifies name of resource
    },
    "resolution": {                  # defines how to resolve problem
        "patch": [{"op": "replace",
                   "path": "spec/containers/2/imagePullPolicy",
                   "value": "Always"}],
        "message": "Change imagePullPolicy to Always"  # Optional
    }
}

Note that the patch field is an array of JSON patch operations so that if multiple operations are necessary to express the required change, the user has the full power of JSON patch at their disposal.
As usual, these decisions are generated by OPA policies and the rules they contain.  Here are examples of OPA rules generating the decisions shown above.  Notice that both of them contribute to the deny set.
# Reject any ingress with a bad FQDN
deny[x] {
    ingress := data.kubernetes["ingresses"][namespace][name]
    host := ingress.spec.rules[_].host
    endswith(host, "hooli.com")     # In reality a more comprehensive test
    msg := sprintf("invalid ingress host fqdn %q", [host])
    x := {
        "id": "ingress-host-fqdn",  # identifies type of violation
        "resource": {
            "kind": "ingresses",    # identifies kind of resource
            "namespace": namespace, # identifies namespace of resource
            "name": name            # identifies name of resource
        },
        "resolution": {
            "message": msg          # human-readable message to display
        }
    }
}

# Patch any pod so that imagePullPolicy is Always
deny[x] {
	pod := data.kubernetes["pod"][namespace][name]
	pod.spec.containers[i].imagePullPolicy != "Always"
	path := sprintf("spec/containers/%v/imagePullPolicy", [i])
    x := {
        "id": "image-pull-policy",  # identifies type of violation
        "resource": {
            "kind": "pod",          # identifies kind of resource
            "namespace": namespace, # identifies namespace of resource
            "name": name            # identifies name of resource
        },
        "resolution": {
            "patch": [{"op": "replace",
                       "path": path,
                       "value": "Always"}],
            "message": "Change imagePullPolicy to Always"  # Optional
        }
    }
}

The policy author now has a unified way of authoring both validation and mutation decisions.  The Validating and Mutating admission controllers have a unified query interface as well.  Below are several examples of relevant queries.
# all types of violations on all resources with all resolutions
deny[{"id": id,
      "resource": resource,
      "resolution": resolution}]

# all "image-pull-policy" violations
deny[{"id": "image-pull-policy",
      "resource": resource,
      "resolution": resolution}]

# all violations on namespace "retail"
deny[{"id": id,
      "resource": {"kind": kind,
                   "namespace": "retail",
                   "name": name},
      "resolution": z}]

# all violations for ingresses in namespace "retail" named "frontdoor"
deny[{"id": id,
      "resource": {"kind": "ingresses",
                   "namespace": "retail",
                   "name": "frontdoor"},
      "resolution": z}]

The results of any of these queries is a collection of the decisions shown earlier, some of which should be rejected with a message sent to the user, and some of which have a patch that should be applied to fix the problem.

The validating admission controller will reject any resources that DO NOT INCLUDE a resolution/patch field.

The mutating admission controller will apply patches for the resources that DO INCLUDE a resolution/patch field.

Going one step further, we need to identify mutation conflicts (e.g. where 2 different decisions on the same resource have conflicting patches) and resolve them.  A reasonable choice is to reject any resource that has conflicting patches and synthesize an appropriate error message from the multiple resolution/message fields.

@timothyhinrichs Thanks combining the policy decision in a response makes perfect sense. And not allow conflicts sounds reasonable to me to as it will make system more predictable.


Hierarchy docstring

#48

I believe we can add an __init__.py file underneath the devices/ directory with a docstring and it should add the docstring to the devices section

yes haha that's why I created the issue I think...


Page numbers out of order after converting

After using "ocrocis convert" to binarize my .tif images in my book/directory file (all images saved as 1.tif, 2.tif, 3.tif etc.) my images were spit out in a random order. Anyone know what I am doing wrong? Sorry for the rookie question...

You should ask that question maybe directly at the ocrocis repo: https://github.com/kaumanns/ocrocis . I don't know much about their CLI.

Oops! Sorry. Thank you for the help!

Oops! Sorry. Thank you for the help!

Cont'd in https://github.com/kaumanns/ocrocis/issues/2

Cont'd in https://github.com/kaumanns/ocrocis/issues/2


Datasets admin projects endpoints

Changes

Added endpoint to count how many projects are on a dataset
endpoint to get all projects on a specific dataset already existed
Added check on if the editor can see all projects and warning on editing if they cannot

Migrations
Screenshots

Notes

Need to check if the existing endpoint that gets all projects in a specific dataset or if it only gets what the user is permitted to see

Checks
Important: please complete these before merging.

[ ] Run migrations, if any.
[ ] Update changelog.md, including migration instructions if any.
[ ] Run unit tests.

Question: why in this screenshot does it say it's in 3 other projects? Did you create 3 other projects beyond CO-CONNECT, and TestProject2 to put it in?

Ok thanks!


darwin-amd64 binary missing from release v.0.6.0

See above

I don't know why, but I uploaded a darwin-amd64 binary from v0.6.0 version. Thanks for your report


low views

shits giving me 20 views per

its showing me : Captcha invalid | check access to zefoy. can anyone help?

text annotations error? help me


INFO exited: blueiris (exit status 1; not expected)

Using the vaapi dockerfiles, after creating the container and running it, I`m getting that message. Also using -v /volume1/docker/blueiris:/home/wineuser/prefix:rw, container is not creating files inside mapped volume
edit: all my containers are created with my main user uid/gid and all are working correctly but this one refuse to create files inside mapped volume. I tried to change /home/wineuser/prefix with /root/prefix on all files inside docker before creation and now is semi populating the mapped volume.

Good Evening I am experiencing a similar issue.

Can you both try to manually chown the data directory from the host to be the proper user and then restart the container? I think the issue is that the permissions aren't being set properly on first run.
e.g. chown -R 1000:1000 ./data

I'm using unraid and had the same issue. Thank you @jshridha for the obvious fix :)

Thanks jshridha I will try tonight, all my containers are created with main root user uid/gid so mybe this is the problem.

This is fixed in: 651429e1688232489729beda836bf271719a0b94 Permissions on data directory are now set at startup. Builds take a while but should hit docker hub soon.
Feel free to reopen if the issue persists for you. Thanks for posting!


Suggestion for reorganization

We could simplify things a bit by making the test system be now "cl-json/test" instead of ".test".  The new "slashy" package names in ASDF would mean that ASDF would know to find that system definition in "cl-json.asd", instead of the user having to make sure it happens. E.g., one could simply say
(asdf:load-system "cl-json/test")
and ASDF would find the system, instead of having to remember to LOAD-SYSTEM "cl-json" first.
I'd be willing to do the reorg, and submit a pull request, but wanted to check and see if it would be OK before proceeding.

Yes sure, it sounds like a good idea.