Implement DNS validation as optional for ACME providers leveraging externalAccountBinding

Please see https://github.com/Netflix/lemur/issues/3129 for additional details.
When I use ACME_DIRECTORY_URL, ACME_REGR and ACME_PRIVATE_KEY information obtained from another ACME client like Certbot, Lemur failed with the following error:
Traceback (most recent call last):
File "/opt/app/lemur/lemur/plugins/lemur_acme/plugin.py", line 561, in get_ordered_certificates
acme_client, order, order_info
File "/opt/app/lemur/lemur/plugins/lemur_acme/plugin.py", line 305, in get_authorizations
dns_provider.options,
File "/opt/app/lemur/lemur/plugins/lemur_acme/plugin.py", line 110, in start_dns_challenge
raise Exception("Unable to determine DNS challenges from authorizations")
Exception: Unable to determine DNS challenges from authorizations
2020-09-24 19:46:44,388 ERROR: {'function': 'lemur.common.celery.fetch_acme_cert', 'message': 'Pending certificate creation failure', 'task_id': '84b6997f-1729-464e-be42-bbdad5b24436', 'id': 3, 'pending_cert_id': 3, 'last_error': Exception('Unable to determine DNS challenges from authorizations',), 'cn': 'FQDN'} [in /opt/app/lemur/lemur/common/celery.py:320]
I noticed this behavior with some ACME providers, they simply ignore having DNS challenge as long as EAB Key and EAB ID match.

Do you know, if it's common for certificate authorities to forgo validation, if the external Account binding is used?
Since right now it's not possible to provide the EAB at all (unless using the pre-configured ACME account), I'd propose to implement a new EAB enabled IssuerPlugin if we can waive the validation in these cases. (Since it would also need a different way of creating a new ACME account, where the EAB is transmitted as well)
If one would still need to be able to do the validations, it might be more useful to refactor the ongoing implementation with two dedicated IssuerPlugins for DNS & HTTP challenges, into a single one, where you can choose between validation methods (e.g. either DNS, HTTP, None) and make the EAB a configuration option there as well.

I tested this again today. My current AWS Route53 profile was for a sub domain. I used certbot and EAB to get the certificate. With Certbot, if I do not provide a valid DNS service option it was failing. However, the AWS account I used does not have ability to update the DNS entries of the domain I was trying. It is kind of tricky that certbot appears to use DNS validation. But with just login credentials it worked just fine.
I noticed it works as long as the domains are pre-authorized. When I tried  to test with an external domain with certbot and my EAB, I got the following error:
The identifiers are not all linked to the same preauthorized Subject organization name/address

Do you know, if it's common for certificate authorities to forgo validation, if the external Account binding is used?

This is currently the case for DigiCert.
With external account binding support added in https://github.com/Netflix/lemur/pull/3611 (thanks!) I think it would be beneficial if we revisited adding optional support to skip validation for a configured ACME authority. If there are no major objections, I can take a shot at contributing a patch. cc @hosseinsh just in case you have any thoughts here...

Yeah, I noticed the flow of DigiCert for ACME in that they skip the validation part, which might be typical for OV-validated domains.
I shared some thoughts on this other issue, but it boils down to if DigiCert is following the protocol correctly, and sending back Status Valid Challenges, in which case Lemur would be able to skip the validation since https://github.com/Netflix/lemur/pull/3666
otherwise, we need a small variation, controlled via options, to skip validation and move right to finalizing the order.
https://github.com/Netflix/lemur/issues/3669#issuecomment-878671476

I see, thanks for the insight. I haven't tested #3666 yet (just learned about it now) but will do so this week and report back whether they are sending Status Valid challenges correctly or not.


Passenger model in Django

Create Passenger model in Django.
Required fields:

first name
last name

(What with GDPR?)

Can you assign me to this?


Weekly Permissions sync 2024-12-28

Weekly Permissions sync 2024-12-28

superseded by https://github.com/microsoftgraph/microsoft-graph-devx-content/pull/936


Add TS to PR checklist

What:

Why:

How:

Checklist:

[ ] Documentation
[ ] Tests
[ ] Ready to be merged

[ ] Added myself to contributors table

:tada: This PR is included in version 5.7.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


fix(repl): create history file if doesn't exists

Closes https://github.com/adonisjs/adonis-framework/issues/642

Coverage remained the same at 46.359% when pulling 924247e526101539f541d0cf9d19f3f40f9d0345 on RomainForks:fix-#642 into 67ca54d2bfd936436a8ba28fb63bb19eef357c30 on adonisjs:develop.

Thanks 😄


MqttDecoder IndexOutOfBoundsException

https://github.com/netty/netty/blob/bc49c4b1464c8345bf87b3053b3adefa64f91530/codec-mqtt/src/main/java/io/netty/handler/codec/mqtt/MqttDecoder.java#L231
Please ask, is there no exception here?
I see that the readable bytes are not checked in front of the code.
for example：
remaining：  0x80, 0x80, 0x01
But tcp is delivered for the first time ：0x80, 0x80

It's using the ReplayingDecoder so there is no problem.


Trying to make this work on Grafana Cloud+Agent

I have my Grafana dashboards running on Grafana Cloud - it's just easier for me. I use Grafana Agent to ship my metrics from my LAN to Grafana Cloud and I have setup the scrape config in my config file, and restarted the Grafana Agent. However the metrics explorer in the Grafana dashboard visualisation editor doesn't seem to know what speedtest_download_bytes and the other speedtest exporter metrics actually are. I have tried entering them anyway and will leave the dashboard overnight to see if I get any data. Any ideas?

Have left the dashboard overnight, there still seems to be no data coming in? I have checked the Prometheus exporter is working, it's giving me Speedtest results when I request /probe on the docker container.

Closing as this works after all, it just needed a bit of time!!


External time sliding window does not remove expired events

You can only see AttributeAggregator.processAdd but not AttributeAggregator.processRemove in external time sliding window. The issue is caused by we don't reset expiredEventChunk before removing expired events, and the lastReturned event does not have next event.

Thanks suhothayan, closed issue https://github.com/wso2/siddhi/issues/263


[relay] Simulating lenet, add Relay to fud, other bug fixes.

Lenet simulation

ONNX model used: pico-cnn/lenet.onnx
Input:

#
TVM Softmax
Calyx Softmax

0
7.1270239e-07
-0.00006103515625

1
1.1549375e-06
0.0002288818359375

2
3.7355226e-06
-0.0000457763671875

3
7.9318892e-07
0.0000457763671875

4
9.9998534e-01
1.0002899169921875

5
3.1714912e-08
-0.000030517578125

6
2.1043759e-06
-0.00006103515625

7
1.4363920e-06
-0.0004425048828125

8
3.1603065e-07
0.000030517578125

9
4.3458881e-06
0.0000457763671875

Given that our softmax output is producing negative numbers, there is still something wrong. I presume it is partly due to the fact that our current exp operator computes e^-x as 1/e^x.  Tracking in #505.
Script to run ONNX models
More generally, I've written a script that takes in:

The name of the net, e.g. lenet
The file path to the input (i.e. the image).
The dataset for which the input will be classified against. This is necessary to determine what preprocessing should be done on the image. e.g. "mnist", "imagenet".
The file path to the ONNX model.
Which output you want:
(1) tvm, Running the Relay program on the TVM executor and printing the final softmax array.
(2) calyx, Writing the Calyx program and data to files.
(3) relay, Writing the Relay program to a file.

Example:
# Writes the Calyx program and data to separate files for simulation.
python3 frontends/relay/onnx_to_calyx.py -n "lenet" -d "MNIST" -i "/nets/four.png" -onnx "/nets/lenet.onnx" -o calyx

Other updates

Add a function to get a dictionary of memories from the Relay -> Calyx program.
Avoid overflow in nn.softmax by subtracting the max value first. As mentioned above, the exp operator still needs to be revamped.
Fix a bug in batch_flatten.
Add a subset of reshape (necessary for lenet), and dropout for future use.
Add relay to fud.
Add a round_float_to_fixed flag to the verilog stage that is currently defaulted to True. This flag essentially says any input from .data files will automatically be rounded to the nearest fixed point number.
Closes #494.

LGTM! If there are new python libraries needed to get the ONNX net stuff working, we should document that.

Added documentation the Relay frontend docs.


[feature-request]: After metadata extraction, show a sample of how the data will/does look (especially for date/time variables)

What feature would you like to request?
I think this is less important for other data types, but for date and time, I know there are sooooo many ways people input them into files and spreadsheets, so having some display after metadata extraction for these data types to say "hey, this is what it will look like, is that correct? If not, please fix it yourself" type thing.
Maybe within the metadata extraction section, for these data types, we can right away show a small sample of what the data looks like after parsing and conversion.
Optional: Suggest a solution
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

This has been done by @philter87 already 🎉


Geowave 859: HBase Performance Improvments

Coverage decreased (-0.08%) to 52.007% when pulling 068f8ce7115f8553938ffa130359dd3f0f2f8733 on GEOWAVE-859-rebase2 into 63f57e35ffc9312facfa37fdec2493b3d05867a8 on master.


Crashing

When I'm in game and trying to use the recipe calculator it crashes the game.
I'm using the ATM 9 (0.2.37) modpack, it was woking fine on ATM 9 (0.2.36)
https://pastebin.com/xgkcqprM

This is gregtech crashing, report to them.


Request: Update F-Droid version

Hi there,
is it possible to get the current version of Riot.im Android messenger also via F-Droid?
Currently the up-to-date version (0.9.1) is only available via Google Play Store.
The F-Droid version is still on version 0.8.28a.
Thanks, Mathias

You can track the issue here. Generally the F-droid builds are done by the F-droid volunteers. Currently this hangs on a compilation issue with jitsi, see here.
If anyone has time to dig in to the issue, that would be great.

This seems to have happened! :D
(unfortunately push still doesn't work)

Note that push can't work with the f-droid version, because that uses FCM which is proprietary and thus can't be included in f-droid.
The f-droid version regularly checks the server to see if there are new messages. If you don't get notifications it's possible that android killed riot-android.


Driveby refactor

Trello card
Some refactors that I've pulled out of a different PR I'm working on, so that I can get feedback on the changes early.

Forgot to run the linter locally, will fix the issues now

Thanks @ChrisBAshton!


Support telemetry v1.0

Is it possible to push a 0.8.1 version to hex with this?

@joshnuss Done!


FreeBSD Error: ENOENT: no such file or directory, open '/proc/cpuinfo'

Another Sentry call ... freebsd :-))
https://sentry.iobroker.net/share/issue/90b41e7c686d4ddd88aac1aa8db2c97d/
Error: ENOENT: no such file or directory, open '/proc/cpuinfo'
  File "fs.js", line 447, col 3, in Object.openSync
  File "fs.js", line 349, col 35, in Object.readFileSync
  File "/usr/local/iobroker/node_modules/iobroker.info/node_modules/systeminformation/lib/system.js", line 618, col 31, in null.<anonymous>
    const linesRpi = fs.readFileSync('/proc/cpuinfo').toString().split('\n');

@Apollon77 will fix it during next week … currently on holiday.

No stress, all good, take a rest and fix afterwards ... was only one user at all

@Apollon77 fixed. Version 5.8.2 just released


Detect current directory when invoking venv-set-location

Using per-diretory local variables doesn't work for me
((python-mode . ((eval . (venv-workon "myproject-env")))))
It always detect the default dir. Not the dir inside that project.
File local-variables error: (file-error Opening directory No such file or directory /home/username/.virtualenvs)

Can we detect current directory location when invoking venv-set-location. (I have Helm installed so it easy for me to jump up and down dirs).
But everytime I want to choose venv-location. I have to start from home dir, duh :(. Can we use something like helm-visit-buffer so it always start from current dir ?

Or maybe. Using something like command in Elisp prompt

current-dir $ set-location "venv-in-this-current-dir"

Thanks :)

Oh no.
I am new to python. I know virtualenv but don't know the intent of virtualenvwrapper.

Organizes all of your virtual environments in one place.

The lack of understanding about virtualenvwranpper intent take me to a lot of confusion (e.g why would add python env dir location to init.el, set venv location every time, etc)
Now I know the workflow. Just type workon "foo" and boooms!. venv-current-name become very handy to put in EShell prompt.
Then I am suprised that this package is reimplementation of virtualenvwrapper. Thanks a ton.


Dev-guide: Kernel build fails on ppc64le

The kernel config file name prefix changed from
ppc64le to powerpc. This change broke the kernel
build on ppc64le. Fix the kernel build steps
accordingly.
Fixes: #207
Signed-off-by: Nitesh Konkar<EMAIL_ADDRESS>

and will note, we can fast track this through the docs review as it is a single code line change.

lgtm, Not needed to to this PR but this we should change this instructions to use https://github.com/kata-containers/packaging/blob/master/kernel/build-kernel_test.sh.

@jcvenegas : Is it a lgtm for this patch? I see Approval pending still.

looks like @jcvenegas ack was not in a form github/pullapprove captured. pls re-ack. Maybe @jodh-intel or @chavafg can also have a peek - it is a simple change...

lgtm

Probably @chavafg is not in the list of approvers. cc @jodh-intel

I find the kernel_dir variable name confusing, but as it's only being reused...
lgtm


Update agent module for py3 encoding issues

When using stdin and stdout use the buffer attribute in order to be able
to send binary strings rather than text.
Required for all non-trivial PRs

[ ] Rebased/mergable
[ ] Tests pass
[ ] CHANGELOG.md updated
[ ] Sign CLA (if not already signed)

Required only if applicable
You can erase any checkboxes below this note if they are not applicable to your Pull Request.

[ ] TICKscript Spec updated

Local Check
@rca 's patch in agent.py works for me on a local Windows 10 machine for a custom UDF. Refer to #2170

As a Proof of Concept I have documented the complete process on a Repository @rca

This works for me under Python 3.6.8.
When merging this (which already includes #1355), you'll probably want to also include the from __future__ import absolute_import line from #1799. That PR is otherwise identical to #1355, but this import ensures backwards compatibility to python 2.7 (which will retire soon, anyways).


fixes RemovedInDjango110Warnings in urls.py

Thanks ✌
I will make a release as soon as the build is green again. However, I don't have time to fix the broken build this week.


Front LEDS?

I don't even know if you still reply to these, but I'm giving it a shot. MSI Mystic Light really interferes with my coolermaster fan LED controller program. Your program works to switch my GTX 1070 logo to my desired color, but the red front fan LEDs remain on. On mystic light, you can turn off the front fans while leaving the logo light on, but it messes with my other RGB programs. Is there any way to turn off the front fan while the logo stays on with your program or only shut them both off?

hi, i'm trying to control my led light on my msi gtx 1080. i installed the file from your 2017 post but i can't access the app. the original color changed to blue. can you help me get all set. trying to get rainbow effect. tx


Duration Timer

It would be great if there could be a parameter named duration, wherein anyone would pass the duration for which the timer is supposed to run.

It would be great if there could be a parameter named duration, wherein anyone would pass the duration for which the timer is supposed to run.

Duration as in milliseconds?

Yes

Cool. But it will be issue when user reload the page and every time duration will be the same. Unless duration is in unix timestamp.

Ya right, in that case I think it's best to configure UNIX timestamp.


报错： Right operand to ? is void, but left operand is of type 'int'

There comes an error in "UIScrollView+MJRefresh.m" file when I drag the MJRefresh file into my project.
I tried to fix it in many ways ,however, it did't work!
Specific information is as follows。
UIScrollView+MJRefresh.m

(void)executeReloadDataBlock
{
!self.mj_reloadDataBlock ? : self.mj_reloadDataBlock(self.mj_totalDataCount);//here comes an error.The specific information is below!
}
error： Right operand to ? is void, but left operand is of type 'int'


Unable to override clipboard permissions

Steps to reproduce
Tell us about your environment:

Puppeteer version: 3.0.0
Platform / OS version: OSX
URLs (if applicable):
Node.js version: 12.16.1

What steps will reproduce the problem?
Try to override clipboard permissions. They won't work.
const context = browser.defaultBrowserContext();
context.overridePermissions('https://example.com', ['clipboard-read']);

Seems like this is happening because of change in underlying CDP.
In code the clipboard permissions are being converted to an older version while the updated value is supposed to be: clipboardReadWrite (from CDP)
Current code: https://github.com/puppeteer/puppeteer/blob/master/src/Browser.js#L347-L348

@mathiasbynens I have created a quick PR. Not sure if you also want to expose clipboardSanitizedWrite as an option.


new priority messages

First try at at adding bus priority messsage to the 1.1 draft.

The result can be viewed (rendered by Github) at https://github.com/rsmp-nordic/rsmp_sxl_traffic_lights/blob/priority_messages/source/sxl_traffic_light_controller.rst

@rsmp-nordic/tech-board

@otterdahl does the tables look about right? Then I can merge this into draft-1.1 and we can work from there with the proposal rendered to the online version

@otterdahl does the tables look about right? Then I can merge this into draft-1.1 and we can work from there with the proposal rendered to the online version

Yes. It think the tables are OK. There are some issues left, but it is easier to fix them once merged. I think it is OK to merge with 1.1-draft

merged to 1.1-draft


Systemd stage 1 iscsi and multipathd

Description of changes
This enables iscsi and multipath with systemd stage 1. It also includes a fix for the iscsi-multipath-root test, which has been broken since #199710.
This is currently marked as a draft because the multipath test doesn't work for unknown reasons.
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Any status on this?

@nyabinary I haven't looked at this since opening it. At the time, the multipath test wasn't working and I wasn't really sure why. Feel free to investigate


Implement Version-Aware Indexing Strategy

Description:
Implement a version-aware indexing strategy for Kyma documentation to handle versioned content effectively. This involves preprocessing documentation for each version, splitting documents into chunks, and adding version metadata to each chunk. Ensure that the indexing job can manage multiple versions simultaneously and provide accurate documentation retrieval based on the specific version of the Kyma modules installed by the end user.
Reasons
Chatbot needs to give response for a Kyma module version installed.

Think about it, might be good idea for a spike


check min_rtt value in collector/extra_data feed

For the purposes of checking the data feed values, check the values of min_rtt in the postgres table to confirm whether they have decimal places. If they do, confirm the values are not being truncated in the data feed.

The database only has integer values.  The values for min_rtt stored in the extra_data table are cast to an integers before insertion.  I think my reasoning for doing this was  a) to simplify form validation b) because RTT is measured in milliseconds and it isn't clear that small fractions of one millisecond are really useful for this purpose.

I think we should cast values for these metrics in postgres based on the format they are stored in within Big Query. Data consistency would be the reason. The aggregation and/or visualizations can reduce values to integers if that's what we want to display.
I'm going to run a query now to see what min_rtt is stored in to confirm whether this issue can be closed.

minrtt is stored as an integer in Big Query. re-closing.


WIP: Add Color mark

When you pick a color via the colorpicker and save it. After reloading the page the whole span tag is removed.
It's saved correctly, I have this value in my database
Test a color

But when Tiptap is rendered again or {!! tiptap_converter()->asHTML($post->content) !!} is used, the value of the tiptap editor becomes
Test a color
.
This can be reproduced in the Peek demo.
I already figured it out with a missing Tiptap PHP extension (https://github.com/ueberdosis/tiptap-php/pull/24), so I wonder if it can be added here.
I already gave it a try, but can't get it to work, the HTMLAttributes in renderHTML are always empty. Also the extension in the pull request above doesn't seem to work (anymore).
With this code I already managed to get back
Test a color
, but I have no clue how to get the color value in the renderHTML method.

I can look into this as soon as I can, but looking at your code your parseHtml method is incomplete.

Closing this PR as it is fixed with #166
Thank you.


Feature: Handler Style

A handler should be declared in different ways/styles.
Style:

mutiple arguments
single argument

The single argument style should have the following properties:

request/req
response/res
next
(method)
(path)
(params)

method, path, params should be evaluated on demand. Otherwise execution overhead and perfomance issues.

72af754000eb1987229d721626bad1bcf20d0e56


Murmur3 Hash with Guava dependency

Part of supporting KIP-213 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+Support+non-key+joining+in+KTable ). Murmur3 hash is used as a hashing mechanism in KIP-213 for the large range of uniqueness. The Murmur3 class and tests are ported directly from Apache Hive, with no alterations to the code or dependencies.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@ijuma  , do you have any concerns about bringing this in? Not sure who else's opinion to seek.

We don't want to add a dependency to Guava in the clients module.

@ijuma - Two questions - who is the we in "we don't want to add..."? Secondly, is there a particular reason that Guava is not acceptable considering it is only as part of test and not part of source? I'm just trying to get an understanding here as to how dependency-inclusion/exclusions are made.

Sorry, I didn't notice it was a test only dependency. The general policy for the clients module is to have minimal dependencies. For tests, we are not as strict. Having said that, guava causes huge issues because it's widely used and different projects rely on different versions. So the motivation has to be strong to include it. What is it in this case?

Also, did we consider https://cyan4973.github.io/xxHash/ ?

@ijuma I was looking at Murmur3 for a hash because it is very important for us to be able to detect changes, as we are using the hash to ensure that only the correct event is propagated and ones that do not match the hash are discarded. Minor changes are very likely in these events, as we're joining entities that may have a single property changed in rapid succession, requiring a very high likelihood of uniqueness for even very minor changes in the event.
The speed of xxhash isn't particularly important in this scenario because the events are serialized and sent over the wire to other partitions after being hashed, so the overhead from that is many times larger than any hash function speed. Uniqueness is pretty much the biggest criteria here, and Murmur3's 128-bits gives an extremely high likelihood that unique events will be correctly hashed as such. If we have to drop down to 64-bits and use xxHash I am fine with that as it's still unlikely we'll have a problem, but the likelihood is way higher than it is with Murmur3's 128-bit hash.

Thanks for the explanation, @bellemare . Just adding my +1; we discussed different hash algorithms in the KIP-213 discussion and settled on 128-bit Murmur3.
I don't recall if we discussed xxHash, but I do agree with Adam's assessment, speed is important of course, but not as important as certainty. In any case, I'd be surprised if we are CPU bound for KIP-213 anyway.

@guozhangwang @bbejeck ,
Do you mind giving this a quick review?
@ijuma ,
It sounds like your major concern is around adding Guava as a test dependency. I had the same concern. It seems like a bit of a trap, because the Murmur3 implementation is copy/pasted. We should try not to modify the source of the main code or the tests, so that the project can stay mergeable with upstream.
Some options I see:

just pull in guava as a test dependency. Note that once it's included, it will most likely spider its way through the whole project's tests, so we should evaluate this as a project level decision to use Guava for tests.
move Murmur3 to a separate module (aka gradle project, but still in the AK repo/build). This way, we can isolate Guava only to the test for Murmur3.
implement identically named test utility methods to the ones that this one depends on from Guava. That way, only the imports would change, and for the most part, it would be easy to merge changes from upstream.

WDYT?

Retest this, please.

I'm not too concerned about guava version conflict since so far I've only seen jersey-guava used by tools in our repo.

Hey, all, the original PR (https://github.com/apache/kafka/pull/5527) is getting kind of close to mergeable. Can we make a call on this PR before it blocks 5527?
Thanks,
-John

/cc @ijuma  and @guozhangwang ^^

I'm +1 on bringing in the dependency.

I'm +1 as well, as it's a test dependency.  But I don't know all the implications on the project.
@ijuma WDYT?

I don't understand why we need the dependency. It seems trivial to copy the expected values in our test.

Btw, also see https://fastcompression.blogspot.com/2019/03/presenting-xxh3.html
It may be true that performance is not needed in this case, but it may be needed in others. From a maintenance perspective, it's easier to rely on a hash that is both fast and provides the uniqueness we need.

@ijuma Not sure about this one (yet) as according to the last paragraph:

XXH3 is now released as part of xxHash v0.7.0. It’s still labelled “experimental”, and must be unlocked using macro XXH_STATIC_LINKING_ONLY. It’s suitable for ephemeral data and tests, but avoid storing long-term hash values yet. This period will be used to gather user’s feedback, after which the algorithm will transferred into stable in a future release.

Is this not of some concern if I want to use this to get 128-bit hashes?

Since the test we are copying here is actually just check that the logic of Murmur3 is consistent with the Murmur3_32HashFunction and Murmur3_128HashFunction which is relying on the const 0xcc9e2d51 etc as well, moving the static values and removing seeds to not introduce the dependency makes feel that we are effectively defeating the purpose of copying the test anyways.
I'd suggest we just reduce the test to UtilTest.java where we just tested a few const with hard-coded expected values:
public void testMurmur2() {
        Map<byte[], Integer> cases = new java.util.HashMap<>();
        cases.put("21".getBytes(), -973932308);
        cases.put("foobar".getBytes(), -790332482);
        cases.put("a-little-bit-long-string".getBytes(), -985981536);
        cases.put("a-little-bit-longer-string".getBytes(), -1486304829);
        cases.put("lkjh234lh9fiuh90y23oiuhsafujhadof229phr9h19h89h8".getBytes(), -58897971);
        cases.put(new byte[]{'a', 'b', 'c'}, 479470107);

        for (Map.Entry c : cases.entrySet()) {
            assertEquals((int) c.getValue(), murmur2((byte[]) c.getKey()));
        }
    }

And remove the Murmur3Test, @bellemare does that sound good to you?

@guozhangwang that would work for me - @ijuma ?

Sounds good to me also. I'd just go for it, since it removes the Guava concern. IMHO, we can bring in xxHash as well (or replace Murmur3 with it) later on with not too much wasted effort.

@bellemare Okay let's do this and I will review again and merge when you're ready.

Btw, is there any reason why this is in clients and not Streams?

Streams already depends on clients but not vice versa, I think it is better to put it in clients so that in the futures if murmur3 want to be used elsewhere we can avoid possible code duplicates -- we have already observed such issues in the past, e.g. connect's JsonDeserializer etc are hard be leveraged by other modules without introducing extra dependency.

Okay, done the changes for murmur3 and removed the guava dependency. Does this work?

@bellemare the checkstyle failed:
clients:checkstyleTest

Also, after syncing with @ijuma offline I think we can move murmur3 into streams for now. The rationales behind it is that:

murmur3 is no longer the best performant hashing function as of today, and at the same time any migrations away from the existing murmur2 to new libraries would be backward incompatible; so in the future when we want to use new hashing function elsewhere we would consider a different function -- i.e. it is not very likely to be shared outside Streams
This class is not a public API, so keeping in Streams does not really prevent the future possibility if we wants to move it elsewhere -- we just need to make sure that any dependencies of AK would not refer to such internal classes, which should be very, very rare.

WDYT? I can help moving them to Streams when merging your PR if it looks fine to you.

@guozhangwang - sounds good to me. I fixed the checkstyleTest issues.
So you're going to move the files under streams when this is committed?

So you're going to move the files under streams when this is committed?

Yes, that's my plan.

Merged to trunk with Murmur3 moved to Streams, thanks @bellemare for your patience!


tsconfig.json target lead to an error

TypeScript Version:  3.8.3

Search Terms:
Code
https://github.com/rabbitooops/lerna-test
More
I am not sure if it's a bug, please help me, thank you!
When target is ES5, it works properly.

But if I set target to ES6 or higher version, it will throw an error.

Uncomment "moduleResolution": "node" so it won't automatically change when you move between targets.

@AviVahl This answer is very helpful! Thanks!


关于训练轮数和loss不变问题

感谢作者的分享，这里想请教几个问题？
（1）我尝试进行训练，用训练后的模型进行测试后结果不好，形状没有成好，是因为训练的不够久吗，需要训练多久呢?
（2）在训练的过程中，我发现loss并没有下降，并且用的是shapenet数据集，这是因为什么呢，形状没有成好，和这是否有关呢？

which model are you training? autoencoder or generator?

I trained the gen model~

I trained the gen model~

which model are you training? autoencoder or generator?

I trained the Generator model.

Hi,
I tested the training script and there seems to be no problem. Can you provide us with more details? My email address is<EMAIL_ADDRESS>

Hi, I ran the code directly and encountered a similar problem with the loss curve shown below, can you point out the problem? What should a normal loss curve look like?


Add k8's service bindings for dependencies in Build resource

Hello,
We want to make all dependencies air gapped so we download it to artifactory and create a secret for all the dependencies. We want to use this as k8's service bindings. From the documentation, i see build resource will not support service binding but image resource does.
Our requirement is to do one-off builds and avoid rebuilds. so dont want to use image resource. The limitation we are facing is that we dont know how to make it air-gapped for build resource by mapping dependencies using k8's secret.
Can someone provide some input ?

Is this what you're looking for https://paketo.io/docs/howto/configuration/#dependency-mappings?

Does  Image Resource support dependency-mapping  bindings?

Yes it is. You can add it under build section.
build:
services:
- name: dependencies
kind: Secret
apiVersion: v1


chore(main): release 1.0.3

:robot: I have created a release beep boop
1.0.3 (2024-11-28)
Bug Fixes

handle null byIso requests (e9af13c)

This PR was generated with Release Please. See documentation.

:robot: Created releases:

v1.0.3
:sunflower:


pybids shipped/used by 20.2.1 LTS is still curious about all things .git/

What version of fMRIPrep are you using?
20.2.1-LTS
What kind of installation are you using? Containers (Singularity, Docker), or "bare-metal"?
I believe a container (dunno details)
crashes with
OSError: Error occurred while trying to decode JSON from file '/SENSORED/.git/annex/transfer/download/590b4fd0-0142-4e9d-8964-d1158c242c6a/MD5E-s102739--2349cac048f4ccf45bcb15278a783d26.json'.

within pybids -- not sure why to look under .git/ at all.

here is the traceback user provided
/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/validation.py:46: UserWarning: The ability to pass arguments to BIDS

Layout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filen

ame`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutInd

exer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.

  warnings.warn("The ability to pass arguments to BIDSLayout that control "

Traceback (most recent call last):

  File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/index.py", line 280, in _index_metadata

    payload = json.load(handle)

  File "/usr/local/miniconda/lib/python3.7/json/__init__.py", line 296, in load

    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)

  File "/usr/local/miniconda/lib/python3.7/json/__init__.py", line 348, in loads

    return _default_decoder.decode(s)

  File "/usr/local/miniconda/lib/python3.7/json/decoder.py", line 340, in decode

    raise JSONDecodeError("Extra data", s, end)

json.decoder.JSONDecodeError: Extra data: line 1 column 18 (char 17)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "/usr/local/miniconda/bin/fmriprep", line 10, in <module>

    sys.exit(main())

  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/run.py", line 17, in main

    parse_args()

  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/cli/parser.py", line 602, in parse_args

    config.from_dict(vars(opts))

  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/config.py", line 617, in from_dict

    execution.load(settings)

  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/config.py", line 218, in load

    cls.init()

  File "/usr/local/miniconda/lib/python3.7/site-packages/fmriprep/config.py", line 453, in init

    re.compile(r"^\."),

  File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 155, in __init__

    indexer(self)

  File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/index.py", line 111, in __call__

    self._index_metadata()

  File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/index.py", line 284, in _index_metadata

    raise IOError(msg) from e

OSError: Error occurred while trying to decode JSON from file '/SENSORED/.git/annex/transfer/download/590b4fd0-0142-4e9d-8964-d1158c242c6a/MD5E-s102739--2349cac048f4ccf45bcb15278a783d26.json'.

didn't try yet to dig deeper, but may be something (like "aha -- was addressed in pybids X which isn't included in this LTS, or didn't make it into LTS) rings a bell?

@effigies et al -- any guidance to guide a user would be appreciated (insofar it was datalad clean && rm -rf .heudiconv)

We may be able to specifically ignore these things, since we'll have a pinned pybids in the lts.
But thanks for looking into this. Have a bit of a backlog...

Trying to reproduce inside a docker container fails. Are you able to provide a minimal reproduction?

will try later on, meanwhile received another user's report with
>    OSError: Error occurred while trying to decode JSON from file
>    '/dartfs-hpc/rc/lab/XXXX/.heudiconv/sid00YYYY/info/filegroup.json'.

Hey @yarikoptic, did you finally find the issue or further info for @effigies to replicate?

will try to provide a complete reproducer today but I guess any broken symlink'ed .json under .heudiconv or .git/annex would trip it.

I think it might have been resolved as of 20.2.3
here is a quick and dirty dump from bash history on how I reproduced first in 20.2.1:
mkdir reproducers
cd reproducers
datalad install ///openneuro/ds000001 && cd ds000001
mkdir -p .git/annex/transfer/download/
ln -s heavens.json .git/annex/transfer/download/inferno.json
ls -l .git/annex/transfer/download/inferno.json
mkdir -p .heudiconv/01/info/
ln -s heavens.json .heudiconv/01/info/wishful.json
datalad get -J4 sub-01

so I got a "random" (first) dataset from openneuro, created broken links to .json files in locations implicated in the issue report, and then (since still lacking space to docker pull the beast even after I freed up over 40GB on /, might be more to it) just used singularity container to reproduce one of the crashes using
mkdir tmp
singularity run -B $PWD:$PWD /home/yoh/proj/repronim/containers/images/bids/bids-fmriprep--20.1.1.sing $PWD $PWD/tmp/out participant --participant-label 01 -w $PWD/tmp/

and now running 20.2.3 using
cd ..
mkdir tmp
rm -rf ds000001/tmp
mkdir licenses
cp ~/.freesurfer-license licenses/
singularity run -B $PWD:$PWD /home/yoh/proj/repronim/containers/images/bids/bids-fmriprep--20.2.3.sing $PWD/ds000001 $PWD/tmp/out participant --participant-label 01 -w $PWD/tmp/ --fs-license-file $PWD/licenses/.freesurfer-license

and it looks like doing useful stuff now (might leave this session open overnight to see it complete), but staying positive, let's consider this resolved for now.


Font Opacity on 2.0 does not work in the same way?

[X] I am on the latest Hyper.app version
[X] I have searched the issues of this repo and believe that this is not a duplicate

OS version and name: Windows 10 x64 (Redstone 3)
Hyper.app version: 2.0
Link of a Gist with the contents of your .hyper.js: https://gist.github.com/iamthemovie/71c924d36400df5d582f93adbf87d104
Relevant information from devtools (CMD+ALT+I on macOS, CTRL+SHIFT+I elsewhere):
The issue is reproducible in vanilla Hyper.app: Using the hyper-material-box and hyperlinks.

Issue

Just updated to 2.0 and it looks like the font opacity is stuck. I was running the latest of 1.X hyper and this was working fine but literally just updated and my font is no longer the correct opacity. I realise this plugin maybe at fault but since it worked on 1.X and now has issues on 2.X I wondered if anyone could point in the direction where this problem might stem from?
1.0 (Nice and lovely)

2.0 (Still nice, not as easy to love)

Edit: The colours look darker between versions on this also https://github.com/zeit/hyper/issues/2821

Yes default colors has been changed: https://github.com/zeit/hyper/pull/2670
We are now using iTerm2 defautls and it permits usage of bright colors.
You can restore previous color scheme by setting in your config, old color scheme:
    colors: {
      black: '#000000',
      red: '#ff0000',
      green: '#33ff00',
      yellow: '#ffff00',
      blue: '#0066ff',
      magenta: '#cc00ff',
      cyan: '#00ffff',
      white: '#d0d0d0',
      lightBlack: '#808080',
      lightRed: '#ff0000',
      lightGreen: '#33ff00',
      lightYellow: '#ffff00',
      lightBlue: '#0066ff',
      lightMagenta: '#cc00ff',
      lightCyan: '#00ffff',
      lightWhite: '#ffffff'
  }

Even themes look dimmer than they did in 1.x. It's not just a matter of the default color scheme changing.

Totally agree, this didn’t fix anything for me.
On Tue, 17 Apr 2018 at 16:46, Dillon<EMAIL_ADDRESS>wrote:

Even themes look dimmer than they did in 1.x. It's not just a matter of
the default color scheme changing.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/zeit/hyper/issues/2827#issuecomment-382041652, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAWhfCjCkmpyj_rLu6ThsNxbppAVX-g4ks5tpg5ygaJpZM4TW3Ih
.

I am experiencing the same issue -- it seems the culprit is a line of css is setting my active window to .5 opacity. I've disabled all of my plugins and issue still persists.
#hyper .term_fit:not(.term_term) {
  opacity: .5
}

@mashaal does this that take into account active pane? For instance if you split then before 2.0 the opacity would be 0.6 on the non-active window (which is super useful). I know it's a temporary fix, I'm just putting this comment for the guys at zeit's benefit 😄

@iamthemovie i saw on a higher up node there was an active class with an opacity set -- not totally sure what the term_fit opacity is meant to be controlling. I feel like this may be an issue with conflicting css -- please see gif below -- in the inspector it shows the css line being overridden, but the .5 opacity is still being rendered regardless?

Admittedly a hacky fix as I haven't done my due diligence on the use of these class selectors in the css but the following works for me on Hyper 2.0.0 (stable):
css: `
  .terminal:not(.focus) {
    opacity: 0.6;
  }
  .terminal.focus {
    opacity: 1;
  }`,

For some reason in my hyper this css is in style tag (I've changed 0.6 to 1 but still WTF? why is it here?):

I added inactive pane dimming in hyper-pane plugin recently.
CSS class selection should be done carefully. Please see:
https://github.com/chabou/hyper-pane/commit/75aba86318259c8df00d9ba0713b4e2de9ba9a80

I'm using this to debug this issue.
// Custom css to embed in the main window.
    css: `
      .term_fit:not(.term_term) {
        opacity: 1 !important
      }
      .terminal: not(.focus) {
        opacity: 1;
        }
      .terminal.focus {
        opacity: 1;
      }
    `,


[KYUUBI #3186] Support applying Row-level Filter and Data Masking policies for DatasourceV2 in Authz module

Why are the changes needed?

RuleApplyRowFilterAndDataMasking in AuthZ currently applys row-level filters correctly on both HiveTableRelation fro Hive tables and LogicalRelation, but is not affcecting the execution plan for Iceberg tables.
As Iceberg table is accssesed via DataSourceV2Relation, RuleApplyRowFilterAndDataMasking is adopted to apply row filtering on TableIdentifier for different relation (including HiveTableRelation and LogicalRelation) and fetch correct table identifier from DataSourceV2Relation.
Considering an iceberg table gftest.sampleice, with an row-level filter as id='1'.
SQL：
EXPLAIN EXTENDED select *from gftest.sampleice limit 5;
Execution Plan before this PR：
== Parsed Logical Plan ==
'GlobalLimit 5
+- 'LocalLimit 5
   +- 'Project [*]
      +- 'UnresolvedRelation [gftest, sampleice], [], false

== Analyzed Logical Plan ==
id: bigint, data: string
GlobalLimit 5
+- LocalLimit 5
   +- Project [id#13332L, data#13333]
      +- SubqueryAlias spark_catalog.gftest.sampleice
         +- RelationV2[id#13332L, data#13333] spark_catalog.gftest.sampleice

== Optimized Logical Plan ==
GlobalLimit 5
+- LocalLimit 5
   +- RelationV2[id#13332L, data#13333] spark_catalog.gftest.sampleice

== Physical Plan ==
CollectLimit 5
+- *(1) ColumnarToRow
   +- BatchScan[id#13332L, data#13333] spark_catalog.gftest.sampleice [filters=] RuntimeFilters: []

Execution Plan before this PR：
== Parsed Logical Plan ==
'GlobalLimit 5
+- 'LocalLimit 5
   +- 'Project [*]
      +- 'UnresolvedRelation [gftest, sampleice], [], false

== Analyzed Logical Plan ==
id: bigint, data: string
GlobalLimit 5
+- LocalLimit 5
   +- Project [id#142L, data#143]
      +- SubqueryAlias spark_catalog.gftest.sampleice
         +- Project [id#142L, data#143]
            +- Filter (id#142L = cast(1 as bigint))
               +- RowFilterAndDataMaskingMarker
                  +- RelationV2[id#142L, data#143] spark_catalog.gftest.sampleice

== Optimized Logical Plan ==
GlobalLimit 5
+- LocalLimit 5
   +- Filter (isnotnull(id#142L) AND (id#142L = 1))
      +- RelationV2[id#142L, data#143] spark_catalog.gftest.sampleice

== Physical Plan ==
CollectLimit 5
+- *(1) Filter (isnotnull(id#142L) AND (id#142L = 1))
   +- *(1) ColumnarToRow
      +- BatchScan[id#142L, data#143] spark_catalog.gftest.sampleice [filters=id IS NOT NULL, id = 1] RuntimeFilters: []

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

Extended PR title, as data masking policies has been also tested locally.

Codecov Report

Merging #3197 (1d47630) into master (a46d655) will decrease coverage by 0.03%.
The diff coverage is 29.41%.

@@             Coverage Diff              @@
##             master    #3197      +/-   ##
============================================
- Coverage     51.40%   51.36%   -0.04%
  Complexity        6        6
============================================
  Files           456      456
  Lines         25422    25432      +10
  Branches       3540     3543       +3
============================================
- Hits          13067    13064       -3
- Misses        11097    11106       +9
- Partials       1258     1262       +4

Impacted Files
Coverage Δ

.../kyuubi/plugin/spark/authz/PrivilegesBuilder.scala
69.37% <ø> (-0.07%)
:arrow_down:

...uthz/ranger/RuleApplyRowFilterAndDataMasking.scala
80.00% <28.57%> (-13.55%)
:arrow_down:

...he/kyuubi/plugin/spark/authz/util/AuthZUtils.scala
43.85% <30.00%> (-2.95%)
:arrow_down:

...rg/apache/kyuubi/ctl/cmd/log/LogBatchCommand.scala
78.00% <0.00%> (-2.00%)
:arrow_down:

...ache/kyuubi/operation/KyuubiOperationManager.scala
80.82% <0.00%> (-1.37%)
:arrow_down:

...ain/scala/org/apache/kyuubi/engine/EngineRef.scala
75.23% <0.00%> (-0.96%)
:arrow_down:

...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala
83.12% <0.00%> (-0.63%)
:arrow_down:

...a/org/apache/kyuubi/service/TFrontendService.scala
91.17% <0.00%> (-0.30%)
:arrow_down:

...org/apache/kyuubi/operation/ExecuteStatement.scala
80.51% <0.00%> (ø)

...in/scala/org/apache/kyuubi/config/KyuubiConf.scala
97.34% <0.00%> (+0.09%)
:arrow_up:

... and 1 more

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

@bowenliang123 thanks for the contribution, and the patience all the way through.
Merged to master.

@bowenliang123 thanks for the contribution, and the patience all the way through.
Merged to master.

Very inspiring and delightful PR experience here. Thanks for all the guidances from feature idea to details of the implementation and even specific code samples.
Also many thanks to my colleague @jeanlyn from Guangfa Securities envolved in this feature.


updating dependencies

Coverage remained the same at 100.0% when pulling 8734bad8d2a76599956ba016bdb755a4380fadad on deps into 616e067daa048fcd75000371b7002caca47a39a1 on master.


Make parameters and variables constant when they are not mutated

Hey, can you please merge this?

Please address my review changes from June.

@jasmcaus I don't see any code reviews from you on this pull request? What are you referring to?

Strange. Just made some updates. LMK if you can see those.

I still don't see it. Are you sure you are submitting the code review? (big green "submit review" button :>)

Yes. Strange why it isn't visible to you.
I'll add below what I'd added:

What's the difference between const char*..., char* const... and const char* const...? -- in tauShouldDecomposeMacro (tau.h:435)
Why the const* const? -- in tauCmdLineRead (tau.h:1019)

Sorry about that. Please see above reply.

const char*... and char const*... are equivalent, they mean that the value of the pointer can change, but the pointer may only store pointers to constant values.
char* const... means that the value of the pointer cannot change, but the pointed-to value may change.
const char* const... and char const* const... combines the two, so you get both a pointer which cannot change, and the value pointed to which cannot change.
int foo = 2;
int const bar = 3;

int *const foo_ptr = &foo;
int const* bar_ptr = &bar;

foo_ptr = world_ptr; // Invalid since foo_ptr variable is constant.
*foo_ptr = 0; // Valid since the foo int is not constant.

*bar_ptr = 1; // Invalid since the value pointed to by bar_ptr is constant.
bar_ptr = foo_ptr; // Valid since the bar_ptr variable is not constant.

// You can go a step further though:
int const* const foo_ptr2 = &foo; // or const char* const foo_ptr2 = &foo;
// This prevents us from modifying the foo int using this pointer even though
// the foo variable itself is not a constant.

// You can also do this:
int const* test_ptr = foo_ptr;
test_ptr = bar_ptr;

The reason I added const in so many places was because most of the variables do not change and if someone wanted to pass constant values to Tau macros/functions, they wouldn't be able to compile. This essentially allows more of the code of Tau and projects using Tau to be immutable, so it gives more guarantees to a human reader. As an extra, you might get extra compiler optimizations thanks to the immutable guarantee.

Hey, hope you didn't forget about this :).

Yup, thanks for this!


Dy sample fix 1x

my mistake, I didn’t think about the fact that with 4 groups it doesn’t work with 1x, don’t connect it yet, I want to test groups = 3 with rgb and 1 with gray and is it even worth using dysample in 1x mode

Added on 1a2733e because this PR conflicted with previous changes.


Demonstrate usage of new bitcoind-json-rpc repo

Demo use of the new repo, has bugs in amount parsing somewhere but I wanted to push this up before the weekend.
Uses PR: https://github.com/tcharding/rust-bitcoind-json-rpc/pull/4
Introduces testing against bitcoind versions

v26.0
v25.2
v25.1
v25.0
v24.2
v24.1
v24.0.1
v23.2
v23.1
v23.0
v22.1
v22.0
v0.21.2
v0.20.2
v0.19.1
v0.18.1
v0.17.2

Thanks for looking. Do you use rust analyzer? I don't, but had the thought this morning that the new client and test crate might be brutal for rust analyzer to handle, I vaguely remember it not working so well with macos.

Superseded by #694


support ansible 2.x

In Ansible 2.x the Python API has changed, the relevant docs are here.
A check is made in L21-27 of ansible.py to try and import ansible.runner and an variable HAS_ANSIBLE is set to True or False as a result.
try:
    import ansible.inventory
    import ansible.runner
except ImportError:
    HAS_ANSIBLE = False
else:
    HAS_ANSIBLE = True

The module ansible.runner has been removed from Ansible 2.x (a PITA I know).
Later a check is made on this variable, if it is set to False the program bails out with an error 'You must install ansible package to use the ansible backend'.

Hi, indeed testinfra isn't compatible with ansible 2.0.
I'm a bit afraid by the API v2 example on https://docs.ansible.com/ansible/developing_api.html#python-api-2-0 (compared to the v1 example)...

Yeah... it's hideous... otherwise I would have just hacked a P/R together ;-) Second system syndrome at it's worst..

Some code for inspiration: https://github.com/seantis/suitable
Also we need to support both ansible 1.X and ansible 2.X

I've got about 70% of the way towards getting it working with Ansible 1 and 2 - I'm not very familiar with how the testinfra code is structured though or Ansible internals.

Created one commit

My copy of the file (master branch) here

I'm getting stuck at figuring out the invocation stuff, and what data structures testinfra expects to see. I've run out of time now for the rest of the week as I'm on a tight deadline. Let me know what you think. I've been testing using virtualenv and manually installing/uninstalling Ansible using pip - on a molecule project like so:
testinfra --connection=ansible --sudo \
--ansible-inventory=.molecule/ansible_inventory --ignore=venv

Let me know if you'd like any more info, I'll try to hack away next week as well but time is very short at the moment.
Bryan

Hi nice work !! I was working on it too with a sample code: https://gist.github.com/philpep/a777410a10766e5a9d39
So It seems that we must use the connection = "smart" in Option and create a callback class (one more :D) to get the results.
I think I can finish this work and integrate in a few days.
My idea is to create a compatibility module that handle v1 and v2 API and just expose get_hosts() and run().

 You may want to refactor but hopefully it's some use to you even in it's current state.
https://github.com/packetops/testinfra/blob/master/testinfra/backend/ansible.py
Feel free to use it as you please.

I'm stuck on this bug: https://github.com/ansible/ansible/issues/13862
Workaround example: https://github.com/seantis/suitable/commit/ac27e02872f977b03002f43d73c21bbdd400f117

@binarytemple I've working code in #50 that you can test.

Sorry @philpep  - I've got to get https://github.com/packetops/ansible-etcd up and working today. If I get any time this weekend I'll take a look. B


[RAY] timeout mechanism for job length or long queues

reflected from private internal issue tracker by @chakrn
When we use caikit-ray-backend to submit a new job, that job can run indefinitely with no timeout. We need a way to have a configurable timeout value and cancel the job after exceeding the time.
After having a quick discussion with Dean, this can probably be done simply by changing the ray.get() to a ray.wait() (which should've been the case anyway). Then poll for status and kill the job after a certain elapsed time.

Thanks @ibm-peach-fish
Clarification on following:

Then poll for status and kill the job after a certain elapsed time.

We need to make the time for timeout configurable. We will be using get_config function from caikit for this, example import statement and example usage. This allows configuration of these variables in multiple fashions, i.e one can configure it via config.yml or can set a value via environment variable.


Question about exceptions

There is any possible way of using this lib without exceptions, or it always throws?
It would be nice something like:
json j =R"({,})"_json;
std::cout << j.is_parsed_sucess();

In case of a parse error, the code will always throw an exception. So far, I do not plan to make exceptions optional.
For your use case, you may catch std::invalid_argument and create a wrapper function is_parsed_success yourself.

std::invalid_argument does not work.
On a related note, I don't see a single word about JSON parsing errors in the documentation.

FYI, it is now possible to disable exceptions in the library.

@berkus The last released version works with std::invalid_argument. The develop branch contains an unreleased version with user-defined exceptions. There, the parser throws json::parse_error.


Add doc for updating Node notifier for APM

This doc will be used to show users how to update their notifier
version. The recommended version is needed to take advantage of
Performance Monitoring features.
New doc

Good points. I will make a separate doc for updating from node-airbrake aka

https://www.npmjs.com/package/airbrake
https://github.com/airbrake/node-airbrake

It will keep things cleaner if they are separate docs. We did the same for our Ruby/Rails notifiers:

https://airbrake.io/docs/ruby/upgrading-from-deprecated-versions/
https://airbrake.io/docs/ruby/upgrading-your-notifier/

I'll add a note mentioning the package name change from airbrake-js to @airbrake/node too.

PTAL
I added two new docs:

Updating to @airbrake/node from airbrake-js
Updating to @airbrake/node from node-airbrake

Screenshot: airbrake-js

Screenshot: node-airbrake


acrn_ehl_crb: testcases tests/kernel/smp failed to run on v2.7-branch

Describe the bug
The test case tests/kernel/smp failed on v2.7-branch on the ACRN board. I tested the case on the main branch, and it works.
To Reproduce
twister -p acrn_ehl_crb --device-testing --device-serial-pty=$HOME/acrn-env/acrn-test-pty.exp --west-flash=$HOME/acrn-env/acrn_efi.sh -v -T tests/kernel/smp/ -v
Logs and console output
ACRN:\>
DEBUG   - DEVICE: ACRN:\>vm_console 0
DEBUG   - DEVICE:
DEBUG   - DEVICE: ----- Entering VM 0 Shell -----
DEBUG   - DEVICE: *** Booting Zephyr OS build zephyr-v2.7.3  ***
DEBUG   - DEVICE: Running test suite smp
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: START - test_smp_coop_threads
DEBUG   - DEVICE: PASS - test_smp_coop_threads in 0.563 seconds
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: START - test_cpu_id_threads
DEBUG   - DEVICE: PASS - test_cpu_id_threads in 1.1 seconds
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: START - test_coop_resched_threads
DEBUG   - DEVICE: PASS - test_coop_resched_threads in 0.301 seconds
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: START - test_preempt_resched_threads
DEBUG   - DEVICE: PASS - test_preempt_resched_threads in 0.512 seconds
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: START - test_yield_threads
DEBUG   - DEVICE: PASS - test_yield_threads in 0.301 seconds
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: START - test_sleep_threads
DEBUG   - DEVICE: PASS - test_sleep_threads in 1.1 seconds
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: START - test_wakeup_threads
DEBUG   - DEVICE: PASS - test_wakeup_threads in 0.201 seconds
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: START - test_smp_ipi
DEBUG   - DEVICE: cpu num=2 PASS - test_smp_ipi in 0.301 seconds
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: START - test_get_cpu
DEBUG   - DEVICE: PASS - test_get_cpu in 0.51 seconds
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: START - test_fatal_on_smp
DEBUG   - DEVICE: E: RAX: 0x0000000000000003 RBX: 0x0000000000135168 RCX: 0x00000000000f4240 RDX: 0x0000000d00000000
DEBUG   - DEVICE: E: RSI: 0x0000000000000200 RDI: 0x00000000000007d0 RBP: 0x00000000001203c0 RSP: 0x00000000001203c0
DEBUG   - DEVICE: E:  R8: 0x0000000000000001  R9: 0x00000000001388d8 R10: 0x0000000000000000 R11: 0x0000000000000000
DEBUG   - DEVICE: E: R12: 0x000000000010f360 R13: 0x0000000000000200 R14: 0x000000000010f360 R15: 0x0000000000000000
DEBUG   - DEVICE: E: RSP: 0x00000000001203c0 RFLAGS: 0x0000000000000206 CS: 0x0018 CR3: 0x0000000000140000
DEBUG   - DEVICE: E: RIP: 0x00000000001009bb
DEBUG   - DEVICE: E: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 1
DEBUG   - DEVICE: E: Current thread: 0x10e940 (test_fatal_on_smp)
DEBUG   - DEVICE: PASS - test_fatal_on_smp in 0.5 seconds
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: START - test_workq_on_smp
DEBUG   - DEVICE:
DEBUG   - DEVICE: Assertion failed at WEST_TOPDIR/zephyr/tests/kernel/smp/src/main.c:712: test_workq_on_smp: k_work_busy_get(&work) not equal to 0
DEBUG   - DEVICE:
DEBUG   - DEVICE: FAIL - test_workq_on_smp in 0.51 seconds
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: START - test_smp_release_global_lock
DEBUG   - DEVICE: E: RAX: 0x0000000000000003 RBX: 0x0000000005a995c0 RCX: 0x0000000000000000 RDX: 0x0000000000000000
DEBUG   - DEVICE: E: RSI: 0x0000000000000000 RDI: 0x0000000000000000 RBP: 0x000000000011d7f0 RSP: 0x000000000011d7e8
DEBUG   - DEVICE: E:  R8: 0x0000000000100149  R9: 0x000000000010a862 R10: 0x0000000000000000 R11: 0x0000000000000000
DEBUG   - DEVICE: E: R12: 0x000000008b93a39c R13: 0x0000000000000200 R14: 0x000000000010e940 R15: 0x0000000000000000
DEBUG   - DEVICE: E: RSP: 0x000000000011d7e8 RFLAGS: 0x0000000000000202 CS: 0x0018 CR3: 0x0000000000140000
DEBUG   - DEVICE: E: RIP: 0x0000000000100152
DEBUG   - DEVICE: E: >>> ZEPHYR FATAL ERROR 3: Kernel oops on CPU 1
DEBUG   - DEVICE: E: Current thread: 0x10d760 (unknown)
DEBUG   - DEVICE:
DEBUG   - DEVICE: Assertion failed at WEST_TOPDIR/zephyr/tests/kernel/smp/src/main.c:649: k_sys_fatal_error_handler: (main_thread_id != child_thread_id is false)
DEBUG   - DEVICE: fatal on the same core
DEBUG   - DEVICE: FAIL - test_smp_release_global_lock in 0.23 seconds
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: START - test_inc_concurrency
DEBUG   - DEVICE: type 0: cnt 60000, spend 8 ms
DEBUG   - DEVICE: type 1: cnt 60000, spend 73 ms
DEBUG   - DEVICE: type 2: cnt 60000, spend 123 ms
DEBUG   - DEVICE: PASS - test_inc_concurrency in 0.203 seconds
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: Test suite smp failed.
DEBUG   - DEVICE: ===================================================================
DEBUG   - DEVICE: PROJECT EXECUTION FAILED

Environment (please complete the following information):

OS: Linux
Toolchain: Zephyr SDK 14.1
Commit: b9fc341

cc @cfriedt

This issue is not stable to reproduce, it's difficult to bisect. Still working on looking for what's commit caused this bug.

@Zhaoningx any update?

Tests with a randomized value and tests with SMP enabled can sometimes fail due to race conditions and timeouts. Sometimes we need to increase timeouts as a result. In CI, we explicitly retry tests with --retry-failed $N for some N (I usually choose 3, and most times the test pass, but sometimes they can fail once).
I would be much more concerned if tests fail in a majority of 10 trials.
@Zhaoningx - can you retry with --retry-failed 10?

Thank you @cfriedt , sure, run this test case with "--retry-failed 10" it definitely pass, but "--retry-failed 3" it doesn't always pass.
So if "--retry-failed 10" is acceptable for daily testing, it could be a solution.

CI tries 3 times, for référence. It could be timing related as well, so perhaps increasing some timeout would make the test more reliable
https://github.com/zephyrproject-rtos/zephyr/blob/fad899d2ad5e3c5b857ee0b5a331185addcc8deb/.github/workflows/twister.yaml#L136

acrn_ehl_crb use apic_tsc timer, for acrn, I suspect that CONFIG_SYS_CLOCK_TICKS_PER_SEC=10000 is too high, I set it to 1000, 800, 500, the failed test cases passed.

acrn_ehl_crb use apic_tsc timer, for acrn, I suspect that CONFIG_SYS_CLOCK_TICKS_PER_SEC=10000 is too high, I set it to 1000, 800, 500, the failed test cases passed.

Nice catch @Zhaoningx - do you know if this bug is present or has already been fixed in main? In either case, are you able to make a PR?

This bug is not present in main, not sure it is fixed by some patches or by side effect of some patches.

After debugging, I found issue happens because at certain point of timing, the hypervisor will have a long delay.
(At about uptime at 106 ms, 632 ms, we will get many cycles late when we call k_cycle_get_32/64(). Actually, we found this via  the tests/kernel/timer/timer_behavior test, add some debug info we see the clock cycle we got has large drift at this two timing.
These points affects some timing sensitive testcases (such as tests/timer/timer_behavior, tests/kernel/smp etc) or even some scheduling tests those measure the time.We are still not totally clear why the hypervisor will have such a delay at these point. It is most likely not an issue of Zephyr, because the clock cycle read from rdtsc provided by hypervisor.
Although the failed test are test_workq_on_smp() and test_smp_release_global_lock(), but the failure only have something to do with the timing of the tests executed.
A suggested workaround solution is to add a 700ms boot delay to avoid these point. We can add CONFIG_BOOT_DELAY=700 for it.
After applying the 700ms DELAY for acrn_ehl_crb, test the SMP test pass in 20 times.  Hi, @mengxianglinx , you can verify it, thanks.

Yes @enjiamai , I run twister with CONFIG_BOOT_DELAY=700 15 times,  the failed case passed.


pom not deployed properly for 2.0.4.RELEASE

Building a project against spring-cloud-connectors-core:2.0.4.RELEASE I get
[WARNING] The POM for org.springframework.cloud:spring-cloud-connectors-core:jar:2.0.4.RELEASE is missing, no dependency information available

I don't see a reference to 2.0.4.RELEASE on Maven central
The pom is also missing from the Sonatype's OSS release repository

The .pom file is now in the Sonatype OSS repository and the Maven Central repository.
The Maven Central search index should get updated soon.


Issue on ZipkinSpanCollector and Zipkin collector communication

Hi,
It's been a month that I discovered Brave and I am currently struggling with Brave and the Zipkin backend components.
My architecture is as follows:

Client 1 (inside JVM 1)
Server 1 (inside JVM 2)
Server 2 (inside JVM 3)

Ok so let's say that I have the following calls that are made (my program entry point is Client 1):

Client 1 requests Server 1 (first trace)
Server 1 requests Server 2 (second trace)
Server 2 requests Server 1 (third trace)

My problem is that the second trace and the third trace are correctly displayed on the zipkin UI but the first trace shows only the SS and SR annotations. I added some logging (with log4j) inside the collect method of the ZipkinSpanCollector class and I can see the CS and CR annotation of my Client 1 in my logs (with the same span-id and trace-id as the server-side).
Plus: the spanQueue.offer(span) line returns true.
Do you have any idea where my problem could come from?
Best regards,
PS: I posted my question as an issue since I didn't find any google-group or IRC channel talking only about brave. I hope it's not a problem.
--
Hugo

Hi,
Just so you know I found the problem. I don't use Spring so the ZipkinSpanCollector.close() method was not used and the spans that were put in the spanQueue were never processed by the SpanProcessingThread.
I temporarly fixed that by calling get() on each element of the futures list when Future.isDone method returns true. I don't think it's a good idea but for now that's what I wanted.
Just in case it helps someone.
Thanks for your project btw.
--
Hugo

@arensteak Good that you found the issue! And thanks for explaining the problem. It might indeed help others.  All the best!


feat: adds properties to construct to improve reuse

Fixes #

Let's wait until #182 #183 is merged.


Can't see my profile tooltip (Trp3-Classic)

Describe the bug
A clear and concise description of what the bug is.
To Reproduce
Steps to reproduce the behaviour:

Hover over character frame

Expected behaviour
Instead of getting my profile tooltip in the lower right corner, I get information about party options.
Screenshots or error message

Version of the add-on:
Total RP 3: Classic 1.13.2
Did you try having Total RP 3 as the only enabled addon and everything else (especially something like ElvUI) disabled?
Yes
Additional context
Add any other context about the problem here.

Can you run the following command in the chat frame: /run SetCVar("showNewbieTips", "0")
You may get prompted to enable dangerous scripts, if so accept it and re-run it.

This is not a bug but rather due to the behaviour of a setting, just disable "Show Detailed Tooltips" in the Display category of the Interface Options.

This is not a bug but rather due to the behaviour of a setting, just disable "Show Detailed Tooltips" in the Display category of the Interface Options.

Oh, thank you very much! I tried going through my settings but couldn't find it.


A dynamic link library (DLL) initialization routine failed

SerialPort version:latest
NodeJS Version:7.0.0
Operating System and Hardware Platform:win10
Are you using an alternative NodeJS runtime? (eg Electron):electron
Summary of Problem
When I run the electron, throw such as error!
Steps and Code to Reproduce the Issue
the code:
var SerialPort = require("serialport").SerialPort;
var portName = 'COM1';
var serialPort = new SerialPort(portName, {
  baudRate: 9600,
  dataBits: 8,
  parityL:'none',
  stopBits: 1,
  flowControl: false
}, false);
serialPort.list(function (err, ports) {
  ports.forEach(function(port) {
    console.log(port.comName);
    console.log(port.pnpId);
    console.log(port.manufacturer);
  });
});

Yes,I try to run after packaging，it's normal, but the debug run under electron or DLL error,I has update the electron to 1.6.0,but the debug  run the node is normal

Yes,I try to run after packaging，it's normal, but the debug run under electron or DLL error,I has update the electron to 1.6.0,but the debug run the node is normal

i have the same problem!

same problem here

look at here http://www.jianshu.com/p/696fbdeb5b8a

Hi,
I have the same error!!

same error

same erro here

@Caumaker @Quzal @datamweb
a easy way to do
export npm_config_target=1.2.3
export npm_config_arch=x64
export npm_config_target_arch=x64
export npm_config_disturl=https://atom.io/download/electron
export npm_config_runtime=electron
export npm_config_build_from_source=true

if Windows  , export  use set, like
set npm_config_target=1.2.3
target is electron version
and then
npm install serialport --save
good luck

Hi,@hanfengcan help me step to step.please

@datamweb

export that env_variables if you use linux. or use set  in windows.
use npm install serialport --save

@hanfengcan Thanks for your reply.
But ...

make sure you can build in nodejs.
rebuild in electron
build a new electron project, if you use vue, you can create new project by https://github.com/SimulatedGREG/electron-vue
And add serialport code to main/index.js


Python Version Updated to 3.11 in GoogleApigee and Google Cloud Platform Cloud Monitoring

Required items, please complete
Change(s):
Change in Python Version to 3.11
Repackaged the solution for Data Connector Instructions.
Reason for Change(s):
-As per requirement (Deprecation of 3.8)
Version Updated:
Python version - 3.11
Solution version - 3.0.0
Testing Completed:
Yes

approved

Approving and merging on behalf of Alex Shevchenko.


attempt to index local 'location' error

I don't have anything special in my config except just the plugin string for sg.nvim on top of latest lazyvim distro

Can you run :checkhealth sg for me? And please update to latest plugin, I just pushed some pretty big changes

@tjdevries

and for reference https://github.com/kapral18/dotfiles/blob/3b2e742c50c36c4d6d4e284af8bb9b8b90c2b3cd/home/dot_config/exact_nvim/exact_lua/exact_plugins/ai.lua#L105

The error is still happening?

@tjdevries yepp

Can you run :checkhealth sg for me? And please update to latest plugin, I just pushed some pretty big changes

I noticed and I tried. I also nuked nvim instance with cache and state folders to make sure it's not affecting.
Still the same result unfortunately

Does this PR fix your problem? https://github.com/sourcegraph/sg.nvim/pull/221

PR removes the errors, but at the same time going to definition doesn't happen as well

Also seeing this. Just logging the error in textDocument/definition gives: 'RPC[Error] code_name = ParseError, message = "No corresponding code intelligence"'. Some extra context that may or may not be relevant is that my sg url is not the default one.

I am also seeing this error with the default sourcegraph url and my health check looks healthy. Happy to give any extra info!

I'm using it on a local instance, and getting the same error as well


Add cmp=False to as_bytes and as_str of DataObjxyz's

Including them in the comparison leads to infinite recursion, and they're not relevant to comparisons anyways.
Hopefully greatly simplifies testing, too!  Seems like it should allow a simple == test, rather than iterating through each relevant member.

Note as versionchanged in docs that equality comparison will work on DataObj... instances now.
Also changed Inventory, to allow equality comparisons...add versionchanged to docs for this as well.


cmd/link: rare corruption of ELF binaries

What version of Go are you using (go version)?

$ go version
go version go1.18 linux/amd64

Does this issue reproduce with the latest release?
Yes, Go 1.18 is the latest (stable) release.
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/michael/.cache/go-build"
GOENV="/home/michael/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/michael/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/michael/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/michael/sdk/go1.18"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/michael/sdk/go1.18/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/michael/2022-06-25-squashfs-debug/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1868362460=/tmp/go-build -gno-record-gcc-switches"

What did you do?
I am building and deploying Go software from a cron job every day.
Recently, I noticed that sometimes, some of my executable binary files do not start up because they are corrupt!
The first time I noticed the issue, the init binary of one of my https://gokrazy.org/ installations was affected, resulting in an installation that wouldn’t boot at all.
The other time, it wasn’t the init binary, but a program of mine called regelwerk which is involved in motion sensor/light control in my home, so I noticed that because the lights weren’t working as they should.
It’s possible this happened more times and I just didn’t notice it.
Yesterday, I found someone on twitter who is also running into this issue, but with an entirely different program (not related to gokrazy at all): https://twitter.com/alvs_versteck/status/1546601648532983808
What did you expect to see?
The Go compiler/linker should produce ELF binaries that contain a valid ELF header.
What did you see instead?
The first 4096 bytes of the ELF binary are zeroed out, as well as another block of 4096 bytes at offset 256K.
You can find the files at https://t.zekjur.net/_2022-06-25-init/
In the other occurrence, it was 4096 bytes at the start of the ELF binary, then 4096 bytes at offset 0x9000.
Unfortunately I have no idea how to reproduce this issue.

Are the corrupted binaries deterministic?
If not, I wonder if #51611 is related.

Are the corrupted binaries deterministic?

WDYM by that, if they are always corrupted in the same way? If yes, I think so but currently only have one sample lying around where this happened so would need to wait for it to happen again to be able to tell for sure.

CC @golang/compiler

Is the corruption always on the ELF header?  Is it on the same machine or on multiple machines? What the file system are you running on? Thanks.

Is the corruption always on the ELF header?

I think so, I remember I got a couple of cannot execute binary file: Exec format error errors and rebuilding helped. I only have one lying around right now though and it happens intermittently, so I can't say for sure until it happens again.

Is it on the same machine or on multiple machines?

Multiple, mine and @stapelberg also sees it on his

What the file system are you running on?

xfs

Thanks. Frequent corruption on the ELF header is probably enough information. The ELF header is written by the linker only. The compiler  and (most of) the Go command are unrelated. The one place that the go command touches the binary after linking is stamping the build ID. So it is either  the linker or the that build ID stamping.
Is your program a pure-Go binary or it uses cgo (i.e. whether it is internal linking or external linking)?

Is your program a pure-Go binary or it uses cgo (i.e. whether it is internal linking or external linking)?

Mine is pure go (i.E. CGO_ENABLED=0)

4096 = block size of ext4, file system bug?

4096 = block size of ext4, file system bug?

Again: note that both the source code (generated) and the binary output location are on /tmp, which is a tmpfs mount in my case.
So I don’t think a file system bug is likely here.

Thanks. Pure-Go means that the file is written by the Go linker, not the C linker.
If this is somewhat reproducible, could you try if this patch makes any difference? (You'll need to rebuild the linker,  go install cmd/link)
diff --git a/src/cmd/link/internal/ld/outbuf_mmap.go b/src/cmd/link/internal/ld/outbuf_mmap.go
index 40a3222788..c3cc157579 100644
--- a/src/cmd/link/internal/ld/outbuf_mmap.go
+++ b/src/cmd/link/internal/ld/outbuf_mmap.go
@@ -9,6 +9,7 @@ package ld

 import (
        "syscall"
+       "unsafe"
 )

 // Mmap maps the output file with the given size. It unmaps the old mapping
@@ -55,6 +56,7 @@ func (out *OutBuf) munmap() {
        if out.buf == nil {
                return
        }
+       syscall.Syscall(syscall.SYS_MSYNC, uintptr(unsafe.Pointer(&out.buf[0])), uintptr(len(out.buf)), syscall.MS_SYNC) // XXX
        syscall.Munmap(out.buf)
        out.buf = nil
 }

Also, do you know if this is a new bug, or it occurs with old versions of Go as well? Thanks.

@cherrymui I haven't seen this before using go 1.18, so I guess it is new. I did some more experimenting and found out:

I can somewhat reliably reproduce this by go building in a loop, in five attempts it always took less than 200 builds
It is not reproducible in a docker container, so seems specific to my env
It is reproducible with an empty go cache
It is not reproducible with a hello world go program, so seems to either be specific to my project or maybe just the size of the codebase

Regarding your patch, I suppose I need to replace the link binary in the GOTOOLDIR? I did that and the issue still happens. I then looked more at the link binary and it has the same sha256sum regardless of applying your patch or not...?
$ g diff
diff --git a/src/cmd/link/internal/ld/outbuf_mmap.go b/src/cmd/link/internal/ld/outbuf_mmap.go
index 40a3222788..c3cc157579 100644
--- a/src/cmd/link/internal/ld/outbuf_mmap.go
+++ b/src/cmd/link/internal/ld/outbuf_mmap.go
@@ -9,6 +9,7 @@ package ld

 import (
        "syscall"
+       "unsafe"
 )

 // Mmap maps the output file with the given size. It unmaps the old mapping
@@ -55,6 +56,7 @@ func (out *OutBuf) munmap() {
        if out.buf == nil {
                return
        }
+       syscall.Syscall(syscall.SYS_MSYNC, uintptr(unsafe.Pointer(&out.buf[0])), uintptr(len(out.buf)), syscall.MS_SYNC) // XXX
        syscall.Munmap(out.buf)
        out.buf = nil
 }
<EMAIL_ADDRESS>$ go build -o /tmp/link cmd/link/
<EMAIL_ADDRESS>$ sha256sum /tmp/link
e7e671ba4d83e6f3a820ba3f0269ef21e26f2c477938499d389f58a696ec7e19  /tmp/link
<EMAIL_ADDRESS>$ g reset --hard
HEAD is now at 558785a0a9 cmd/compile: remove -installsuffix flag
<EMAIL_ADDRESS>$ go build -o /tmp/link cmd/link/
<EMAIL_ADDRESS>$ sha256sum /tmp/link
e7e671ba4d83e6f3a820ba3f0269ef21e26f2c477938499d389f58a696ec7e19  /tmp/link

This is weird. The patch does make a difference on my machine
$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
$ git log -1 --oneline
558785a0a9 (HEAD -> master, origin/master, origin/HEAD) cmd/compile: remove -installsuffix flag
$ go build -o /tmp/link cmd/link
$ sha256sum /tmp/link
173de0c0efed5ea3c57ef497effe803f20c60603889de4c64cb1ac726b7ed45f  /tmp/link

$ git apply </tmp/p
$ git diff
diff --git a/src/cmd/link/internal/ld/outbuf_mmap.go b/src/cmd/link/internal/ld/outbuf_mmap.go
index 40a3222788..c3cc157579 100644
--- a/src/cmd/link/internal/ld/outbuf_mmap.go
+++ b/src/cmd/link/internal/ld/outbuf_mmap.go
@@ -9,6 +9,7 @@ package ld

 import (
        "syscall"
+       "unsafe"
 )

 // Mmap maps the output file with the given size. It unmaps the old mapping
@@ -55,6 +56,7 @@ func (out *OutBuf) munmap() {
        if out.buf == nil {
                return
        }
+       syscall.Syscall(syscall.SYS_MSYNC, uintptr(unsafe.Pointer(&out.buf[0])), uintptr(len(out.buf)), syscall.MS_SYNC) // XXX
        syscall.Munmap(out.buf)
        out.buf = nil
 }
$ go build -o /tmp/link cmd/link
$ sha256sum /tmp/link
3237cd32022b5b6cee6a9286cdec0d10e7e24455d9ee610553e8caecd8754c81  /tmp/link

$ git reset --hard
HEAD is now at 558785a0a9 cmd/compile: remove -installsuffix flag
$ go build -o /tmp/link cmd/link
$ sha256sum /tmp/link
173de0c0efed5ea3c57ef497effe803f20c60603889de4c64cb1ac726b7ed45f  /tmp/link

Maybe I messed up tabs and spaces when I first pasted the patch, so it didn't directly apply. Try this.
diff --git a/src/cmd/link/internal/ld/outbuf_mmap.go b/src/cmd/link/internal/ld/outbuf_mmap.go
index 40a3222788..c3cc157579 100644
--- a/src/cmd/link/internal/ld/outbuf_mmap.go
+++ b/src/cmd/link/internal/ld/outbuf_mmap.go
@@ -9,6 +9,7 @@ package ld

 import (
 	"syscall"
+	"unsafe"
 )

 // Mmap maps the output file with the given size. It unmaps the old mapping
@@ -55,6 +56,7 @@ func (out *OutBuf) munmap() {
 	if out.buf == nil {
 		return
 	}
+	syscall.Syscall(syscall.SYS_MSYNC, uintptr(unsafe.Pointer(&out.buf[0])), uintptr(len(out.buf)), syscall.MS_SYNC) // XXX
 	syscall.Munmap(out.buf)
 	out.buf = nil
 }

@alvaroaleman I think the problem you ran into when trying to apply the patch is:

go build -o /tmp/link cmd/link/

Go will not use the local directory, but your installation’s cmd/link source instead. You need to use:

go build -o /tmp/link ./cmd/link/

You can verify the code was updated by using go tool objdump ./pkg/tool/linux_amd64/link. Search the output for outbuf_mmap.go and look for the munmap code. You should see disassembly like:
  outbuf_mmap.go:55     0x5ae5d2<PHONE_NUMBER>              MOVQ AX, 0x50(SP)
  outbuf_mmap.go:59     0x5ae5d7<PHONE_NUMBER>              MOVQ DX, 0x38(SP)
  outbuf_mmap.go:59     0x5ae5dc                488b4818                MOVQ 0x18(AX), CX
  outbuf_mmap.go:59     0x5ae5e0                48c704241a000000        MOVQ $0x1a, 0(SP)
  outbuf_mmap.go:59     0x5ae5e8<PHONE_NUMBER>              MOVQ DX, 0x8(SP)
  outbuf_mmap.go:59     0x5ae5ed                48894c2410              MOVQ CX, 0x10(SP)
  outbuf_mmap.go:59     0x5ae5f2                48c744241804000000      MOVQ $0x4, 0x18(SP)
  outbuf_mmap.go:59     0x5ae5fb                0f1f440000              NOPL 0(AX)(AX*1)
  outbuf_mmap.go:59     0x5ae600                e8bbf0eeff              CALL syscall.Syscall.abi0(SB)
  outbuf_mmap.go:59     0x5ae605                450f57ff                XORPS X15, X15
  outbuf_mmap.go:59     0x5ae609                644c8b3425f8ffffff      MOVQ FS:0xfffffff8, R14
  outbuf_mmap.go:60     0x5ae612                488b442450              MOVQ 0x50(SP), AX
  outbuf_mmap.go:60     0x5ae617                488b5810                MOVQ 0x10(AX), BX
  outbuf_mmap.go:60     0x5ae61b                488b4818                MOVQ 0x18(AX), CX
  outbuf_mmap.go:60     0x5ae61f                488b7820                MOVQ 0x20(AX), DI
  syscall_linux.go:1142 0x5ae623                488b1576be2600          MOVQ syscall.mapper(SB), DX
  syscall_linux.go:1142 0x5ae62a                4889d0                  MOVQ DX, AX
  syscall_linux.go:1142 0x5ae62d                e88eceeeff              CALL syscall.(*mmapper).Munmap(SB)

If you can’t find munmap, look for Close — maybe the code got inlined.
If you only see the syscall.(*mmapper).Munmap call, the patch was not applied correctly. If you see the syscall.Syscall.abi0 call, the patch was applied correctly.
I have applied the patch on my system and will report back if I still encounter the corruption. However, as I can’t trigger it even when building hundreds of times in a loop, it would be better if @alvaroaleman could report back, as it sounds like it happens more frequently for you?
Thanks

We also encountered the same problem (see: #53116),   On some linux/loong64 machines, an "exec format error" error occasionally occurs when executing the make.bash or all.bash. Check these executable files with errors, it is found that the data of 16K(only one pagesize)  at the beginning of the file is missing. The result of hexdump is as follows:

   1  0000000 0000 0000 0000 0000 0000 0000 0000 0000
   2  *
   3  0004000 0020 4c00 3d24 1c01 d084 02fc 2064 29c0
   4  0004010 9004 02c0 4064 29c0 4800 576a 0000 0014
   5  0004020 62d3 28c0 8e73 0012 1260 5c00 0025 0015

This problem is caused by the occasional loss of 16K data at the beginning of the file after the link process calls the syscall.Fallocate function for the second time. However, not all machines can reproduce this error (the file system formats I tested include xfs, ext4, tmfs ).

my kernel version is  5.17.0+, pagesize is 16K。
In the case of ensuring enough disk space, I removed the second fallocate call, the problem no longer occurs.
I tried doing MS_SYNC before munmap, but the problem persists

Thanks for adding to this report, @abner-chenc!
I also just encountered this corruption again today, with @cherrymui’s patch applied — so the extra MSYNC does not help unfortunately.

I tried   using posix_fallocate (the implementation refers to glib's posix_fallocate.c) replaces syscall.fallcate, and this error does not occur on the same machine.
Here is my implementation of posix_fallocate:    https://github.com/abner-chenc/go-abner/commit/c43a4a75a031ac9550eb02697b50032f1dced9ae

@stapelberg thanks for the hint, using go build -o /tmp/link ./cmd/link just resulted in main module (std) does not contain package std/cmd/link and building from the cmd folder as go build -o /tmp/link ./link still ended up using my installations cmd/link. I ended up applying the patch there, but as you already noted, it doesn't fix the issue.

Seems like @abner-chenc patch is indeed fixing this, I couldn't reproduce the issue anymore in 500 builds when I never needed more than 200 for it to occur prior to their patch.

Are all the Exec format error case that the first page is missing? Or (sometimes) just the ELF header?
Does the fallocate syscall not work on your file system? But it also doesn't return an error? The fallocate isn't really necessary unless you're running low on disk space. Is that the case on your system? What happens if you just comment that fallocate call out?
@abner-chenc your patch also reorders  Truncate and fallocate. What happens if you don't reorder them, or just reorder them without doing the rest?

In the case of ensuring enough disk space, I removed the second fallocate call, the problem no longer occurs.

Could you explain what you did exactly? What  is "the second fallocate call"?
@alvaroaleman it looks  like @abner-chenc 's patch only affects  Loong64. Just to make  sure,  did you change  it to apply to your architecture as well?
Thanks.

@abner-chenc are you also running on tmpfs?
@alvaroaleman @stapelberg does it reproduce on a non-tmpfs file system?
Thanks.

@cherrymui I just applied https://github.com/abner-chenc/go-abner/commit/c43a4a75a031ac9550eb02697b50032f1dced9ae.patch without any changes, then built the link binary. While testing it, I output the binary to /tmp so a tmpfs, but when I orginally encountered the error I had targeted an xfs fs. I can check later if it also fixes the issue when targeting the xfs fs.

That patch should not make any difference if you're not running on Loong64. Would be good to try on xfs. Thanks.

Thanks. Could be. Could you try just moving the Truncate call without the rest of the change? Thanks.

@cherrymui yes, reducing the patch to only the changes in outbufs_mmap.go as shown below is enough to make the issue go away. Thanks for taking the time to look into this btw!
--- a/src/cmd/link/internal/ld/outbuf_mmap.go
+++ b/src/cmd/link/internal/ld/outbuf_mmap.go
@@ -20,6 +20,10 @@ func (out *OutBuf) Mmap(filesize uint64) (err error) {
                out.munmap()
        }

+       err = out.f.Truncate(int64(filesize))
+       if err != nil {
+               Exitf("resize output file failed: %v", err)
+       }
        for {
                if err = out.fallocate(filesize); err != syscall.EINTR {
                        break
@@ -33,10 +37,6 @@ func (out *OutBuf) Mmap(filesize uint64) (err error) {
                        return err
                }
        }
-       err = out.f.Truncate(int64(filesize))
-       if err != nil {
-               Exitf("resize output file failed: %v", err)
-       }
        out.buf, err = syscall.Mmap(int(out.f.Fd()), 0, int(filesize), syscall.PROT_READ|syscall.PROT_WRITE, syscall.MAP_SHARED|syscall.MAP_FILE)
        if err != nil {
                return err

Thanks.  If that is the fix then it seems that it should be possible to cause the same problem in a fairly simple C program.

Thanks @alvaroaleman !
I couldn't find the requirement of ftruncate before fallocate being documented anywhere. But it is probably a good thing to do to swap the two calls anyway.
cc @aclements in case you know something about the FS.

@abner-chenc your patch also reorders Truncate and fallocate. What happens if you don't reorder them, or just reorder them without doing the rest?

I have tested and found that by exchanging the calling order of 'fallocate' and 'ftruncate',  the occurrence of this error can be reduced,  but it cannot completely solve the problem.

Could you explain what you did exactly? What is "the second fallocate call"?

In the function copyHeap(outbuf.go), if len(out.heap) > 0,   syscall.Fallocate will be called again to allocate more disk space.
I tested and found that it is this Fallocate call that occasionally causes the loss of existing data in the file, all missing data is the beginning of the file 16K,   exactly one page size.
I tried executing MS_SYNC before munmap, but the problem persists.

@abner-chenc are you also running on tmpfs?
Yes, I also tested on tmpfs

I think mixing file IO syscalls and mmaps on the same file is probably not a good idea. So your f.ReadAt may not work as expected. (For the same reason I think we don't want to use the implementation of posix_fallocate.)  Can you observe the same corruption if you read the first 4 bytes from the mmap immediately after the mmap call?
Also, in the patch above the truncate call  is still after the fallocate. Could you try moving it up?
On the other hand, if fallocate causes occasional data lost, it sounds like an OS bug to me. Could you try writing a C program to see if this happens? Thanks.

The problem still happens with Go 1.19 (just happened to me again).

Has anyone tried to reproduce the failure in C?
It is interesting to know why it seems to only fail for you. Like, it doesn't fail on the builders (except loongarch). It would be good to know what is special on those machines, and why it fails. Thanks.

I tried testing with C code  fallocate.c , but the problem doesn't occur.

I tested and found that when fallocate returns the error code EINTR for the first time, and trying to fallocate again will cause the problem to appear (the second time no error code is returned).

I tried preallocate new space at the end of the file and the problem didn't reappear by testing, but I don't know what the root cause is.  this is my patch

Thanks for trying the C code.

when fallocate returns the error code EINTR for the first time, and trying to fallocate again will cause the problem to appear

Could you try in the C code, having a thread (or process) sending some signals to the fallocate thread, so it may return EINTR?

Thanks for trying the C code.

when fallocate returns the error code EINTR for the first time, and trying to fallocate again will cause the problem to appear

Could you try in the C code, having a thread (or process) sending some signals to the fallocate thread, so it may return EINTR?

@cherrymui

I can reproduce the error of fallocate returning EINTR in the C code by sending a signal to another process, and found that the problem of data loss does occur.  My C code:  https://github.com/abner-chenc/abner/blob/master/fallocate.c

I can reproduce the same error as loong64 by executing make.bash in Golang's linux/amd64 (kernel version 6.1.0-rc2), here are my logs and corrupted files：
bad_go_asm_x86.tar.gz
golang-x86.log

Note: You need to mount /tmp as tmpfs when testing (because some Linux distributions don't do this)

This should be a kernel bug(introduced in linux 5.16-rc4), I have reported this<EMAIL_ADDRESS>

This is the fix patch provided by our colleagues：<EMAIL_ADDRESS>

Thanks very much @abner-chenc for tracking this down!
A fix has landed in the Linux kernel as commit https://github.com/torvalds/linux/commit/44bcabd70cf1425b4243e02251c02b01638a8287, which is included in Linux 6.1 and newer. I’ll update my machine when I get a chance.


Boolean can't have omitempty or it can never be false

Need to strip omitempty from these bool fields or they will never be returned as false.
Types of changes
What sort of change does your code introduce/modify?

[X] Bug fix (non-breaking change which fixes an issue)

Checklist:

[X] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[X] All new and existing tests passed.
[ ] This change is using publicly documented (api.cloudflare.com or developers.cloudflare.com) and stable APIs.

Another option is to make these fields a *bool and only explicitly set them when we want them. Here are some examples of it in action in cloudflare-go
https://github.com/cloudflare/cloudflare-go/blob/b8935550e028d7c46a7adb20ab8c48f8e849bd06/dns.go#L23
https://github.com/cloudflare/cloudflare-go/blob/b8935550e028d7c46a7adb20ab8c48f8e849bd06/dns_test.go#L77

closing to becoming stale. let me know if we still end up needing this.


Your friends section

create a section that displays your friends. it should be visible on profile section to the right


6.0 Poison Leaks

6.0 internal build is reporting 446 poison leaks in this build (internal link)

These are known leaks. The build reports these as poison leaks, but the diff tests show no difference against the current poison baseline. I verified this manually. Closing as a result.


I can't install/modify/uninstall/update PowerToys..... Please help.....

Microsoft PowerToys version
0.68.1
Installation method
Microsoft Store
Running as admin
No
Area(s) with issue?
General
Steps to reproduce
powertoys-bootstrapper-msi-0.80.1_20240418100705.log
I've been using it well... I've been having problems booting lately, but I've been trying to manipulate it because I need administrator privileges. So I'm trying to uninstall/install/update, and nothing works.
✔ Expected Behavior
No response
❌ Actual Behavior
I hope I can use the latest version well instead of the old one...
Other Software
No response

Hi,
you have PowerToys installed for all users in you computer,
then you can update it using the machine installer: https://github.com/microsoft/PowerToys/releases/download/v0.80.1/PowerToysSetup-0.80.1-x64.exe
or, uninstall PowerToys and install the latest version using the user installer: https://github.com/microsoft/PowerToys/releases/download/v0.80.1/PowerToysUserSetup-0.80.1-x64.exe
/needinfo


Convert to Apollo React Hooks

Instead of using the Query component for data fetching, convert to use Apollo React Hooks

I'll give this a go :)


Feat partners ark

add Volcengine ark partner package

ChatArk
ArkEmbeddings

Volcengine Website

hey there! We're not accepting new integration packages in the monorepo like this, and we have a process for launching it yourself and documenting it in the repo!
Future PRs against langchain would just be {docs updates, as well as registering your package in libs/packages.yml, deprecating this community integration in favor of your integration package}
Here's the guide, and if you have questions, feel free to leave them in the comments on those pages so others can see them! https://python.langchain.com/docs/contributing/how_to/integrations/


requests module work slow

I pip install http-request-randomizer then try to use it, but speed was too slow and i decide not to. But then my regular requests by reuqests module get extrimly slow, seems like it is still working through proxi. Then i pip uninstall http-request-randomizer, but it doesnt help, also i try to reinstall requests.
Still requests working so slow. I'm a newbie in python please tell me what to do

Sorry, It was just unbelievable coincidence with low speed of my internet for specific urls

@Spitzbergen I am glad you sorted things out!
I consider this issue settled


scss: Fix misaligned close icon on notification alert panel.

Right now:

Corrected Version:

There already was a PR for this #16195 by @garg3133. While your fix is simpler, I think ideally we shouldn't hardcode the position (so that it's centered no matter the height of the alert).

Yeah, agreed, the more we can avoid coupled settings, the better.  Let's work on #16195 for this issue.


[ISSUE #281]refactor(eventmesh-sdk-java):rename to org.apache

the gradle.properties also need replace 'com.webank'

the problem has been resolved.