Remote sre8033

I have made changes to our golang version of Official Cloudwatch Exporter. Our version needed to be modified to add more features.
Changes to collector.go has been done:
Support for SuccessfulRequests: Number of successful requests by scrape
Support for ScrapeTimeBuckets: Scrape time taken in. buckets
See the following:

We are missing changes where metrics are actually incremented.
Eg, https://github.com/jonathansm1th/cloudwatch_exporter/blob/4d3e05b7fa3a2a6ea54c8ea8a42fc42fca0d472d/aws.go#L118

Metrics defined in change only reflect one event of one scrape. Slo requires you to have totals for all scrapes. https://github.com/qlik-trial/elastic/pull/803/files

Metrics defined in change only reflect one event of one scrape. Slo requires you to have totals for all scrapes. https://github.com/qlik-trial/elastic/pull/803/files

use https://github.com/artemtar/cloudwatch_exporter/pull/1/files to get global counters

missing a totalRequests.Inc() in the for nextToken!=nil loop in aws.go

Hi @venuspathak7522 Some changes:
Let's just add a total error counter to the top level:
To main.go

Add a new counter in below totalRequests:

totalErrors    prometheus.Counter

Init & register it:

	totalErrors = prometheus.NewCounter(prometheus.CounterOpts{
		Name: "cloudwatch_errors_total",
		Help: "Failed API requests made to CloudWatch",
	})

	globalRegistry.MustRegister(totalRequests)
	globalRegistry.MustRegister(totalErrors)

To aws.go

Add

totalErrors.Inc()

@artemtar

3 Requests are done, 2 requests are error and 0 successful. That does not add up. globalRegistry.MustRegister(totalErrors) that puts your errors into the global endpoint.

@jonathansm1th @artemtar

Do we use a global registry for results or a specific scrape endpoint? The way it is implemented now -> it will partially appear in global and partially in scrape per resource. In addition please, send more than two requests and post the screenshot, I am pretty sure it would have same problem as before

@artemtar
After 4 requests,

Or, with a bigger config set; auth'd against sre-services: ( top lines returned metrics from /scrape shortened for brevity )
Scrape No 1:
/metrics

/scrape?

Scrape No 2:
/metrics

/scrape?

Error increment goes in 2s (0,2,4....) I believe there is a retry mechanism.
Assuming I did 12 error requests.

12 error requests should appear here.
https://github.com/jonathansm1th/cloudwatch_exporter/blob/0b06dbd6dede7e2c06ebd5d3e7fce43e1bb12c8d/aws.go#L124
aws.go This value would not save error requests between runs for the particular endpoint.

In further discussion w. @artemtar we want to move the histogram to only the global scope; and only collect it once per /scrape  call. ie, "How long did each /scrape take to complete, regardless of how many individual api calls happened
Also, modify the buckets to be in a range from 1s to 50s ( 60s is the aws api timeout )
Buckets: []float64{1., 2., 4., 8., 16., 32., 48., }

In further discussion w. @artemtar we want to move the histogram to only the global scope; and only collect it once per /scrape call. ie, "How long did each /scrape take to complete, regardless of how many individual api calls happened
Also, modify the buckets to be in a range from 1s to 50s ( 60s is the aws api timeout )
Buckets: []float64{1., 2., 4., 8., 16., 32., 48., }

https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/custom-http.html
Looks like was has inconsistent timeouts between sdks. Request timeout for golang is 30s, so probably the same buckets till 30

@artemtar @jonathansm1th

@artemtar  0 successful and 2 errors for each of the 4 requests i made
that means 0 successful and 8 errors in total

@artemtar 0 successful and 2 errors for each of the 4 requests i made
that means 0 successful and 8 errors in total

I see, that makes a bit more sense so bucket has all requests now.


2404 USR1/USR2: Different formats for queue dumps.

This PR causes USR1 to dump only the debug representation of the event queue. USR2 can be used to obtain the JSON format.
Closes #2404

bors r+

bors r+


How to Add Image art?

Hello,
I want to add the image art, i was try to add in JcAudio (Adapter) but i didnt know for implementation

Where do you want to add the image art, on the list?


Zipline + docker = Not Working

Dear Zipline Maintainers,
Before I tell you about my issue, let me describe my environment:
Environment

Operating System: (Windows Version or $ uname --all)
Python Version: $ python --version
Python Bitness: $ python -c 'import math, sys;print(int(math.log(sys.maxsize + 1, 2) + 1))'
How did you install Zipline: (pip, conda, or other (please explain))
Python packages: $ pip freeze or $ conda list

Now that you know a little about me, let me tell you about the issue I am
having:
Description of Issue

What did you expect to happen?
What happened instead?

Here is how you can reproduce this issue on your machine:
Reproduction Steps

...
What steps have you taken to resolve this already?
...
Anything else?
...
Sincerely,
$ whoami

Have not seen much activity on current issues - so - will try other avenues. If those do not work, will come back here again.


ListBox with many items - Scrollbar too small

Describe the bug
If a ListBox contains a lot of entries (1000+) the scrollbar becomes too small to use
To Reproduce
Steps to reproduce the behavior:

Create ListBox
Add 5000 entries
Try to scroll

Expected behavior
The vertical scrollbar should have a fixed minimum height

You can use the following codes to specify the minimum height or width:
<ListBox.Resources>
    <system:Double x:Key="{x:Static SystemParameters.VerticalScrollBarButtonHeightKey}">40</system:Double>
</ListBox.Resources>


Example on README.md crashes.

Hi,
I'm just learning your great crate k. I tried to build the example on README page. It crashed on
// chain.update_transforms();
let solved_pose = target_link.world_transform().unwrap();

Why do you put the line before it in comment?

Thank you for trying me code.
I fixed it, and added a test for README.md.


error: package ConnectionStateHolder does not exist

Hi. I'm trying build the client with android studio. When I try to generate a new apk the terminal show those errors, among other mistakes which I have could resolved:
error: package ConnectionStateHolder does not exist
error: cannot find symbol variable ConnectionStateHolder
It looks like a package or part of the code is missing or that class is no longer used. I try to search this class in Android's API but doesn't appear. I'm using api level 29.
Did someone have the same thing or know how to fix it?

Thanks.

Did you pull the experimental branche?

Yes, the experimental

Ok, I resolved it. Android studio change automatically to master branche instead experimental. I downloaded the zip file and it build at the first without problem.
Thanks for helping and great work


Terminal stats and status UI

I've been wanting to add a terminal based UI to show server status and stats.  I think I might have gone a little overboard compared to what I was originally thinking, but I think it's kind of nice.

It's using termdash as a console widget framework.  It actually has mouse support and you can scroll text.  Pretty wild.  But because of that you can't highlight text, and that's pretty annoying.

This UI is not enabled by default.  You must pass -enableTerminalUI when running the service.

There's probably lots of improvements that can be made as far as layout, appearance, formatting, etc.

Log messages are intercepted in termui mode and logs them to this new UI instead of stdout.

The X axis on the charts have useless values.  Time would be useful, but those values take up space.

@jeyemwey @graywolf336 I'd love to hear your thoughts if you happen to have a chance to give this branch for a spin and run with -enableTerminalUI.
Some updates I'm going to make

"Clients" title should have the count in it
Memory usage should probably be RAM usage in actual values, probably gigs, instead of % of system memory.
X values I think should be the point in time that the value was drawn, in seconds since the service has started?

I am not sure if I can answer until next week, but from what I see, this looks great!

From: Gabe Kangas<EMAIL_ADDRESS>Sent: Tuesday, August 11, 2020 6:25:06 PM
To: gabek/owncast<EMAIL_ADDRESS>Cc: Jannik<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [gabek/owncast] Terminal stats and status UI (#111)
@jeyemweyhttps://github.com/jeyemwey @graywolf336https://github.com/graywolf336 I'd love to hear your thoughts if you happen to have a chance to give this branch for a spin and run with -enableTerminalUI.
Some updates I'm going to make

"Clients" title should have the count in it
Memory usage should probably be RAM usage in actual values, probably gigs, instead of % of system memory.
X values I think should be the point in time that the value was drawn, in seconds since the service has started?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/gabek/owncast/pull/111#issuecomment-672068404, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAVKV36BURJAIJBAHPCJZBLSAFWGFANCNFSM4P2T5EZQ.

Absolutely no rush, thanks!  I'm going to keep tweaking anyway.

Hi Gabe, overall I really like it! However, I have some remarks:

I don't think that publishing the Stream Key is necassary. Usually, I have set up my streaming box beforehand, and don't need other people to take over the stream.
The refresh rate for window size changes is quite low, can this be improved?
The video variants and the configuration tabs could use some more space, if my terminal is full screen (which i rarely ever do), it still does not have enough space to show all config items:
 For my production servers, I use other services to monitor CPU and RAM load, so I don't need to see these charts.

Thanks for checking it out!

That's fair to remove the stream key, it's a convenience but it's also a security issue if somebody were to gain access to that terminal.
I'll take a look at resizing refresh, but I think it's only refreshing when I update & redraw the screen, and redraws are every few seconds.  It's probably not worth redrawing more often for the rare occasion that somebody is resizing their window.
I'll have to find a new layout, since it's clear the current one doesn't fit for all users.  But in general I don't see wanting to add configuration around showing and hiding things.  I kind of wanted a basic most important stats so out of the box people can see if performance is hurting.  But for people like you who have other methods, I think it's best just to use your existing tooling.

I really like this but I feel like it belongs in a client package for a few reasons..

Why compile all that UI code and carry it around on a server i'll never use it on
It's way more useful as a client connecting to the already running server/daemon, even if that's only local.
Exposing an interface in the 'server' that does this, allows others to just develop their own interface to that

I don't disagree.
Since there's a very real correlation between performance, hardware capabilities and your specific configuration I wanted people to see these things out of the box.  If somebody loads up a bunch of video variants that they can't handle and sees it isn't performing well, they're likely just to give up on the service instead of looking deeper into it or realizing there's anything they can do about it.  Many likely don't even know how their hardware is performing.  So helping people have a good experience "out of the box" is important to me.
An alternative to this instead of built-in proactive monitoring is to enable reactive alerts.  Start throwing errors when the machine is in the red and make some suggestions on where to look to fix it.  Another logical idea would be to have a web status page with the same info as the this terminal version, but that's a whole other thing.  But that's something to think about.
But I don't want to make people install additional secondary tools and add complexity and confusion.  If they require 3rd party monitoring integrations they might be better off looking for something a bit more piecemeal that they can customize on the operations side.
I think I'll give the performance-based errors a spin.  Continue to monitor the hardware silently, but let people know when things get in the red because that means video segment generation will end up suffering, and they can be pointed to some documentation on how to fix it.
Open to what other people think as well!

I think if it's not enabled by default but well documented, I am okay with this and would love to have this so I can get the most out of a machine while also not hurting the performance of a live stream.
One alternative I would like to see is maybe prometheus metrics since that is close to an industry standard. I have an infrastructure which is in place and might be moving my owncast server underneath that and we already have prometheus data scrapers for our other services/servers. Would be amazing to see this data in there!
Which leads me to the following train of thought: what if you created a metrics package which then other packages pulled information from. So the prometheus package could pull from it and so could the terminal ui?

Here's what I put in #78, so I'll just copy it here:

After hearing feedback from different people on #111 I've decided to change gears and go with the following game plan:

Drop the idea of a full terminal performance/configuration dashboard.
Add some simple error events based on hardware utilization to alert the service operator things aren't going well (cpu spikes mostly).
Plan for a future lightweight web dashboard that has more detailed statistics to fill the need I was trying to fill with the terminal version.  This would include creating a new metrics package.
Keep getting feedback on what values would be useful to expose in different ways.  @graywolf336 mentioned prometheus metrics as an option.  But I figure we can wait until we have our own little dashboard that works for everyone out of the box before we start doing specific implementations for more niche use cases.


how does grav make using materialize css easier

I'm somewhat confused as to how to implement this theme.  Do I still have to code the different aspects of materialize css framework?  Or do I use shortcodes to just insert the asset that I want into the markdown?
I'm confused because I want to be able to build a website but not have to copy js/css for each slider or animation that I want to do.

The Materialize Theme depends on the Materializer Plug-in:
https://github.com/bobrocke/grav-plugin-materializer
If that is installed, all the materialcss framework is available.

I understand that.  But basically we still have to know how to use materialize css?
Like there are no shortcodes to implement the various different assets of materializecss?  I'm talking about being able to create pages via grav admin panel.
I'm sorry if the original post was not clear.

Another source of information is the Materialize site itself. It will tell you what classes to use to get what styling:
http://materializecss.com/about.html

I would like to see some sort of shortcode so that I can call an asset at will and switch on/off parameters without having to copy the js/css

You don't have to copy anything. Just apply classes to get the look you want. Dig into the theme's source code to see how the theme has applied Materialize classes.


Pdfstream as cmap

Resolves #210 .
The previous work #264 maps "OneByteEncodingH/V" to default CMap initialization. It also ignores the fact that "DLIdent-H/V" is a valid encoding since it isn't mentioned in any PDF Reference the author can dig in.
While looking for the output of pdf2text.py the OneByteEncoding characters were absent denoting that default CMap encoding is ineffective to capture the glyph.
I have added the decode function for OneByteEncoding and also added DLIdent as there is a possibility of this being in many PDF.
Both OneByteEncoding and DLIdent aren't available in any PDF Reference referred.

@pietermarsman Hi, Have removed the comments. Can you please review it once again and merge if no other changes are required. :)

Hi @fakabbir, sorry for the miscommunication but I do not have commit rights right now. I do my bit by reviewing and creating PR's.
I hope to get commit rights soon though.


fix: useThrottle discard intermediary calls

Fix as described here https://github.com/streamich/react-use/issues/154#issuecomment-477302736. I had to use window.timeout because of a type issue which I think is related to https://github.com/Microsoft/TypeScript/issues/842
Closes #154

:tada: This PR is included in version 6.2.2 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


If message starts with // or #, or surrounded by /* */, it is replaced with a null string.

This is because the code tries to parse the content of the message to see if it is JSON here https://github.com/splunk/splunk-library-javalogging/blob/main/src/main/java/com/splunk/logging/serialization/EventInfoTypeAdapter.java#L36
We should check if the returning object is null, and the message is not null and has data, to then fallback to using the unparsed message.

Hi @larrys,
Thanks for reaching out !
We are open for the PR to support your use case. We would like to review your use case.
Please let us know, if you have any queries.

You can see a "fix" for this (minus unit tests) here: https://github.com/larrys/splunk-library-javalogging/commit/5c905e7cc629560d480584a3531702c0829acaa7

Hi @larrys,
We are concern about the use case for this specific need.
Could you please share any business use case in which this change is required?
It would help us to make a decision of incorporating this change.
Thank you.

I was trying to switch from using a logback server to send logs to, that then got indexed with a UF to using this library to directly talk to Splunk over HEC. I was unable to since it was dropping the log message that happened to start with //, # or was surrounded by /* */.

Hi @larrys .
Can you please provide us more details so we can simulate/reproduce this behaviour on our side?
Thanks,
Marcin

Just have a log.info statement such as:
log.info("#### This is a test ####")

With the fix on my PR it will have a log message sent to splunk with the content of that log statement. With the master branch of this repo, it won't, the message will be evaluated as an empty string.

You can see https://github.com/splunk/splunk-library-javalogging/commit/d6589567b0c252e7c7a0e7abf31186ca14102da7 for a very simplistic unit test that I was able to throw together. It should be expanded on for other edge cases, being passed to the seralize method.

Thanks!


Poorly named driver source release file

In the latest release we have:
opae-intel-fpga-driver-source-2.0.1-6.x86_64.rpm
which appears as a binary x86_64 rpm by naming convention.  Please rename it to opae-intel-fpga-driver-2.0.1-6.src.rpm or opae-intel-fpga-driver-x86_64-2.0.1-6.src.rpm if you insist on having x86_64 in the name.  Even better yet, if you're not including a spec file then just release the source as a compressed tarball.  As it stands, this breaks some of our tools/scriptware.   Thanks.

Following are the instructions for generating binary RPM from the opae-intel-fpga-driver-source-a.b.c-d.x86_64.rpm that we provided with the release.
Download the kernel source package from shareholders. We assume the kernel package name is:
opae-intel-fpga-driver-source-a.b.c-d.x86_64.rpm
Before we install the latest rpm driver, it had better remove the installed driver firstly.
# rpm –e opae-intel-fpga-driver
Install the new OPAE source rpm.
# yum install opae-intel-fpga-driver-source-a.b.c-d.x86_64.rpm
Installing the kernel driver .rpm will cause the driver source (packaged inside the .rpm) to be placed in the /usr/src directory. Change directory there and follow these steps:
# tar zxf opae-intel-fpga-driver-source-a.b.c-d.tar.gz
# rpmbuild –tb opae-intel-fpga-driver-source-a.b.c-d.tar.gz ```

This will create the installable binary rpm in ~/rpmbuild/RPMS/x86_64.

Change directory to there and install the binary rpm.
```# cd ~/rpmbuild/RPMS/x86_64
# yum install opae-intel-fpga-driver-a.b.c-d.rpm ```

That's not what I asked.  The file opae-intel-fpga-driver-source-a.b.c-d.x86_64.rpm is poorly named, it looks like it is a binary rpm by naming convention, but it contains source.  Please re-read what I wrote previously.

Sorry about that. I have uploaded the source tarball here:
https://github.com/OPAE/opae-sdk/releases/download/1.3.6-4/opae-intel-fpga-driver-2.0.1.tar.gz
Hope this helps.

That's great, thanks.  Now can you do the same for the 2.0.4 driver release please (in the 1.4.0 SDK release) and any releases going forward from now on?  Thanks!

Actually, this is named correctly, for the most part.  To truly be accurate, it should be named opae-intel-fpga-driver-2.0.1-6.noarch.rpm and there should be a corresponding opae-intel-fpga-driver-2.0.1-6.src.rpm.  The .rpm is a package installation rpm containing preinstall, postinstall, and postrm scripts as well as the driver source, which is needed to rebuild the drivers outside of the kernel during a kernel update (dkms handles this).  The src.rpm is normally a special rpm package that has the source tarball, teh rpm spec file, and possibly any patches to the original source.
Since I doubt Intel will be supporting these drivers on any other architecture anytime soon (PPC, ARM64, etc), x86_64 is appropriate in this instance, even though technically it could be noarch as there are no real binaries in the package.  Now, if the drivers are using kernel functions that are specific to x86_64, then the x86_64.rpm is in fact the correct naming convention.


feat: Add production server on openapi spec.

Add servers on OpenAPI spec
This add "servers" property on OpenAPI specification.
Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Refactoring / Technical debt

What parts of the code are impacted ?
OpenAPI spec.
Checklist:

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes

PR is good to me, but i'm not sure i understand why we're doing this ! can you elaborate on what the need was ?


fix upgrade kafka user not found

https://issues.redhat.com/browse/ACM-9829

/cc @yanmxa @morvencao @clyang82

/retest

/retest


Return all OrderId using Google Play Developer API

Hello, I am trying to manage subscriptions made by the In-App Billing API. Right now i want to retrieve all orders, just like it can be seen in the google play console.
I can get the last orderId, using the token to get subcription info:
def get_credential():
    # Getting the credentials...
    return credentials

def get_google_payment(token):
    credential = get_credential()
    service = build("androidpublisher", "v3", credentials=credential)
    request = service.purchases().subscriptions().get(packageName="com.company.app", token=token, subscriptionId='subId')
    response = request.execute()
    return json.dumps(response)  # A JSON with all subcription info is returned

My question is, there's a method to get ALL orderIds, or am i using the only way the get them?

Hi @EmiCarrizo. google-api-python-client generates libraries based off the discovery document. reference docs and discovery document.  It looks like the API doesn't have a method for getting all orders, so it won't be possible with the client library.
I'm looking into the right place to direct your question.

@EmiCarrizo, questions with the Google Play Developer API are best directed to the Google Play developer support. Please reach out to them with your question. Thanks!


Fixes #2822

added code to to recognize RNA and DNA bases.
added relevant tests

@ptosco: I assume that you've actually looked at the results of running this and determined that they make sense, right?

@greglandrum Yes, I did, on a couple of examples from the PDB. I took specifications from PDB LIgandExpo, as I guess Roger did for amino acid residues.


Propagate custom cfg directives to rustdoc and doctests

Fixes #1980

r? @wycats
(rust_highfive has picked a reviewer for you, use r? to override)

r? @alexcrichton

Awesome, thanks @gkoz!

Updated for both comments.

@alexcrichton looks like you don't have `Rolling builds" enabled in appveyor project settings. Because of this it doesn't stop building stale commits promptly and takes longer to get to the current ones.

Whoa, I had no idea that option existed! I've turned it on now

I went with the more concise version giving the lock up after iterating over cfgs. At least I believe its lifetime will now be limited to the if let block.

@bors: r+ 807da6b11e9a5821add1438247fbde3a6435dbe0
Thanks!

:hourglass: Testing commit 807da6b with merge 1dc1865...

:sunny: Test successful - cargo-linux-32, cargo-linux-64, cargo-mac-32, cargo-mac-64, cargo-win-gnu-32, cargo-win-gnu-64, cargo-win-msvc-32, cargo-win-msvc-64


Add styled-components example

Following the other examples in the examples directory, we should add an example for Mirrorful + styled-components
Requirements:

use latest version of Mirrorful
change the default index.js to actually consume the Mirrorful theme (don't just install Mirrorful)
explain in the README how it works
add example to top level readme andpackages/mirrorful/editor README

I'm looking for a first issue here and wouldn't mind taking this. One question - do you have a preference on which meta framework the example project will use? I'll likely use Next but curious if there's value in doing it in Vite since there's a CRA example but no Vite currently?

Hey @neilhsmith welcome 👋🏼 , thanks for taking this one. There is no "official" preference. Next is great (as I would imagine that covers the most ground), but if you want to do it in Vite go for it.

Hey @neilhsmith are you still interested in this? Btw, feel free to check out our slack workspace here! Lot of other contributors https://join.slack.com/t/mirrorful/shared_invite/zt-1ps2xtxh0-2NaixFfFzSKZbr5gw_AHfA

Hey, sorry for the delay! I am but if someone else wants to take it, go for it. I'll join the Slack and find something when I free up in a few days if someone else does take this.

Hi, @alexdanilowicz I will like to take up this issue.

@Adefizzy I still haven't started so go ahead :)
I'm still planning on joining the Slack and can find another issue when I free up.

Hi @alexdanilowicz , I have made a PR for this issue

Great work. Merging this in. Thanks @Adefizzy !


Add scoring information and item definitions to result JSON

Everything should be bundled in the JSON that an applicant downloads; this allows reproducibility and avoids conflicts of versions.
The score.R script should take the local scoring information (if not available, fetch from server)

Same as #3


Refactor to add common library for integration test

          We now have 3 copies of this code fragment. Probably time to write a helper function.

Originally posted by @coryan in https://github.com/googleapis/google-cloud-cpp/pull/13251#discussion_r1419056929

Fixed this with #13384 + #13413


Question on notifiers.

For the PR #429, I'm currently working on notifiers. I realized that all launchers, except by the pipeline one, have the argument notifyEndProcess. However, I didn't find this working in the module repairnator-realtime, i.e. I didn't find the notifyEnd method from EndProcessNotifier being called.
Question on the module repairnator-realtime: what I just described is a bug or the notifier is not supposed to be in that module? My point is: either there is code missing (to make the notifier works) or there is more code than it should have (the argument notifyEndProcess and the other ones related to email notification shouldn't be pass to the launcher of that module).

Contrary to other modules of Repairnator such as the scanner or the dockerpool, repairnator-realtime aims at being always running without interruption.
Then it makes no sense to use a notifier for the end of the process. That's why I did not put it in.

Ok. Thus can I remove the arg notifyEndProcess from that module?
Another question: is the notifier engine being used somewhere in that module? The notifier engine has been initialized, but I still didn't find usage of this.

Thus can I remove the arg notifyEndProcess from that module?

Yup. I've been a bit fast when doing my cut/paste there.

is the notifier engine being used somewhere in that module? The notifier engine has been initialized, but I still didn't find usage of this.

Right now, it's not used indeed. You can remove it too.

Deal.


update to ptrref chapter in dev

This PR is a minor update to the guidelines in ptrrefmodule.
It only formats/renames the examples and deletes the unused duplicates.

Thanks!


Error after calling the ios api for mlc-llm

I got into this error while calling api like mention in documentation. Can you please help me?
terminating with uncaught exception of type tvm::runtime::InternalError: [14:31:27] /Users/nileshkatariya/mlc-llm/cpp/llm_chat.cc:500: InternalError: Check failed: (config_istream) is false:
Stack trace:
[bt] (0) 1   Solo Ai Chat App                    0x0000000104de8ba8 tvm::runtime::detail::LogFatal::Entry::Finalize() + 68
[bt] (1) 2   Solo Ai Chat App                    0x0000000104de8b64 tvm::runtime::detail::LogFatal::Entry::Finalize() + 0
[bt] (2) 3   Solo Ai Chat App                    0x0000000104de7b84 __clang_call_terminate + 0
[bt] (3) 4   Solo Ai Chat App                    0x0000000104dffdc0 mlc::llm::LLMChat::Reload(tvm::runtime::TVMArgValue, tvm::runtime::String, tvm::runtime::String) + 11236
[bt] (4) 5   Solo Ai Chat App                    0x0000000104dfcf1c mlc::llm::LLMChatModule::GetFunction(tvm::runtime::String const&, tvm::runtime::ObjectPtrtvm::runtime::Object const&)::'lambda'(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const + 768
[bt] (5) 6   Solo Ai Chat App                    0x0000000104dfcc10 tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<mlc::llm::LLMChatModule::GetFunction(tvm::runtime::String const&, tvm::runtime::ObjectPtrtvm::runtime::Object const&)::'lambda'(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)> >::Call(tvm::runtime::PackedFuncObj const*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) + 40
[bt] (6) 7   Solo Ai Chat App                    0x0000000104da04ec tvm::runtime::TVMRetValue tvm::runtime::PackedFunc::operator()<tvm::runtime::Module&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >&>(tvm::runtime::Module&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator >&) const + 252
[bt] (7) 8   Solo Ai Chat App                    0x0000000104d9ffac -[ChatModule reload:modelPath:appConfigJson:] + 408
[bt] (8) 9   Solo Ai Chat App                    0x0000000104d85574 $s16Solo_Ai_Chat_App12HomeScreenVCC11viewDidLoadyyF + 3572

Hi @rjzsatvara this error seems to indicate that the particular project cannot find the mlc-chat-config.json locally in the folder. Please make sure you place the files in the right place. The example app would be a good tutorial to follow

Hi @rjzsatvara this error seems to indicate that the particular project cannot find the mlc-chat-config.json locally in the folder. Please make sure you place the files in the right place. The example app would be a good tutorial to follow

where does the error indicate it cannot find the mlc-chat-config.json? i see it saying Check failed: (config_istream) is false: which is indicating a different sort of issue from tvm.
where should the files be?
mine is located at dist/bundle/Llama-3.2-3B-Instruct-q4f16_1-MLC/
dist
├── bundle                   # The directory for mlc-app-config.json (and optionally model weights)
    │                        # that will be bundled into the iOS app.
    ├── mlc-app-config.json  # The app config JSON file.
    └──Llama-3.2-3B-Instruct-q4f16_1-MLC
     │
     └──mlc-chat-config.json

Hi @loekTheDreamer, thanks for bringing this up. This issue is a bit outdated so do you mind opening a new issue and describing the problems you met there? If you can also share the contents of your mlc-chat-config.json that might be helpful for us to figure out.

Sure, I can do that 😉


Pathological behavior when http_request_queue grows too big

Hypothetical scenario:

a bug in pg_net causes requests to accumulate in http_request_queue
a patch is released
all requests are fired off at once
DB has trouble keeping up and causes OOM/EMFILE/etc.
worker enters crashloop
DB instance is forever sad

I haven't tried reproducing the above scenario, but we should take preventive measures against this. Some possible actions:

limiting max amount of requests in progress
TRUNCATE all requests on worker restart (so it doesn't enter crashloop)

this happened on a customer instance before - https://supabase.slack.com/archives/C01D6TWFFFW/p1632734535465600?thread_ts=1632712292.453200&cid=C01D6TWFFFW

a bug in pg_net causes requests to accumulate in http_request_queue
all requests are fired off at once

Another measure could be refusing new requests while the worker is down(https://github.com/supabase/pg_net/issues/44#issuecomment-938238307).

a patch is released

Not directly related to this issue, but I found out that upgrading extensions in shared_preloaded_libraries cannot be done unless the database is restarted(downtime), we need to find a way around this.

Limiting the amount of requests succeeds in limiting CPU usage, in this case processing 12000 requests at a rate of 500 consumes < 20% CPU(and this doesn't seem to change with more requests).

I'll go ahead and put a default of 500 and make this configurable.


Table relations not generated

Hi. It generated the dbml and the svg output, however the relations are not present.
I have my schema per file and merged them in an index.ts, does it work with this? or does it need to be in a single schema file?

src/lib/server/db/schema/
`.
├── absences.ts
├── appointmentInformation.ts
├── archive.ts
├── archiveType.ts
├── authHistory.ts
├── authType.ts
├── bankAccount.ts
├── civilStatus.ts
├── employeeSalary.ts
├── employee.ts
├── employmentStatus.ts
├── gender.ts
├── gsis.ts
├── index.ts
├── organization.ts
├── passwordResetToken.ts
├── position.ts
├── role.ts
├── salaryGrade.ts
├── salary.ts
├── session.ts
├── station.ts
├── status.ts
├── tax.ts
├── userPreference.ts
├── userStatus.ts
└── user.ts

1 directory, 27 files
`

src/lib/db/index.ts

import { drizzle } from 'drizzle-orm/node-postgres';
import * as schema from '@/server/db/schema';
import pg from 'pg';
import { pgGenerate } from 'drizzle-dbml-generator';

export const pool = new pg.Pool({
	connectionString: process.env.DATABASE_URL
});

const db = drizzle(pool, { schema });
const out = './schema.dbml';
const relational = true;

console.info(pgGenerate({ schema, out, relational }));

export default db;

"dbml:generate": "bun run -r dotenv/config src/lib/server/db/index.ts && bun dbml:render",
"dbml:render": "bun dbml-renderer -i schema.dbml -o erd.svg"

The way you have it set up should work. Can you show me a snippet where I can see how you're defining relations?

The way you have it set up should work. Can you show me a snippet where I can see how you're defining relations?

Hi, thanks for the response. It works now, it was my fault that I did not include the relations as an export. I thought it works out of the box. I now created a sub directory containing the table relations and it now generates it properly.

import { pgTable, text, timestamp } from 'drizzle-orm/pg-core';
import { relations } from 'drizzle-orm';
import { employee } from '.';
import { createId } from '@paralleldrive/cuid2';

const appointmentInformation = pgTable('appointment_info', {
        id: text('id')
                .primaryKey()
                .$defaultFn(() => createId()),
        createdAt: timestamp('created_at', { precision: 3 }).defaultNow(),
        updatedAt: timestamp('updated_at', { precision: 3 }).$onUpdate(() => new Date()),
        appointmentDate: timestamp('appointment_date', { precision: 3 }).notNull(),
        lastPromotionDate: timestamp('last_promotion', { precision: 3 }).notNull(),
        employeeId: text('employee_id').notNull()
});

export const appointmentStatusRelations = relations(appointmentInformation, ({ one }) => {
        return {
                employee: one(employee, {
                        fields: [appointmentInformation.employeeId],
                        references: [employee.id]
                })
        };
});

export default appointmentInformation;

import { pgTable, text, timestamp } from 'drizzle-orm/pg-core';
import { relations } from 'drizzle-orm';
import { employee } from '.';
import { createId } from '@paralleldrive/cuid2';

const appointmentInformation = pgTable('appointment_info', {
        id: text('id')
                .primaryKey()
                .$defaultFn(() => createId()),
        createdAt: timestamp('created_at', { precision: 3 }).defaultNow(),
        updatedAt: timestamp('updated_at', { precision: 3 }).$onUpdate(() => new Date()),
        appointmentDate: timestamp('appointment_date', { precision: 3 }).notNull(),
        lastPromotionDate: timestamp('last_promotion', { precision: 3 }).notNull(),
        employeeId: text('employee_id').notNull()
});

export const appointmentStatusRelations = relations(appointmentInformation, ({ one }) => {
        return {
                employee: one(employee, {
                        fields: [appointmentInformation.employeeId],
                        references: [employee.id]
                })
        };
});

export default appointmentInformation;

this was fixed by separating the relations to a new file.


Global timeline randomly forgets selected relay

Describe the bug
The app randomly forgets which relay you selected on the global timeline.
To Reproduce
Steps to reproduce the behavior:

Select a relay on the global timeline.
Navigate around app in a normal way for a while.
Go back to global timeline.
Relay selection has reverted back to default.

Expected behavior

The selection should be remembered.
Have a dropdown with checkboxes so you can pick one or more relays to compose your own timeline, like Damus for iOS does, and have the app consistently remember that.

Desktop (please complete the following information):

OS: macOS 12.6
Browser: Chrome
Version: 110.0.5481.177

Just realised I was running a slightly older build because I didn't hit Shift+Refresh. This bug might have been fixed. Will watch it and post an update or close the ticket, depending on what I find out.

It seems that the value in the drop-down is preserved in more recent builds.

It seems to forget the setting when you reload the page.


How can I open a new tab in Chrome?

I use   5.19.0
public void TestDemo2(){
Configuration.browser = "chrome";
open("https://www.abc.com");
sleep(1000);
switchTo().window(1);
}
But IDE :
No window found with index: 1
Caused by: TimeoutException: Expected condition failed: waiting for window to be available by index: 1 (tried for 4 second(s) with 200 milliseconds interval)

@CloudWzhZzz This is a bug reporting area. Next time ask in stackoverflow
String selectLinkOpeninNewTab = Keys.chord(Keys.CONTROL,Keys.RETURN);
$(By.linkText("urlLink")).sendKeys(selectLinkOpeninNewTab);

Thank you！
After reading the official api documentation of selenide, I didn't understand the usage of "switchTo". I thought it was possible to open a new tab page, so I mistakenly thought it was a bug, so I came here to ask questions.

@CloudWzhZzz method switchTo is used to switch between existing tabs/windows/frames. It cannot open a new window.
Most probably you don't really need to open a new window. Can you specify why you think you need it?

@CloudWzhZzz method switchTo is used to switch between existing tabs/windows/frames. It cannot open a new window.
Most probably you don't really need to open a new window. Can you specify why you think you need it?

I need to open two tabs, log in on tab 1, then switch to tab 2 and refresh the page, so that tab 2 can use the cookie of tab 1 to log in automatically.
You may ask why you don’t log in directly on the  tab2, because the page opened on the  tab2 needs to be verified by the machine verification code if you want to log in, but the machine cannot pass the verification code verification, so you can only use the methods I mentioned above. Sign in to bypass the verification code verification process.

Sorry, i still have some questions :)

How do you bypass the verification code check in tab1?
If you can login in tab1 somehow, why cannot you use the same mechanism for logging in tab2?
When you have logged in in tab1, why can't you just continue the test in tab1 without opening any new tabs?

1.The process of bypassing the machine verification code is to use the internal login service. This is not universal, so the detailed process is not described.You can understand it as opening the back door on the login machine and directly obtaining the cookie of the account.
2.As mentioned in 1, tab1 is only used to obtain the cookie of the account, and has no other use, so after tab1 obtains the cookie, open tab2 and enter the website, you can use the cookie obtained by tab1 under the same browser to log in directly.
3.emmmm,this is indeed a way, I didn't expect it at the time, thank you for your reminder.(>_<|||)


Mrcjkb/annotation auto detect color

This PR adds an API endpoint to Styler that allows the client to configure an auto-detection of the annotation font color based on the background color it is painted over.
An example is included with the demo bar chart BarChart13.

This is very cool. I think this should be the default feature though and if people want to change it, they use the standard setcolor method in the styler.

I would have to look into it.
It makes sense to make it the default behaviour for charts on which the annotations are always on top of the chart (like category charts and heat maps),. But it wouldn't make sense for pie charts with the annotations outside.

I have set the default behaviour for category charts and disabled it for other types for now (as I don't know what side effects it will have due to the merge conflict resolution).

I have set the default behaviour for category charts and disabled it for other types for now (as I don't know what side effects it will have due to the merge conflict resolution).

Thanks. BTW, I'm calling the old "annotations" now "labels" to differentiate the from "Chart annotations". See the readme about what those are. I'm going to update your method names to reflect this change.

Thanks. BTW, I'm calling the old "annotations" now "labels" to differentiate the from "Chart annotations". See the readme about what those are. I'm going to update your method names to reflect this change.


What does "weights_scaling_factor_2" mean in safetensor results of awq_w4a8

I follow this step to do quantization for qwen2 model.
Then I got the safetensor results like

What does  ' prequant_scaling_factor',  'activation_scaling_factor', 'weights_scaling_factor', 'weights_scaling_factor_2' mean. And how are they used in the w4a8 gemm?

For a linear layer with GEMM shape [M, N, K], we need these components in the TRT-LLM layer:
Name	Dtype	Shape	Layout
{LAYER_NAME}.weight	float16	[K, N / 4]	Interleaved and packed INT4
{LAYER_NAME}.weight_scaling_factor	float16	[K / group_size, N]	Row-major
{LAYER_NAME}.activation_scaling_factor	float16	[K]	Row-major
{LAYER_NAME}.alpha	float32	[1]	-
The calculation process is: output = FP16(FP8(act * activation_scaling_factor) * FP8(weight * weight_scaling_factor) * alpha)
However, the checkpoint will have more parameters, here is how they are converted when building the engine.

Thanks!!!

Exactly.
For clearer understanding, you can consider W4A8_AWQ as W4A16_AWQ + FP8.
In addition to the components of W4A16_AWQ, i.e., prequant_scaling_factor, weight_scaling_factor, FP8 will provide 2 more per-tensor scaling factors activation_scaling_factor and weight_scaling_factor_2. In order to having them combined in one GEMM, we have:

Multiplied the per-tensor activation_scaling_factor into prequant_scaling_factor
Multiplied the per-tensor weight_scaling_factor_2 into weight_scaling_factor
Exposed alpha as layer parameters for de-quantization


fix(userspace/engine): properly include stdexcept header to fix build.

What type of PR is this?
/kind bug

Any specific area of the project related to this PR?
/area build
/area engine

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

NONE

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, jasondellaluce
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP,jasondellaluce]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LGTM label has been added.  Git tree hash: d82787ae0c558abddfd158f85347b8e7159d658f

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, FedeDP, jasondellaluce
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Andreagit97,FedeDP,jasondellaluce]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Use environments to set hit card time

Previously it's hard to personalize hit card time since modifying action.yml will introduce dirty commit to branch. Now we can use Repo Environments to avoid modifying action.yml.

I see no good reason not to use cron itself but to add an extra layer of delay to implement a customizable report time. Introducing commit into your branch is inevitable due to monthly update. All you have to do is to merge upstream changes into your fork. This should be handled decently even without command line git operations.


1.2.1: .done is not set to true for null Checkpoint

const {
Checkpoint
} = require('crtk');
var cp = Checkpoint.allOf();
// cp.done is false, and remains false even after awaiting

fixed in 1.3.0


External link boolean is required on globals in CMS

External link is required, it shouldn't be! required: false looks to be the solution

Cheers, Jonny. This should be a straight-forward fix.

Hi, Andy. Thanks for building this theme, I'm happy to jump on this!


Fixes bug where we didn't merge extra fields from implicit context

While unusual, and likely a leak, it is possible to have a span in scope
when extracting a remote context. This fixes the code to merge the extra
fields in the current span with extracted fields and backfills tests.
Thanks to @aldex32 and @aukevanleeuwen for hunting this down

@llinder or @devinsba this is waiting for zipkin-reporter-java 2.3.2 to sync to central (should happen soon, but I'll be on the plane).
When that's in, can you merge this and cut release-4.15.0? Note that brave uses travis for release.
cc also @jcchavezs

maven just synced I'll cut the release!

@adriancole thanks for the quick fix!

@adriancole https://github.com/adriancole thanks for the quick fix!

so you are all good now? no more 3GB of trace contexts?

@adriancole it's not any more in {{PROD}}, now it needs to pass through all environments again. But I will let you know!
Thanks again!

@adriancole now this fix is running on our PROD environment for one month, and till now seems everything good.


Failed connect() does not throw an error.

When calling connect(), it sometimes fails like:
Connecting...
Connected

Often followed by some other error:
Connecting...
Disconnect error
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/crownstone_ble-1.0.0-py3.7.egg/crownstone_ble/core/ble_modules/BleHandler.py", line 172, in getCharacteristics
  File "/usr/local/lib/python3.7/dist-packages/bluepy-1.3.0-py3.7.egg/bluepy/btle.py", line 507, in getServiceByUUID
    self._writeCmd("svcs %s\n" % uuid)
  File "/usr/local/lib/python3.7/dist-packages/bluepy-1.3.0-py3.7.egg/bluepy/btle.py", line 315, in _writeCmd
    raise BTLEInternalError("Helper not started (did you call connect()?)")
bluepy.btle.BTLEInternalError: Helper not started (did you call connect()?)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "example_setup_and_reset.py", line 54, in <module>
    ibeaconMinor=456
  File "/usr/local/lib/python3.7/dist-packages/crownstone_ble-1.0.0-py3.7.egg/crownstone_ble/core/CrownstoneBle.py", line 77, in setupCrownstone
  File "/usr/local/lib/python3.7/dist-packages/crownstone_ble-1.0.0-py3.7.egg/crownstone_ble/core/ble_modules/SetupHandler.py", line 20, in setup
  File "/usr/local/lib/python3.7/dist-packages/crownstone_ble-1.0.0-py3.7.egg/crownstone_ble/core/ble_modules/BleHandler.py", line 174, in getCharacteristics
crownstone_core.Exceptions.CrownstoneBleException: (<BleError.CAN_NOT_FIND_SERVICE: 'CAN_NOT_FIND_SERVICE'>, 'Can not find service: 24f10000-7d10-4805-bfc1-7663a01c3bff')

Here is a hackish way to see if the connect was successful: https://github.com/Kvanr23/CrownstoneTestSuite/blob/7ef388ecd7ab6ce0d58f13a6a4e4ae49e87f3677/Switching.py#L184

Has it really failed to connect in the first example or did it immediately lose connection afterwards.
I think many of these issues relate to just crappy bluetooth hardware. We know that this is highly dependent on the usage of supported BLE hardware, like an external BLE dongle. Most laptops or raspi native bluetooth chips are just unreliable to use together with bluepy.
Additionally, there might be a version mismatch with the version of bluez on de host and the one that's bundled with bluepy.
Before handling the issues in code, maybe first check that to see if the problem is at a lower level.

I would like to remark that most of our current hardware is newer than the software that comes with bluepy. If you look at https://github.com/IanHarvey/bluepy you see that there is a version of bluez-5.47 which is 3 years old.
Hence, regarding the possible source of bugs it is in this order:

our own libraries
the bluepy library (or in particular the bluepy-helper code)
the bluez libraries
the hardware

I've seen bug reports where bluepy-helper became a zombie, however, I'm assuming here a simple bug. Just run with strace and you'll probably find a mismatch in reporting connection state or something like that.


Update fxmanifest.lua

dependencies

Also needs to be this:
server_scripts  {
<EMAIL_ADDRESS><EMAIL_ADDRESS>'server/main.lua'
}
the<EMAIL_ADDRESS>is missing


Support explain query when running dfbench

Which issue does this PR close?
Closes #13941
Rationale for this change
What changes are included in this PR?
Add the explain option.
Are these changes tested?
yes:
cargo run --release --bin dfbench clickbench --iterations 5   --query 35 --explain
    Finished `release` profile [optimized] target(s) in 0.18s
     Running `target/release/dfbench clickbench --iterations 5 --query 35 --explain`
Running benchmarks with the following options: RunOpt { query: Some(35), common: CommonOpt { iterations: 5, partitions: None, batch_size: 8192, debug: false }, path: "benchmarks/data/hits.parquet", queries_path: "benchmarks/queries/clickbench/queries.sql", output_path: None, explain: true }
Q35: SELECT "ClientIP", "ClientIP" - 1, "ClientIP" - 2, "ClientIP" - 3, COUNT(*) AS c FROM hits GROUP BY "ClientIP", "ClientIP" - 1, "ClientIP" - 2, "ClientIP" - 3 ORDER BY c DESC LIMIT 10;
Query 35 iteration 0 took 1202.7 ms and returned 10 rows
Query 35 iteration 1 took 1092.0 ms and returned 10 rows
Query 35 iteration 2 took 1021.7 ms and returned 10 rows
Query 35 iteration 3 took 1022.2 ms and returned 10 rows
Query 35 iteration 4 took 982.8 ms and returned 10 rows
+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type     | plan                                                                                |
+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| logical_plan  | Sort: c DESC NULLS FIRST, fetch=10                                                                                |
|               |   Projection: hits.ClientIP, hits.ClientIP - Int64(1), hits.ClientIP - Int64(2), hits.ClientIP - Int64(3), count(*) AS c                                                                                |
|               |     Aggregate: groupBy=[[hits.ClientIP, __common_expr_1 AS hits.ClientIP - Int64(1), __common_expr_1 AS hits.ClientIP - Int64(2), __common_expr_1 AS hits.ClientIP - Int64(3)]], aggr=[[count(Int64(1)) AS count(*)]]                                                                                |
|               |       Projection: CAST(hits.ClientIP AS Int64) AS __common_expr_1, hits.ClientIP                                                                                |
|               |         TableScan: hits projection=[ClientIP]                                                                                |
| physical_plan | SortPreservingMergeExec: [c@4 DESC], fetch=10                                                                                |
|               |   SortExec: TopK(fetch=10), expr=[c@4 DESC], preserve_partitioning=[true]                                                                                |
|               |     ProjectionExec: expr=[ClientIP@0 as ClientIP, hits.ClientIP - Int64(1)@1 as hits.ClientIP - Int64(1), hits.ClientIP - Int64(2)@2 as hits.ClientIP - Int64(2), hits.ClientIP - Int64(3)@3 as hits.ClientIP - Int64(3), count(*)@4 as c]                                                                                |
|               |       AggregateExec: mode=FinalPartitioned, gby=[ClientIP@0 as ClientIP, hits.ClientIP - Int64(1)@1 as hits.ClientIP - Int64(1), hits.ClientIP - Int64(2)@2 as hits.ClientIP - Int64(2), hits.ClientIP - Int64(3)@3 as hits.ClientIP - Int64(3)], aggr=[count(*)]                                                                                |
|               |         CoalesceBatchesExec: target_batch_size=8192                                                                                |
|               |           RepartitionExec: partitioning=Hash([ClientIP@0, hits.ClientIP - Int64(1)@1, hits.ClientIP - Int64(2)@2, hits.ClientIP - Int64(3)@3], 14), input_partitions=14                                                                                |
|               |             AggregateExec: mode=Partial, gby=[ClientIP@1 as ClientIP, __common_expr_1@0 - 1 as hits.ClientIP - Int64(1), __common_expr_1@0 - 2 as hits.ClientIP - Int64(2), __common_expr_1@0 - 3 as hits.ClientIP - Int64(3)], aggr=[count(*)]                                                                                |
|               |               ProjectionExec: expr=[CAST(ClientIP@0 AS Int64) as __common_expr_1, ClientIP@0 as ClientIP]                                                                                |
|               |                 ParquetExec: file_groups={14 groups: [[Users/zhuqi/arrow-datafusion/benchmarks/data/hits.parquet:0..1055712604], [Users/zhuqi/arrow-datafusion/benchmarks/data/hits.parquet:1055712604..2111425208], [Users/zhuqi/arrow-datafusion/benchmarks/data/hits.parquet:2111425208..3167137812], [Users/zhuqi/arrow-datafusion/benchmarks/data/hits.parquet:3167137812..4222850416], [Users/zhuqi/arrow-datafusion/benchmarks/data/hits.parquet:4222850416..5278563020], ...]}, projection=[ClientIP] |
|               |                                                                                |
+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Are there any user-facing changes?
yes

cc @alamb It's a small change for explain support for dfbench, thanks!

Perhaps we can use an existing common option to print plans
https://github.com/apache/datafusion/blob/ab69bb04bbcf1b4fd6b528a8433056e9adae67a8/benchmarks/src/util/options.rs#L40
tpch benchmark uses this flag, but it's not used in clickbench yet
https://github.com/apache/datafusion/blob/ab69bb04bbcf1b4fd6b528a8433056e9adae67a8/benchmarks/src/tpch/run.rs#L194-L232

Thank you @2010YOUY01 for review and good suggestion! Addressed in latest PR.


Filter Accounts by Missing Password Policies

Requesting a non-existant password policy causes an exception when used with the existing password-policy - but serves as a good check "do they have a password policy at all?
The existing password-policy filter is based on the ValueFilter - and so I can't easily add an extra flag to is (as far as I know?). I could modify the data it returns to have a single value (configured or the like) not present in the actual response, and check that value - but that risks breaking backwards compatibility by catching the edge case (but unlikely).
This adds a new, simple filter that just checks for the presence of a password policy and is intended to be easily combinable . Tests included.
Please note: This may hit the object twice in the process of getting it, so may be worth also adding the password policy on to the object given we're fetching it?

so i think we should be doing this try except for get account password policy in the password-policy filter, effectively if there is no password-policy we can just default it to the default values of those policy attributes if no explicit password policy is set (per https://docs.aws.amazon.com/IAM/latest/APIReference/API_UpdateAccountPasswordPolicy.html) or alternatively to {}. afaics this would remove the need entirely for the has-password-policy as a separate filter, since it would allow verifying actual contents of the policy irrespective of mere existence (ie strong verification). . ie a password policy with a password length of 1, doesn't really say much if just verifying existence vs  actually checking the min length (for the record min length that can be set is 6). ie. do you really want to merely check existence of custom, or verify contents.

I think there exists value in being able to exist "Have I set the default policy" vs. "do I have a custom policy" (especially in combined / and'ed filters) - but I think doing so using the existing filter makes more sense.
I'll rework this PR and send it through - think it's worth adding a flag denoting if it has been set / customised, in addition to the default maybe? That may make more sense once I add more detail.

sounds good, an option on the extant for policy-present as a boolean or maybe custom-policy, thanks.

Sounds good - will fix it shortly

How does this look? We add an extra field PasswordPolicyConfigured to the object which signifies it's configured? we match case etc to keep in line with the existing structure of the password policy object.


One picture mistakenly rejected as a duplicate

On some of my uploads I'm seeing the following output from the script:
No sequence file existing
Found 447 pictures to upload
processing G0013736.JPG
skipping - duplicate image
processing G0013738.JPG
Uploaded - 2 of total :447, percentage: 0.45%
processing G0013739.JPG
Uploaded - 3 of total :447, percentage: 0.67%
processing G0013737.JPG
Uploaded - 4 of total :447, percentage: 0.89%
processing G0013741.JPG
Uploaded - 5 of total :447, percentage: 1.12%
processing G0013742.JPG

When this particular upload completed, it only contained 446 images. Here is picture 1 of the sequence: http://openstreetview.com/details/11477/1
If you try to go to picture 0, it forwards to 1.
The first picture is about 20 meters north of the second picture and has a different file name. I'm not sure if this is the upload script or the server responding with an error.

I added some more print statements in the place where it prints out the duplicate image message. The data variable which I believe contains the response from the server is:
{'status': {'apiMessage': ' You are not allowed to add a duplicate entry (sequenceIndex)', 'httpMessage': 'Bad Request', 'apiCode': '660', 'httpCode': 400}}

But according to the file names in the output, it is uploading different files. So it seems like either there is some kind of off-by-one error in the upload script that causes the same contents to be uploaded twice at the beginning of the sequence or the server is mistakenly flagging images as duplicates somehow.

I added more debugging during the upload process. I printed out both the photo and photo_data variables on line 113. Here is the result:
No sequence file existing
Found 522 pictures to upload
photo:  {'photo': ('G0014448.JPG', <_io.BufferedReader name='/G0014448.JPG'>, 'image/jpeg')}
data_photo:  {'sequenceIndex': 0, 'coordinate': '38.65599554001036,-96.48843597000031', 'headers': 59.11930483795209, 'sequenceId': '11489'}
photo:  {'photo': ('G0014449.JPG', <_io.BufferedReader name='/G0014449.JPG'>, 'image/jpeg')}
photo:  {'photo': ('G0014450.JPG', <_io.BufferedReader name='/G0014450.JPG'>, 'image/jpeg')}
photo:  {'photo': ('G0014452.JPG', <_io.BufferedReader name='/G0014452.JPG'>, 'image/jpeg')}
data_photo:  {'sequenceIndex': 3, 'coordinate': '38.656040309998644,-96.48828589000176', 'headers': 92.21664626682987, 'sequenceId': '11489'}
data_photo:  {'sequenceIndex': 1, 'coordinate': '38.656004559988375,-96.48839636003116', 'headers': 67.82361963190183, 'sequenceId': '11489'}
data_photo:  {'sequenceIndex': 2, 'coordinate': '38.656011709996626,-96.48835445003417', 'headers': 71.5006605019815, 'sequenceId': '11489'}
photo:  {'photo': ('G0014453.JPG', <_io.BufferedReader name='/G0014453.JPG'>, 'image/jpeg')}
processing G0014452.JPG
data_photo:  {'sequenceIndex': 4, 'coordinate': '38.6559758199916,-96.48820831999146', 'headers':<PHONE_NUMBER>9733448, 'sequenceId': '11489'}
Uploaded - 1 of total :522, percentage: 0.19%
photo:  {'photo': ('G0014454.JPG', <_io.BufferedReader name='/G0014454.JPG'>, 'image/jpeg')}
processing G0014448.JPG
data_photo:  {'sequenceIndex': 5, 'coordinate': '38.655864060051144,-96.48816235003925', 'headers':<PHONE_NUMBER>1161474, 'sequenceId': '11489'}
data:  {'status': {'httpMessage': 'Bad Request', 'apiMessage': ' You are not allowed to add a duplicate entry (sequenceIndex)', 'httpCode': 400, 'apiCode': '660'}}
name:  G0014448.JPG
skipping - duplicate image

The error message seems to indicate that there is a duplicated sequenceIndex being sent with one of the pictures but I'm not seeing any evidence of this in the output. Maybe there is some threading problem that is changing something being sent across the wire at the last second?
Or what happens on the server if (as in this case) the picture 0 is not the first picture to finish uploading? Could that mess with something?

It is looking to me like this is a server-side problem. I just did some packet captures and everything looks to be in order on my end of the wire. All the uploads have a unique sequenceIndex and I can see the file name in the EXIF data being uploaded. The order of the file names matches the expected sequence index number. The server just seems to be incorrectly returning the duplicated error on the first file uploaded.
As I said on the first comment, it isn't happening every time either. But often enough that I was able to reproduce it for my tests.

Can you send me a link with o couple of photos so i can make a test with them.
Thank's for reporting

I am now pretty certain that the cause of this error is a picture other than picture 0 being finished uploading before picture 0 finishes. At work I wasn't seeing it happen nearly as much as at home. But now I made a test sequence where the 2nd file is tiny (220 KB instead of 2 MB) and now it happens every time. You can grab this test sequence here: https://dl.dropboxusercontent.com/u/1475575/OSM/test_sequence.zip

... Which I suppose means that this issue should actually be moved to the openstreetview.org repository since it seems to be a bug on the server side, not the upload script.

This issue was moved to openstreetview/openstreetview.org#43

@ToeBee i've tested this zip that you create but it work's fine ...

Oh, it is probably ordering it wrong for you. I had to modify the script to order by file name. Ordering by mtime would put the tiny file at the very end of the upload. So either switch the script to file name ordering or use touch to change the mtime on the files. Either way, make sure that G0016050 is the second file to be uploaded.

@ToeBee We managed to solve the problem. Thanks for reporting.
Explanation:
When you upload using threading it's possible to upload first the photo with index 1,2,3... not the foto with index 0 so there is created the sequence. After that you try to upload the photo with index 0, at this point PHP recognized the 0 as Null so it's rejected. But we fixed the problem. Today we hope to make a deploy with this commit.
Thanks again for helping.


Moved format argument in Exporter class

Allowed setting the format of the exportObject function to be changed at runtime.

BC break: NO


Improve developer experience for new contributors

What can we do to make it easier to contribute to this repo?

Hi! I have contributed to another open source organization before. The thing that got me started with github and OpenSource was their first-timers-only issue template.
These are issues specifically made for newcomers who know how to code but are not familiar with github and OpenSource.
Such issues with the specific label first-timers-only gain a lot of github trafic and call a lot of newcomers. I have seen issues being claimed within hours of publishing.
These issues aren't necessarily very difficult or coding related, they can even be a small typo fix but they include instructions (sometimes step by step) on how to use github and explain its principles. These issues bring a ton of newcomers.
Example: https://github.com/publiclab/image-sequencer/issues/1527

I recently started contributing to this repo a little while ago and here is some feedback:
Things I found frustrating:

There is a repo named lbrytv that says it is "LBRY on the web", which I assumed was all the code for lbry.tv, but it is actually the backend written in go. Granted, there is a step in setup instructions that say you must clone lbry-desktop and then "Launch UI in lbry-desktop". So I went through the process of setting up docker and the backend just to run the frontend of the website. It would be helpful if at the top of the readme file for lbrytv that it says this is the repo for the backend and if you just want to modify the frontend, then you should look in lbry-desktop.

The setup instructions for frontend in the the lbrytv repo fails to mention that you need to run yarn compile:web the first time you run the web app.

The setup instructions for lbrytv say "Open http://localhost:8081/ in Chrome" but I believe the correct url is http://localhost:9090/

The contributors guide (https://lbry.tech/contribute) fails to mention anything about the lbrytv repo.

This repo summary says it is "A browser and wallet for LBRY, the decentralized, user-controlled content marketplace." It isn't really clear from this that it is both an app and also the front-end of lbry.tv.

Also at the top of the readme file for this repo it isn't clear that this is used as the frontend for lbry.tv.

I was having problems with a git precommit hook failing due to some flow issues in files that I wasn't working on, but those problems seem to have gone away now.

Things I found helpful:

Everyone I have interacted with from the team has been very helpful and encouraging!

The lbrytv repo description was changed to "API Server for LBRY on the web", and I just opened https://github.com/lbryio/lbrytv/pull/207 to update the readme there which includes a link to the lbry-desktop repo.

I'll give another pass at the readme for this repo sometime next week. I agree it's not very clear this is the UI for lbry.tv and the desktop app.

If I have permissions, then may I convert some of the issues marked with
help-wanted to the first-timers-only issues as I suggested above? I
guarantee that will instantly pull a lot of new contributor traffic.
On Fri, 24 Apr, 2020, 11:50 PM Sean Yesmunt<EMAIL_ADDRESS>wrote:

I'll give another pass at the readme for this repo sometime next week. I
agree it's not very clear this is the UI for lbry.tv and the desktop
app.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/lbryio/lbry-desktop/issues/4036#issuecomment-619171602,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIJI5H22P5XRYYRP3SJAL4TROHJ6VANCNFSM4MNPUD6Q
.

I don't think we will be spending very much time setting up issues for first timers only. The issue you linked looks like it would be very helpful for new contributors, but would take a really long time to build, much longer than actually fixing the issue.
At first I think we should focus on improving the experience for developers with some knowledge of git/github.
Feel free to ping me in any issues you think might be labeled wrong, or could use new labels.

The issue can be generated by a bot too. Also, most of the issue is just a
template, I probably edited only 20 or so lines.
On Sat, 25 Apr, 2020, 12:22 AM Sean Yesmunt<EMAIL_ADDRESS>wrote:

I don't think we will be spending very much time setting up issues for
first timers only. The issue you linked looks like it would be very helpful
for new contributors, but would take a really long time to build, much
longer than actually fixing the issue.
At first I think we should focus on improving the experience for
developers with some knowledge of git/github.
Feel free to ping me in any issues you think might be labeled wrong, or
could use new labels.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/lbryio/lbry-desktop/issues/4036#issuecomment-619185061,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIJI5HZTJW25NPZJLTIKY43ROHNU7ANCNFSM4MNPUD6Q
.

Yes I understand that, but finding an issue as simple as that, and creating a new issue for it will always take longer than just fixing it.

I do think it's a great idea though!

New contributors will be allowed to resolve only one such issue and then once they get comfortable, they can continue with solving other issues. These issues do the first step. New contributors are like lost when they see github issues. They get overwhelmed. They don't know what a PR is or how to edit code and stuff, so they need a step-by-step tutorial like this issue. Once that is done, it's no big deal for them to continue.
I have been contributing to OpenSource for only about a year now. The only thing that made this possible was one these issues. The story that I generalized for new contributors was my story too.

And I am saying that I can do it for the organisation. The core lbry team doesn't have to.

Yes I understand that, but finding an issue as simple as that, and creating a new issue for it will always take longer than just fixing it.

The idea is not to get the issue solved but to attract new people who can later solve issues. This process can be slow at the beginning but once new people start contributing, more people start flowing in. I am pretty sure that a lot of newcomers are browsing the issues from the lbry repos but they just can't get started so they move along.
Also, newcomers just search for first-timers-only issues directly. It can make (and made me) a proper contributor from 0. I didn't even know what github was.

Oh I misunderstood! If you would like to try taking this on go for it! I just added you to the lbry-desktop team. You should be able to edit issues/add labels once you accept the invite. Can you see if it works?

Yeah it worked. Ty. I will get started on it soon.
On Sat, 25 Apr, 2020, 12:35 AM Sean Yesmunt<EMAIL_ADDRESS>wrote:

Oh I misunderstood! If you would like to try taking this on go for it! I
just added you to the lbry-desktop team. You should be able to edit
issues/add labels once you accept the invite. Can you see if it works?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/lbryio/lbry-desktop/issues/4036#issuecomment-619190952,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIJI5H54Y62NCOKGDMZ75KLROHPIVANCNFSM4MNPUD6Q
.

@HarshKhandeparkar Great! Let me know if you have any issues.


feat: update notifications for ceph image

I would like to get some early feedback on this, so opening this for review


Feature/Add Policies to Relationships

Signed-off-by: Björn Müller<EMAIL_ADDRESS>Add Policies to Relationships in Topology Modeler

[x] Ensure that you followed http://eclipse.github.io/winery/dev/ToolChain#github---prepare-pull-request. Especially, we require a single commit
[x] Ensure that the commit message is a good commit message
[x] Ensure to use auto format in all files
[x] Ensure that you appear in NOTICE at Copyright Holders
[ ] Tests created for changes
[ ] Documentation updated (if needed)
[ ] Screenshots added (for UI changes)

Can one of the admins verify this patch?


Partially revert “Support @objc(CustomName) on extensions”

Revert portions of 07b9fe9ce6692da25f386fb933deffb71c21b8a2 to temporarily avoid applying @objc(CustomName) extensions to generated header category names.
Fixes rdar://135924149.

@swift-ci test

@swift-ci test Windows platform


Make openstack target for installing openstack operators only

This pr makes the certmanager operator installation conditional by introducing a new var INSTALL_CERT_MANAGER (default to true).
It also moves the certmanager target under openstack_prep as Network related vars are placed under openstack_prep.
Now by setting BMO_SETUP=false, INSTALL_CERT_MANAGER=false, NETWORK_ISOLATION=false, NETWORK_BGP=false, we can install the openstack operator independently.
It will help to use this target to just install openstack operator.
This feature will be useful for va testing where all the infra/networking stuff will done by ci-framework role.

/retest

recheck

recheck

recheck

recheck bm failure

recheck

recheck


Zoom-in with two side-by-side floating window

Sometimes we need to compare. (I haven't thought about the details.)

Say you have two floating windows, then why not just create a tab for this?


[SP-5856] Backport of PDI-18303 - Transformation Step: Get Sub folder…

… names fails to retrieve any folders if user does not have access to a sub folder (9.1 Suite)
Original PR: pentaho-kettle#7803
@pentaho-lmartins @ssamora @bcostahitachivantara

:white_check_mark: Build finished in 23m 19s
Build command:
mvn clean verify -B -e -Daudit -amd -pl core

:ok_hand: All tests passed!
Tests run: 4940, Failures: 0, Skipped: 7    Test Results

:information_source:  This is an automatic message

:white_check_mark: Build finished in 23m 19s
Build command:
mvn clean verify -B -e -Daudit -amd -pl core

:ok_hand: All tests passed!
Tests run: 4940, Failures: 0, Skipped: 7    Test Results

:information_source:  This is an automatic message


Local address at <IP_ADDRESS> with a HOSTS blocking file using <IP_ADDRESS>

Sorry if the question is naive.
I'm running dnscrypt-proxy with the default <IP_ADDRESS> local address (and <IP_ADDRESS> for second instance).
I happen to use a HOSTS file dedicated to blocking which may be set to block addresses with <IP_ADDRESS> or <IP_ADDRESS>
Should dnscrypt-proxy using <IP_ADDRESS> be considered as an argument to avoid <IP_ADDRESS> on the HOSTS file? If so, dnscrypt-proxy set to use <IP_ADDRESS> and <IP_ADDRESS> or set HOSTS to use <IP_ADDRESS> ?
Thanks.

Redirecting domains you want to avoid to <IP_ADDRESS> or <IP_ADDRESS> is a terrible idea, as connections will still happen, and will be directed to your local machine. The hosts file also involves a linear scan for every query, which is slow if you have a long list.
You can use dnscrypt-proxy to block domains. It's much easier, faster and powerful than a hosts file, as it can match suffixes, prefixes and patterns. It can also log the queries that have been blocked, along with the rule that cause the block.
If you are using dnscrypt-proxy, there are no reasons to use a hosts file.

Thanks, @jedisct1
I've always used a HOSTS file here with Windows, manged by an application called HostsManager which allows to concatenate several HOSTS sources, always with Windows DNS Client service (dnscache) set to disabled and the blocking has always been free of any problems, speed included.
I understand your argument. The fact is,

I use several hosts sources all managed by HostsMan
Using dnscrypt-proxy to block domains is easy with a local file, in my case, set in the conf file:

BlackList domains:"d:\My Data\DNSCrypt\dnscrypt-dom.txt" logfile:"d:\My Data\DNSCrypt\dnscrypt-blocked.log"

But mixing external sources with my own local blocked domains file is more problematic for me. It's easy with Simple DNSCrypt, less with dnscrypt-proxy. I've read the documentation here but as I understand it I'd have to use .py scrpting to handle those external sources... that's what I'm facing at this time.
I'm discovering a new world, I appreciate,  always, your help, and so promptly proposed. But I cannot ask here for personal school-type assistance! So I try to manage and ask for advice/help only when I believe the answer is out of my present possibilities.
Of course using dnscrypt-proxy's built-in features to handle all domains and IPs is the most clever thing to do.

@zymase So what is the problem to use Simple DNSCrypt if it makes all easier? Do you lose something?
@jedisct1 I'm using W10Privacy which contains lists of blocked IPs and domains. It uses the hosts file to block domains, and the firewall to block IPs.

Is it faster to use DNSCrypt to block both? You confirm for the hosts file but for the firewall? Even if we take a classic list without pattern?
If yes, after entering it in DNSCrypt, should we remove the entries from hosts/firewall to be faster, or can we keep it without performance lost?

@TroudhuK I'd rather run whatever app free of .net Framework, because it is dependency and because it is heavy.
Also, Simple DNSCrypt seems unable to log the blocked domain/ips, as with

BlackList domains:"d:\My Data\DNSCrypt\dnscrypt-dom.txt" logfile:"d:\My Data\DNSCrypt\dnscrypt-blocked.log"

I saw no way of getting this dnscrypt-blocked.log
Third, I cannot get Simple DNSCrypt to add my local domain blocking file to external ones (which it handles nicely) : local filters have to be added 1 by one. My domain blocking file is important, 300+ entries so i've tried to have SDNSCrypt add it via a "local url" such as file:///D:/My%20Data/DNSCrypt/dnscrypt-dom.txt : it's accepted but the content is not added!
@jedisct1 is right, everything can and should be operated via dnscrypt-proxy's blocking features. But ley this feature be considered to its full potential : I will not add an external source built up as a hosts file when,dnscrypt can handle with one rule, i.e. 302br.* ... hundreds of the family in an external source. What i'm doing right now is to consider the heavyweights of my HOSTS file (such as 302br.) which can be considered not as exceptions to a valid domain but the very expressions of unhealthy domains. Once done I add their short form to my local domain blocking file and ... add them to my HostsMan applications 'exceptions' (HostsMan then removes them from my Hosts file. I did so and, just with 302br. my HOSTS file went from 80,000 to 65,000 or so entries!
My rhetoric is not elegant and even less technical but I hope you get the idea. Moreover I'm French so English is expressed with many approximations.
Thanks.

Je suis français aussi, je crois qu'on l'est tous en fait, qu'est-ce qu'on s'embête avec une langue étrangère ? Enfin bon.

Third, I cannot get Simple DNSCrypt to add my local domain blocking file to external ones (which it handles nicely) : local filters have to be added 1 by one.

You can edit domains-blacklist.yaml in Simple DNSCrypt\data folder. Simple DNSCrypt is just a configurator GUI for DNSCrypt-proxy, which evolves well to support all the possibilities, but you can edit the configuration yourself, waiting for the support of more features from the GUI.

I saw no way of getting this dnscrypt-blocked.log

Simple DNSCrypt configure at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\dnscrypt-proxy\Parameters, then you can edit Plugins, and at the line libdcplugin_ldns_blocking.dll add :
,--logfile=d:\My Data\DNSCrypt\dnscrypt-blocked.log

Merci, @TroudhuK (original comme pseudo!)
OK. For editing the .yaml file directly, I ignored that. Cumbersome nevertheless.
As for editing the Reg entry, you do realize I'd have to do it every time I uninstall a dns server or whenever the connection to that server is closed (this happens sometimes when editing the settings): cumbersome as well.
Cumbersome but feasible. I still prefer rolling over dnscrypt-proxy (not to mention Beethoven!).
Thanks again, compatriote :)


feat(lib): Support comments in opt-in/out files

Current coverage is 100.00%

Merging #15 into master will not affect coverage as of 1506b37

@@            master     #15   diff @@
======================================
  Files            2       2
  Stmts           68      68
  Branches         0       0
  Methods          0       0
======================================
  Hit             68      68
  Partial          0       0
  Missed           0       0

Review entire Coverage Diff as of 1506b37

Powered by Codecov. Updated on successful CI builds.


Upgrade i18next to latest

Description
Upgrades i18next to v19, and change the config/set up to match the new API.
Related Issues / PRs
Closes https://github.com/overleaf/issues/issues/3276
Review
Probably best reviewed commit-by-commit.
I don't believe there are any changes needed on the web side. I've tried to keep this backwards compatible.
Manual Testing
Installed the branch to a local version of web and checked

[x] English translations work
[x] French translations work
[x] Interpolation works

[x] Basic
[x] With multiple interpolations (e.g. Hello __foo__-world __bar__)
[x] With escaping disabled

[x] showUserOtherLng flag works

Yes I just pushed a commit to match that, this PR is still WIP.
I was leaning towards a change in ExpressLocals to disable where needed (hence separate commits). But then I realised there's also direct calls to req.i18n.translate, where we might be relying on the lack of escaping. So I've switched back to disabling it by default.

Fwiw, I also decided to leave off https://github.com/i18next/i18next/blob/5fb4dbcb7ccdcfc78192e69ef3291d2ace6814b4/test/backward/compatibility/v1.js#L13-L14 since I can't find any usage of the nesting syntax.

We probably want to disable the nested interpolation as well? It can be used to blow-up translations/mislead users quite a bit: https://repl.it/repls/SimilarImpressiveScriptinglanguages (forked from your repl, nice tool for this kind of thing!)
https://www.i18next.com/translation-function/nesting Oh and it supports passing your own context in -- hello prototype pollution.

We probably want to disable the nested interpolation as well?

To be clear - I meant that I couldn't find any usage of nesting in our existing translations, and so we don't need to add backwards compatibility for it.
However as you point out there is a security angle here. I've added the skipOnVariables flag which prevents nesting inside interpolated values.

I think this is now ready for review. Assigning @mserranom for fresh eyes and @das7pad since you've already looked at it.

i18n.translate is not exported

Ah right, I hadn't realised that it was used directly. I'll fix that now.

So I looked into the failing test (should not try to extract the project name from the URI), with the fix for i18n.translate and it's still failing (locally and on CI).
I believe it may actually be a problem with the test, and the old behaviour is only "accidentally" working. This is what I think is happening:

The test bootstrap initialises the server: https://github.com/overleaf/web-internal/blob/35c87efcbc1add4d4731c371262c69f5ab646b9a/test/acceptance/src/Init.js#L1
This ends up running the translation package's setup with some settings: https://github.com/overleaf/web-internal/blob/35c87efcbc1add4d4731c371262c69f5ab646b9a/app/src/infrastructure/Server.js#L31
The test then runs setup again, this time with no settings: https://github.com/overleaf/web-internal/blob/35c87efcbc1add4d4731c371262c69f5ab646b9a/modules/open-in-overleaf/test/acceptance/src/OpenInOverleafTests.js#L18

With the old version, I suspect that the translations are cached so that on the second call the lack of settings is ignored, and the English translations are actually loaded within the test.
In the latest version, no translations are found within the test (due to the lack of settings). This means that translate will only echo back the key. The test relies on the English translation being loaded (and not being changed), so fails.
There are a few options here:

Switch the assertion to translations.i18n.translate('new_snippet_project'). This is what we are doing in at least 1 other test: https://github.com/overleaf/web-internal/blob/35c87efcbc1add4d4731c371262c69f5ab646b9a/modules/open-in-overleaf/test/acceptance/src/OpenInOverleafTests.js#L1539
Pass some minimal config to the setup call within the test, so that the English translations are loaded correctly
Create some mechanism for accessing the i18n instance created in the server initialisation

I am currently leaning towards option 1, as this would protect the test against any changes to the underlying translation string. However in the long term, I think we should also consider option 2 or 3.
cc @gh2k @jdleesmiller

I went ahead a made a PR with the test fix proposed above: https://github.com/overleaf/web-internal/pull/3076. This is blocked until that is resolved.

OK, option 1 seems fine for now. I would lean toward option 2 in the longer term --- having actual (English) strings hopefully makes the tests a bit easier to read and write and brings them closer to what the user actually sees (even though that does mean that changing translations can break tests).

I was thinking more about this and I realised that there's an option 4: remove the translations package entirely and test only against the English strings.
This has a few benefits:

Satisfy the desire to make the tests closer to what the user sees
Bring the tests more inline with the other tests (AFAICT this is the only instance of the package being imported to the tests)
We could remove the i18n.translate alias that was added only for this one test

This is another longer term fix as it would require changing the existing tests to use the English strings.

The setLng logic seems to be broken

Thanks good catch. Fixed in https://github.com/overleaf/translations/pull/31/commits/4b958d0d04c2afc6797bc864d77b70357da033e8, and added test for this behaviour.

Oh actually, before you approve, I have some tests that I wanted to add.

Oh actually, before you approve, I have some tests that I wanted to add.

Snap I already submitted, but you can probably add them w/o my approval :) (Or just ask for another review, I do not mind.)

Ok, I've added some tests for the translate function, which I believe addresses most of https://github.com/overleaf/issues/issues/3390. I talked it through with @jdleesmiller and we think it's fine to just test for the actual translation string (even though there's some risk of the translation string changing, causing the test to fail).


How should missing values be handled for Jaccard metric?

My case is some roll call data where the members of parliament can be away (in a non-meaningful way) or just be on their seat temporarily, so that it is encoded as a missing value. However, I get errors due to missing data. Since the Jaccard metric itself is agnostic to the amount of available information, is there any way to handle this?

I ended up precalculating a distance matrix and providing that, such that I only compare actual voting decisions i.e. not absences which are not meaningful in a Swedish context, in contrast to abstentions which have an intentional meaning. Here is my R code which may benefit someone else (I am using umap through reticulate):
# vectors a and b should be equal length, e.g. the full voting record of an individual
# this calculates the distances pairwise
jaccard_dist <- function(a, b) {
  if(length(a) != length(b)){stop("Unequal lengths")}
    intersection = sum(Vectorize(`==`)(a, b), na.rm = TRUE)
  union = length(na.omit(a))+length(na.omit(b)) - intersection
  output = 1-(intersection/union)
  return(output)
}

# to create a distance matrix
usedist::dist_make(dat_mat, jaccard_dist) -> dist_output

And here is ChatGPT's translation into Python 😀
import numpy as np
from scipy.spatial.distance import pdist, squareform

def jaccard_dist(a, b):
    if len(a) != len(b):
        raise ValueError("Unequal lengths")

    intersection = np.sum(np.equal(a, b), where=~np.isnan(a) & ~np.isnan(b))
    union = (len(a) - np.isnan(a).sum()) + (len(b) - np.isnan(b).sum()) - intersection
    output = 1 - (intersection / union)
    return output

def dist_make(dat_mat, distance_function):
    dist_output = squareform(pdist(dat_mat, metric=distance_function))
    return dist_output

# To create a distance matrix
# Replace 'data_matrix' with the actual data matrix you are working with.
dist_output = dist_make(data_matrix, jaccard_dist)


improved seo and add favicon closes #9

Improved seo by adding some keywords and added new logo due to courtesy of mohuxaiphasalik

Also , added some new labels and label generator action


esri leaflet live sample breaks when updated to 0.4.0

http://esri.github.io/esri-leaflet/examples/webmap.html?id=22c504d229f14c789c5b49ebff38b941
bumped cdn on the sample above to 0.4.0

Uncaught TypeError: L.esri.webMap is not a function(…)

@jgravois Thx for letting me know!
it requires leaflet-omnivore to parse CSV in 0.4.0.
would you add the cdn url below and try it again?
<!-- Load Leaflet Omnivore -->
<script src='https://api.tiles.mapbox.com/mapbox.js/plugins/leaflet-omnivore/v0.3.1/leaflet-omnivore.min.js'></script>

worked like a charm. thanks!


Allow workspace to be configured when deploying ckcp

The workspace in which the service is deployed can now be configured by changing the kcp.workspace value in config/config.yaml.
The structure of the config.yaml file has been modified to improve readability by grouping values together when possible.
Signed-off-by: Romain Arnaud<EMAIL_ADDRESS>

/retest

/retest

@xinredhat I can't reproduce the error on plnsvc-ckcp-test in my dev env using a cluster-bot hypershift cluster. I tried to access the test cluster, but while I can login, I don't seem to have the rights to see anything. Could you please have a look and help me troubleshoot the issue?

/retest

@Roming22 the ckcp-test error was a flaky error which I too have faced earlier. Triggered a retest and looks to be good. Merging the PR.


[cluster_test] Check ac status in tx emitter

Previously we ignored returned submission status, as long as grpc succeeded.
This can complicate debugging if transactions are rejected, so we do need to distinguish accepted and rejected transactions. For now we just print error if transaction was not accepted.

@bors-libra r+

:pushpin: Commit 5ec2e07 has been approved by dmitri-perelman

:lock: Merge conflict

:umbrella: The latest upstream changes (presumably ed3f33d53ab6076ef55db139e018cf84c456b7b4) made this pull request unmergeable. Please resolve the merge conflicts.

@bors-libra r=dmitri-perelman

:pushpin: Commit 6ea3f37 has been approved by dmitri-perelman

:hourglass: Testing commit 6ea3f373b937a45df89609d38ca942f5a5973f6f with merge c09b822abee105ea27667424e40e22f5006ce766...

:sunny: Test successful - checks-circle_commit_workflow
Approved by: dmitri-perelman
Pushing c09b822abee105ea27667424e40e22f5006ce766 to master...


llvmdev 14.0.0.rc4

Let's see if the CMake stuff now works without workarounds.

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.


support m1

This PR resolves two compile errors on m1/arm architecture. Tested on both x86_64 and arm.

keccakf1600

{:error,
{:load_failed,
'Failed to load NIF library: 'dlopen(/Users/wenchen/Work/vivo_wallet_lib/_build/dev/lib/keccakf1600/priv/keccakf1600.so, 0x0002): tried: '/Users/wenchen/Work/vivo_wallet_lib/_build/dev/lib/keccakf1600/priv/keccakf1600.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/wenchen/Work/vivo_wallet_lib/_build/dev/lib/keccakf1600/priv/keccakf1600.so' (no such file), '/Users/wenchen/Work/vivo_wallet_lib/_build/dev/lib/keccakf1600/priv/keccakf1600.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/Users/wenchen/Work/vivo_wallet_lib/deps/keccakf1600/priv/keccakf1600.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/wenchen/Work/vivo_wallet_lib/deps/keccakf1600/priv/keccakf1600.so' (no such file), '/Users/wenchen/Work/vivo_wallet_lib/deps/keccakf1600/priv/keccakf1600.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))''}}

libsecp256k1

c_src/secp256k1/src/num_gmp.h:10:10: fatal error: 'gmp.h' file not found
#include <gmp.h>
^~~~~~~

Fixed were pulled from https://github.com/diodechain/libsecp256k1 and https://github.com/hayesgm/erlang-keccakf1600

Thanks @wchenNL :). I will update upstream with a few newer libraries, one to replace keccakf1600 that I also wrote (https://github.com/tzumby/ex_keccak) and the secp256k1 the Rust version with pre-compiled binaries (https://github.com/ayrat555/ex_secp256k1). We're both very active in maintaining those two

Thanks @wchenNL :). I will update upstream with a few newer libraries, one to replace keccakf1600 that I also wrote (https://github.com/tzumby/ex_keccak) and the secp256k1 the Rust version with pre-compiled binaries (https://github.com/ayrat555/ex_secp256k1). We're both very active in maintaining those two

👍 That would be much better. Right now I am just pulling some fixes from a fork of the original libsecp256k1 and erlang-keccakf1600` library.


longevity panic: unwrap() in source_executor.rs

Describe the bug
thread 'risingwave-streaming-actor' panicked at 'called `Option::unwrap()` on a `None` value', /risingwave/src/stream/src/executor/source/source_executor.rs:258:14

https://github.com/risingwavelabs/risingwave/blob/66df2f77a63458e78b0e7d6b2be1e3e69f0f5a0f/src/stream/src/executor/source/source_executor.rs#L258
Full logs:
https://rqa-logs.s3.ap-southeast-1.amazonaws.com/longevity/222_logs.txt
OR
https://ap-southeast-1.console.aws.amazon.com/cloudwatch/home?region=ap-southeast-1#logsV2:log-groups/log-group/$252Frisingwave$252Frls-apse1-eks-a$252Frwc-3-longevity-20221214-180848/log-events/ip-10-0-74-173.ap-southeast-1.compute.internal-risingwave.var.log.containers.risingwave-compute-1_rwc-3-longevity-20221214-180848_compute-5a7b1ee2a90c3f8ddf2cfea2c1993d7918d26f1aa3d199281c26bb3d4017855e.log$3FfilterPattern$3D
To Reproduce
N/A
Expected behavior
No response
Additional context
No response

Hi! Any updates? Please don’t mind to share some discoveries or handling it to someone else.

#7086 just replaced the panic into the error returned to the upper layer, in fact, if there is a problem the upper layer will still panic, this issue also needs to continue to investigate, the main problem is that there is no very effective way to reproduce the scene

Any updates?

The problem has not recurred since then, so it seems that it can be closed now.


Inference of base-type for boolean database-type columns not working

This issue might be related with issue #88
Inference of base-type for boolean database-type columns is still not working. Given a boolean column with database-type "Bool", the base-type metadata is recognized as :type/* instead of :type/Boolean (e.g. see GUI-->Admin-->DataModel-->Table-->Original schema).
I will propose a PR with fix and tests to solve this issue...

@s-huk released as https://github.com/ClickHouse/metabase-clickhouse-driver/releases/tag/1.0.1