Error when Loading Compressed JSON DataSet through the Catalog

I have a dataset defined in my DataCatalog which is a gzip compressed JSON lines file that I would like to load through Kedro using a JSONDataSet. I have the following definition in my DataCatalog:
test_json:
  type: JSONDataSet
  filepath: "data/01_raw/test.json.gz"
  load_args:
    lines: true
    compression: 'gzip'

However, when I try and load this dataset through a Kedro ipython session with the command:
df = catalog.load('test_json')

I get the following error message:
DataSetError: Failed while loading data from data set JSONDataSet(filepath=data/01_raw/test.json.gz, load_args={'compression': gzip, 'lines': True}, protocol=file).
'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

I did some digging and it turns out GZIP adds an additional byte character, see here.
Then, I decided to look under the hood of JSONDataSet's load function and I found the following in kedro/io/json_dataset.py lines 138-142: https://github.com/quantumblacklabs/kedro/blob/master/kedro/io/json_dataset.py#L138-L142
    def _load(self) -> Any:
        load_path = get_filepath_str(self._get_load_path(), self._protocol)

        with self._fs.open(load_path, mode="r") as fs_file:
            return pd.read_json(fs_file, **self._load_args)

It looks as through by forcing the file to be read initially through the filesystem, it is complaining about the startbyte. Can I ask why there is an explicit need to read the file first and then invoke pandas? Couldn't this be done directly through
return pd.read_json(load_path, **self._load_args)

That would resolve the issue I am currently having with reading the gzip compressed json file

Hi @uwaisiqbal , good to see you back here! Thanks for a really well-written issue!
I believe the ability to pass a path object into pandas.read_json() is only possible since pandas 1.0.0. The older versions, which Kedro is compatible with, expect a string which is a valid JSON or a file-like object, which is why we go through _fs.open(...). The string can also be a URL, but that's not something get_filepath_str returns, except  if there's an HTTP protocol.
We have an item on our backlog to make Kedro work with new, stable pandas, which will be included in the next release. Immediate workaround would probably be to have a custom dataset class to handle your case?

Hey @lorenabalan, thanks for getting back to me.
Yup I have a custom dataset as a quick fix for this. Guess I'll just have to wait for the next release.
Thanks!

Hi @uwaisiqbal,
Thank you that you reported this issue.
Answering your question: we are using fsspec for filesystem related operation to have a general approach in all the supported datasets. The gzip compressed JSON file is actually a binary file but we open it as usual file mode="r" that causes the problem.
We are planning to fix this issue in the nearest future. Now we decide how to fix it in the best way since there are a few approaches. One of them is to change the mode to mode="rb".

Hi everyone, this should be fixed by this commit and made available in the next release: https://github.com/quantumblacklabs/kedro/commit/8329f452b96bda4ca3a3cd2b30f71f261e6a8af8
I'll go ahead and close this issue for now, feel free to come back if it's still causing you problems.


⚠ ClamAV has degraded performance

In 5e8dbd7, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 476 ms

Resolved: ClamAV performance has improved in ffdec82.


Introduce "useVisitorData" hook

I'm wondering if we could introduce a similar hook as we have in https://github.com/fingerprintjs/fingerprintjs-pro-react for this package as well?
Something like this:
import {useVisitorData} from '@fingerprintjs/fingerprintjs-pro-react-native';

  const {
    isLoading,
    error,
    data,
  } = useVisitorData('PUBLIC_API_KEY', 'REGION');

I believe that it would provide a better developer experience, and make it more consistent with our react library.
To keep the consistency we can also omit the parameters here, and add a provider as well.
Things to consider:

would we also want to use the @fingerprintjs/fingerprintjs-pro-spa for caching?
maybe in order to avoid duplicating the code, we can use the logic from https://github.com/fingerprintjs/fingerprintjs-pro-react, while providing custom agent implementation, in order to use the native libraries under the hood?

WDYT?

Hook added in v1.0.0


use CF conventions to enhance plot labels

Elsewhere in xarray we use CF conventions to help with automatic decoding of datasets. Here I propose we consider using CF metadata conventions to improve the automatic labelling of plots. If datasets declare long_name, standard_name, and units attributes, we could use these instead of the variable name to label the relevant axes / colorbars. This feature would have helped me avoid several past mistakes due to my failure to examine the units attribute (e.g. data given in cm when I assumed m).
Code Sample, a copy-pastable example if possible
Here I create some data with relevant attributes
import xarray as xr
import numpy as np
ds = xr.Dataset({'foo': ('x', np.random.rand(10),
                         {'long_name': 'height',
                          'units': 'm'})},
                coords={'x': ('x', np.arange(10),
                              {'long_name': 'distance',
                               'units': 'km'})})
ds.foo.plot()

Problem description
We have neglected the variable attributes, which would provide better axis labels.
Expected Output
Consider this instead:
def label_from_attrs(da):
    attrs = da.attrs
    if 'long_name' in attrs:
        name = attrs['long_name']
    elif 'standard_name' in attrs:
        name = attrs['standard_name']
    else:
        name = da.name
    if 'units' in da.attrs:
        units = ' [{}]'.format(da.attrs['units'])
    label = name + units
    return label
ds.foo.plot()
plt.xlabel(label_from_attrs(ds.x))
plt.ylabel(label_from_attrs(ds.foo))

I feel like this would be a sensible default. But it would be a breaking change. We could make it optional with a keyword like labels_from_attrs=True.
 For publication/production purposes, I suspect labels are almost always been explicitly set, anyways.
So I think this convention makes sense. Let's just document it directly in the plotting docs, maybe even by including your label_from_attrs() function to make it fully explicit. That way non-weather/climate users can also make use of it, without referring to the full CF conventions spec.

It would take me some time to wrap my head around the plotting code again, and as a result I don't anticipate having time to work on this any time soon.
I know @dcherian has been digging into the plotting functions a lot. I imagine that he would be able to implement this much faster than I would. Just leaving that suggestion out there... ;)
Obviously this is not an urgent issue.

I can do it. are we agreed that long_name, standard_name, and units are the attributes we want to follow? I think we should also add support for upper-case versions of these (I think I've seen those in the wild)


chore: forward-port leanprover-community/mathlib#17483

This forward ports the changes intoduced by #17483.

There is one file touched by #17483, data.real.ereal, which needed no change (the relevant proofs had been changed to not use the lemmas that have been removed). I assume I should still update the SHA there?


Various improvements for TitleColorMatch

Black logos will now have their color inverted so that they're readable on the black gradient
Cleaned up some code

This should probably have the font_color set to auto by default since that's the main appeal of using this over GradientLogoTitleCard.

I need to nail down a better minimum luminosity still. Also some of the code returns on the first loop of the luminosity check every time even if its too low, I will fix that.


Unable to import on docker

Summary of Problem
(Please answer all 3)

What are you trying to do?
import the library inside docker container
What happens?
throws;

/workspaces/ed64/node_modules/bindings/bindings.js:91
        throw e
        ^

Error<EMAIL_ADDRESS>invalid ELF header
    at Object.Module._extensions..node (module.js:681:18)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at bindings (/workspaces/ed64/node_modules/bindings/bindings.js:84:48)
    at Object.<anonymous> (/workspaces/ed64/node_modules/@serialport/bindings/lib/linux.js:1:98)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)

What should have happened?
Warn if it is not available?

Code to Reproduce the Issue
const SerialPort = require('serialport')

Versions, Operating System and Hardware

SerialPort@? 7.1.0
Node.js v? v8.10.0
Windows? Linux? Mac? Ubuntu 18.04 on docker
Hardware and chipset? (Prolific/FTDI/Other) N/A

Ok, I'm pretty sure that's it thank you!


syscall.Stat_t with different fields' name between Darwin and Linux

What version of Go are you using (go version)?

$go version

go version go1.12.6 darwin/amd64

Does this issue reproduce with the latest release?
YES!
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/MC/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/MC/Golang/pkgs"
GOPROXY=""
GORACE=""
GOROOT="/Users/MC/Golang/go"
GOTMPDIR=""
GOTOOLDIR="/Users/MC/Golang/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/MC/Workspaces/qutoutiao/godis/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/th/22jxv0q12c5c23v02rt67gg00000gp/T/go-build623000608=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?
Access file ctime from os.FileInfo;
https://play.golang.org/p/LD80qQy5qyC
What did you expect to see?
All os.FileInfo.Sys()  should have the same fields' name across OS.
What did you see instead?
Within Linux OS, it should call Stat_t.Ctim.Sec; But under Darwin OS, you have Stat_t.Ctimespec.Sec in hand.

Unfortunately I don't think we can change this now.  It would break existing programs.

In https://github.com/golang/go/issues/18130#issuecomment-267703067, I suggested struct field aliases as a next step after type aliases. That would allow the two platforms to be unified by defining one of the field names as an alias for the other.
That would, however, require a language change.

There are three ways to archive this:
1, We could and extra fields to Stat_t without break anything, but a little
confusing;
2, We could add some Getter methods to Stat_t, which can provide universal
accessor for underline;
3, The last and I preferred! We should fix this at go1.13, prepared for go2.
Otherwise, developers have to use build tag to deal with the issue.
On Tue, Jul 2, 2019 at 9:25 PM Ian Lance Taylor<EMAIL_ADDRESS>wrote:

Unfortunately I don't think we can change this now. It would break
existing programs.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/32887?email_source=notifications&email_token=AAAB3YTFMKNAJVVLJFWLJF3P5NJNLA5CNFSM4H4XYPX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZBIATI#issuecomment-507674701,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAB3YRHWL4ZPAWOYZM733DP5NJNLANCNFSM4H4XYPXQ
.

--
Best Regards!
- Spring


Wavelet transform bias

Hi
I have come across an apparent issue with the current implementation of the Morlet complex wavelet function used in elephant.signal_processing.wavelet_transform(). For my somewhat circuitous working, see [1] and [2].
According to the documentation, this implements the Morlet wavelet function described in Le Van Quyen et al. (2001) J Neurosci Meth 111:83-98. However, it appears the current code does not, as in Eqn 4 of Le Van Quyen et al., scale the function by the square root of the wavelet frequency. The result of this is that the wavelet power calculated from the output of the transform is increasingly underestimated at higher frequencies. A workaround for this issue seems to be to multiply the calculated wavelet power by its corresponding frequency.
Used scipy (1.2.1), elephant (0.6.2) and neo (0.7.1).
[1] cwt_power_test.pdf
[2] cwt_power_test.ipynb

Thank you for pointing that out. We'll take a look and let you know.

On request of the poster, temporarily closing the issue for now for some further cross-checking of the issue. Re-opening if necessary.


test-branch/execute-job/fork-branch/bc8264ef-e54a-40bf-9953-151a15031228

PR for testing 3fb2bce97319f5c802202bbb20ae5cc9279e8785

padding /canonical/self-hosted-runners/run-workflows 3fb2bce97319f5c802202bbb20ae5cc9279e8785 padding


ES256Algorithm Returns Result.Ok() Even If  this.PublicKey.VerifyData() Returns False

Hi @angusmillar,
I noticed an issue in the ES256Algorithm. It looks to return Result.OK(...) regardless of the outcome of the verification of the payload/signature.
https://github.com/angusmillar/SmartHealthCard/blob/a600df729d3408694258ff6ded81246faaab38b4/SmartHealthCard.Token/Algorithms/ES256Algorithm.cs#L66-L72
I don't see anything in the JwsSignatureValidator that reacts to that boolean passed back either.
https://github.com/angusmillar/SmartHealthCard/blob/a600df729d3408694258ff6ded81246faaab38b4/SmartHealthCard.Token/JwsToken/JwsSignatureValidator.cs#L12-L30
Unless I'm missing something else, it looks like it's always returning that the SmartHealthCard is valid even if the payload/signature doesn't validate correctly. Let me know if I'm mistaken.
Again, thanks for your hard work. Your code has been really easy to work with.

Hi @DandiAndy , Thank you for reporting this, rather embarrassing bug.
You were correct, the library was ignoring invalid signatures and not reporting any issue. I have just released a new Nuget package 'SmartHealthCard.Token' V1.0.2 that resolves this issue. The new version will now throw a new exception type of  SmartHealthCardSignatureInvalidException when the signature is found to be invalid. This new exception type inherits from the older type that was documented to be thrown, so it is backward compatible in that sense. Yet this new type now gives you a way to single out an invalid signature as opposed to any other form of error. I've also put in some bloody test cases to ensure this never happens again.
I also added another exception type 'SmartHealthCardJwksRequestException' which singles out the issue of the JWKS (public key) not being accessible, likely due to no internet connection.
I'm also going to report this bug on the Zulip https://chat.fhir.org/ 'Smart/Health-Cards' channel as it is rather critical for anyone using the library.
Thanks again and sorry for any inconvenience this has caused.

Thanks @angusmillar!


feat: feast component

Create a feast component to be able to add a feature store to a stack.

Fixed in #20


VCS reported an error, NYI-CSTR-SYS-FTC

I am totally a rookie and learning this.
I did nothing but download the toolchains and build the spike under the user guide.
When I type "make" in the terminal.
My VCS just report an error like this.

NYI-CSTR-SYS-FTC
riscv_instr_pkg, "$clog2(((XLEN * 4) / 8))"
System function calls are not yet implemented in constraints.
Remove the function call or if possible replace it with an integral state
variable assigned in pre_randomize().

My Environment

lowrisc-toolchain-rv32imc-20200904-1.tar.xz
from
https://github.com/lowRISC/lowrisc-toolchains
spike
from
https://github.com/riscv/riscv-isa-sim
EDA tool and version:

VCS L-2016.06
uvm1.2
Operating system:

"Ubuntu Linux 20.04"
Version of the Ibex source code:

latest, just download today 3.23.2021

I suspect we just require a newer version of VCS than that. I see that we don't list a minimum version in our tool requirements page (sorry!), but we've been using the "Q-2020.03" series of releases to test things locally. I'll file a follow-up issue to track adding a version to the requirements page.

I suspect we just require a newer version of VCS than that. I see that we don't list a minimum version in our tool requirements page (sorry!), but we've been using the "Q-2020.03" series of releases to test things locally. I'll file a follow-up issue to track adding a version to the requirements page.

thanks, I will updata my VCS and try again.


CriticalTemperatureGauge mod repo changed

The second commit would have updated the existing PR if you'd left it open, by the way. GitHub is pretty cool that way. This looks all fine to me, and Jenkins is happy, too!
Merging with thanks!


removed unused assets.

according to the feedback on https://github.com/expressjs/expressjs.com/pull/1489 removed the unused assets

and sorry for the inconvenience with the old pull request the vs code was asking me to fork the repo I already had forked so I deleted the fork and then forked again and that fixed it.

I checked out the fork/branch and ran it locally to make sure there were no issues, so LGTM.
I just want to make sure that there's no reason to keep these file around that I'm not thinking of, so I've asked for another review from @jonchurch just to be sure.


Add features to generic callable bond

Currently, the generic callable bond example only contains one coupon, so it does not show any nesting of claims. I have added a 2nd coupon to make the example a bit more realistic and to show claims nesting.
I also added a separate principal amount in case of a call, to make it possible to add a call premium if desired.

I would test if the lifecycle works as expected (I believe it does not in its current form)

I added a test for not calling as well, and the coupon payment seems to be lifecycled fine.


Feature  : Add a Carousel in the hero section rather than a still picture !.

Category

[ ] Documentation
[ ] Resource Addition
[ ] Codebase
[ ] User Interface
[X] Feature Request

Description
Add a Carousel in the hero section rather than a still picture !.

@SohanRC A carousel looks nice, but in the case of this website, we want to keep a single picture for now. So, we're closing this issue.


Allow toggling the transform control

Always displaying the transform control makes it easy to accidentally move the objects while rotating the camera. Other CAD software do not activate the transform control by default. I think we should follow this behavior.

Makes Sense! Will be happy to do it :)

I updated the issue about the viewport config. The transform control merits its own toggle button on the toolbar

Toggle object highlight mode: using a bounding box or using highlight color
Highlight or bounding box color selector
Viewport background color, it would be great if we could support gradient color

I'm thinking these should be using normal JupyterLab settings.
There may be a way to add a shortcut in the toolbar to go to the settings page for JupyterCAD?

Toggle axes helper

I feel like this one should instead be stored on a per-file basis in StateDB like the other view states listed in https://github.com/jupytercad/JupyterCAD/issues/601

I'm thinking these should be using normal JupyterLab settings.

There may be a way to add a shortcut in the toolbar to go to the settings page for JupyterCAD?

If we put it into JupyterLab settings, having the shortcut is a must. I know lots of users do not know about the setting page of JupyterLab.

I feel like this one should instead be stored on a per-file basis in StateDB like the other view states listed in https://github.com/jupytercad/JupyterCAD/issues/601

just checked FreeCAD, it's at application level. I would follow them for these configs


ORT: Removing unnecessary errors

The diff was not showing the source/destination due to improper handling of notes.
Fix will...

Prevent wrong ERROR printing on the screen due to traffic ops comments
Prevent having to run traffic_ctl config reload when not needed

ERROR Lines for /opt/trafficserver/etc/trafficserver/volume.config from Traffic Ops do not match file on disk.
ERROR Config file volume.config line only in TrOps:	# TRAFFIC OPS NOTE: This is running with forced volumes - the size is irrelevant
ERROR Lines for /opt/trafficserver/etc/trafficserver//hosting.config from Traffic Ops do not match file on disk.
ERROR Config file hosting.config line only in TrOps:	# TRAFFIC OPS NOTE: volume 1 is the Disk volume
ERROR Config file hosting.config line only in TrOps:	# TRAFFIC OPS NOTE: volume 2 is the RAM volume

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/incubator-trafficcontrol-PR/1561/
Test PASSed.


[Feature Request]: Add funny quotes

Is there an existing issue for this?

[X] I have searched the existing issues

Feature Description
I would like to add a new section : fun within which I would like to add some quotes
Kindly assign me this task @JayShukla8
Record

[X] I agree to follow this project's Code of Conduct
[X] I want to work on this issue
[X] I'm willing to provide further clarification or assistance if needed.

Hey! Thanks but we don't require any new section on our webpage. You can add your favorite quote if you want.

Sure @JayShukla8 I will then a new quote under tags of  fun and humor. Then can you please assign me this task under hacktoberfest.

Can you please add the hacktoberfest label @JayShukla8

Can you please assign it to me.


Can you pleae make it Linux compatible?

Hello,
I really like your solution however I can't run it on Linux.  Is it possible to make it cross platform?

Fetching https://github.com/gereons/AoCTools from cache
Fetching https://github.com/lukepistrol/SwiftLintPlugin from cache
Fetched https://github.com/lukepistrol/SwiftLintPlugin from cache (0.43s)
Fetched https://github.com/gereons/AoCTools from cache (0.43s)
Computing version for https://github.com/gereons/AoCTools
Computed https://github.com/gereons/AoCTools at 0.0.45 (0.02s)
Computing version for https://github.com/lukepistrol/SwiftLintPlugin
Computed https://github.com/lukepistrol/SwiftLintPlugin at 0.54.0 (0.02s)
Creating working copy for https://github.com/lukepistrol/SwiftLintPlugin
Working copy of https://github.com/lukepistrol/SwiftLintPlugin resolved at 0.54.0
Creating working copy for https://github.com/gereons/AoCTools
Working copy of https://github.com/gereons/AoCTools resolved at 0.0.45
Downloading binary artifact https://github.com/realm/SwiftLint/releases/download/0.54.0/SwiftLintBinary-macos.artifactbundle.zip
error: failed downloading 'https://github.com/realm/SwiftLint/releases/download/0.54.0/SwiftLintBinary-macos.artifactbundle.zip' which is required by binary target 'SwiftLintBinary': downloadError("Error Domain=NSURLErrorDomain Code=-1001 \"(null)\"")
error: fatalError

I'm sorry, I don't have access to a Linux machine with Swift installed, so I wouldn't be able to run/test in such an environment. Also, as far as I can tell, neither SwiftLint nor SwiftLintPlugin are currently available for Linux, and that's where your error comes from, and I would rather like to keep using SwiftLint.
I would, however, welcome PRs that add Linux capabilities while preserving the current API and usage of SwiftLint when building on macOS.

Okay. Thanks
On Wed, 31 Jul 2024, 21:43 Gereon Steffens, @.***>
wrote:

I'm sorry, I don't have access to a Linux machine with Swift installed, so
I wouldn't be able to run/test in such an environment. Also, as far as I
can tell, neither SwiftLint nor SwiftLintPlugin are currently available for
Linux, and that's where your error comes from, and I would rather like to
keep using SwiftLint.
I would, however, welcome PRs that add Linux capabilities while preserving
the current API and usage of SwiftLint when building on macOS.
—
Reply to this email directly, view it on GitHub
https://github.com/gereons/AoCTools/issues/1#issuecomment-2261306260,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AEH2G2XU7BZKAUYSQZTG7ILZPE44PAVCNFSM6AAAAABLUB37LGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRRGMYDMMRWGA
.
You are receiving this because you authored the thread.Message ID:
@.***>


proposition to help Coexya in development

We suggest they do quick demo for a next sprint review. Organize this with them.


Vendor CDI 1.7.0

Signed-off-by: Alexander Wels<EMAIL_ADDRESS>
What this PR does / why we need it:
Update CDI version to 1.7.0
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
Special notes for your reviewer:
Release note:

NONE

@awels could you run make generate?

@rmohr Done, did you know that make deps-update no longer works, it keeps failing on some package deep in the bowels of something vendored in.

@awels, no did not know. That can sadly always happen with deps-update. In your case, changine glide.yaml and running deps-install should be sufficient.

Yeah, that is what I eventually did, I thought make deps-update also runs make generate and I simply forgot that step when I did make deps-install.

ci test please

Hm, travis still thinks that make generate && make should be run.

That is weird, I just re-ran make generate and make, and there we no changes at all. I rebased, lets see if it passes now.

ci test please

That is weird, I just re-ran make generate and make, and there we no changes at all. I rebased, lets see if it passes now.

That can be. Thanks @cynepco3hahue  for pushing. A rebase should have solved it. We updated goimports and changed how goimports is invoked a few days ago.

Failures under the CI unrelated, lgtm.


Add ParquetMetadata::memory_size size estimation

~Draft as I need to implement memory size calculations for the schema structs as well~
Which issue does this PR close?
Closes https://github.com/apache/arrow-rs/issues/1729
Rationale for this change
For systems that want to do low latency queries on parquet files stored in
object store, it is important to somehow provide the ParquetMetadata to the reader to
avoid the overhead of fetching the file footer and re-parsing the metadata.
For example, when using the ArrowReaderMetadata API:
https://github.com/apache/arrow-rs/blob/ed018a34d996590544fe5e833cb601bf46e9758e/parquet/src/arrow/arrow_reader/mod.rs#L293-L315
One way to provide ParquetMetadata to the Arrow reader is to cache it in memory and for large numbers
of parquet files this can consume non trivial memory. Thus accurately understanding the memory requirements
is important
What changes are included in this PR?

Add ParquetMetadata::memory_size(), named similarly to arrow::Array::get_array_memory_size
Add a non public trait to help calculate memory usage
Tests

Are there any user-facing changes?
There is a new function in the API

I plan to merge this in later today as well unless there are any additional comments or people would like additional time to review


Update environment variables for REMS-644

Describe your changes
Please include a summary of the changes and the related issue/task. Please also include relevant motivation and context. List any dependencies that are required for this change, including links to other pull requests/branches in other repositories if applicable.
Issue ticket number and Jira link
Please include the Jira Ticket Number and Link for this issue/task.
https://jira.mitre.org/browse/REMS-644
Checklist before requesting a review

[ ] I have performed a self-review of my code
[x] Ensure the target / base branch for any feature PR is set to dev not main (the only exception to this is releases from dev and hotfix branches)

Checklist for conducting a review

[ ] Review the code changes and make sure they all make sense and are necessary.
[ ] Pull the PR branch locally and test by running through workflow and making sure everything works as it is supposed to.

Workflow
Owner of the Pull Request will be responsible for merge after all requirements are met, including approval from at least one reviewer. Additional changes made after a review will dismiss any approvals and require re-review of the additional updates. Auto merging can be enabled below if additional changes are likely not to be needed. The bot will auto assign reviewers to your Pull Request for you.

FYI this PR does not touch Playwright tests


Don't send "terms" to transifex for unsearchable presets

I don't think unsearchable terms are ever used by iD so there's no point in asking people to translate them. I'm posting this issue first in case someone is actually using them.

I think we stopped doing this a long time ago and there was an issue for it. Will look when I’m at my desk later.

Oh you're right. What I'm seeing is the terms appearing in dist/en.json.
https://github.com/openstreetmap/iD/blob/7c846baa317e845bbd1dafd5e7cdd0140569e37a/dist/locales/en.json#L8802-L8805


Reorganize Package.from_string.

#962  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @rohinb2 and the rest of your teammates on  Graphite

Merge activity

Jul 4, 10:25 AM EDT: @rohinb2 started a stack merge that includes this pull request via Graphite.


[TECHDEBT][Shared] Address legacy Run golangci-lint errors

Objective
Improve unit tests by addressing legacy linting errors
Origin Document
Raised by review of https://github.com/pokt-network/pocket/pull/235#issuecomment-1270639788

Goals

Block PRs that break unit tests
Block PRs that break linting errors
Display code changes that do not follow style guidelines

Deliverables

[ ] A PR that tends to all of the legacy linting errors in the repo
[ ] A PR that updates the linting github workflow to block (rather than simply display a warning) on golang and repo linting errors

Non-goals / Non-deliverables

Changing any business logic in the codebase

General issue deliverables

[ ] Update the appropriate CHANGELOG
[ ] Update any relevant READMEs (local and/or global)
[ ] Update any relevant global documentation & references
[ ] If applicable, update the source code tree explanation
[ ] If applicable, add or update a state, sequence or flowchart diagram using mermaid

Creator: @jessicadaugherty
Co-Owners: @Olshansk

@jessicadaugherty Can we include this in iteration 4? It's a lower priority, but it blocks us from enabling the autolinters

@jessicadaugherty I'll do this (I don't know when iteration 4 is but can have it done this week). I find these errors annoying anyway so it'll be nice to get rid of em!

Amazing, thanks @h5law! I assigned this out to you on DeWork. LOL iteration 4 was in like October or something so this was originally a priority backlog item that kept getting refactored. Agreed it would improve contributor QoL! Appreciate it.

Just to confirm - fix the errors from the golanglint-ci test throughout the codebase and make the workflow blocking on failure?

Exactly! I'd take a look at the linked PR in "Origin Document" for more context on where we stumbled on this need. And, of course, lmk if you have any questions!

Hey @h5law, thank you for looking into this!
Here is a related thread: https://github.com/golangci/golangci-lint-action/issues/23
Basically, from what I understand, the packages are getting downloaded by golangci-lint-action are already present because of golang cache. Or due to the fact we need to generate mocks and all that stuff that requires us to download the packages prior to executing golangci-lint-action. Lmk if you have any questions — I've looked into that briefly a couple of months ago but never committed a change.

Just wanted to provide some context to what @okdas mentioned.
@h5law is working on enabling the golang CI linter by tending to ALL OF the issues it raises throughout the codebase.
One of the many such issues is the fact that there is a ❌ when the test_results.json is being uploaded. The following screenshot is taken from this link

The important to note is that this is a false positive. The file gets uploaded and there is no real issue. However, seeing a red X can cause confusion and lead to developers spending time looking in a place where there is no problem.
The information Dima provided is a potential starting point to understand how/if we could resolve this as well.


Use lbry.tv stream URLs when launching the app from a notification upon a cold start

In cases where a user receives a notification when the sdk service isn't running in the background, the app takes too long to start up waiting for the sdk. To speed up the app load time for a better experience, we can use the lbry.tv stream url on the media player, and then the sdk can be used when it's ready.

Please make sure that when this is released we will be able to clearly compare behavior of people who entered via these notifications vs. those who entered via the previous mechanism.


Raise ValueError when loss function is incompatible with activation function of output layer.

This is an alternative fix to #4744. I prefer this one.

Many of the failing tests are actually additional evidence of the sneakiness of bug #4744. Let's take a closer look at the test_masking_layer test.
import numpy as np
from numpy.testing import assert_allclose

from keras.layers import Dense, recurrent, TimeDistributed
from keras.models import Sequential
from keras.layers.core import Masking

from keras import backend as K

nb_samples, timesteps, embedding_dim, output_dim = 2, 5, 4, 3
embedding_num = 12

# @keras_test
# def test_masking_layer():
''' This test based on a previously failing issue here:
https://github.com/fchollet/keras/issues/1567

'''
I = np.random.random((6, 3, 4))
V = np.abs(np.random.random((6, 3, 5)))
V /= V.sum(axis=-1, keepdims=True)

model = Sequential()
model.add(Masking(input_shape=(3, 4)))
model.add(recurrent.LSTM(output_dim=5, return_sequences=True, unroll=False))
# model.add(TimeDistributed(Dense(output_dim = 1, activation='softmax'))) # This line should be here.
model.compile(loss='categorical_crossentropy', optimizer='adam')
model.predict(I)

Notice how the original code uses a categorical_crossentropy loss even though the output from a vanilla LSTM ranges from -1 to 1 because of the gated hyperbolic tangent activation function. Keras currently clips the output values before calculating the loss, so it's not clear what the model would be learning here.
For the test_sequential_count_params test, the code uses a binary_crossentropy loss even though the model is framed as a softmax over two output units. I think it would be a better practice to use the categorical_crossentropy in these situations.
import numpy as np

from keras.layers.core import Activation, Dense
from keras.models import Sequential

from keras import backend as K

# @keras_test
# def test_sequential_count_params():
input_dim = 20
nb_units = 10
nb_classes = 2

n = input_dim * nb_units + nb_units
n += nb_units * nb_units + nb_units
n += nb_units * nb_classes + nb_classes

model = Sequential()
model.add(Dense(nb_units, input_shape=(input_dim,)))
model.add(Dense(nb_units))
model.add(Dense(nb_classes))
model.add(Activation('softmax'))
model.build()

assert(n == model.count_params())

model.compile('sgd', 'binary_crossentropy')
# model.compile('sgd', 'categorical_crossentropy') # Should be this.
assert(n == model.count_params())

This is far too restrictive. In particular binary_crossentropy should be usable with nearly any activation function (the only constraint being that the output should be between 0 and 1). E.g. it makes sense to use binary_crossentropy together with softmax (albeit it is unusual and generally suboptimal, depending on what you are trying to do).

I have mixed feelings about this. On the one hand I see the value of preventing incompatible output nonlinearity / loss function combos, on the other hand there are an endless number of naive mistakes one can make and it doesn't seem reasonable to expect Keras to catch them all.

@the-moliver - I'm less interested in preventing "naive" mistakes and more interested in preventing unexpected behavior, e.g., a model continuing to "train" even though the pairing between the activation function on the output layer and the loss function doesn't make sense (e.g., a regression that's producing values far outside of zero and one being paired with a binary cross entropy loss). I think the "naive" mistake here would be to not clip your outputs when training with a, e.g., binary_crossentropy loss. With that in mind, I've implemented a Clip layer to give the user control over the clipping process.

I disagree with almost all of that. As @fchollet mentioned, by definition only values between 0 and 1 should be fed to binary_crossentropy loss. The best way to accomplish this is with a differentiable non-linearity such as a sigmoid, that maps all incoming values to 0->1. (Also, the binary_crossentropy loss-sigmoid nonlinearity is the cannonical pairing as in logistic regression, and should be the clear choice, unless one has a really good reason to do otherwise.) Clipping should only really be used for numerical stability, not to ensure the input to binary_crossentropy  is between 0 and 1. One could do that, but your gradients are likely to get messed up, since anything in the clipped region will have 0 gradient. A sigmoid will give nice gradients over a much larger range (although not infinite range, hence the clipping).
For those interested, a nice post on related issues is here<EMAIL_ADDRESS>

@the-moliver: I should've clarified that the "naive" mistake would be to train a model that is using a sigmoid activation and a binary cross entropy loss and then not clipping the values to ensure numerical stability. I've edited the original comment to reflect that. I agree with your objections to using clipping to ensure the inputs to the binary_crossentropy loss lie between 0 and 1, but that's exactly what Keras is currently doing and it's the whole reason I opened issue #4744 in the first place.

Yes, and my point is that if you're doing things that make sense (like properly pairing loss functions and output nonlinearities), then there is no issue with the status quo. There's only aberrant behavior if you do things that don't make sense, like not mapping to 0-1 before using 'binary-crossentropy`, hence my original comment.

Agreed, the current setup is fine. As @the-moliver points out, the current clipping happening with crossentropy exists for numeric stability: without it, a perfectly valid architecture would be very likely to end up with a NaN loss.


Feat(line): event on whole line

Brief Information
This pull request is in the type of:

[ ] bug fixing
[x] new feature
[ ] others

What does this PR do?
series should response to event. Add extra check for seriesIndex
Fixed issues
Close #13637
Close #13843
Details
Before: What was the problem?

After: How is it fixed in this PR?

Misc

[ ] The API has been changed (apache/echarts-doc#xxx).
[ ] This PR depends on ZRender changes (ecomfe/zrender#xxx).

Related test cases or examples to use the new APIs
NA.
Others
Merging options

[x] Please squash the commits into a single one when merge.

Other information

This is an underlying change that will affect all series. Could you please do a double-check what's the behavior of the other series after this change @susiwen8 @Ovilia? For example: are there other series that will also have elements only having seriesIndex and will trigger events.


Please add the rest of the chapters in the tutorial

Please add the rest of the chapters in the tutorial
https://ice1000.org/llvm-cs/en/

I did some work here supporting up to Chapter 7. I followed a different approach to parsing but LLVM part should be decent.


explore getting data from new federal site

http://open.canada.ca/data/en/dataset/5133b19b-6a92-44b2-8b12-a49731b56421

got - I like it


⚠ SpeakUpConference.com has degraded performance

In 237f0aa, SpeakUpConference.com (https://www.speakupconference.com) experienced degraded performance:

HTTP code: 200
Response time: 8182 ms

Resolved: SpeakUpConference.com performance has improved in 594c84c after 24 minutes.


Global enabling of direct access grant

Done for AGPUSH-1149
W/ KC 1.3.0 this will be the default (see http://lists.jboss.org/pipermail/keycloak-dev/2015-June/004587.html), and it does not hurt doing this now. Actually adds more options for full integration of UPS into other systems

@abstractj @stianst @sebastienblanc any comments ?

@smiklosovic @kpiwko FYI

Finally, it took just 1 year or so ;-)

+1
On Wed, Jun 3, 2015 at 3:40 PM, Karel Piwko<EMAIL_ADDRESS>wrote:

Finally, it took just 1 year or so ;-)
—
Reply to this email directly or view it on GitHub
https://github.com/aerogear/aerogear-unifiedpush-server/pull/585#issuecomment-108418761
.

@TadeasKriz FYI, I know that you would appreciate this as well ;-)

@kpiwko a lot of mindset changes L(

Nice! :)

Going -- https://github.com/keycloak/keycloak/pull/1325

@kpiwko I knew you were waiting, that's what make everything better :)


Trying to disable Auto Start - command line documented NOT working

Teams_Windows_x64.msi... using the following command line via MEMCM - deployed as Application:
msiexec /i Teams_windows_x64.msi /QN OPTIONS=”noAutoStart=true” ALLUSERS=1
Its autostarting on our test machines. Since the commandline is not working, do we need to delete the REgistry key instead? and why cant it work as is documented?!?!?
this key
Key: HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Run
Value: com.squirrel.Teams.Teams
Data: %LocalAppData%\Microsoft\Teams\Update.exe --processStart "Teams.exe" --process-start-args "--system-initiated"

turns out, I cannot read... the MSI must be run as system(via memcm in our case) in order to have the noAutoStart=true actually work... This is noted in the documentation - even a blue colored box - yet i missed it..


Update content of tools-ui-components.md

Hello,
I would like to add the library react-toastify to the free UI component section.

Thank you @bvaughn 😄


Neo4j 5

Seems like we've been doing this for long enough for there to be a new major release of Neo4j!
Need to work out what's changed and how easy the upgrade path will be, lest we end up in the situation with OLS3 where our database is 7 years out of date and can't be easily upgraded.

Thanks @giraygi , I wasn't aware and that's super interesting.
For future reference (mainly for @henrietteharmse) I think it would be a fairly major change in OLS4 to go back to incremental loads, as although it would be trivial to hash owl files and consider the hashes when loading ontologies, we would also need to consider the linker which would potentially have stale outputs (as new links would not be discovered).


bugfix: fix did-resize modifier conditional

Summary
Fixes #97.
#92 added the ability to install the did-resize modifier conditionally. This PR fixes an upstream error caused by this change when _shouldUseNativeTextOverflowCSS is true by adding an additional conditional to the didResize() getter.
Miscellaneous

Add test case for Native Text Overflow and did resize
Update htmlSafe import in test file to @ember/template
Fix test-conatiner typos in test file.
Add @babel/plugin-proposal-private-property-in-object and @ember/string as devDeps to fix issues when integrating with applications and a couple ember-try scenarios.

Testing Done

Repo tests pass
Application tests pass when integrated via yarn link

@lstrrs Would appreciate it if you could review and get this published as a new release. Thanks!

@lstrrs Bumping this. Would appreciate a review and a new release after this lands. Thanks!


Couldn't initiliaze test case and test data

Why it is looking in my user specific folder even though I have given proper target folder where I have copied my binaries.
Assert.Inconclusive failed. Couldn't initiliaze test case and test data. Error message: 'C:\Users\vsrivastava16\AppData\Local\Temp\UI\UIControls.xlsx' could not be found. Check the spelling of the file name, and verify that the file location is correct. If you are trying to open the file from your list of most recently used files, make sure that the file has not been renamed, moved, or deleted.

@vicky6nov
Which task are you using to run the tests? Does it work if you directly call into vstest.console.exe with the binaries?

We have made it work while changing our app.config file and provincial no
input file path as as absolute path.
Still I want to use relative path instead of absolute. How can it work with
relative path?
On Thursday, July 7, 2016, Atin Bansal<EMAIL_ADDRESS>wrote:

@vicky6nov https://github.com/vicky6nov
Which task are you using to run the tests? Does it work if you directly
call into vstest.console.exe with the binaries?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/Microsoft/vsts-tasks/issues/2035#issuecomment-231149440,
or mute the thread
https://github.com/notifications/unsubscribe/ATYpiomrtru4bKzDwGDU4X4z9e3pw3LQks5qTTdZgaJpZM4JGeRh
.

This seems to be the same issue as #1638. We are considering our default working directory to be %temp% instead of the folder specified in the task. Opened a bug to track on our side.

I am still facing the same issue. Can you please let me know if any workaround?


Houdini: Fix single frame publishes

Changelog Description
The instance.data["frames"] in the CollectFrames plug-in stores a str if it's a single frame but a list if it's multiple files, similar to how the Integrator currently likes it.
However, there was some scattered logic that seemed to assume that instance.data["frames"] was always a List[str] which isn't expected anymore now.
Additional info
Fixes #672
I feel like it'd still be worth it to start moving into logic where there's a single ExtractROP plug-in that works the same for all ROP nodes because the shared logic across them is 99% the same, and where it currently isn't it should become the same. So I may follow up with another PR after this to start working to a single ExtractROP plug-in.
Testing notes:

All Houdini publishes types should allow to publish single frame and multi-frame publishes without issue.

Closing this in favor of #676


Little confusing about setting specific default node version

In numerous blog posts regarding the configuration of the default nodejs version using nvm, the following command is often cited:
nvm alias default node

While this command might work in some cases (as node will point to the latest version installed on your system), it might not behave as intended if you want to default to a specific version of Node.js.
This command is actually referring to the Set default node version section in the nvm's README. It's crucial to note that the instructions suggest setting a specific node version of your choosing, rather than using the generic node.
So if you're aiming to set a particular version (for example, 14.15.1) as your default, the correct command would be:
nvm alias default 14.15.1

This ensures that the specified version of Node.js will be used by default. If you simply use node as in the first command, it could point to any installed version, which might not be what you intend.

This PR's base branch outdated, so closing

that's not a reason to close a PR; that's a reason to rebase it.

// OPEN SOURCES ™ ******
``` parga 08.6.1
 contract public string (name)) 9-00)

.
###


expr/*: do some rename to make it in correspondence with tidb

What have you changed? (mandatory)
This PR does some rename to make the expression module  in correspondence with tidb
What are the type of the changes? (mandatory)

Improvement (non-breaking change which is an improvement to an existing feature)

How has this PR been tested? (mandatory)
unit test
Does this PR affect documentation (docs/docs-cn) update? (mandatory)
no
Does this PR affect tidb-ansible update? (mandatory)
no

LGTM

ping @hicqu


Hey

Where's the website? I need the MusicBot for The Dingus Bungalow. :)

The.... website?

If jagrosh doesn't have a public bot for this. Send me a DM: Matthww#5032


adding action 'encrypt' to log-group

Regarding #2242

adding ability to encrypt cloudwatch log-group

Appears this is never going to get implemented...  :(

@joshuaroot  Is this something you are going to finish?  I have been asked for this functionality about 15 times or more from my security team and coworkers.  If you aren't going to be able to finish it then is this something I can try and pick up?  Might be way over my head but it's really needed

@kapilt Is this going to be a lot of work to finish this PR up?  If Joshua is not able to complete it, is this something I could try?


Guard Vim keybindings with if executable('fzy')

This prevents errors if you sync your dotfiles to a computer that doesn't yet have fzy.

Is there any harm in defining the keybinding? It won't error unless the user is trying to run it. I'd rather get "error fzy not found" than wonder why my keybinding isn't working on the new machine.

Thanks for taking a look at this, @jhawthorn !
I’m inclined to think an explicit error is a good idea now, too. I think the condition that inspired me to send this PR (sharing a .vimrc between machines where it’s easy to install fzy, and machines where it isn’t) is likely less common than being on a new machine where you can install fzy and forgot.


Email received during Sign Up (email valid...) is having 'Password change' instructions

Currently the email validation send while new user sign up, is having the content related to password change request.

The expected content for email validation should be like:
Please click the button to validate your account.
 button.

I think this is fixed by changing Auth0 template but there is some other unexpected staging issue I need to look into.

@joshmorel seems to be working fine on my end, are able to reproduce the issue @rabbi08  or it only happened once?

@rabbi08 - Should be fixed please confirm and then close this issue.

Validate with new user account, It is working fine as shown in below screenshot.


chore: enable the setOption, longLine, missingEnd linters globally

With Lean 4.11.0-rc1, downstream projects can disable these linters by setting (e.g.) linter.setOption false in their lakefile.lean.

Not tested on downstream projects yet: help doing so is welcome.

[x] depends on: #15616

I added the dependency that was suggested in the other PR. Will revisit once the other one is reviewed!

bors d+

Thanks!
bors merge


CDK Params not written to param store

In our self-hosting docs, we note a few params that are setup by CDK deploy:

signingKeyId
botLambdaRoleArn
database
redis

However, when validating against some of our demo deployments, these are not currently being set by our CDK code.

signingKeyId is currently broken.  Good bug.  It looks like it was originally set, but then there was a regression when we added support for regions other than us-east-1.
One major CDK limitation is that all "resources" must be within the same AWS region.  Some resources must be in us-east- (i.e., CloudFront distributions).  The way we handle that is by creating 2 "stacks".  That creates an unfortunate situation for signingKeyId, because it is created in us-east-1, but the Parameter Store entry must be created in the primary region (which may or may not be us-east-1).
For that, I'm inclined to move the signingKeyId logic out of CDK and into the Medplum CLI aws init command, similar to how we pre-create SSL certificates.
Alternatively, the servers could look for config settings from multiple regions at startup, but that feels less clean.

botLambdaRoleArn is set explicitly: https://github.com/medplum/medplum/blob/main/packages/cdk/src/backend.ts#L422-L427

database and redis are documentation issues.  Our CDK code writes parameters for DatabaseSecrets and RedisSecrets, which are IDs for AWS Secrets Manager.  Our config loader understands that, recognizes DatabaseSecrets/RedisSecrets, and loads the appropriate values from AWS Secrets Manager.  For better or worse, CDK only supports exposing database credentials via AWS Secrets Manager, so we have to do it this way.

https://github.com/medplum/medplum/pull/2728


Remove duplicated flows from flow lists before matching process

Closes #66

remove duplicate flows
remove duplicated flows from nomatch lists
add tests

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (8a71ed5) 93.30% compared to head (af7731f) 93.35%.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #75      +/-   ##
==========================================
+ Coverage   93.30%   93.35%   +0.04%
==========================================
  Files          12       12
  Lines         553      557       +4
==========================================
+ Hits          516      520       +4
  Misses         37       37

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Allow randomization of read candidates using Redis Cluster

Bug Report

Current Behavior
I am trying to connect to Redis cluster with 5 slave nodes and 1 master. All read queries are redirected to only one slave throughout the application scope.
Input Code

Java/Kotlin/Scala/Groovy/… or Repo link if applicable:

// connection pool creation:
GenericObjectPool<StatefulRedisClusterConnection<String, String>> connectionPool;
GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
            poolConfig.setMinIdle(8);
            poolConfig.setMaxIdle(8);
            poolConfig.setMaxTotal(16);
            poolConfig.setMinEvictableIdleTimeMillis(1000*30);
            poolConfig.setSoftMinEvictableIdleTimeMillis(1000*30);
            poolConfig.setMaxWaitMillis(0);
            connectionPool = ConnectionPoolSupport.createGenericObjectPool(() -> {
                logger.info("Requesting new StatefulRedisClusterConnection "+System.currentTimeMillis());
                return redisClient.connect();
                }, poolConfig);

//
public StatefulRedisClusterConnection<String, String> getConnection() throws Exception{
   StatefulRedisClusterConnection<String, String> connection =connectionPool.borrowObject();
   connection.setReadFrom(ReadFrom.SLAVE);
    return connection;
}

Expected behavior/code
It should distribute read queries on all slaves.
Environment

Lettuce version(s): 5.1.0.M1
Redis version: 3.2.10

Possible Solution

Additional context

Lettuce selects a slave per slot hash and uses the selected slave for read operations until the next topology refresh. This is by design to reduce CPU overhead.

In high-concurrency, reading data from a slave node can improve request effectiveness if a master node in the cluster has multiple slave nodes and we can share read requests on these slave nodes.
but the lettuce now just select one of the slaves for read request, it would not balance the read requests among the slaves, one of the slave would be busy and the others are not. And the request effectiveness can not be improved a lot.

You can mitigate this behavior by enabling periodic topology refresh. Each time the topology is refreshed, Lettuce discards the slot hash connection cache and determines a different slave. This also works if there were no changes to the topology.

How does lettuce do the refresh, by "cluster slot" command? if so, it also increase the host pressure and the  "cluster slot" command spend a lot of time. I think it would be better to provide interface that we can DIY by ourselves.

That's a neat approach to apply randomization without creating objects. Currently, what's missing is the ability to configure this behavior. ReadFrom contradicts randomization to some extent as a specific node type preference is no longer given if we apply randomization.
We could introduce a boolean randomize() method to ReadFrom: This would basically signal to perform what the code does above (without the second randomize fallback).

I pushed a change to our feature branch for this ticket and the branch build that has produced artifact snapshots with version 5.2.0.gh-834-SNAPSHOT. Please upgrade to this version and run your tests to evaluate how the change affects your throughput/Redis utilization.
Make sure to include our snapshot repository:
<dependencies>
  <dependency>
    <groupId>io.lettuce</groupId>
    <artifactId>lettuce-core</artifactId>
    <version>5.2.0.gh-834-SNAPSHOT</version>
  </dependency>
</dependencies>

<repositories>
  <repository>
    <id>sonatype-snapshots</id>
    <url>https://oss.sonatype.org/content/repositories/snapshots</url>
    <snapshots>
      <enabled>true</enabled>
    </snapshots>
  </repository>
</repositories>

thanks for your feature branch
you add the random branch to get the read node when the connection is not cached.
how about the conenctions are cached ?
as I tested, the node just change when a topology refresh happens, but between two refreshes, the read requests would always go to the same node.
I think we can add the same random branch for the cached connections

if we go to the random branch, the connections would not be cached, does it affect performance?

if we go to the random branch, the connections would not be cached, does it affect performance?

Connections are held in AsyncConnectionProvider. The thing that is not cached is the candidate list and I think there is a penalty associated with this but that is the price to pay as we no longer can apply certain optimizations.

@mp911de As I see, the support for the read distribution from replicas will be added in the 5.2.x. When this going to be released?

Probably September, see https://github.com/lettuce-io/lettuce-core/milestone/51.

@mp911de Thanks for the quick reply. We are really waiting for this to be released.

@mp911de @smoomrik AWS has recently introduced reader endpoints for Redis which manages distribution of reads among the slaves. You can check more details here: https://aws.amazon.com/about-aws/whats-new/2019/06/amazon-elasticache-launches-reader-endpoint-for-redis/
Might want to check this out. Using this should help avoid having to provide individual replica endpoints in the application giving more flexibility.
Not sure about exact underlying implementation of this on AWS side. Waiting for more clarification. Initial tests which we have done on test environment (1 master, 2 replicas) showed roughly equal distribution of load among the 2 replicas

hey @adityak93 , we have been using reader endpoint, but its not distributing the traffic equally among read replicas for us.sometimes, even when other read replicase are under heavy load, few read replicase have barely any requests coming their way. can you tell us if this is consistent with your observations , or if you used any additional configuration apart from read preferred and periodic topology refresh.


Implemented missing QueueEntry event

QueueStatus action also returns QueueEntry event.
See:
http://www.asteriskdocs.org/en/2nd_Edition/asterisk-book-html-chunk/asterisk-APP-F-32.html

Hey thanks! :) Could you throw in a unit test too?

I wasn't able to run the tests. I hope the test is OK.

So we can't merge unless the build passes. Make sure that running ./vendor/bin/phing build yields no errors in your box. Travis will run the same command (more information is available in the README).


Formatting Issue with PR

Hi there! :) I've been trying to add a resource to the  HTML & CSS section and the GitHub action keeps saying that formatting is wrong even though I followed the rules. I'm at a lost. Can you help me?
My PR

I have left a comment on the PR with the correct syntax; however we do not accept email address required cheatsheets.


[Feature] support max_by built-in function (#3288)

[Feature] support max_by built-in function (#3288)

[FE PR Coverage Check]
:disappointed: fail : 26 / 35 (74.29%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/catalog/Function.java
0
2
00.00%
[184, 185]

:large_blue_circle:
com/starrocks/sql/analyzer/FunctionAnalyzer.java
3
10
30.00%
[202, 203, 204, 207, 209, 210, 211]

:large_blue_circle:
com/starrocks/sql/optimizer/rule/transformation/MergeTwoAggRule.java
2
2
100.00%
[]

:large_blue_circle:
com/starrocks/catalog/FunctionSet.java
17
17
100.00%
[]

:large_blue_circle:
com/starrocks/sql/analyzer/ExpressionAnalyzer.java
4
4
100.00%
[]

[FE PR Coverage Check]
:disappointed: fail : 26 / 37 (70.27%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/catalog/Function.java
0
2
00.00%
[184, 185]

:large_blue_circle:
com/starrocks/sql/analyzer/FunctionAnalyzer.java
3
12
25.00%
[202, 203, 204, 207, 208, 211, 213, 214, 215]

:large_blue_circle:
com/starrocks/sql/optimizer/rule/transformation/MergeTwoAggRule.java
2
2
100.00%
[]

:large_blue_circle:
com/starrocks/catalog/FunctionSet.java
17
17
100.00%
[]

:large_blue_circle:
com/starrocks/sql/analyzer/ExpressionAnalyzer.java
4
4
100.00%
[]

[FE PR Coverage Check]
:disappointed: fail : 24 / 33 (72.73%)
file detail

path
covered_line
new_line
coverage
not_covered_line_detail

:large_blue_circle:
com/starrocks/sql/analyzer/DecimalV3FunctionAnalyzer.java
1
6
16.67%
[77, 78, 79, 80, 82]

:large_blue_circle:
com/starrocks/sql/analyzer/FunctionAnalyzer.java
9
12
75.00%
[203, 204, 215]

:large_blue_circle:
com/starrocks/sql/analyzer/ExpressionAnalyzer.java
9
10
90.00%
[787]

:large_blue_circle:
com/starrocks/catalog/FunctionSet.java
5
5
100.00%
[]

run starrocks_admit_test


Update Astro+Keystatic: document to markdoc field

Description (required)
This PR updates the Astro + Keystatic integration documentation to reflect a recent change in the Keystatic API. The fields.document field type has been deprecated and replaced with fields.markdoc.
Specifically, this PR modifies the following code snippet:
From:
content: fields.document({
  label: 'Content',
  formatting: true,
  dividers: true,
  links: true,
  images: true,
}),

to:
content: fields.markdoc({
  label: 'Content',
}),

Related issues & labels (optional)

Closes #
Suggested label:

For Astro version: 4.11.0. See astro PR #. --> https://github.com/amirzezo201/docs.git
First-time contributor to Astro Docs?
Yes

amir_zezo2001  ( discord Username )

Hello! Thank you for opening your first PR to Astro’s Docs! 🎉
Here’s what will happen next:

Our GitHub bots will run to check your changes.
If they spot any broken links you will see some error messages on this PR.
Don’t hesitate to ask any questions if you’re not sure what these mean!

In a few minutes, you’ll be able to see a preview of your changes on Vercel 🥳

One or more of our maintainers will take a look and may ask you to make changes.
We try to be responsive, but don’t worry if this takes a few days.

Lunaria Status Overview
🌕 This pull request will trigger status changes.

Learn more
By default, every PR changing files present in the Lunaria configuration's files property will be considered and trigger status changes accordingly.
You can change this by adding one of the keywords present in the ignoreKeywords property in your Lunaria configuration file in the PR's title (ignoring all files) or by including a tracker directive in the merged commit's description.

Tracked Files

Locale
File
Note

en
guides/cms/keystatic.mdx
Source changed, localizations will be marked as outdated.

Warnings reference

Icon
Description

🔄
The source for this localization has been updated since the creation of this pull request, make sure all changes in the source have been applied.


ImportError: cannot import name safe_indexing

Hey,
I'm currently trying to use sklearn_theano through dl-docker. I've installed it with a regular setup:
GeForce GTX 660 Ti
Ubuntu 16.04
Nvidia drivers 367.57
CUDA 7.5
and using the nvidia-docker to initialize a session. Unfortunately, the following error occurs when trying to run the OverFeat/GoogleNet example:
In [1 ]: from sklearn_theano.feature_extraction import OverfeatClassifier
Using gpu device 0: GeForce GTX 660 Ti (CNMeM is enabled with initial size: 85.0% of memory, cuDNN 4008)
/usr/local/lib/python2.7/dist-packages/theano/tensor/signal/downsample.py:6: UserWarning: downsample module has been moved to the theano.tensor.signal.pool module.
  "downsample module has been moved to the theano.tensor.signal.pool module.")

ImportError                               Traceback (most recent call last)
<ipython-input-1-96d87d778fe1> in <module>()
----> 1 from sklearn_theano.feature_extraction import OverfeatClassifier

/root/sharedfolder/sklearn-theano/sklearn_theano/feature_extraction/__init__.py in <module>()
----> 1 from .overfeat import OverfeatTransformer
      2 from .overfeat import OverfeatClassifier
      3 from .overfeat import OverfeatLocalizer
      4 from .overfeat import fetch_overfeat_weights_and_biases
      5 from .overfeat_class_labels import get_all_overfeat_labels

/root/sharedfolder/sklearn-theano/sklearn_theano/feature_extraction/overfeat.py in <module>()
     16 from ..base import (Convolution, MaxPool, PassThrough,
     17                     Standardize, ZeroPad, Relu, fuse)
---> 18 from ..utils import check_tensor
     19
     20

/root/sharedfolder/sklearn-theano/sklearn_theano/utils/__init__.py in <module>()
      1 """This module holds various development utilities."""
      2 from .validation import check_tensor, get_minibatch_indices
----> 3 from .ports import train_test_split
      4
      5 __all__ = ['check_tensor',

/root/sharedfolder/sklearn-theano/sklearn_theano/utils/ports.py in <module>()
      2 from sklearn.cross_validation import ShuffleSplit
      3 from itertools import chain
----> 4 from sklearn.utils import safe_indexing
      5 import numpy as np
      6 import scipy.sparse as sp

ImportError: cannot import name safe_indexing

I couldn't find anything exactly similar anywhere, so I thought I'd drop it here.
Any help would be appreciated, thanks.

What is the sklearn version used? It seems that sklearn.utils.safe_indexing exists at least since version 0.15


Add _get_comp_words_by_ref

This patch fixed completion on my MacOS machine. Previously it would just list the available commands, but not actually complete them based on the current fragment. Since I'm the sort that hammers tab, this is what I was looking for. I haven't done that many bash completion functions so I'm not certain about how cross-platform this solution is.
Sorry about the whitespace changes, my editor does that by default.

@JonathanUsername Thanks for the PR.
Very strange. What version of bash-completion do you have installed? What version of bash are you running?
I actually just removed the call to _get_comp_words_by_ref (11516545d23f8f54006590286949fc9b92e8b892) because it's called in the _init_completion function.

I installed bash-completion via homebrew. I just updated it and I still don't see the _init_completion function anywhere, neither available from the command line, nor in the main source of the bash-completion script.
$ brew upgrade bash-completion
$ . $(brew --prefix)/etc/bash_completion

I should have looking in the closed issues, is this what you're referring to? https://github.com/dsifford/yarn-completion/issues/1
I was going on this similar issue with another CLI (a quick google dredged it up for me) https://github.com/kubernetes/kubernetes/issues/12400#issuecomment-137567835

@JonathanUsername Aha! Thanks for clarifying. That's helpful. I assume you're using the default mac OS bash version then (which I believe is < v4.x)?
So now it's my understanding that _init_completion came after _get_comp_words_by_ref. Good to know.
Would you mind moving your call to _get_comp_words_by_ref to between these two lines just so we can keep those function calls near each other?
After that, I'd be happy to merge. 😄

Is that ok @dsifford ?

Looks great to me. Thanks!


Vardict filtering fails on bcftools AF subfield issue

bcbio-1.1.5, running Vardict in tumor-only config.
Failing with error:
The FORMAT tag AF can have multiple subfields, run as AF[sample:subfield]

Fixed in variation/vardict.py line 220 (_lowfreq_linear_filter definition):
           """-e  'FORMAT/AF[{tumor_index}:0] < 0.02 && FORMAT/VD[{tumor_index}] < 30 """

I'm not sure if my fix is the correct behaviour, but I'm getting past the error now.

Heya, Brad fixed that issue here:
https://github.com/bcbio/bcbio-nextgen/commit/1a5343c094b7a5cd79b7b4eebd527b92392831c6
if you update to the latest development version you should be all set.

@roryk @chapmanb long time no see, hope you're well.
We came across this same issue running through bcbio_vm on Google Cloud. Is there a way to get the new version of VarDict into the vm Docker(?) image?

Hi Miika! I am glad to see you. :)
@chapmanb how do you update the docker image? It would be helpful for me to figure out how you are doing that.

Miika -- great to hear from you. Hope things are going well.
Rory -- thanks for looking at this. I just pushed the necessary change to bcbio_docker to rebuild the Docker image. All you need to do is bump the git hash it references and everything else happens as part of the TravisCI build. Hope this fixes it and gives you a path for updating going forward.

Hi Brad, thanks! Made a note for myself.


ResponseConvertible -> ResponseRepresentable

Vapor is my favorite web framework written in Swift.
Thanks for the great work!
I just fixed typo.

Thanks!


n/a

n/a

Timeout is set by default to the following, according to AWS doc:
:http_idle_timeout (Integer) — default: 60 —
The number of seconds a persistent connection is allowed to sit idle before it should no longer be used.
:http_open_timeout (Integer) — default: 15 —
The number of seconds before the :http_handler should timeout while trying to open a new HTTP session.
:http_read_timeout (Integer) — default: 60 —
The number of seconds before the :http_handler should timeout while waiting for a HTTP response.

(and there are no more)

Branch: 5784-check-S3-timeouts-and-https

After some research, S3 by default always uses HTTP for working with buckets internally.
In order to setup HTTPS for internal traffic (remember, not the urls served to the resques, those are already HTTPS),  seems that you need to setup an ssl ca bundle, or store or something similar so that the bucket_name (e.g. my.cartodb.imports) is called using https and not http.
There is no straightforward way of just saying "hey, I want https here".
So for internal https calls we'll need an SSL certificate for staging and other for production, as for sure one per bucket is required.
So... your call @santisaez, if you think is worth to have the internal traffic this safe, setup the keys and bring back to me the issue so I can setup it and do some tests to confirm.

BTW, the other commit shouldn't do much as only makes explicit some config values for timeouts and SSL on download urls that were in theory already there...


Get connections synchronized in MEMORY_STRICTLY mode

In MEMORY_STRICTLY mode to avoid dead lock in connection pool, we must meet the condition:
ds.maximumPoolSize >= acceptor.size * ds.table_count.
If we get all connections needed synchronized between queries, we just need to meet the condition:
ds.maximumPoolSize >= ds.table_count.
According to the performance test, there's little degrade of the QPS.

fixed at 3.0.0.M3


Failed to execute "requestDevice" on USB error

When integrating with useWallet, and Ledger wasn't connected (Previously it did). There are times this issue shows up. in dev environment.

Fixed this issue


Add class to compute statistics of the whole dataset

At my best knowledge it is not currently possible to compute statistics on the whole dataset, e.g., per-pixel mean and standard deviation. This wouldn't fit well in the Transformer class as it is not an on-the-fly transformation of the data, so I think it should be a separate class.
The best would be to be able to apply this class either on a Stream coming from a Dataset, or from one coming from a Transformer (I hope my terminology is sound here, correct me if I am wrong), as in some cases preprocessing of the data is required before it is possible to collect statistics.
This issue naturally pairs with https://github.com/mila-udem/fuel/issues/230 , as I can imagine one would want to save the result of this computation in a (potentially new) dataset. In this case, it might be convenient to define how these statistics should be saved in an h5 file, to enforce consistency among the datasets.

There was talk of using Blocks Aggregators for this.
it's a good idea, though we should think about how to store this
information in a standardized way.
@rizar thoughts?
On Tue, Sep 15, 2015 at 3:05 PM, Francesco<EMAIL_ADDRESS>wrote:

At my best knowledge it is not currently possible to compute statistics on
the whole dataset, e.g., per-pixel mean and standard deviation. This
wouldn't fit well in the Transformer class as it is not an on-the-fly
transformation of the data, so I think it should be a separate class.
The best would be to be able to apply this class either on a Stream
coming from a Dataset, or from one coming from a Transformer (I hope my
terminology is sound here, correct me if I am wrong), as in some cases
preprocessing of the data is required before it is possible to collect
statistics.
This issue naturally pairs with #230
https://github.com/mila-udem/fuel/issues/230 , as I can imagine one
would want to save the result of this computation in a (potentially new)
dataset. In this case, it might be convenient to define how these
statistics should be saved in an h5 file, to enforce consistency among the
datasets.
—
Reply to this email directly or view it on GitHub
https://github.com/mila-udem/fuel/issues/231.

I am not sure I understand what kind of support from Fuel you guys want. Yes, statistics can be computed by iterating dataset. Indeed, DatasetEvaluator from Blocks might be a great way of doing this and not  worrying about memory usage. But the output can be whatever...
On the other hand, I find it not very nice that I can not use Fuel to create a whitened version of my dataset.

This is what I would like from Fuel:

an offline preprocesser to iterate over the whole dataset and compute statistics. It is an operation that has to be done every time a new dataset wrapper is created, it would be nice to have an utility class to standardize the way it's done and to avoid to reinvent the wheel in each converter;
a standardized way to store this information in the dataset file;
once you have 1) and 2), objects that use that information on the fly (e.g. normalize, whiten, etc, ..)

@fvisin I think we should split this ticket up. We should try and tackle the offline preprocessor first, as it is potentially useful without the other two pieces. My next priority would be accompanying transformers that can use these statistics to do things like #242 (your number 3), so at least the user can store this stuff manually while we figure out 2.
We should design the API such that it can be used as an optional step at dataset conversion time, so you can say something like "also, store a whitening matrix for this foo as foo_whitening_matrix in the HDF5 file". For large streaming datasets we should think about some way to deal with a very generic conversion pipeline that invokes the aggregation methods of any of these things you specify as the data is being stored/converted.

@dwf I agree on everything. The transformers and the offline preprocessor can indeed be tackled independently, provided that we design them keeping in mind that we want them to work together at some point.
Having transformers that use global statistics could be very handy even without the offline preprocessor, as it is quite common to compute and store global statistics and I would expect people to already have them somewhere for a bunch of datasets, e.g., I pickled the pixel mean and variance of Imagenet when I worked on it, I could plug them into Fuel easily with the Transformer.


Type safe routes

Currently, I'm doing this:
enum AppRoutes {
    static let splash = "/"
    static let login = "/login"
    static let feed = "/feed"
}

SwitchRoutes {
    Route(AppRoutes.splash) {
        SplashScreenView()
    }
    Route(AppRoutes.feed) {
        FeedView()
    }
    ...
}

Is there any way to make this properly type safe, or at least safer? It seems rather fragile to rely on strings in this manner. Ideally, I wouldn't want to define my own "route enum" and have navigator.navigate accept only that type, etc.

@frzi Makes sense, thank you!

@frzi I can't seem to figure out what the Route init would look like when using enum AppRoute with associated values on some of the cases. Am I right in suspecting that it's not simple?

You'll probably want the paths for Routes to be formed differently. As an example:
enum AppRoute {
	case news(id: UUID? = nil)

	// Read this property in `NavLink` and `Navigator.navigate()`
	var path: String {
		switch self {
		case .news(let id): return "/news/\(id?.uuidString ?? "")"
		}
	}

	// Read this property in `Route`
	var route: String {
		switch self {
		case .news(let id): return "/news/\(id?.uuidString ?? ":id?")"
		}
	}
}

Route(.news()) { /* ... */ } // For `/news/:id?`
Route(.news(someConstantUUID)) { /* ... */ } // For `/news/edc54f97-ff00-463a-8897-f75144e41b7b` or something

With a bit of imagination I'm sure you'll be able to come up with a solution that fits your needs perfectly. 😄


release: Installation on Windows x64 in C:\Program Files\Go redirects to C:\Program Files (x86)\Go\

Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (go version)?

Windows 10 64-bit
What did you do?
Download latest install .MSI from Golang site. Double click to install. Override default C:\Go\ path to C:\Program Files\Go\ path. Installer redirects to C:\Program Files (x86)\Go\
What did you expect to see?
The installer should install to the path asked, not the 32-bit variant.
What did you see instead?
 Sorry.
Any help on this would be great.

If I had to guess, the installer executable is a 32-bit program so part of windows magic redirects it to use the 32-bit program files folder. I again would guess that the fix would be to make the installer 64-bit for 64-bit installs.

How do we get a 64-bit installer? We don't compile the installer.

I'm looking into it now. I can confirm this happens with the go1.8beta1 msi.

Okay, all yours. This is still tagged Go 1.9, but if you have a fix, we can try it out earlier.

In golang.org/x/build/cmd/release/releaselet.go Line 545
<Package
    Id='*'
    Keywords='Installer'
    Description="The Go Programming Language Installer"
    Comments="The Go programming language is an open source project to make programmers more productive."
    InstallerVersion="300"
    Compressed="yes"
    InstallScope="perMachine"
    Languages="1033" />
    <!--    Platform="x86 or x64" -->

Currently we don't specify in the wxs file if the platform is 32-bit or 64-bit. The msiexec will assume it is 32-bit if not specified. The fix will be to set the attribute. I'll send a CL later, though probably not in time for the beta2.

Ah, great.
/cc @broady


Faster interactive python workflow

Need to make this better, and then split out remaining items into additional TODOs
P1: The interactive workflow is too slow. The following 10 rotation chain takes 4.8 seconds to generate

2.9 seconds is rendering template, of which 2 seconds is calling black (lol), .8 seconds is calling string join inside of the template, and a small fraction is actually rendering

We’re calling black.format_str 38 times. we should probably disable this for interactive use - flag in the language args of codegen?

I made an option to LanguageArgs to not autoformat. Perhaps controversial, but I’d propose to turn it off for the python Factor by default.

Now str join is taking all the time of render template. What’s up with that?
Why 38 times? We have 19 factors, so must be twice per factor.

What’s the second python file per factor? An __init__.py file from symforce/symforce/codegen/python_templates/function/__init__.py.jinja. Gross.
All prior and between factors should be the same expression. Can we only generate two?

2 seconds is in codegen_util.print_code. Seems like a lot of subs operations and list comprehensions, as well as some time in the sympy printer tree. Probably lots to optimize.

A ton of time is being spent in subs in codegen_util.perform_cse, basically all initialization._get_subs_dict → _flatten_storage_type_subs → mostly python_util.scalar_like. We should work on reducing the calls to these or making them faster.

Fixed (for now) by the various performance improvements in v0.3.0


📊 energy: Add data on battery prices

I will handle this in the PR about energy prices: https://github.com/owid/etl/pull/3499


[Website] Make use of the Project Logos

Suggestion
Currently there are a couple of logos attached in the repository (which aren't used).
Let's contact the people of these projects and ask if they want to be on our website as a token of which projects support this idea and want to collaborate to make the best spec out there to benefit us all.
Use case
Seeing major platforms they all tell who are using their product.. we could do the same and build a "Used by" section on the website. This will make more of an impact than written links to the respective project
Extra info/examples/attachments
No response

Issue title
Use of Logo for UltraStar Format Specification
Issue Body
UltraStar Format Specification
We are from the UltraStar Format Specification project trying to create a specification for the UltraStar text format. This specification contains a set of rules and requirements that dictate what an UltraStar textfile is and how each item should be interpreted.
The endgoal
The endgoal for our project would be to:

[ ] Bring the different communities on par with eachother to increase the quality of the produced textfiles
[ ] Versionise the format so we can improve and add features to it
[ ] Give each Tag or NoteType a definition and how it should be working

The question
Since  uses the UltraStar Format we'd like to show your logo onto our website to tell people which projects are using the Specification and which projects don't.
For any questions please reply on this issue and we'll try our best to answer the question.
Issue Body as Markdown

Open Markdown for copy paste
## UltraStar Format Specification
We are from the [UltraStar Format Specification project](https://github.com/UltraStar-Deluxe/format) trying to create a specification for the UltraStar text format. This specification contains a set of rules and requirements that dictate what an UltraStar textfile is and how each item should be interpreted.

### The endgoal
The endgoal for our project would be to:
* [ ] Bring the different communities on par with eachother to increase the quality of the produced textfiles
* [ ] Versionise the format so we can improve and add features to it
* [ ] Give each Tag or NoteType a definition and how it should be working

### The question
Since <this project> uses the UltraStar Format we'd like to show your logo onto our website to tell people which projects are using the Specification and which projects don't.
For any questions please reply on this issue and we'll try our best to answer the question.

All communities (with repositories) are contacted and have an issue about FORMAT-Project.
We have many on board, not all but many ✅


update latest benchmark results

@guolinke Which commits of xgboost / LightGBM did you use for the benchmarks?

@guolinke Very impressive!
What about Parallel Experiment? I think it should be updated as well or be marked as from first version.

@StrikerRUS as we didn't improve parallel algorithm, we can mark them to first version


feat(checkbox, radio): Provide a way to vertically align checkboxes and radios in stacked layouts

Feature Description
MDC increased checkbox and radio touch areas. It means that in a stacked layout, the checkboxes/radios are visually indented to the right.
It can be seen in the checkbox docs: https://material.angular.io/components/checkbox/examples (the "Checkboxes with reactive forms" example).
It makes forms visually unbalanced - because other elements (labels, text inputs) don't have such indentation.
While a workaround with custom CSS is possible, it would relies on internal details of the Material components (sizes).
It would be great to have the official / documented way to do that. Maybe expose a variable as public API, add an Input to the components, or something else.
Use Case
No response

Are you asking to have a supported way to apply negative margin to the checkbox/radio? Or have a variable that provides how far other elements should be padded to line up?

Perhaps the latter. The actual margin to apply depends on how the material components are structured internally, and I wanted to avoid hardcoding it in a project.
Also wondering whether such negative margin should be applied by default e.g. in a form field. That's what React MUI seems to do:

I would love to avoid negative margins if I knew a better way to align checkboxes with the rest of the form.
Wondering whether it's just "us" problem, or if it is going to be a typical design challenge across different projects that adopt the MDC components


[$500] LHN - Selecting Mark as unread not displaying chat in bold

If you haven’t already, check out our contributing guidelines for onboarding and email<EMAIL_ADDRESS>to request to join our Slack channel!

Version Number: 1.4.21
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Expensify/Expensify Issue URL:
Issue reported by: Applause - Internal Team
Slack conversation:
Action Performed:

Go to https://staging.new.expensify.com/
Long press the chat which is not in bold and select "mark as unread"

Expected Result:
Selecting "Mark as unread" must display chat in bold in LHN.
Actual Result:
Selecting "Mark as unread" not displaying chat in bold in LHN.
Workaround:
Unknown
Platforms:

Which of our officially supported platforms is this issue occurring on?

[ ] Android: Native
[x] Android: mWeb Chrome
[ ] iOS: Native
[x] iOS: mWeb Safari
[x] MacOS: Chrome / Safari
[ ] MacOS: Desktop

Screenshots/Videos
Add any screenshot/video evidence
https://github.com/Expensify/App/assets/93399543/b51dcb21-8313-4ddc-aca7-cb1000757e78
View all open jobs on GitHub
Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~010c13f72ee15736be
Upwork Job ID:<PHONE_NUMBER>596718080
Last Price Increase: 2024-01-03

Proposal
Please re-state the problem that we are trying to solve in this issue.
Selecting "Mark as unread" must display chat in bold in LHN.
What is the root cause of that problem?
Currently in the OptionRowLHN menu context here, we pass '0' as the reportActionID
https://github.com/Expensify/App/blob/3826e5dd01cde3ae14d2f258a05a9514fa11e158/src/components/LHNOptionsList/OptionRowLHN.js#L130-L136
So that here it will take current time.
https://github.com/Expensify/App/blob/2fe11741539c96dadd2ce109b8538b01d02a60cb/src/libs/actions/Report.ts#L954-L955
What changes do you think we should make in order to solve the problem?
We should pass the last report action ID here.
Get reportActions from OptionRowLHNData.
const sortedReportActions = ReportActionsUtils.getSortedReportActionsForDisplay(props.reportActions);
const lastReportAction = _.first(sortedReportActions);

What alternative solutions did you explore? (Optional)

Proposal
Please re-state the problem that we are trying to solve in this issue.
Mark as unred is not marking the report as unread (no bold effect)
What is the root cause of that problem?
the function markCommentAsUnread from report.ts needs data from Onyx.
The moments we try to mark the report as unred, the data of this report, is not available on Onyx
If we check from dev tool, application, Onyx database, we will not find the "report_Actions_xxxxxxxxx" of the report.
If we open the report (click on the lis), then mark it unred, it works, because the Onyx data is updated.
What changes do you think we should make in order to solve the problem?
We need to update Onyx data on markCommentAsUnread before calling the data from the local storage.
https://github.com/Expensify/App/blob/3b385efb37c2d35965dbc58469a7725da1f673aa/src/libs/actions/Report.ts#L943
We can use openReport(reportID); from the same file to update the OnyxData, then reportActions will not turn undefined.

Looks like a regression from https://github.com/Expensify/App/pull/32958.

I am not able to reproduce on latest main

You need to mark it as unread on a report with undefined report actions (not loaded to Onyx yet). Try relogin first.

@situchan @allroundexperts are either of you able to reproduce this still?

still not

@situchan try to clean cache from dev tool, logout then login and try this from LHN without opening a report and for old chat. I just tested on staging and I can see the issue.
https://github.com/Expensify/App/assets/12425932/2f40a965-5962-44a6-9cb1-f9ce2b20e915

@dragnoir Your solution works partially. Adding openReport call inside the markAsUnread function doesn't make it bold on the first time. You have to mark it as unread, wait for a few seconds and then mark is as unread again in order to make the bolding work.

Still looking for better proposals.

Proposal
Please re-state the problem that we are trying to solve in this issue.
Mark as unread on some chat doesn't turn the title to bold/
What is the root cause of that problem?
The title will turn to bold if ReportUtils.isUnread returns true. It compares if lastReadTime < lastVisibleActionCreated
https://github.com/Expensify/App/blob/fd45dea98390e74e7953a764a56c32c4da541a68/src/libs/ReportUtils.ts#L3350
When pressing the mark as unread, it will take the report action created time subtracted by 1.
https://github.com/Expensify/App/blob/fd45dea98390e74e7953a764a56c32c4da541a68/src/libs/actions/Report.ts#L955-L968
The report action created time that we take is:

The report action that we mark as unread, but this is not the case for us because we mark it from LHN, so it's undefined
The last report action of other users from the report actions onyx
Fallback to current time

But in our case, the report's report actions Onyx is not available yet, so the 2nd case fails too and falls back to the current time, so lastReadTime is always > lastVisibleActionCreated.
This is the regression from https://github.com/Expensify/App/pull/32958 where we remove the fallback of the current report's lastVisibleActionCreated.
What changes do you think we should make in order to solve the problem?

Add back the fallback of the report's lastVisibleActionCreated
const actionCreationTime = reportActionCreated || (latestReportActionFromOtherUsers?.created ?? allReports?.[reportID]?.lastVisibleActionCreated ?? DateUtils.getDBTime(0));

If we decide to fix this here, then @bernhardoj's proposal looks good to me!
🎀 👀 🎀 C+ reviewed

This is no longer reproducible.

@youssef-lr This is easily reproducible. You need to logout and then login. Without opening the report, try marking it as unread.

You need to logout and then login. Without opening the report, try marking it as unread.

I personally don't see a real world scenario for this. We typically mark messages as unread to get back to them later, after reading them, which means the report actions will be loaded. We also use infinite sessions so our typical users will only need to log in once.

@youssef-lr I agree that it would be rare for any real user to encounter this case, but this works fine before https://github.com/Expensify/App/pull/32958, so I think it makes sense if we fix it here.

If this should be fixed, should be treated as regression from https://github.com/Expensify/App/pull/32958.
cc: @MonilBhavsar

Thanks @situchan, I reopened https://github.com/Expensify/App/issues/31748.


feat: Update XDS icon classes

EX-7776

Renamed some icons to follow the naming convention x-icon-${iconName}
Updated size classes
Added some stroke classes when needed
Fix clear input btn in mobile to set it squared

Update x-components to the latest version, if not, you are going to mix the old styles with the new ones.

Change the filtersIcon to use lg size:
<FiltersIcon class="x-icon-lg" /> in desktop-toolbar and mobile-open-aside


TokenizeText func doesn't return all sentences

I expect text.parsedSentences should contain all sentences. let me explain the problem with code :)
place it in parse/tokenizer_test.go file
func TestTokenizeText(t *testing.T) {
	rule := NewRule()

	text := TokenizeText("Hi!!!", rule)
	assert.Equal(t, "Hi!", text.parsedSentences[0].original)
	assert.Equal(t, "!", text.parsedSentences[1].original)
	assert.Equal(t, "!", text.parsedSentences[2].original)
}

I expect this test should be passed, but apparently, it is not!

Hi, I'm a beginner to open source. Could I have some pointers on where to start contributing to this project?

Hi,
Any contribution is welcome.
I recommend creating an issue here on GitHub then fork the repo and create a branch to that issue.
If you want to fix a logical error in the text ranking that's fine, but if you wish to add a new feature that should be discussed before.
I also recommend creating very small pull requests to make the whole process faster and more efficient.
Make sure GoReport is always on 100% https://goreportcard.com/report/github.com/DavidBelicza/TextRank and code coverage is near to 100% https://coveralls.io/github/DavidBelicza/TextRank?branch=master and please don't add 3rd party libraries to the project.


#67 Functions for unpacking based on byte arrays

This adds some new functions for packing/unpacking can messages based on byte arrays and does also support message lengths != 64 bit.
For me this is considered a base implementation that could be used to build on. (E.g. Message encode/decode function)
I also added a new benchmark project to the solution using dotnetbenchmark. This is optional but could help to tune performance in the future.

Hi @Uight ,
Thanks for the PR. You said there was something strange with the GetStartBitLE function. Did you manage to figure it out?
I am in parallel checking another implementation in js:
https://github.com/bit-dream/candied/blob/4e4077131d4104f417e750f916edae5cd9e76eb1/src/can/Can.ts#L167-L183
https://github.com/bit-dream/candied/blob/4e4077131d4104f417e750f916edae5cd9e76eb1/src/can/BitUtils.ts#L90-L102

@EFeru i figured that out i was just being stupid ;)
i now did check everything against the python cantools as described in the issue and everything seems to be the same. Assuming its done right in the can tools its also done correctly here.
If this PR goes through it would probably still be usefull to discuss how the final packer interface should look like in the end and do some more stuff there

@Uight still need to look into this. I did not forget :)


Docs for Commands API reference refers to waypoints not commands

http://python.dronekit.io/automodule.html#dronekit.CommandSequence.count
The count is a count of commands, which may include both waypoint and non-waypoint commands.

Should also be changed for CommandSequence.add and CommandSequence.next


Refactoring to turn into proper Python module package

Here's my new attempt at solving #59 which seems to be working better now that I rebranched off master to start. Needed to do some of the refactoring that was in the issue-59-pip branch and some other tweaks.
Right now it's available in PyPI as juriscraper 1.1.0 and installable via pip, but I believe will fail with easy_install. (Haven't tested with it...and not sure any users use it.)
If this looks good and gets merged, should probably start using tags on "releases" of Juriscraper as to make things a little less dependent on the setup.py packaging we can point the download to the tagged versions in github.

@mlissner any chance you can review?

Oof, looks like Github is reporting conflicts, but I'll try to take a look tomorrow.


Fix price calculation for non-reducible ingredients with modified res…

…ourceCostMultiplier
Resolves #38

Does the modifier also affect the work input?  I noticed the price goes slightly down with the modifier set to 0.5 now, and it seems to be calorie-cost-related.

Is the modifier supposed to affect the work input? I noticed the price goes slightly down with the modifier set to 0.5 now, and it seems to be calorie-cost-related.

I am not exactly following. A quick test seems to show craft resource modifier having no effect on calorie cost.
100x modifier:

0.1x modifier:

Default modifier (1):

Modifier = 0.5:

Hmm that is strange. It only seems to apply when the ingredient cost is non-zero. If you set all the ingredient costs to zero, and just leave the calorie cost, changing the craft resource modifier has no effect. Need to investigate more.

@rannmann are you testing on currently deployed environment or with the Preview for this PR?

@rannmann are you testing on currently deployed environment or with the Preview for this PR?

I was using the preview for this PR.

@rannmann are you testing on currently deployed environment or with the Preview for this PR?

I was using the preview for this PR.

I looked at this quite a bit until I came to a realization. Changing the craft resource modifier SHOULD actually increase the price of an AU3 somewhat, given that the ingredients (iron bar, nail) are in the outputs section. This is because those outputs have a nonzero price (e.g. 0.01). When craft resource modifier is increased, the number of that ingredient (e.g. Iron Bar) required for the AU3 recipe goes up. So the price should go up slightly.

Ah right, that makes sense.


If a package is installed with -i, do not download its dependencies on scoop update *

Would it be possible for scoop to remember which packages were installed with and without the -i flag, and update them accordingly? If I have a mix of packages installed with and without dependencies, it would be very nice if scoop would update those which were installed normally including dependencies, while not trying to install dependencies for those that were installed with the -i flag.
Of course I could always use update * -i, but then I might miss out on new dependencies for packages that are install without the -i flag. However if it as exceedingly rare case that is not worth handling, then please disregard this.

An extra flag in install.json file should be do-able. Just like architecture  check on update, check the other flags too.


[OpenTelemetry] Implement async gauges

Note that there are still a couple of more substantive comments remsining unaddressed from my earlier review pass.