Tentative plan for creating gtc pipeline

Even with an existing R package that has similar functions, a working command line pipeline starting with idats (ideally) or gtc files could make this process a lot easier to run.  Wrapping this in a workflow management system like Snakemake, which can submit jobs to an HPC cluster and track I/O will make parallel processing easy and run time should be really quick, after the initial upfront cost of generating an egt file in Genome Studio.  Luckily, groups dealing with Illumina data will already be comfortable with Illumina files so generating an egt and using a bpm should be pretty straightforward.  Hopefully, after that the pipeline will run with essentially one command.
Another plus for this approach is that if we contact Illumina about our tool, as Ben suggested, they may like that we are using some of their existing software as part of it (like the gtc parsing library).  He's right, if they mention our tool it could mean a lot of exposure.  In addition, I think this approach has a clearer implementation than the R approach.  And since we need to get most of this project done tomorrow, that's appealing to me.
I have some experience with PennCNV, so I'll start with that.  But adding other command line CNV callers should be pretty easy as well.  And since they will all run in parallel, it doesn't add much cost (if you have the compute) to run more CNV callers.  Since CNV calling in general has a fairly high FP rate, using multiple callers may be the way to go.
Though a lot of the code will be python, it will also involve a lot of testing of command line programs.  Both testing CNV callers that we download, like PennCNV, and also testing scripts that we write.  So knowing how to write python code is not the only thing needed in order to get this pipeline working.
Here's a rough idea of what the pipeline would look like:
wine AutoConvert.exe /path/to/idats /path/to/out/dir manifest.bpm clusterFile.egt  ##hopefully this works
        |
        |
       \/
  gtc files
        |
        |
       \/

"scripts/gtc2PennCNV.py"  ##I've already written this script, tested it,  and pushed it to this repo
        |
        |
       \/

run PennCnv ##There will also be generation of a couple of PennCNV specific reference files which is another up front cost.
        |
        |
       \/

There are some existing scripts for post-processing on PennCNV output that we can try if we have time.

I agree that having a simple pipeline is more appealing, and having illumina advocate it is a huge plus. I’m gonna get through what I can tonight and then we can decide tomorrow where to focus our efforts. I think for this project having something that takes advantage of HPCs and can scale is imperitive, and R unfortunately isn’t the most scalable environment. However it is great for data science-y/visualization methods, so like you were saying for downstream things, I think R would come in handy there. Maybe we can make it I/O compatible i.e. make python output able to be loaded into R.
From: Eric Karlins<EMAIL_ADDRESS>Reply-To: NCBI-Hackathons/Global_Screening_Arrays<EMAIL_ADDRESS>Date: Monday, March 20, 2017 at 7:59 PM
To: NCBI-Hackathons/Global_Screening_Arrays<EMAIL_ADDRESS>Cc: Subscribed<EMAIL_ADDRESS>Subject: [NCBI-Hackathons/Global_Screening_Arrays] Tentative plan for creating gtc pipeline (#8)
Even with an existing R package that has similar functions, a working command line pipeline starting with idats (ideally) or gtc files could make this process a lot easier to run. Wrapping this in a workflow management system like Snakemake, which can submit jobs to an HPC cluster and track I/O will make parallel processing easy and run time should be really quick, after the initial upfront cost of generating an egt file in Genome Studio. Luckily, groups dealing with Illumina data will already be comfortable with Illumina files so generating an egt and using a bpm should be pretty straightforward. Hopefully, after that the pipeline will run with essentially one command.
Another plus for this approach is that if we contact Illumina about our tool, as Ben suggested, they may like that we are using some of their existing software as part of it (like the gtc parsing library). He's right, if they mention our tool it could mean a lot of exposure. In addition, I think this approach has a clearer implementation than the R approach. And since we need to get most of this project done tomorrow, that's appealing to me.
I have some experience with PennCNV, so I'll start with that. But adding other command line CNV callers should be pretty easy as well. And since they will all run in parallel, it doesn't add much cost (if you have the compute) to run more CNV callers. Since CNV calling in general has a fairly high FP rate, using multiple callers may be the way to go.
Though a lot of the code will be python, it will also involve a lot of testing of command line programs. Both testing CNV callers that we download, like PennCNV, and also testing scripts that we write. So knowing how to write python code is not the only thing needed in order to get this pipeline working.
Here's a rough idea of what the pipeline would look like:
wine AutoConvert.exe /path/to/idats /path/to/out/dir manifest.bpm clusterFile.egt ##hopefully this works
        |
        |
       /
  gtc files
        |
        |
       /
"scripts/gtc2PennCNV.py" ##I've already written this script, tested it, and pushed it to this repo
|
|
/
run PennCnv ##There will also be generation of a couple of PennCNV specific reference files which is another up front cost.
|
|
/
There are some existing scripts for post-processing on PennCNV output that we can try if we have time.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@ngiangre, Snakemake can take R code directly, though I think behind the scenes it's calling it with the python package rpy2.  R code can definitely be part of this "gtc" pipeline if certain steps make the most sense in R.  And submitting small R jobs to the cluster is a good way to use R and parallelization, since, as you mention, R itself is not very scalable.

@ekarlins , after reviewing snakemake, PennCNV and gsrc, I think this pipeline is our best shot at getting ultimately a streamlined, testable raw cnv --> cmv calling and downstream analysis workflow.

What do you mean by 'this'?
Fwiw, we've built several successful projects based on snakemake.
Cheers!
Ben
On Mar 20, 2017 11:16 PM, "Nick Giangreco"<EMAIL_ADDRESS>wrote:
@ekarlins https://github.com/ekarlins , after reviewing snakemake,
PennCNV and gsrc, I think this pipeline is our best shot at getting
ultimately a streamlined, testable raw cnv --> cmv calling and downstream
analysis workflow.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/NCBI-Hackathons/Global_Screening_Arrays/issues/8#issuecomment-287965022,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFePtcWC8WQZ4MxuBRm2mHEsVYSJVcvxks5rn0EOgaJpZM4MjKso
.

I’m advocating snakemake. What I mean by this is by using snakemake to collate together the gtc -> pennCNV input files, PennCNV, maybe a gsrc cnv calling wrapper and other command line tools, I think that would make the most streamlined and possibly parallelizable program.
From: DCGenomics<EMAIL_ADDRESS>Reply-To: NCBI-Hackathons/Global_Screening_Arrays<EMAIL_ADDRESS>Date: Monday, March 20, 2017 at 11:36 PM
To: NCBI-Hackathons/Global_Screening_Arrays<EMAIL_ADDRESS>Cc: Nick Giangreco<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [NCBI-Hackathons/Global_Screening_Arrays] Tentative plan for creating gtc pipeline (#8)
What do you mean by 'this'?
Fwiw, we've built several successful projects based on snakemake.
Cheers!
Ben
On Mar 20, 2017 11:16 PM, "Nick Giangreco"<EMAIL_ADDRESS>wrote:
@ekarlins https://github.com/ekarlins , after reviewing snakemake,
PennCNV and gsrc, I think this pipeline is our best shot at getting
ultimately a streamlined, testable raw cnv --> cmv calling and downstream
analysis workflow.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/NCBI-Hackathons/Global_Screening_Arrays/issues/8#issuecomment-287965022,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFePtcWC8WQZ4MxuBRm2mHEsVYSJVcvxks5rn0EOgaJpZM4MjKso
.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

I’ve been looking at the snakemake functionality and it looks really elegant
From: DCGenomics<EMAIL_ADDRESS>Reply-To: NCBI-Hackathons/Global_Screening_Arrays<EMAIL_ADDRESS>Date: Monday, March 20, 2017 at 11:36 PM
To: NCBI-Hackathons/Global_Screening_Arrays<EMAIL_ADDRESS>Cc: Nick Giangreco<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Subject: Re: [NCBI-Hackathons/Global_Screening_Arrays] Tentative plan for creating gtc pipeline (#8)
What do you mean by 'this'?
Fwiw, we've built several successful projects based on snakemake.
Cheers!
Ben
On Mar 20, 2017 11:16 PM, "Nick Giangreco"<EMAIL_ADDRESS>wrote:
@ekarlins https://github.com/ekarlins , after reviewing snakemake,
PennCNV and gsrc, I think this pipeline is our best shot at getting
ultimately a streamlined, testable raw cnv --> cmv calling and downstream
analysis workflow.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/NCBI-Hackathons/Global_Screening_Arrays/issues/8#issuecomment-287965022,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFePtcWC8WQZ4MxuBRm2mHEsVYSJVcvxks5rn0EOgaJpZM4MjKso
.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Cool.
On Mar 20, 2017 11:42 PM, "Nick Giangreco"<EMAIL_ADDRESS>wrote:

I’ve been looking at the snakemake functionality and it looks really
elegant
From: DCGenomics<EMAIL_ADDRESS>Reply-To: NCBI-Hackathons/Global_Screening_Arrays<EMAIL_ADDRESS>Date: Monday, March 20, 2017 at 11:36 PM
To: NCBI-Hackathons/Global_Screening_Arrays <Global_Screening_Arrays@
noreply.github.com>
Cc: Nick Giangreco<EMAIL_ADDRESS>Mention <
<EMAIL_ADDRESS>Subject: Re: [NCBI-Hackathons/Global_Screening_Arrays] Tentative plan for
creating gtc pipeline (#8)
What do you mean by 'this'?
Fwiw, we've built several successful projects based on snakemake.
Cheers!
Ben
On Mar 20, 2017 11:16 PM, "Nick Giangreco"<EMAIL_ADDRESS>wrote:
@ekarlins https://github.com/ekarlins , after reviewing snakemake,
PennCNV and gsrc, I think this pipeline is our best shot at getting
ultimately a streamlined, testable raw cnv --> cmv calling and downstream
analysis workflow.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<https://github.com/NCBI-Hackathons/Global_Screening_
Arrays/issues/8#issuecomment-287965022>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/
AFePtcWC8WQZ4MxuBRm2mHEsVYSJVcvxks5rn0EOgaJpZM4MjKso>
.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/NCBI-Hackathons/Global_Screening_Arrays/issues/8#issuecomment-287968383,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFePtU7ZWFRZeEHgsngWdfsItg-O-Phrks5rn0cbgaJpZM4MjKso
.

I'll keep this ticket open for general comments about the pipeline.  We're going to move forward with this Snakemake pipeline and hopefully add some modules (Snakemake rules) using the R package "gsrc" as well.
Let's try to split up features into as many issues as we can so we can clearly divide and conquer.  To assist in getting stuff done, I'll assign issues to who I think it makes sense to work on it.  But feel free to let me know if it's not in your comfort zone or if there are other issues you want to work on.

https://www.gliffy.com/go/html5/11884987

updated
https://www.gliffy.com/go/publish/11884987


the prompt in handleSelectedCode is flaky

Hi!
I am testing your extension and I really like all the improvements you added 🌟
I wanted to report a bug/improvement that also affect the original fork.
The problem is that the prompt used for the Refactor/Modify command is not taking into consideration the selected text, and I found that it is often prone to modify the code outside the selection:
const prompt = `${task}\n\nFile: ${relativePath}\nLine: ${lineNumber}`;

For example if I select a portion of code, click Modify and instruct "Add comments" it often add comments outside the selected area.
I use gpt-4o and I noticed that with this model this behavior happens a lot.
I think the prompt need to be tuned to be more strict.
My suggestions would be to embed the selected code in the prompt and explicitly command to only modify what is in scope.
For example:
${task}
In the file ${relativePath}

IMPORTANT: perform this task by strictly editing only the code in scope:
`
${text}
`

I did some digging and found out that in the original code from @filip-koch the selected text was originally embedded in the prompt but has been removed due to unspecified issues.
In alternative it would be very cool to allow users to modify the prompt as they please from the settings page.

Hey there.
I will look into it, but I'm sort of preoccupied with other projects. However, in the meantime, you can try and pull the extension, test locally your changes, and contribute with a PR. That way we can ensure that it won't break anything.
In any case, big thanks for trying out the extension.


StyleMixin type

[x] right click context menu: "convert style to mixin"

[x] should move style to new mixin-specific frame

[x] only elements & text nodes can be converted to atoms
[x] Atom element types cannot be changed (because atoms don't have element types)
[x] atoms can only be used in the "mixin section"
[ ] menu bar needs to have "convert to atom" option

LATER:

[ ] atom children can have defined "parts" (like css3 ::parts spec)
[ ] elements that use mixins can point to parts
[ ] atoms can be swapped out

menu bar needs to have "convert to atom" option

Not done, but also not necessary for Alpha. Will add later on.


import_realm: Adapt message upload file relative path depending on whether the backend is local or S3.

This is a new reimplementation of #19478.
Testing plan:

[x] Local. Tested on rein.zulipdev.org. The upload file path in ~/zulip is correctly made as var/uploads/files/2/53/M1oUYZH8JfVgwdR3qq4bVReW/stuff.txt for local upload backend.
[x] S3. Already covered in the unit tests.

GIFs or screenshots:

This is great, merged after some small tweaks to the comments/commit messages.  Thanks @rht!


Handle alerts when the back button is pressed

Currently the form shown is changed when the back button is pressed, but if an alert is showing the alert remains showing.  Hash routing prevented the back button from working when a non-dismissible alert was shown, and dismissed ones that could be dismissed.
I figure this is already on the list, but ran across it today so submitted it just in case.

@jshaffstall - try 0.3.0
you'll need to do as hash routing does
import anvil
from routing import router

anvil.alert = router.alert
anvil.confirm = router.confirm

That works, thanks!  That should probably be a standard addition to the example startup module in the docs (or at least a note about that being where it would go if you want alerts to prevent navigation automatically)

yeah - the docs have been updated
You can find it in the api reference and in blocking navigation
There is also a "Kitchen Sink" app in the works that will include this in the startup module


Use Xcode 8 to open the project as the following error:"Use of unresolved identifier 'kSecTrustResultInvalid' "

Alamofire/Source/ServerTrustPolicy.swift:243:41: Use of unresolved identifier 'kSecTrustResultInvalid'

Update your library,
updated now and in ServerTrustPolicy file i can see:
private func trustIsValid(_ trust: SecTrust) -> Bool {
var isValid = false
    **var result = SecTrustResultType.invalid**
    let status = SecTrustEvaluate(trust, &result)

    if status == errSecSuccess {
        let unspecified = SecTrustResultType.unspecified
        let proceed = SecTrustResultType.proceed

        isValid = result == unspecified || result == proceed
    }

    return isValid
}

Please ensure you are using the appropriate version of Xcode for the version of Alamofire you're trying to use. Alamofire 4 requires Xcode 8, Alamofire 3.5 is compatible with both Xcode 7.3 and Xcode 8 with the "Use legacy Swift version" setting set to true.

I am using Appropriate version. I tried Alamofire 3.1.4, 3.5.0, 3.5.1 with Xcode 8. Use Legacy swift version = YES. Still unresolved identifier error is appearing !
Any other solution ?

Try Below Code. This worked for me. You've to capitalised enum type for swift 2.3 and Smaller for 3.0
var result = SecTrustResultType.Invalid
        let status = SecTrustEvaluate(trust, &result)

        if status == errSecSuccess {
            let unspecified = SecTrustResultType.Unspecified
            let proceed = SecTrustResultType.Proceed

            isValid = result == unspecified || result == proceed
        }

        return isValid


Added pill shaped buttons tip

Added a tip which shows how to easily make a pill shaped button

UNDER MATCH PROGRAM BY FILLED BY


Tree: Synchronous delete operation(current node = deleted node)

Please make sure these boxes are checked before submitting your PR, thank you!
Fix the bug: #13756

[x] Make sure you follow Element's contributing guide (中文 | English | Español | Français).
[x] Make sure you are merging your commits to dev branch.
[x] Add some descriptions and refer relative issues for you PR.

duplicate of https://github.com/ElemeFE/element/pull/14604


Nancy 2.0 Razor viewer has issues with .NET standard assemblies

Description
RazorAssemblyProvider.cs line 83 only checks the name and not assembly version.
As soon as some nuget references have more modern dependencies you get multiple assemblies like system.runtime which makes the .SingleOrDefault fail.
Steps to Reproduce
.NET 4.7 solution with rancy & razor viewengine - add nuget for sendgrid 9.9.
sendgrid has dependencies for System.Net.Http, which needs
System.Security.Cryptography.X509Certificates
which needs:
System.Security.Cryptography.Algorithms
which needs:
System.Runtime (>= 4.3.0)
which will cause the error as System.runtime 4.0 also is loaded.

Nancy version: 2.0
Nancy host ASP.NET
.NET Framework version: 4.7

+1
        private IEnumerable<Assembly> GetDefaultAssemblies()
        {
            var loadedAssemblies = new HashSet<Assembly>();

            foreach (var assemblyDefinition in this.defaultAssemblyDefinitions)
            {
                var assembly = AppDomain.CurrentDomain.GetAssemblies()**.SingleOrDefault(**x => x.GetName().Name.Equals(assemblyDefinition, StringComparison.OrdinalIgnoreCase));

                if (assembly != null)
                {
                    loadedAssemblies.Add(assembly);
                }
            }

            return loadedAssemblies;
        }


npm-base: Move cache dir to persistent location

Pointing NPM_CACHE_DIR_NATIVE to store cache under DL_DIR.
It wouldn't be required anymore to download all the packages after
removing TMPDIR before doing a clean build.

@imyller any news on this one?
It was a pain to track down this issue on our servers.

(same story here: https://github.com/imyller/meta-nodejs-contrib/pull/5)

I think this makes sense. As far as I can tell the cache really only stores the downloaded tarball and the package.json.

I'm not saying no to this, but I'd like to explain earlier reasoning for placing npm cache to workdir:
It's not quite that simple. Earlier I learned by error-and-trial that npm cache is not guaranteed to be cross-platform portable and has limited portability between npm versions (especially if switching backwards).
This caused issues in environments where packages with different Node.js version dependencies exist and/or build is made for several machine architectures using same OE environment (e.g. repository builders). Sharing common npm cache for these situations caused unexpected package build failures. Especially for npm packages containing native libraries. npm cache simply is not designed to be quite that portable.
Safe solution, and the "OE way", is to make sure that each and every package build cycle/workdir gets clean starting point without leftovers from earlier builds on another architecture or build tool version. One way to safely do this with npm is to place npm's cache to workdir. Downside is that cache lifetime is only for the package build (especially if if oe_rmwork is enabled).
To be honest, I haven't checked the situation with npm cache portability for a while, so I'm open for suggestions how to get the best of the both solutions. Maybe prefix cache dir with proper {target/build}_arch and.. maybe if required.. npm version?
Your thoughts?

I'd like to get this in too along with #68, but issues remain with supporting multiple Node.js and NPM versions. NPM does not really handle well jumping versions back and forth with common cache dir.
With large builds having packages depending on different Node.js versions this causes very hard to debug build failures.
So far only reliable solution has been placing cache in (temporary) workdir.
Shared npm cache dir would be nice, but I'd hate to cause breakage.
Suggestions are welcome.
/cc @KurtSchluss @mdavis777

How does OE support different nodejs versions within one build? - Is that supported at all?
I am not familiar with the download dir structure which is created for or from npm, but it sounds to me like there has to be version and architecture dependencies introduced.

I'm thinking we could append target arch and npm version to cache dir name? If those two remain the same, it should be safe to assume full compatibility.

That would be my first guess on a solution.  Target Arch is easy to add, but NPM version would be more difficult.  I would assume you would have to have the nodejs recipe set some new global var and npm.bbclass then use it.

Since all npm usage goes through oe_runnpm wrapper we could do version detection there with simple npm -v?

Going further, we could also add variable like NPM_SHARED_CACHE ?= "0" (or "1") allowing bypassing this automatic cache dir prefixing.
That way you could have fully common/shared npm cache (with your own risk) or safely arch+npmver automatically prefixed cache dir.

Did some rough work on it.  Seemed to work fine.
https://github.com/mdavis777/meta-nodejs/commit/c6a034cd66017db32cc4f5b175e9a9f8a31137ae

Exactly what I had in mind. I'll test this while jumping between major Node.js versions, but I do not expect any problems.

Looks nice. Just one question: Shouldn't npm -v be called via ${NPM}?

Yes, with ${NPM}. Good catch.
Also in my testing this breaks compatibility with recipes that define their own custom cache dir by overriding NPM_CACHE_DIR. We have to check if NPM_CACHE_DIR is defined when calling oe_runnpm. If it is, use that. If not, generate version prefixed dir.

1e68570079ca7b3d34e781a3fd5daa0a04e0fc3b

Looks good to me. How can we proceed here? May you create a pull request?

Replaced by PR #70


libwasmtime: extract reusable functions from the binary

The intention is kick of the process of building an abstraction library
over all subcrates, which can be embedded in any application which wants
to run WASM modules. The wasmtime binary acts as the first consumer of
this library.

AFAICT this is has been superseded by #287. @yurydelendik what could I add here (after a rebase) that's still missing in the new API?

what could I add here (after a rebase) that's still missing in the new API?

I'm not sure what can be added to this PR. The intent of #287 to allow use classes from wasmtime-api instead of relying on internal classes from wasmtime_jit or wasmtime_runtime, which addresses the same issue: "building an abstraction library over all subcrates". Additional goal is also to provide standard meaning of embedding wasmtime into the C application.
@steveeJ Is that anything missing in wasmtime-api that included in this PR? I'll be glad to quickly add it to the wasmtime-api.

@steveeJ Is that anything missing in wasmtime-api that included in this PR? I'll be glad to quickly add it to the wasmtime-api.

I'm building an example on top of the current API. I'll report back here if I found anything missing or otherwise close the PR.

I'm sorry for the confusion. The wasmtime binary's status at this momement is similar to the example that demonstrate usage of the wasmtime-api, so there is no much sense to wrap it into the library.
If you need more examples and "roadmap features" for wasmtime-api, check out https://github.com/CraneStation/wasmtime/blob/master/wasmtime-api/examples/gcd.rs and C/C++ examples at https://github.com/WebAssembly/wasm-c-api/tree/master/example .

I'm sorry for the confusion.

I'm not aware of any confusion, Still, no worries at all :-) What I meant by my last comment is that I'm currently building an example on the new API you wrote which was merged in #287 to see if it satisfies all the requirements that we currently have on wasmtime.
I've found the GCD example also, thanks! I've used to get started after I saw the PR.


SQL syntax highlighting breaks with string inside f-string

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: Version: 1.92.2
OS Version: macOS Sequoia 15.0 public beta

Steps to Reproduce:

When inside a cell of a jupyter notebook, create an f-string to store SQL query text.
As soon as a string is used inside of the f-string, the syntax highlighting breaks

Attached image:

Examples of broken syntax, non broken syntax without a string, and .sql file with correct syntax of this example
syntax_highlighting_bug.pdf

are you able to paste the code-snippets here please

Query with string criteria in the WHERE statement:
SELECT
  col_a,
  col_b,
  col_c
FROM
  table_a
WHERE
  col_a = 'example'
GROUP BY
  ALL
ORDER BY
  col_a, col_b;

Query with string criteria in the SELECT statement:
SELECT
  col_a AS "a",
  col_b,
  col_c
FROM
  table_a
GROUP BY
  ALL
ORDER BY
  col_a, col_b;

Query with string criteria in the SELECT & WHERE statement inside of an f-string:
query = f"""
SELECT
  col_a AS "a",
  col_b,
  col_c
FROM
  table_a
WHERE
  col_a = "a"
GROUP BY
  ALL
ORDER BY
  col_a, col_b;
"""


[Campaign] Ensure Github workflow runs on docker image used by Production Distribution Build

Hi All,
This is coming from the campaign here:

https://github.com/opensearch-project/opensearch-build/issues/3966

Overview
We would like your CI check (specifically plugin build) in GitHub Repo to run on top of the Build Docker Images from production distribution pipeline.
This is to ensure every plugin repo will use the exact docker images we used in Jenkins build, to check their PRs and run tests before merging the code, so that issues can be detected earlier, and environment can be identical across teams.
Solutions
The Build Team has created a simple script to dynamically retrieve the current docker image name/tag, so everyone can easily pull the images for their CI checks.
We have a trial run of the above with k-NN team. The script retrieves the docker image dynamically, save output, and use it as the docker image to pull for the upcoming run:

https://github.com/opensearch-project/k-NN/pull/1099

Note that GitHub Actions only support LINUX docker container at the time of this writing, so we will add Windows containers later on as well as macOS.
Implementation Notes
We would like you to review above PR and implement similar changes. Note on line 33 of the above k-NN PR, -u and -p parameters needs to assign values accordingly.

OpenSearch Plugin:

          CI_IMAGE_VERSION=`opensearch-build/docker/ci/get-ci-images.sh -p centos7 -u opensearch -t build | head -1`

OpenSearch-Dashboards Plugin:

          CI_IMAGE_VERSION=`opensearch-build/docker/ci/get-ci-images.sh -p rockylinux8 -u opensearch-dashboards -t build | head -1`

Note that in the above k-NN PR, despite it being OpenSearch plugin, it still uses rockylinux8, as we initially plan to upgrade to rockylinux. We have since revert back to centos7 to support older versions of systems running k-NN lib. As a result, all OpenSearch plugins still uses centos7 for the time being, and all OpenSearch-Dashboards plugins can go to rockylinux8.
Completion Date
The above should be implemented by Nov. 1, 2023 (2023-11-01) by Plugin Owners to their repository.
And backport the changes to 2.x branch after merging in main branch.
Contacts
Please contact @peterzhuamazon for any questions on this campaign.
cc: @bbarani
Thanks.

@peterzhuamazon, @bbarani completing this by 11/1 is a short notice with all the ongoing work. What OpenSearch release are we targeting this for?

@peterzhuamazon our CI workflow doesn't run in an image. Is there a standard we need to update to?

@peterzhuamazon our CI workflow doesn't use the container/image keywords at all. Is there a standard we need to update to? We have a JENKINS_URL env var set.

I will work with you on that soon.
Github actions itself is using containers anyway.
It is very easy to onboard a new container image and run the same.

We have a new approach established here.
Please see this sample workflow file from Alerting for example:
https://github.com/opensearch-project/alerting/blob/main/.github/workflows/multi-node-test-workflow.yml
Thanks.

@peterzhuamazon I'm still having troubles using that alerting change as reference. I've copied the Get-CI-Image-Tag, copied the job container configuration, and copied the shell cmds, but I'm still getting GLIBC errors. Ref: https://github.com/opensearch-project/anomaly-detection/actions/runs/6568513863/job/17843044510?pr=1080


fix: replace merge-graphql-schemas with @graphql-tools/merge

Hi, I tried to fix the issue #142 I've opened previously by following the migration docs pointed by the deprecation notice on the merge-graphql-schemas repository.

Also excuse me, but I have no idea of what I did wrong.
Can someone explain what's up with the "Semantic Pull Request" check?

Thank you @adriantodt
Semantic Pull Request just checks that the commit follows this pattern
This LGTM! Just update the snapshots

@bboure snapshots updated!

Thank you @adriantodt !

@all-contributors add @adriantodt for code

:tada: This PR is included in version 0.19.3 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Caching distinct function results

Hi, I have two methods under the same service as below:
@Cacheable({
    cacheBusterObserver: myCacheBuster
})
listAll() {
    // return response data here
}

@Cacheable({
    cacheBusterObserver: myCacheBuster
})
list(filter) {
    // return filtered response data
}

The cache is only working if I hit subsequent calls to the same function. By having different functions, wasn't it supposed to have both results cached?
Apparently it works if I set the maxCacheCount to more than 1, but I understand that it should affect only combinations of cache under the same function. Am I right?
GlobalCacheConfig.maxCacheCount = 2;

Thanks!

Not sure I understand. Both methods are independent in terms of caching.
If you call once listAll, you obviously wont' have cached results. Every other call of the same parameters will be. Same for list

They are independent functions, but they are not behaving as expected.
The behaviour:

Call listAll function -> executes HTTP request
Call listAll function again -> Data from cache (OK)
Call list function with some param -> executes HTTP request
Call listAll function -> executes HTTP request (ERROR).

@fabioseno just curious, how do you check whether data is returned from cache or not?

@fabioseno Can you provide an example in stackblitz?
Also, why do you need the cacheBusterObservers?

@fabioseno Can you provide more information?

@deyanpeychev00 I just got some time to get back to this issue and realized it's working as expected now! I created a stackblitz (https://stackblitz.com/edit/angular-vzxqvv) and everything is alright.
Thank you all!

Closing as resolved.


Attaching extra contextual information to REQUIRE() macro.

Is there any (easy) way to attach contextual information to a REQUIRE macro?
I'd like to be able to do something like:
REQUIRE(expected_prog.size() > 0, "Program loaded from disk is empty! Either it's missing, or the file is damaged.");

Where the string "Program loaded from disk is empty! Either it's missing, or the file is damaged." is additionally printed to the output if the macro fails.
INFO() kind of does this, but the "INFO" stack is not cleared by a REQUIRE macro, so if I have a number of tests in the setup of a complex object, and the last one fails, it dumps all the INFO() messages to the console.
I can force scoping by putting each INFO()/REQUIRE() pair in a separate set of blocks, but when my setup procedure can involve 10+ calls that must be executed in sequence, it quickly gets hideous.

Before someone derails with comments about overly complex setup stuff, this is the setup procedure for a specialized piece of embedded hardware. Yes, it is that complicated, and I'm testing and exercising the low-level API. It's unavoidable.

You can write your own macro that wraps the other two in a block:
#define IREQUIRE(cond, msg) do { INFO(msg); REQUIRE(cond); } while(0)

IREQUIRE(expected_prog.size() > 0, "Program ...");

That works, but this seems like something that should be part of the core library. Python asserts work that way, though I don't know any other languages with this feature off the top of my head. C asserts don't do this, but then C asserts are a lot lower level.

That's a common idiom to swallow the semicolon.

Ah, I see, it prevents the macro from expanding into two statements. Cool!


Do not auto-add from http: hosts

Do not auto-add server aliases to an MD when the virtualhost that provides these is clearly running on the http: port.

Fixed in v1.0.1


Implementation of top-k accuracy function

Current accuracy function can't show top-k accuracy for voluntary k.
My simple implementation using numpy follows.
# by this line, same as accuracy.py
# xp == numpy
# if you use gpu, use cupy.asnumpy(x)

# Assuming that module of t and y is numpy
argsorted_y = xp.argsort(y)[:,-self.top_k:]
return xp.asarray(xp.any(argsorted_y.T == t, axis=0).mean(dtype='f')),

Do you plan to implement argsort() for CuPy?

related to https://github.com/pfnet/chainer/issues/602?

@wkentaro
I cannot judge it, but as you mean, I think #602 is generalization of this issue.

Yes, #602 is related to yours, and you can use argpartition to get top-k elements.
This issue is about Chainer function, and #602 is related to CuPy, isn't it?
By the way, I made the issue, but we don't have a plan. Implementation of effective sort program in GPU is difficult for me :-(
@okuta knows about it.

Until we have a cupy-supported solution, a naive implementation is to run use chainer.functions.softmax(x), argsort it and search for a match within the k top results.

There is currently a pull request for CuPy argsort.

This PR should be closed because of https://github.com/cupy/cupy/pull/67 and https://github.com/chainer/chainer/pull/2057. Thanks.


WriteBool

AbiSerializationProvider.WriteBool method should be:
WriteByte(ms, (bool)value ? (byte)1 : (byte)0);
or it will write wrong bytes into MemoryStream.

https://github.com/GetScatter/eos-sharp/blob/master/EosSharp/EosSharp/Providers/AbiSerializationProvider.cs#L245
ms.Write(new byte[] { (byte)value }, 0, 1);
Converts the value to byte and only writes one byte to the memstream.

When sending a "delegatebw" transaction, there is a "transfer" parameter, type of "transfer" is bool, and only one byte in memsteam can run or it will throw a exception.

It should work. Try it and give feedback if is ok. :)
WriteByte will always write 1 byte or throw a exception trying to convert to byte.


Boss Hiscores not updated with The Nightmare

Describe the bug
The Nightmare being added to the boss hiscores has shifted all boss kc after The Nightmare alphabetically by one.
To Reproduce
Lookup any account on the hiscore plugin
Expected behavior
A clear and concise description of what you expected to happen.
Additional context
runelite-client/src/main/java/net/runelite/client/plugins/hiscore/HiscorePanel.jav
To fix, add The Nightmare to the BOSSES list in that file.

Fixed in #10783


Ia 1639 fix cypress ci

Cypress binary was missing when running cypress manually in the CI
Self proof reading checklist

[NA] Did I use eslint and black formatters
[x] Is my code clear enough and well documented
[NA] Are my typescript files well typed
[NA] Did I add translations
[NA] My migrations file are included
[NA] Are there enough tests

Changes

Replaced the caching of node_modules as per guidelines
Cached cypress binary and .npm folder as in recommended example

How to test
Run Iaso manual cypress testing in Github actions
Print screen / video
NA
Notes
I just fixed the missing binary issue. the tests still take a long time (20min+), but this can probably be optimized further

@cypress

@quang-le I've pushed the correct fix. It worked here: https://github.com/qgerome/iaso/actions/runs/3513836648

@quang-le I've pushed the correct fix. It worked here: https://github.com/qgerome/iaso/actions/runs/3513836648

I'm curious as to why it didn't work. It worked when I tested it on Friday

@cypress

It has to be merged to work.Quentin Gérôme Let’s Build It SRLLe 21 nov. 2022 à 13:23, github-actions[bot] @.***> a écrit :
Test failed: https://github.com/BLSQ/iaso/actions/runs/3514318230
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because your review was requested.Message ID: @.***>

It works when you trigger the workflow manually (as did the config I pushed on Friday), so I'm merging this


[WIP] Improved version of menu screen.

Replacement of #61
Currently there is a menu screen but it is not functional yet. The UI is made with PyImGUI. I will add an image of the menu tomorrow. Using PyImGUI means we no longer have resizing issues that were present in #61, in addition, we do not need to code the UI elements that we want in the future.
Also, if you do find a way into the world, the F3 screen is rendered with PyImGUI, in addition, this fixes a bug where the F3 text could go behind blocks a certain camera angles.

Here's an image of the menu so far.

Looks great! I'll review this PR shortly... the only issue I have right now is that I'm not able to run it on my PC (I'll send over the details next time I'm on my PC) - could you clarify what exactly needs to be installed, and add that to the readme?

Looks great! I'll review this PR shortly... the only issue I have right now is that I'm not able to run it on my PC (I'll send over the details next time I'm on my PC) - could you clarify what exactly needs to be installed, and add that to the readme?

The community/start.bat has been modified to have all dependencies and I will update the readme. This PR is not finished yet.

I think we should probably delete start.bat and replace it with a requirements.txt.

I agree.

This is the issue I'm getting w.r.t. not being able to run the program. It seems to be ImGUI-related more than anything, but I'm not excluding the possibility that something is being done in the MCPY code to alter the state of the OpenGL context s.t. it causes issues in ImGUI:
Traceback (most recent call last):
  File "/tmp/python-minecraft-clone/community/main.py", line 460, in <module>
    main()
  File "/tmp/python-minecraft-clone/community/main.py", line 457, in main
    game.run()
  File "/tmp/python-minecraft-clone/community/main.py", line 436, in run
    pyglet.app.run(interval = 0)
  File "/home/obiwac/.local/lib/python3.11/site-packages/pyglet/app/__init__.py", line 76, in run
    event_loop.run(interval)
  File "/home/obiwac/.local/lib/python3.11/site-packages/pyglet/app/base.py", line 163, in run
    timeout = self.idle()
              ^^^^^^^^^^^
  File "/home/obiwac/.local/lib/python3.11/site-packages/pyglet/app/base.py", line 224, in idle
    self.clock.call_scheduled_functions(dt)
  File "/home/obiwac/.local/lib/python3.11/site-packages/pyglet/clock.py", line 180, in call_scheduled_functions
    item.func(dt, *item.args, **item.kwargs)
  File "/home/obiwac/.local/lib/python3.11/site-packages/pyglet/app/base.py", line 118, in _redraw_windows
    window.dispatch_event('on_draw')
  File "/home/obiwac/.local/lib/python3.11/site-packages/pyglet/window/__init__.py", line 671, in dispatch_event
    super().dispatch_event(*args)
  File "/home/obiwac/.local/lib/python3.11/site-packages/pyglet/event.py", line 387, in dispatch_event
    if getattr(self, event_type)(*args):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/python-minecraft-clone/community/main.py", line 420, in on_draw
    self.impl.render(imgui.get_draw_data())
  File "/home/obiwac/.local/lib/python3.11/site-packages/imgui/integrations/pyglet.py", line 219, in render
    super(PygletProgrammablePipelineRenderer, self).render(draw_data)
  File "/home/obiwac/.local/lib/python3.11/site-packages/imgui/integrations/opengl.py", line 149, in render
    common_gl_state_tuple = get_common_gl_state()
                            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/obiwac/.local/lib/python3.11/site-packages/imgui/integrations/opengl.py", line 347, in get_common_gl_state
    last_texture = gl.glGetIntegerv(gl.GL_TEXTURE_BINDING_2D)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/obiwac/.local/lib/python3.11/site-packages/OpenGL/latebind.py", line 43, in __call__
    return self._finalCall( *args, **named )
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/obiwac/.local/lib/python3.11/site-packages/OpenGL/wrapper.py", line 678, in wrapperCall
    raise err
  File "/home/obiwac/.local/lib/python3.11/site-packages/OpenGL/wrapper.py", line 671, in wrapperCall
    result = wrappedOperation( *cArguments )
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/obiwac/.local/lib/python3.11/site-packages/OpenGL/error.py", line 230, in glCheckError
    raise self._errorClass(
OpenGL.error.GLError: GLError(
        err = 1280,
        description = b'invalid enumerant',
        baseOperation = glGetIntegerv,
        pyArgs = (
                GL_TEXTURE_BINDING_2D,
                <object object at 0x83e0c7d60>,
        ),
        cArgs = (
                GL_TEXTURE_BINDING_2D,
                array([3], dtype=int32),
        ),
        cArguments = (
                GL_TEXTURE_BINDING_2D,
                array([3], dtype=int32),
        )
)

That error could be happening because internally imgui uses PyOpenGL instead of Pyglet's gl, so I should probably modify my extra image loading function to use PyOpenGl instead of Pyglet's gl. For me I do not get this error, maybe we have different graphics drivers.

I don't think it's to do with a difference between different gl bindings. The OpenGL state is common between the two, and is stored in the graphics driver. If you enable debug_gl you do see errors in community even without this PR's additions. These probably have to be tracked down and fixed first. (And yeah, a difference in graphics driver probably explains the difference in behaviour; for reference I'm using NVIDIA's X driver.)

Didn't see your message until now, but I have made my extra load function use PyOpenGL. Could you try if that works for you?

Could you provide your system information (found in the latest log from the logs directory).
My system information is:
[2023-12-27 12:12:35,759] [MainProcess/MainThread/INFO] (main.py/__init__) System Info: Python: CPython 3.10.2
System: AMD64 Windows 10 10.0.19045
CPU: Intel64 Family 6 Model 165 Stepping 3, GenuineIntel
Display: NVIDIA GeForce GTX 1650/PCIe/SSE2
(4, 6)


Consider mipmapping ui.Image's generated by the engine.

Currently, the engine does not enable mipmapping on any images generated for the framework. Skia does support building mipmaps. Using mipmaps will lead to better quality in rendering of scaled images and better per frame performance because of fewer texel fetches from texture memory at non-base mipmap levels. This is at the cost of ~1.3x per texture memory usage.
We should consider either enabling mipmapping for all images or provide a flag to the framework for images it intends to render at a scaled size.
Mipmapping should always be disabled for images in the raster cache because rendering at resolution is the entire point for images generated by the cache.

We do use mipmapping on cross context images. Closing this on the assumption that that's all that was intended here.
Specifically, https://github.com/flutter/engine/blob/17e07c537e136d92f1f3f6957531c81b8b4374af/lib/ui/painting/image_decoder.cc#L289


OSSM-5301 Support multiple default profiles

Do you think another alternative would be change spec.profile to spec.profiles and allow the user to pass an array? pros? cons?

It wouldn't fix the problem. The reason we're adding support for multiple default profiles is so that we can set default+openshift as the default on OpenShift. Changing Istio.spec.profile to profiles wouldn't fix the problem, as users would have to always add the "openshift" profile to their Istio resource. Also, let's keep profiles as simple as upstream instead of making them too complex like we did in 2.x. At least until there's an actual need for specifying multiple profiles in the Istio resource.


Setting up Containers without Internet Connectivity

Does anyone know a method to install and configure containers on Windows 2016 TP4 without Internet  connectivity?  I've gotten as far as installing docker but don't know how to pull down the docker images manually.  Any insight would be appreciated.
This what I see basically
Querying status of Windows feature: Containers...
Feature Containers is already enabled.
Querying status of Windows feature: Hyper-V...
Feature Hyper-V is already enabled.
Waiting for Hyper-V Management...
Networking is already configured.  Confirming configuration...
Getting Container OS image (WindowsServerCore) version 10.0.10586.0 from OneGet (this may take a few minutes)...
Install-ContainerImage : Unable to download.
At C:\Scripts\Install-ContainerHost.ps1:461 char:17
+ ...              Install-ContainerImage $imageName -Version $imageVersion
+                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Install-ContainerImage

Container base image install complete.  Querying container images...
Waiting for VMMS to return image at (02/04/2016 06:08:16)...
Waiting for VMMS to return image at (02/04/2016 06:08:18)...
Waiting for VMMS to return image at (02/04/2016 06:08:20)...
Waiting for VMMS to return image at (02/04/2016 06:08:22)...
Waiting for VMMS to return image at (02/04/2016 06:08:24)...
Waiting for VMMS to return image at (02/04/2016 06:08:26)...
Waiting for VMMS to return image at (02/04/2016 06:08:28)...
Waiting for VMMS to return image at (02/04/2016 06:08:30)...
Waiting for VMMS to return image at (02/04/2016 06:08:32)...
Waiting for VMMS to return image at (02/04/2016 06:08:34)...

and it never grabs the image..  Is there a manual work around?  Download files on another box and copy to the server?

I am using the Install-ContainerHost.ps1 script by the way. @mebersol @StefanScherer

Looks like there is a bug in Install-ContainerHost.ps1 - we should throw if the download fails.
@planet - you can use Save-ContainerImage to copy down the container base image to a .wim on another machine, and then copy it to the container host and use Install-ContainerOsImage to install this image as a container base package.

@neilpeterson -- we should answer this somewhere.  Maybe FAQ or as an inline note?

@scooley - agreed, Ill get it in.

@planet Haven't tried it yet, but perhaps with Install-ContainerHost.ps1 -WimPath ... as mentioned here: Install-ContianerHost.ps1#L49-L50

@planet Haven't tried it yet, but perhaps with Install-ContainerHost.ps1 -WimPath ... as mentioned here: Install-ContianerHost.ps1#L49-L50

I will try and get back to you.

Resolved my issue.  It turned out to be a certificate related issue on my end.  Thanks everyone.


Non-primitive spreads to empty object

Fixes #20635
Specifically, when spreading object on the right, spread now returns the left of the spread; previously it returned object, which loses everything to the left.
The type baselines are not quite as nice now, but that isn't actually visible to users.

Isn't this unsound?
interface I { x: string };
function f(p: I, o: object): I {
    return { ...p, ...o };
}
console.log(f({ x: "" }, { x: 0 }).x.toUpperCase());

This kind of unsoundness is inherent in typescript and isn't particularly special to spreads:
let o: object = { x: 12 }
let xq: { x?: string } = o;
xq.x && xq.length

In addition, the unsoundness isn't any different between object and {}.


What's this based on...not seeing sourcecode.

Hello,
I'm fascinated by this project, but I don't see any of the source code even though there's a "src" folder and you can install it on Pypi.  What is this based on?  I'm unable to replicate your test results saying that it's 6x faster than faster-whisper or what not?  Is this just faster-whisper with an increased batch_size or does it actually innovate something new?
You're making bold claims about its speed that I have not been able to verify so...
Thanks!  Always looking for latest and greatest.  I'd also like to know how you think it compares to the Jax implementation located here https://github.com/sanchit-gandhi/whisper-jax

Thanks, I used the non-flash attention approach since I couldn't get flash attention v2 to install correctly...and I used a batch size of 1 to get a fair comparison with faster-whisper...and I did notice a 2x speedup compared...although, I did not have have a chance to manually compare the actual transcriptions to verify any quality differences.  Are you contending that it's just as accurate as well as the transcription itself being faster?  Also, any particular reason you don't release the source code?
But if I understand you correctly, it's faster than faster-whisper out of the box...not faster than the Jax implementation...but faster than Jax if the flash-attention v2 approach is used?
I modified your example as follows when doing my test:
pipe = pipeline("automatic-speech-recognition",
                "openai/whisper-large-v2",
                torch_dtype=torch.float32,
                device="cuda:0")

pipe.model = pipe.model.to_bettertransformer()

# Process the audio file
outputs = pipe("[REDACTED PATH FOR PRIVACY REASONS]",
               chunk_length_s=30,
               batch_size=1,
               return_timestamps=True)

Are you contending that it's just as accurate as well as the transcription itself being faster?

Transformers implementation matches one to one with openai/whisper. So I'm quite confident that the transcription is at the same level.

Also, any particular reason you don't release the source code?

Our codebase is fully open source and licensed under Apache 2.0 - https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/modeling_whisper.py

But if I understand you correctly, it's faster than faster-whisper out of the box...not faster than the Jax implementation...but faster than Jax if the flash-attention v2 approach is used?

Nope. It should be faster than Whisper-Jax (which Transformers implementation of Whisper using the JAX backend) since that implementation does not leverage Flash Attention yet.

btw just a side note and I used a batch size of 1 to get a fair comparison with faster-whisper
The batching is a core part of what makes this implementation fast!

Right...and even the Jax implementation states that you get at least a 10x speedup with batching with approximately 1% max loss in WER so...not sure if Jax would still be faster if your guys' approach and Jax did a comparison with the same batch size...But I don't think Jax uses flash attention v2 so...I'm not doubting you.  And I'm not an expert on this so...
But with Ctranslate2-based "faster-whisper" beam size of 5 (default) and batch size of 1 with this implementation, I saw a 2x speedup.  So I guess it falls upon me to try a beam size of 1 for a true comparison...Very interesting though, thanks.  So ya'll are part of the same team at Huggingface as I understand it?  I've been seeing some problems with the release of large-v3, but that's another topic for another day.  lol
THANKS for the sourcecode, thought it was being obscured.
Lastly, you are not converting the models to a proprietary format are you (like with ctranslate2/faster-whisper)?  I mean, if you use torch dtype16, it converts in that sense, but not a proprietary converter like ctranslate2...I noticed that it just downloaded the vanilla model from Openai...

btw just a side note and I used a batch size of 1 to get a fair comparison with faster-whisper
The batching is a core part of what makes this implementation fast!

That's kind of what I figured!

Yeah! Sanchit and I work on the same team. We're just trying to maximise the available open source options :)
Btw, I have some old benchmarks with various setting in the notebooks section:
faster-whisper - https://github.com/Vaibhavs10/insanely-fast-whisper/blob/main/notebooks/infer_faster_whisper_large_v2.ipynb
transformers - https://github.com/Vaibhavs10/insanely-fast-whisper/blob/main/notebooks/infer_transformers_whisper_large_v2.ipynb
Re: issues with large-v3 please feel free to open an issue in transformers and tag me. I'll look into it promptly!
I'll close this issue now (feel free to re-open)

That's cool to know.  I've noticed cool work from him and now I've met you so...Let me ask you though, is the default behavior of your implementation (and if you know Jax's) a beam size of 1 or 5 or what?

It means a lot to get a knowledgeable response.  Appreciate your guys' hard work.  Issue closed!


How to set adaptor to ultimateRecyclerView

simpleRecyclerViewAdapter = new SimpleAdapter(arrayList); //array List is not allowed in SimpleAdaptor.
ultimateRecyclerView.setAdapter(simpleRecyclerViewAdaptor);

Please help me fast i got stuck in between. Thanks in advance.

you need to extend anything from UltimateViewAdapter


[event-fetcher] Implement a binary to fetch events on chain.

Motivation
Implement a binary tool to fetch all payment events related to a set of accounts.
cargo run -- --url <JSON_RPC_ENDPOINT> get-payment-events <account1> <account2> ...
Have you read the Contributing Guidelines on pull requests?
Yes
Test Plan
Implemented a smoke test. Tested this tool with libra-node running locally as well.

Implemented with async api. Updated the test.

/land

/abort

:exclamation: Invalid command

Bors help and documentation

Bors is a Github App used to manage merging of PRs in order to ensure that CI is always green. It does so by maintaining a Merge Queue. Once a PR reaches the head of the Merge Queue it is rebased on top of the latest version of the PR's base-branch (generally master) and then triggers CI. If CI comes back green the PR is then merged into the base-branch. Regardless of the outcome, the next PR is the queue is then processed.
General

This project's Merge Queue can be found here.
This project requires PRs to be Reviewed and Approved before they can be queued for merging.
Before PR's can be merged they must be configured with "Allow edits from maintainers" enabled. This is needed for Bors to be able to update PR's in-place so that Github can properly recognize and mark them as "merged" once they've been merged into the upstream branch.

Commands
Bors actions can be triggered by posting a comment which includes a line of the form /<action>.

Command
Action
Description

Land
land, merge
attempt to land or merge a PR

Canary
canary, try
canary a PR by performing all checks without merging

Cancel
cancel, stop
stop an in-progress land

Cherry Pick
cherry-pick <target>
cherry-pick a PR into <target> branch

Priority
priority
set the priority level for a PR (high, normal, low)

Help
help, h
show this help message

Options
Options for Pull Requests are configured through the application of labels.

          Option
Description

Indicates that the PR is high-priority. When queued the PR will be placed at the head of the merge queue.

Indicates that the PR is low-priority. When queued the PR will be placed at the back of the merge queue.

Before merging the PR will be squashed down to a single commit, only retaining the commit message of the first commit in the PR.

/cancel

/land

/cancel

/land


gulp to-html doesn't produce usable html

Hi Guys,
Great project! In trying to make a non-universal app I found that gulp to-html doesn't produce usable html at the moment. index.html contains this script tag:  <script src="http://<IP_ADDRESS>:8080/build/app.js"></script> which should probably be something like <script src="assets/app-abcd.js"></script>.
Steps to reproduce
Run NODE_ENV=production gulp to-html
Expected Behavior
script/style tags with relative url.
Actual Behavior
script tags with absolute paths, and no style tags.

I just deployed a new este.firebaseapp.com without any problem. Btw, you don't have to set NODE_ENV to production, just gulp to-html. I don't know why it doesn't work for you but there is no magic. It's only 66 lines of code, so you should be able to debug it easily. https://github.com/este/este/blob/master/gulp/to-html.js
I am closing it because I can't reproduce it. But feel free to reopen once you know more.

@steida I stripped out all the firebase stuff, might be the cause. Would 'gulp build -p' be my best bet?

Just read the source code. It's really simple. You don't need my advice. Just learn Node, that's all :-)

I find that this error occurs when some previous steps in the to-html sequence fails, like eslint, jest or flow, which results in the rest of the code not getting executed.


Issues when trying to connect to server with enforce-secure-profile=true

Modloader
Forge
Minecraft Version
1.19.1
Modloader Version
Forge: 42.0.1
No Chat Reports Version
1.19.1-v1.3.2
Modpack Info
No response
The latest.log File
Client log: https://pastebin.com/bjTasu14 , server log: https://pastebin.com/ThUErjDS
Bug Description
It is currently not possible (or at least haven't managed to do it) to join a server with enforce-secure-profile=true, regardless of whether the server's IP is whitelisted in NCR's config file.
Steps to Reproduce

Start a Forge client with only this mod installed, and a local Forge server without any mods and the enforce-secure-profile property set to true.
Try to join the local server for the first time. The "Caution: Unsafe Server" screen of NCR should open, telling you about it not being able to protect you on such a server.
Try pressing the "Proceed" button to continue, and observe the screen continuing to open itself without letting you join the server.
Check the "Don't show again" checkmark and press "Proceed". This adds the server's IP to the whitelist, however you still get disconnected with a disconnection message of "Invalid signature for profile public key. Try restarting your game." or similar.
Go back to the server list and try to join again, and observe the same disconnection message appearing every time without you being able to join the server.

Other Information
No response

This was caused by my reliance on ClientPlayerNetworkEvent.LoggingOut event from Forge, which, as it turns out, is not dispatched at disconnect time as one would expect.


PGGrantTable.from_database requires grantor = CURRENT_USER

https://github.com/olirice/alembic_utils/blob/76607483cb9f3d8de9fb1aba4c0409c513da3668/src/alembic_utils/pg_grant_table.py#L151
More a question than an issue. I'm guessing there's good reason for it.
While creating the grant tables for my database, I ran into some trouble as some of my tables had different owners. I was able to resolve this by making them all owned by my migration user. I assume that is what this line in the documentation is explaining:

PGGrantTable requires the role name being used to generate migrations to match the role name that executes migrations.

Why is this constraint necessary?

The main thing it does is enables uniquely identifying the records in information_schema.role_column_grants which is needed for reflection and diffing
it also avoids avoids runtime (migration time) permissions errors b/c we've checked that the migration role is able to perform the grant


[WIP] JAMES-2586 Guide binding Postgres Message/Mailbox mapper

+1 one for a PR on PostgresMailboxSessionMapperFactory first

+1 one for a PR on PostgresMailboxSessionMapperFactory first

https://github.com/apache/james-project/pull/1829

@vttranlina I let you rebase?

Rebase please :)


Attributor throws syntax error when a header is defined with hypens

As a user, I should be able to define headers with hyphens in the name like "Content-Type", "X-Api-Version", etc.
But attributor throws a Syntax Error if I try to define a header with hyphens.
headers do
  header :'X-Api-Version', String, values: ['1.0'], required: true
end

SyntaxError: .../ruby/2.1.2/lib/ruby/gems/2.1.0/gems/attributor-2.1.0/lib/attributor/types/model.rb:38: syntax error, unexpected '-', expecting ';' or '\n'
        def X-API-VERSION

The attributor should be able to catch such exceptions.

The headers in praxis default to using hashes now, so this should be fixed.


Make no-restricted-paths more generic

no-restricted-paths seems like a great rule.
For example, it lets you ensure nothing in ./a could reach into ./b.
My use case is a bit different. I want to ensure that nothing in ./a could import anything outside of ./a except for node_modules. So if I’m in ./a, I want it to be impossible to import ../index, ../b, or ../../whatever. However I want to preserve the ability to import Node modules, like 'my-package' which can technically occur at any level above, and could even symlink to ./b.
Does this request make sense? Basically I’m trying to confine the use of relative paths to a specific folder, and disallow any relative imports outside of it.

I think you could do .eslintrc inside the a dir with "no-restricted-imports": [2, { "patterns": ["../*"] }], "no-restricted-modules": [2, { "patterns": ["../*"] }] (core rules), but that would prohibit any backtracking, even from deeply inside a up to a higher level that was still within 'a'.
tbh I think this is probably better as a separate rule, that ensures exactly what you're trying to achieve. It'd be a good rule to enable by default in every dir with a package.json, for example - or better, the rule could have a mode that prevents any relative imports that come from outside a package.json hierarchy?

Yes you're right I want to allow backtracking inside of it. I agree this could be a separate rule.
I'd rather have flexibility over just relying on package.json. For example in case of Create React App the sources are confined to ./src, and while package.json is one level above, you're still not supposed to import from, for example, ./public

Sure, I agree there's a case where you want to define a more restrictive jail than "package.json", and that absolutely should be doable - but I think the default/base case is "jailing to package.json", which limiting to "src" would trump within that directory.


Clean up publishing set up and add ambry-all

Remove publishing of test jars. This is no longer needed after setting
up the ambry-test-utils subproject. Move the testJar task into
build.gradle since that is the only place its needed now.
Add an "ambry-all" subproject that makes it easier to depend on the
entirety of ambry.

Testing:
publish locally:
./gradlew -Dmaven.repo.local=$HOME/local-repo assemble publishToMavenLocal

checked pom:
❯ cat ~/local-repo/com/github/ambry/ambry-all/0.3.146/ambry-all-0.3.146.pom
<?xml version="1.0" encoding="UTF-8"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.github.ambry</groupId>
  <artifactId>ambry-all</artifactId>
  <version>0.3.146</version>
  <licenses>
    <license>
      <name>The Apache License, Version 2.0</name>
      <url>https://github.com/linkedin/ambry/blob/master/LICENSE</url>
      <distribution>repo</distribution>
    </license>
  </licenses>
  <scm>
    <url>https://github.com/linkedin/ambry.git</url>
  </scm>
  <issueManagement>
    <system>GitHub issues</system>
    <url>https://github.com/linkedin/ambry/issues</url>
  </issueManagement>
  <ciManagement>
    <system>TravisCI</system>
    <url>https://travis-ci.org/linkedin/ambry</url>
  </ciManagement>
  <dependencies>
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>
      <version>1.7.5</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>com.github.ambry</groupId>
      <artifactId>ambry-frontend</artifactId>
      <version>0.3.146</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>com.github.ambry</groupId>
      <artifactId>ambry-server</artifactId>
      <version>0.3.146</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>com.github.ambry</groupId>
      <artifactId>ambry-tools</artifactId>
      <version>0.3.146</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>com.github.ambry</groupId>
      <artifactId>ambry-cloud</artifactId>
      <version>0.3.146</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>com.github.ambry</groupId>
      <artifactId>ambry-test-utils</artifactId>
      <version>0.3.146</version>
      <scope>compile</scope>
    </dependency>
  </dependencies>
</project>

Codecov Report

Merging #1428 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #1428      +/-   ##
============================================
+ Coverage     72.98%   72.98%   +<.01%
- Complexity     7564     7566       +2
============================================
  Files           541      541
  Lines         42185    42185
  Branches       5296     5296
============================================
+ Hits          30787    30789       +2
- Misses         9867     9872       +5
+ Partials       1531     1524       -7

Impacted Files
Coverage Δ
Complexity Δ

.../com.github.ambry.network/BlockingChannelInfo.java
80.95% <0%> (-3.18%)
16% <0%> (ø)

...rc/main/java/com.github.ambry.store/BlobStore.java
88.8% <0%> (-0.95%)
108% <0%> (-2%)

...va/com.github.ambry.replication/ReplicaThread.java
86.38% <0%> (-0.95%)
93% <0%> (-1%)

...java/com.github.ambry.network/SSLTransmission.java
70.46% <0%> (-0.93%)
76% <0%> (ø)

...ain/java/com.github.ambry.router/PutOperation.java
91.23% <0%> (-0.5%)
114% <0%> (-1%)

...n/java/com.github.ambry.store/PersistentIndex.java
89.96% <0%> (-0.12%)
243% <0%> (-1%)

...in/java/com.github.ambry.store/BlobStoreStats.java
71.66% <0%> (+0.2%)
104% <0%> (+1%)
:arrow_up:

...c/main/java/com.github.ambry.network/Selector.java
79% <0%> (+0.78%)
78% <0%> (+1%)
:arrow_up:

....ambry.clustermap/DynamicClusterChangeHandler.java
88.66% <0%> (+1.21%)
58% <0%> (+2%)
:arrow_up:

...b.ambry.network/BlockingChannelConnectionPool.java
75.78% <0%> (+2.1%)
12% <0%> (+1%)
:arrow_up:

... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2f60a66...ed9e779. Read the comment docs.


Fixed #682: Restoring the ability to provide an empty "newVersion" argument

Looks like folks were using the ability to clear a property value. Restoring it + updating the unit test suite.

@slawekjaranowski please review

Could you restart the jobs?


avoid running a full GC cycle on every FFI closure allocation

This PR makes make_function_pointer faster by avoiding a full GC cycle on every invocation.
The documentation of caml_alloc_custom (section 9.2 of the manual) says:

Another way to describe the effect of the used and max parameters is in terms of full GC cycles. If you allocate many custom blocks with used / max = 1 / N, the GC will then do one full cycle every N allocations.

So for used=1, max=1 we make a full GC cycle on every call (or within a small constant factor of that). This is clearly excessive and can be very slow.
I think caml_alloc_custom_mem can be used to specify the object size more precisely, but I don't know enough about the code to make that change. Additionally, I'm expecting 0-size to be at most a constant-factor error, anyway, because the custom block itself still counts toward the heap usage, so allocating these in a loop will eventually run a GC.
We at Jane Street have used the patched version of the code for a long time and it works well, so the code is tested by production use.
cc @tiash in case you'd like to add any background on what motivated the change (e.g. how bad the slowness was)

Thank you for the clear description and the fix, @aalekseyev !
I'm going to merge this now, since it's obviously correct, but I'd be interested to hear more from @tiash about how bad the slowness was, if he'd like to add more detail.


feat(53736): Switch statement over a union of literals is not exhaustive when checking against enum members

Fixes #53736

@typescript-bot perf test this

@jakebailey could you review this PR?


x/pkgsite: package removal request for [github.com/cosmiqal/cosmiqal-apis*]

What is the path of the package that you would like to have removed?
github.com/cosmiqal/cosmiqal-apis
github.com/cosmiqal/cosmiqal-apis-go
Are you the owner of this package?
Organization is cosmiqal and I am the owner.
What is the reason that you could not retract this package instead?
The repositories no longer exist.

These packages have been removed.


UnhandledPromiseRejectionWarning: Error: Failed to launch chrome!

Steps to reproduce
Tell us about your environment:

Puppeteer version: 1.3.0
Platform / OS version: Ubuntu 16.04.3 LTS
URLs (if applicable): https://github.com/Astro36/KoreanSchool/blob/master/lib/index.js#L238 (line:238)
Node.js version: v9.6.0

I already install Ubuntu dependencies by below command.
sudo apt-get install gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
What steps will reproduce the problem?
Please include code that reproduces the issue.

Run const browser = await puppeteer.launch();
Error occured.

What is the expected result?
Launch the chrome.
What happens instead?
(node:9952) UnhandledPromiseRejectionWarning: Error: Failed to launch chrome!
/home/odroid/Programs/KoreanSchool-ScheduleBot/node_modules/korean-school/node_modules/puppeteer/.local-chromium/linux-549031/chrome-linux/chrome: 1: /home/odroid/Programs/KoreanSchool-ScheduleBot/node_modules/korean-school/node_modules/puppeteer/.local-chromium/linux-549031/chrome-linux/chrome: Syntax error: Unterminated quoted string
TROUBLESHOOTING: https://github.com/GoogleChrome/puppeteer/blob/master/docs/troubleshooting.md
at onClose (/home/odroid/Programs/KoreanSchool-ScheduleBot/node_modules/korean-school/node_modules/puppeteer/lib/Launcher.js:255:14)
at Interface.helper.addEventListener (/home/odroid/Programs/KoreanSchool-ScheduleBot/node_modules/korean-school/node_modules/puppeteer/lib/Launcher.js:244:50)
at Interface.emit (events.js:130:15)
at Interface.close (readline.js:379:8)
at Socket.onend (readline.js:152:10)
at Socket.emit (events.js:130:15)
at endReadableNT (_stream_readable.js:1101:12)
at process._tickCallback (internal/process/next_tick.js:152:19)

(node:13165) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:13165) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

+1 (Also having this issue)

@Astro36 there’s some syntax error in the output provided. Can you run pptr’s chrome manually? Does it start?
Sent with GitHawk

@KevinNovak pptr doesn’t support ARM out of the box. You’ll have to compile chromium for arm on your own and point pptr to use it with executablePath
Sent with GitHawk

@aslushnikov Didn't realize that. I ended up doing what you mentioned with the executable path using the Chromium that comes with Raspbian. Thanks!

Ran into same issue. I have Ubuntu 14.04.5 LTS server.

I'm also receiving this error when I try to run chrome without headless and without sandbox with the following command:
const browser = await puppeteer.launch({
headless: false,
args: ['--no-sandbox', '--disable-setuid-sandbox']
});

I also get the same issue But it solves in my case.
const browser = await puppeteer.launch({
    headless: true,
   executablePath:'/home/sagar/workplace/scraping-demo/node_modules/puppeteer/.local-chromium/linux-599821/chrome-linux/chrome',
    args: ['--no-sandbox', '--disable-setuid-sandbox'],
  });

When you are using headless: true options then also use the option executablePath give the accurate path of your chrome file in node_modules folder, like above. In my case i am putting whole path of my system, where my chrome file.

I also get the same issue But it solves in my case.
const browser = await puppeteer.launch({
    headless: true,
   executablePath:'/home/sagar/workplace/scraping-demo/node_modules/puppeteer/.local-chromium/linux-599821/chrome-linux/chrome',
    args: ['--no-sandbox', '--disable-setuid-sandbox'],
  });

When you are using headless: true options then also use the option executablePath give the accurate path of your chrome file in node_modules folder, like above. In my case i am putting whole path of my system, where my chrome file.

Also worked for me!

I also get the same issue But it solves in my case.
const browser = await puppeteer.launch({
    headless: true,
   executablePath:'/home/sagar/workplace/scraping-demo/node_modules/puppeteer/.local-chromium/linux-599821/chrome-linux/chrome',
    args: ['--no-sandbox', '--disable-setuid-sandbox'],
  });

When you are using headless: true options then also use the option executablePath give the accurate path of your chrome file in node_modules folder, like above. In my case i am putting whole path of my system, where my chrome file.

Also worked for me!

I love you man!

Hi im having the same issue I tried everything and nothings working, im so confused. can someone please help. I ran npm install pupteer and put this in my main file bot.js
const browser = puppeteer.launch({
headless: true,
executablePath: 'node_modules/puppeteer/.local-chromium/win64-686378/chrome-win/chrome',
'args': ['--no-sandbox', '--disable-setuid-sandbox']
});
its still not working when I put it on Heroku, what can I do to fix it? Ps. If you need to see more code just let me know! Thanks!

Hi im having the same issue I tried everything and nothings working, im so confused. can someone please help. I ran npm install pupteer and put this in my main file bot.js
const browser = puppeteer.launch({
headless: true,
executablePath: 'node_modules/puppeteer/.local-chromium/win64-686378/chrome-win/chrome',
'args': ['--no-sandbox', '--disable-setuid-sandbox']
});
its still not working when I put it on Heroku, what can I do to fix it? Ps. If you need to see more code just let me know! Thanks!

Did you figure it out? I'm having the same issue. Thanks

Hi im having the same issue I tried everything and nothings working, im so confused. can someone please help. I ran npm install pupteer and put this in my main file bot.js
const browser = puppeteer.launch({
headless: true,
executablePath: 'node_modules/puppeteer/.local-chromium/win64-686378/chrome-win/chrome',
'args': ['--no-sandbox', '--disable-setuid-sandbox']
});
its still not working when I put it on Heroku, what can I do to fix it? Ps. If you need to see more code just let me know! Thanks!

Did you figure it out? I'm having the same issue. Thanks
Hey sorry for the late reply, ive been busy lately. No unfortunately I was never able to fix it.  I tried to run these commands:
heroku plugins:install heroku-repo
$ heroku repo:purge_cache -a appname
$ git commit --allow-empty -m "Purge cache"
$ git push heroku master
with no luck,
Then I tried this
$ heroku plugins:install https://github.com/heroku/heroku-repo.git
$ heroku repo:purge_cache -a appname
and still didn't work.
TBH  ive just left this alone for a wile and started working on other things, however id love to figure o this out. If you have any qus, you can DM me on instagram @ joey2031 hopefully we can fix this one day. Good luck!

I'm also receiving this error when I try to run chrome without headless and without sandbox with the following command:
const browser = await puppeteer.launch({
     headless: false,
     args: ['--no-sandbox', '--disable-setuid-sandbox']
  });

Not sure why but this didn't work for me. I am running Ubuntu 18.04.3 LTS on WSL2. I was however able to get it to work but adding the puppeteer version of Chrome to my PATH environment variable.

Yeah same issue. Works perfectly on local environment. but fails to deploy properly on production Ubuntu (Heroku). I followed the docs on how to launch to heroku from pupetteer but that's still not working. I think it has something to do with that interactive_test.exe file which I've pushed via Git LFS. but somehow all the contents within node_modules/pupetteer aren't being pushed properly from my local env.
lmk if you find a fix.
Cheers.

UPDATE: I was never able to fix it, im just going to try and use jimp, can someone help me center text on the image?

joey2031  try to launch the chrome.exe like a standalone application once before using it in a script.

If you are running an ARM processor, e.g. Raspberry PI, as stated in the begning of this thread I solved the issue with the following:
Install Chromium by running:
sudo apt install chromium-browser chromium-codecs-ffmpeg
It usually installes Chromium in /usr/bin/ -folder. So then we change the executablePath to the below:
  const browser = await puppeteer.launch({
          headless: true,
          executablePath: '/usr/bin/chromium-browser',
          args: ['--no-sandbox', '--disable-setuid-sandbox']
  });


Add --smooth option, which will collect X samples (one per --interval…

I've added a new --smooth option, which will collect X samples (one per --interval) before logging/plotting; this will reduce the spikey-ness of the graph.
Here's an example of the difference using --interval 0.1 and --smooth 50 (= one graph/log point every 5 seconds).

And here's the same process without --smooth

The approach taken in this PR rolls the time window in window width increments (instead of in increments corresponding to the original time difference between samples), which adds a non-natural frequency component to the output, and after all doesn't smooth as "effectively" as it could.
I think smoothing should be done using a "proper" rolling time window analysis. That's what I have made good experiences with over the years. I've implemented this approach in https://github.com/jgehrcke/goeffel.


[ADAP-882] [Feature] Allow base64-service-account-json key auth

Is this your first time submitting a feature request?

[X] I have read the expectations for open source contributors
[X] I have searched the existing issues, and I could not find an existing issue for this feature
[X] I am requesting a straightforward extension of existing dbt-bigquery functionality, rather than a Big Idea better suited to a discussion

Describe the feature
Allow setting gcp service account as base64 value. currently it only supports setting file path. it should be possible to set base64 SA value using environment variable.
similar to https://github.com/dbt-labs/dbt-snowflake/issues/436
implementation https://github.com/dbt-labs/dbt-snowflake/pull/292
https://github.com/dbt-labs/dbt-core/issues/6018
Describe alternatives you've considered
No response
Who will this benefit?
No response
Are you interested in contributing this feature?
No response
Anything else?
No response

Thanks for this suggestion @ismailsimsek !
Currently, there are two service account methods available in dbt-bigquery:

Service Account File (via keyfile)
Service Account JSON (via keyfile_json)

Did you already see the second method?
Assuming so, how would you imagine your suggestion being implemented?
Here's the relevant code within dbt-bigquery that interfaces with BigQuery:
https://github.com/dbt-labs/dbt-bigquery/blob/27ade3df84b040d4140c1870a033b20c2495b5cd/dbt/adapters/bigquery/connections.py#L330-L336
Here are the BigQuery docs for the two different methods:

from_service_account_file
from_service_account_info

@dbeatty10  from dbt side, this should work:
# new method
 elif method == BigQueryConnectionMethod.SERVICE_ACCOUNT_BASE64_JSON:
     sa_json_base64 = profile_credentials.keyfile_base64_json
     sa_json_content = base64.b64decode(details_base64.encode("utf-8")).decode("utf-8")
     return creds.from_service_account_info(info=sa_json_content)

from bigquery lib this method could be used
    def from_service_account_info(cls, info, **kwargs):

What do you think of this @dataders ?
The proposal is to add a third method named  keyfile_base64_json that is related to BigQuery service account credentials stored in JSON:

keyfile
keyfile_json
keyfile_base64_json ← Newly proposed

Two alternatives to consider:

Allow Base64-formatted JSON data within keyfile_json and/or keyfile (and then do fancy footwork within the implementation to recognize this case, decode it, and call the applicable method)
Choose not to support Base64-formatted JSON data and direct folks to use pre-existing keyfile_json or keyfile methods instead

@dbeatty10 I support this. In fact, I'd love to mainstream key handling into the default implementation. imo, a single implementation reduces risk. Happy to accept a PR adding support for keyfile_base64_json for the short- and medium-term, but long-term is that this lives in a common implementation
In a larger theme of authentication, I'd love to handle token handling as it related to cloud providers such as AWS and Azure. https://github.com/dbt-labs/dbt-core/issues/8186 is one example and MSFT adapter family token-handling (source) is another.

I saw this issue and was wondering if you would accept a reimplementation of BigQueryConnectionMethod.SERVICE_ACCOUNT_JSON?
I can implement an is_base64_encoded method that will check if the string is base64 encoded. If it returns True, then the input will be decoded.
`    import base64
import binascii
def is_base64(s: str | bytes) -> bool:
    """
    Checks if the given string or bytes object is valid Base64 encoded.

    Args:
        s: The string or bytes object to check.

    Returns:
        True if the input is valid Base64, False otherwise.
    """

    if isinstance(s, str):
        # For strings, ensure they consist only of valid Base64 characters
        if not s.isascii():
            return False
        # Convert to bytes for decoding
        s = s.encode('ascii')

    try:
        # Use the 'validate' parameter to enforce strict Base64 decoding rules
        base64.b64decode(s, validate=True)
        return True
    except binascii.Error:  # Catch specific errors from the base64 module
        return False`

Then a simple if-else will decode the base64 or leave the string as is.


header scan fails incorrect for little endian segy

In segyio, the buf is ALWAYS big endian regardless of the segy bytorder. Thus, def header_scan_worker() will result in incorrect header values for the snippet below from src/mdio/segy/_workers.py:
formats = [type_.numpy_dtype.newbyteorder(endian) for type_ in byte_types]

An issue was raised with segyio issues #559. Confirmed in discussion on Slack.
Example of behavior can be found at anthonytorlucci/segyio_header_buf.

Thanks for raising the issue. We will take a look before the next release.

I believe the fix is very simple (line 86 in /src/mdio/segy/_workers.py)
formats = [type_.numpy_dtype.newbyteorder('>') for type_ in byte_types]

since the byte order in buf is always BIG.
I forked the project, but unable to run nox without errors so I haven't submitted a pull request.

@anthonytorlucci I made the fix and its on the main branch. Can you test it either:

test pypi: https://test.pypi.org/project/multidimio/
clone main and install locally

Thanks


diagnostic plots for lm and roblm objects

adjust plots to show and adjust plot captions
server.R line 194, make multiple cases for OLS model estimation lm()
plot(fit_rlm, which=c(1,2,3,5), caption = list("Residuals vs Fitted", "Normal Q-Q vs. Residuals",
       "Scale-Location", "Cook's distance", "Residuals vs Leverage",
       expression("Cook's dist vs Leverage  " * h[ii] / (1 - h[ii]))),

and robust linear model estimation roblm()
caps = c("Standardized residuals vs. Robust Distances", "Normal Q-Q vs. Residuals",
"Response vs. Fitted Values", "Scale-Location" , "Residuals vs Fitted")
plot(fit_rlm, 5, caption = caps)
plot(fit_rlm, 2, caption = caps)
plot(fit_rlm, 4, caption = caps)
plot(fit_rlm, 1, caption = caps)

from line260 - line307 in server.R: changed plotting order in diagnostic plots
OLS and MM-type approach now show plots in  same order.
  #plot2 (model diagnostics)
  output$plot2 <- renderPlot({
    # define data
    data <- data()
    # Plot the kept and excluded points as two separate data sets
    keep    <- data[ vals$keeprows, , drop = FALSE]
    exclude <- data[!vals$keeprows, , drop = FALSE]
    # MM-type regressor: SMCcalibration::fitSMDM
    if (input$robust) {
      # robust model estimation
      fit_rlm <- SMCcalibration::fitSMDM(formula = meansample ~ meanstation,
        data = keep)
      # caption for MM-type dignostic plots
      caps = c("Standardized residuals vs. Robust Distances",
        "Normal Q-Q vs. Residuals", "Response vs. Fitted Values",
        "Residuals vs Fitted", "Scale-Location")
      # define visualization paramter
      op <- par(mfrow=c(2,2))
      # Residuals vs Fitted
      plot(fit_rlm, 4, caption = caps)
      # Normal Q-Q vs. Residuals
      plot(fit_rlm, 2, caption = caps)
      # Scale-Location
      plot(fit_rlm, 5, caption = caps)
      # Standardized residuals vs. Robust Distances
      plot(fit_rlm, 1, caption = caps)
      # return to standard par
      par(op)
    # OLS method: stats::lm
    } else {
      # linear model estimatin (OLS)
      fit_rlm <- lm(formula = meansample ~ meanstation, data = keep)
      # define visualization paramter
      op <- par(mfrow=c(2,2))
      # plot model dignostics - OLS
      plot(fit_rlm, which=c(1,2,3,5),
        caption = list("Residuals vs Fitted", "Normal Q-Q vs. Residuals",
        "Scale-Location", "Cook's distance", "Residuals vs Leverage",
        expression("Cook's dist vs Leverage  " * h[ii] / (1 - h[ii]))))
      # return to standard par
      par(op)
    }


v 4+ (SUPER URGENT - REGRESSION) Fix 'Explore's' 'VALIDATE THIS TREE' call to the iTree API

ToDO: As of 4.0.7 (build 74) there is a bug in the iTree API call when the user try's to use 'Validate This Tree' on an 'Unknown' tree. The problem is the API calls the location of the phone validating the tree - instead of using the coordinates already available in the database.
This is an error because the location of the tree has already been recorded - and it is the location of the tree that should generate the API results (not the validators phone location).
Note: as of 7/18/2022 you CAN use the lat/long which are stored in 'coords' to generate an iTreeAPI results - please refer to these API docs. (note: this is similar to a change we made in #371 ).

This issue is back - looks like the PR from #419 ( https://github.com/TreeMama/iSeaTree-React-Prototype/pull/491 ) did NOT sync with /develop/ before submitting. @dceu @bestw07 we need to make sure to take care of this ASAP

this looks fixed. Closing again.


Remove version: CrystalDewWorld.CrystalDiskInfo.KureiKeiEdition version 8.11.0

[Automated] It returns code over 400 in all urls
Microsoft Reviewers: Open in CodeFlow

URL: https://osdn.net/frs/redir.php?m=ipconnect&f=crystaldiskinfo%2F74602%2FCrystalDiskInfo8_11_0KureiKei.exe
Response Error: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
(Automated message - build 923)


 You have to buy a plan for which there is no price listed and then also pay an add-on fee? Will update to reflect what sounds like 171 -> 1500 (500 for pro + 1000 for SAML?) and can fix later if that's wrong. I'll actually reach out to them directly to give them a chance to provide new numbers.

E-mailed, let's see what they say.

Well, they had a chance:

Thanks for reaching out. Our pricing for SSO and other features can vary based on each contract. Unfortunately we wouldn't be able to comment in more detail.


Fixes issue where .mention-bot file shouldn't be required

This allows the work process to continue when a .mention-bot file doesn't exist in the target repo.

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please sign up at https://code.facebook.com/cla - and if you have received this in error or have any questions, please drop us a line at<EMAIL_ADDRESS>Thanks!

Thanks! Sorry about this


Missing Literature

Hi, thanks for your efforts. This is a great work!
Recently, we have been working on using non-autoregressive models for streaming generation tasks, particularly for simultaneous translation. These efforts seem to have been ignored in this overview. Due to the fast generation property of non-autoregressive models, we believe they have great potential in real-time streaming generation scenarios. If convenient, could you please add the following work to a new  Simultaneous Translationcategory?
[23EMNLP] Non-autoregressive Streaming Transformer for Simultaneous Translation [paper] [code]
[24ACL] A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation [paper][code]
Thank you!

@Paulmzr Hi, I have mentioned the new efforts and advance in Simultaneous Translation.
Thank you for reminding me of my negligence. I have added them on GitHub, and will be supplemented in the next version of my survey.
Feel free to contact me again.


Should all links of activated routes be set to the selected state?

Is this the www repository?

This is some old code I don't seem to find anywhere on the current repositories. Please help!

OOPS. Found this from page-router. Not sure why I could not find this with my text editor search. Fixing!

Now fixed. Thank you and sorry for the noise!


Should we add a triple-dot menu to the admin-view form if there’s a route mapped? Or some way to get to the contextual form and/or site?

Should we add a triple-dot menu to the admin-view form if there’s a route mapped? Or some way to get to the contextual form and/or site?

This was answered both in the original discussion about the change in workflows surrounding the CMS and once more as we were nearing release - we'd like to strive for simplicity:  A single link that routes to either the basic editor or the contextual editor depending on how the site is configured.  For a user, having to choose between editors can introduce friction.
In the future, we may still want to introduce a triple-dot menu for things like Delete or other Document actions.


Crank Program

What is the reason for this?
Can we clean it up before merging? Just on a quick first pass, linting is disabled heavily, not all functions are used like getMultipleAssociatedTokenAddresses, the tsconfig points to files that dont exist
If this is needed, can review now, but would be a lot easier if it got cleaned up and self reviewed first.

Running ad hoc locally.  Just wanted to get a working version on the repo. Can keep it off main for awhile

Lets keep it just on a feature branch until it needs to productionized.


Dataset contains very large unmasked value

One of the datasets has a very large unmasked value (5.813931e+36), which causes issues with the multimodel statistics.
# Loading #15: esmvaltool_sample_data/data/timeseries/CMIP6/CMIP/E3SM-Project/E3SM-1-1/historical/r1i1p1f1/Amon/ta/gr/v20191211
#     cube.shape=(780, 2, 2, 2) cube.data.min()=235.27495 cube.data.max()=5.813931e+36
#     cube.coord("time").units.calendar='365_day'

Good find, could you please open a dataset issue in the ESMValCore about it too?


Gui

Can you add gui features?
I was testing tkinter with x terminal, but it says have no display to export

Yes, I'm working on this.
Actually termux doesn't have any display server/protocol to display gui apps

Check v3.1.8


deleting manually all cache files "prioritize_critical_css" causes Firefox to render only half of the page

Hello,
my very first post on GitHub. Don't know if it is the right way to ask questions, but... ;-)
Installed mod_pagespeed yesterday and was quite satisfied with the results.
Then I wanted to change some CSS, this wasn't shown in firefox, so wanted to clear pagespeed cache.
touch /var/cache/mod_pagespeed/cache.flush didn't seem to work (at least I thought so because the cache wasn't emptied), so I manually deleted the files and restarted apache.
Since that, when I look at my pages in firefox, it somethimes shows me the content correctly, sometimes it doesn't load the images and the layout looks bad too. It looks as if pieces of CSS were missing (only in firefox and edge, chrome works fine) disabling prioritize_critical_css results in the page being loaded correctly in firefox (and chrome of course) but still not in edge.
NB: Refreshing the page once helps, but when I click trough the site and come back to these pages the problem reoccurs.
my pagespeed.conf looks like this:
ModPagespeed on
ModPagespeedRewriteLevel PassThrough
ModPagespeedEnableFilters collapse_whitespace
ModPagespeedEnableFilters remove_comments
ModPagespeedEnableFilters rewrite_css
ModPagespeedEnableFilters combine_css
# ModPagespeedEnableFilters prioritize_critical_css

the rest of the values are untouched (default). BTW: Apache version is 2.4.10
Might it have to do, with me deleting the cache files? Is there a way to get pagespeed to regenerate them?
Anyway, trying to figure it out for some hours now, but can't find a clue online.
So thanks a lot for your help!
Regards, biiissen

Have you configured the cache directives?
Try to use ?PageSpeedFilters=+debug with your url, this filter inject debug comments in the html code, perhaps you get any clue reading it (viewing the code of the page)

Also, to properly purge the cache, see
https://www.modpagespeed.com/doc/system#purge_cache
Otto
On Mon, 27 Nov 2017 at 05:34, biiissen<EMAIL_ADDRESS>wrote:

Hello,
my very first post on GitHub. Don't know if it is the right way to ask
questions, but... ;-)
Installed mod_pagespeed yesterday and was quite satisfied with the results.
Then I wanted to change some CSS, this wasn't shown in firefox, so wanted
to clear pagespeed cache.
touch /var/cache/mod_pagespeed/cache.flush didn't seem to work (at least
I thought so because the cache wasn't emptied), so I manually deleted the
files and restarted apache.
Since that, when I look at my pages in firefox, it somethimes shows me the
content correctly, sometimes it doesn't load the images and the layout
looks bad too. It looks as if pieces of CSS were missing (only in firefox
and edge, chrome works fine) disabling prioritize_critical_css results in
the page being loaded correctly in firefox (and chrome of course) but still
not in edge.
NB: Refreshing the page once helps, but when I click trough the site and
come back to these pages the problem reoccurs.
my pagespeed.conf looks like this:
ModPagespeed on
ModPagespeedRewriteLevel PassThrough
ModPagespeedEnableFilters collapse_whitespace
ModPagespeedEnableFilters remove_comments
ModPagespeedEnableFilters rewrite_css
ModPagespeedEnableFilters combine_css
ModPagespeedEnableFilters prioritize_critical_css
the rest of the values are untouched (default). BTW: Apache version is
2.4.10
Might it have to do, with me deleting the cache files? Is there a way to
get pagespeed to regenerate them?
Anyway, trying to figure it out for some hours now, but can't find a clue
online.
So thanks a lot for your help!
Regards, biiissen
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/pagespeed/mod_pagespeed/issues/1688, or mute the
thread
https://github.com/notifications/unsubscribe-auth/ACIsRHvbkdo-vW7dA5jzfZberDEMeCoHks5s6ju7gaJpZM4QrHKB
.

Thanks a lot for your hints!
I'll try it and see what it gives me.

Mea culpa, mea maxima cupla!
Sorry for wasting your time guys.
<script **async** src="//code.jquery.com/jquery....js"></script>
Turned out to be the problem.
Thanks for helping me anyway. Got the browser interface configured now. Read most of the documentation. Purged once more correctly. Removed the async and everything seems to work properly.
Will have to review the settings at some point, when I have more people visiting my site to optimize de last bit. For now pingdom.com says: Performance grad A/93 and faster than 90% of the tested sites. Not that bad to start with I would say.
And congratulations for this more than useful mod!
Thanks again!
Bye, biiissen


Not iOS 7 compatible

The podfile says iOS 7.0 compatibility, but viewForKey: is only available on iOS 8.0+
UIView *presentedControllerView = [transitionContext viewForKey:UITransitionContextToViewKey];

(line 40 of YPBubbleTransition.m)

Fixed. Thank you!


Document requirements for statusbar methods

cordova plugin add org.apache.cordova.statusbar is required to use the statusbar methods

thanks!