Ignore decode errors when parsing files

Background for PR: Suppose you're testing a site that has a CAPTCHA that to some extent blocks tools like SubDomainizer from getting any useful responses. My workaround for this situation so far has been to manually browse the site with an intercepting proxy like ZAP or BurpSuite and manually click myself past the CAPTCHAs. Once I'm happy with my exploration of the site I'll dump all the response data to file and run python3 SubDomainizer.py -f <dir with files> on it. That initially failed for me because the response data has binary content that breaks "file.read()" ('t' flag/text). I added errors='ignore' to it to resolve that problem.
If you've got a better way or any thoughts or ideas about this I'd love to hear it. Thanks for making this tool!
Commit comment:
This change only applies to when using the "-f" argument for parsing
files in a directory. Suppose you have some binary data in your file(s)
you will then end up with decoding errors when reading the file(s).
Ignoring errors lets you move past the binary data and continue
processing the remainder of the file(s).

Thanks for creating a pull request. Did you got some examples of responses for which you're getting responses. Also how you're getting a binary file?

It could be for example PDFs or images. I think I've seen font data, like "woff2" (also binary) somewhere too...

Yeah I am talking about that only. How're you getting that data? I am neither downloading pdf nor fonts, but only files inside script tag?

Ah, I see. I've come to pass all my response data from ZAP/Burp to 'SubDomainzer.py -f ' because if I just set 'SubDomainzer' on a site with a CAPTCHA then the responses returned to 'SubDomainizer' won't be the actual site contents. And either way, just letting 'SubDomainzer' work through the data from a browsing session, looking for subdomains and secrets seems like a good idea to me...

Ah got it. I still doubt on the decoding thing. Although merging your pull request.
Thanks for that.


Errors when running vim with Pathogen installed

I'm trying to install pathogen, but am getting a string of errors.  I've installed autoload/pathogen.vim and added the following to .vimrc (before the 'filetype plugin indent on' directive'):

call pathogen#helptags()
call pathogen#runtime_append_all_bundles()

But when I open vim, I get the following command line errors:

me@me-desktop:~/.vim$ vim autoload/README
Error detected while processing /home/me/.vim/autoload/pathogen.vim:
line    4:
E492: Not an editor command: ^M
line    8:
E492: Not an editor command: ^M
line    9:
E15: Invalid expression: exists("g:loaded_pathogen") || &cp^M
line  143:
E171: Missing :endif
line    4:
E492: Not an editor command: ^M
line    8:
E492: Not an editor command: ^M
line    9:
E15: Invalid expression: exists("g:loaded_pathogen") || &cp^M
line  143:
E171: Missing :endif
Error detected while processing /home/me/.vimrc:
line   72:
E117: Unknown function: pathogen#helptags
Error detected while processing /home/me/.vim/autoload/pathogen.vim:
line    4:
E492: Not an editor command: ^M
line    8:
E492: Not an editor command: ^M
line    9:
E15: Invalid expression: exists("g:loaded_pathogen") || &cp^M
line  143:
E171: Missing :endif
Error detected while processing /home/me/.vimrc:
line   73:
E117: Unknown function: pathogen#runtime_append_all_bundles
me@me-desktop:~/.vim$

I've tried to get rid of the ^M with a global replace: :%s/^V^M//gbut the replace doesn't see any of them.
Any suggestions what I'm doing wrong?

With Notepad++ I set the encoding to UTF-8 and it worked for me.


Life pass

Life is wonderful and beautiful people helping people is all about the potential of experiences and priorities are in place to ensure that you can be sure of experiences that are positive for the world keep the faith in yourself and God


fix checked test when @INC lacks .

The path function won't include a leading ./ even if we give it a . path
component.  Just use a normal string.  Using slashes for path separators
will always work anyway.

when, realistically, will @INC lack '.'?  It is still there in 5.24.1-RC4 and only locally removed during some modules' functions.

Debian testing already shipped a perl 5.24.1 that doesn't have . in @INC under normal circumstances.  5.26 most likely will make that change as well.  There are special exceptions made for the module installation process, but that means running the tests manually (perl -Ilib t/checked.t) will fail.

ahhh, I didn't grasp that debian's 5.24.1 was baked with non-default options.
I'll create a perl alias that sets PERL5OPT=-lib:. for easier testing.

thanks, merged and released!


Issues since version 6.3.0

Hi,
I was using version 6.2.2 and wanted to upgrade to 7.2 but I got this issue
Unable to resolve system scoped dependency: p2.eclipse.plugin:org.apache.xerces:eclipse-plugin:2.12.1.v20210115-0812:system
exception: org.owasp.dependencycheck.exception.DependencyNotFoundException: Unable to resolve system scoped dependency: p2.eclipse.plugin:org.apache.xerces:eclipse-plugin:2.12.1.v20210115-0812:system

org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.collectMavenDependencies(BaseDependencyCheckMojo.java:1292)
org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.collectDependencies(BaseDependencyCheckMojo.java:1437)
org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.scanArtifacts(BaseDependencyCheckMojo.java:1098)
org.owasp.dependencycheck.maven.AggregateMojo.scanDependencies(AggregateMojo.java:73)
org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.runCheck(BaseDependencyCheckMojo.java:1689)
org.owasp.dependencycheck.maven.BaseDependencyCheckMojo.execute(BaseDependencyCheckMojo.java:950)
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:210)
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:156)
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:148)
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
org.apache.maven.cli.MavenCli.execute(MavenCli.java:972)
org.apache.maven.cli.MavenCli.doMain(MavenCli.java:293)
org.apache.maven.cli.MavenCli.main(MavenCli.java:196)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.base/java.lang.reflect.Method.invoke(Method.java:566)
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:282)
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:225)
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:406)
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:347)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.base/java.lang.reflect.Method.invoke(Method.java:566)
org.apache.maven.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:39)
org.apache.maven.wrapper.WrapperExecutor.execute(WrapperExecutor.java:122)
org.apache.maven.wrapper.MavenWrapperMain.main(MavenWrapperMain.java:61)

So I tried different versions and this issue first occures with version 6.3.0.
I do not have the slightest clue on why this happens.
The build uses tycho pomless , maybe this is related.
Thanks for this great plugin andyour work maintaining it.

Any chance you can provide an example project that causes this issue?

Likely related to https://github.com/jeremylong/DependencyCheck/pull/3501 - an upgrade of the maven internal libraries

In one or more of your project's dependencies there is a transitive dependency with scope system that is listing a path (potentially with an environment variable) that does not resolve to the given dependency.

I think we can improve the error message somewhat by also listing what according to the dependency is the system path where it was supposed to be found.

I just tried to create a small project, but this is not as easy as I thought.
But I then saw, that xerces has a dependency to org.apache.xml.resolver and that again has a dependency to
org.eclipse.osgi. And this dependency is written as (at least in the eclipse editor I use) system.bundle [org.eclipse.osgi]

@aikebah Hi, would it be possible to add your change to the error message, so I could maybe dig deeper into the problem?
Thx.

@StefanHirche would you be able to build a snapshot from branch issue-4969 to see if added logging does shine more light on the subject?
If it turns out there is no info to show in these cases I'd rather not confuse users with strange log messages.

Hi, I build your branch and run the plugin-in. I got this
[ERROR]          DependencyNotFoundException: Unable to resolve system scoped dependency: myplugin:myplugin.source:eclipse-plugin:libs/3rdparty.jar:8.1.0.qualifier:system at path null
I use a 3rdparty plugin withoutputting it into maven, just into the libs folder of my plugin, because it is actively developed and changes very fast at the moment.
But this is a new construct, so I'll try to remove this to look at the older issues mentioned in this ticket.

@StefanHirche how is the system-path setup in the pom? Because it looks like how we currently collect/resolve dependencies does not resolve it to the actual configured location, but to null.

There is no configuration for the system path in the pom.

I now also get this error after removing the local lib:
[ERROR]         DependencyNotFoundException: Unable to resolve system scoped dependency: p2.eclipse.plugin:jakarta.activation-api:eclipse-plugin:1.2.2:system at path null
Seems to be the same issue, but I don't even have a clue what to set the system path to.

Well, a system-scoped dependency (while deprecated) is always supposed to have a systemPath telling maven where to find the jar representing/containing that dependency so that it can be included in the classpath.
Are you sure these should be system-scoped? Or is there maybe some eclipse/p2 plugin-magic happening behind the scenes that injects those system-paths into maven's internals after maven has done it's dependency resolution and failed to resolve the system dependencies?

@StefanHirche googled a bit on tycho-pomless and think the two are related indeed. It looks as if the tycho-pomless core extension does not inject the proper system-path when using modern versions of maven shared libraries for dependency-collection/resolution.

@StefanHirche If you are able to provide a (dummy would be good enough, but real-life public project would also be fine) reproducer project using tycho-pomless and triggering this behaviour that would help in getting me started to dig deeper into this with mvndebug (as I have no experience on tycho / tycho-pomless I don't want to start out on the adventure from scratch)

Yes, but it might take a little time.

NP... would be nice to try and resolve this so that it also works properly for tycho-pomless, but not in any hurry on that.

Hi,
OK, with 5220 this gets more urgent. Or is there a chance to get the fix backported to 6.2.2?
If not I will start to create an example project tomorrow.
Thanks.

@StefanHirche Assuming that your CVE database is persistent between builds you should be able to resolve the issue, while continuing to scan with 6.2.2 by running a dedicated job for CVE database update (mvn org.owasp:dependency-check-maven:7.4.4:update-only) as a one-off job.
6.2.2 should run fine with the updated database schema that 7.4.4 will upgrade to.

Hi,
OK, with 5220 this gets more urgent.
Here you can find my project, I hope it works (or better not)
https://github.com/StefanHirche/OWASPTest

@StefanHirche Thanks, can confirm that it works for me as a reproducer. Had to fix some configurations that leak your internal setup (mvnw JAVA_HOME, pointing to a windows path and your private maven distribution-url inside the .mvn folder (may I suggest upgrading to Maven 3.8.7?)
[ERROR] Failed to execute goal org.owasp:dependency-check-maven:7.4.4:aggregate (default-cli) on project root: One or more exceptions occurred during dependency-check analysis: One or more exceptions occurred during analysis:
[ERROR] 	DependencyNotFoundException: Unable to resolve system scoped dependency: client:client.bundle.source:eclipse-plugin:libs/uls-client_<IP_ADDRESS>.jar:1.0.0.qualifier:system
[ERROR] 	DependencyNotFoundException: Unable to resolve system scoped dependency: client:client.bundle.source:eclipse-plugin:libs/uls-common_<IP_ADDRESS>.jar:1.0.0.qualifier:system

@StefanHirche Hope to dig a bit deeper after finishing up reviews of the currently pending 8.0.0-targeted PRs, but initial looks in debug-mode make me suspect that it should be resolvable by some further updates to the resolution logic (won't make it into 8.0.0 yet which we intend to release this week, but hope to have it available for 8.1.0 (release date still to be decided on, depends on what else gets fixed in the near future))

Thanks,
just in case you still need this example, I just recreated the repo without the leaks, thanks for the notice.
I also upgraded maven and tycho (to 3.0.0), the issue still remains, but Java 17 is now needed.

@StefanHirche Can you validate my resolution from this branch (working for your 'pet-project') to be also working for the real-life Tycho pomless projects that you have before I file the PR for integrating it into the main development branch?

@jeremylong I think it would be good that this lands in the next release, so I marked it for 8.0.2 milestone, but unsure whether we should call it a fix (of broken setup with Tycho pomless) which would mean 8.0.2, or a feature/enhancement (now also supporting Tycho pomless extension to maven) which would mean 8.1.0
I tend mostly to a fix as in: we should not assume that maven extensions add system-scoped dependencies with a systemPath prefilled, but should rely on the extensions to also properly extend the artifact resolution for anything added as system-scoped.

@StefanHirche Can you validate my resolution from this branch (working for your 'pet-project') to be also working for the real-life Tycho pomless projects that you have before I file the PR for integrating it into the main development branch?

OK, I tested your noew changes and it seems to work. I don't get any errors anymore and all dependencies, also those provided directly in a plugin, show in the list of all dependencies.
Thanks for your work.

@aikebah I vote fix and just bump the patch level.


text2workspace speed improvements (81X)

Same as #393

There is a conflict now with this merge but it looks fairly trivial to solve. Are there updates (along with #393) yet?


libopenssl build failure

I am facing this build error on Windows machine.....
CMake Error at C:/amazon-kinesis-video-streams-producer-sdk-cpp/open-source/local/libopenssl/build/tmp/project_libopenssl-gitclone.cmake:52
(message): Failed to update submodules in: 'C:/amazon-kinesis-video-streams-producer-sdk-cpp/open-source/local/libopenssl/build/src/project_libopenssl'ninja: build stopped: subcommand failed.
We are trying to build SDK through MSVC (Visual Studio 2019).

Not really sure what might cause OpenSSL build failures. We have had some issues with parallel build built that usually gets past the failure after the retry. Are you able to replicate the script as in Travis.yaml file when the Travis Ci gets build for Windows?

Below is output log...
[CMake] fatal: unable to access 'https://boringssl.googlesource.com/boringssl/': SSL certificate problem: unable to get local issuer certificate
[CMake] fatal: clone of 'https://boringssl.googlesource.com/boringssl' into submodule path 'C:/amazon-kinesis-video-streams-producer-sdk-cpp/open-source/local/libopenssl/build/src/project_libopenssl/boringssl' failed
[CMake] Failed to clone 'boringssl'. Retry scheduled

@amitrjaiswal1 it seems that the issue is within your system, it failed to get public certificates. Could you try to use curl https://boringssl.googlesource.com/boringssl/? It should give you the same error.

@lherman-cs Got below error for 'curl https://boringssl.googlesource.com/boringssl/'
curl: (35) schannel: next InitializeSecurityContext failed: Unknown error (0x80092012) - The revocation function was unable to check revocation for the certificate.

@amitrjaiswal1 the build/runtime failure is not really related to KVS and we won't be able to help much. Perhaps you could ask the question in OpenSSL forums and they might help.
Alternatively, you could clean up your system and ensure you have a non-broken OpenSSL libraries..

@MushMal will do. Thanks for prompt update.

Did you manage to find a solution? Facing the same problem...


Parameterize ONNX model tests.

Progress on https://github.com/iree-org/iree-test-suites/issues/6. See how this is used downstream in https://github.com/iree-org/iree/pull/19524.
Overview
This replaces hardcoded flags like
iree_compile_flags = [
    "--iree-hal-target-backends=llvm-cpu",
    "--iree-llvmcpu-target-cpu=host",
]
iree_run_module_flags = [
    "--device=local-task",
]

and inlined marks like
@pytest.mark.xfail(raises=IreeCompileException)
def test_foo():

with a JSON config file passed to the test runner via the --test-config-file option or the IREE_TEST_CONFIG_FILE environment variable.
During test case collection, each test case name is looked up in the config file to determine what the expected outcome is, from ["skip" (special option), "pass", "fail-import", "fail-compile", "fail-run"]. By default, all tests are skipped. This design allows for out of tree testing to be performed using explicit test lists (encoded in a file, unlike the -k option), custom flags, and custom test expectations.
Design details
Compare this implementation with these others:

https://github.com/iree-org/iree-test-suites/tree/main/onnx_ops also uses config files, but with separate lists for skip_compile_tests, skip_run_tests, expected_compile_failures, and expected_run_failures. All tests are run by default.
https://github.com/nod-ai/SHARK-TestSuite/blob/main/alt_e2eshark/run.py uses --device=, --backend=, --target-chip=, and --test-filter= arguments. Arbitrary flags are not supported, and test expectations are also not supported, so there is no way to directly signal if tests are unexpectedly passing or failing.

I'm imagining a few usage styles:

Nightly testing in this repository, running all test cases and tracking the current test results in a checked in config file.

We could also go with an approach like https://github.com/nod-ai/SHARK-TestSuite/blob/main/alt_e2eshark/utils/check_regressions.py to diff test results but this encodes the test results in the config files rather than in external reports. I see pros and cons to both approaches.

Presubmit testing in https://github.com/iree-org/iree, running a subset of test cases that pass, ensuring that they do not start failing. We could also run with XFAIL to get early signal for tests that start to pass.

If we don't run with XFAIL then we don't need the generalized tests_and_expected_outcomes, we could just limit testing to only models that are passing.

Developer testing with arbitrary flags.

Misc
TODO:

[x] Write PR description
[ ] Add docs to README files
[ ] Add job matrix to workflow (needs runners in this repo with GPUs)
[ ] Iterate on tests_and_expected_outcomes ergonomics (nightly runs in this repo with xfails, then only passing tests in iree-org/iree? script to update the config with the latest xfails like for the onnx operator tests?)
[ ] Triage some of the failures (e.g. can adjust tolerances on Vulkan)

I'm looking for some early feedback, @zjgarvey or others. Early testing shows that running these tests on HIP would have caught some recent regressions. The specific mechanics used for setting flags, choosing which tests to run, and checking or reporting which stages passed/failed can be implemented in multiple ways.

I think it will probably be helpful to have the ability to specify extra test-specific options for an individual test through the json file.

Yes! However, I'm on the fence about specifically where to support extra options. We can add options in all these places:

In the test runner (conftest.py)
In individual test cases within files (test_foo.py)
In configuration files (config.json)
As parameters on environment variables (e.g. IREE_HIP_TEST_TARGET_CHIP)

I don't want to confuse developers with too many choices, but flexibility can helpful for a variety of situations.

E.g. for large models, it might be useful to pass additional importer and runtime flags for externalizing params.

For example here, that sounds like those options should apply regardless of the backend configuration, so the flags could go in the test cases.

Although I'm not sure if models that large are going to be in the scope of this testing suite right now.

I think they should be, as we can choose which tests to run on what schedule. How I have the PR right now allows for the list of tests to be fully opt-in using the default "skip" behavior. You can also filter with pytest:
# run only tests matching a string
-k resnet

# skip all large tests
-m "not size_large"

We can also use something like https://pypi.org/project/pytest-shard/ to run across multiple machines.

One thing I think will be rather helpful is to have a way to take a list of urls and automatically generate the test functions.

Yeah! I like how easy it looks to add test cases when they are in files like https://github.com/nod-ai/SHARK-TestSuite/blob/main/alt_e2eshark/onnx_tests/models/external_lists/onnx_model_zoo_computer_vision_1.txt. One of the tradeoffs I'm considering here is how it isn't obvious at all from a single test case name what the test is actually doing. I'd like for test suites to be forkable into user code, not be their own world of meta programming. The new tests that Rob added in https://github.com/iree-org/iree-test-suites/blob/main/sharktank_models/llama3.1/test_llama.py are on the other side of that spectrum:
Resemble user code                                               Templated to run lots of tests
<--------------------------------------------------------------------------------------------->

sharktank_models/                                          onnx_models/           alt_e2eshark/

What's in this test suite right now is closer to alt_e2eshark in that there is code for each test case... but that code is nearly all boilerplate:
def test_age_gender_gender_googlenet(compare_between_iree_and_onnxruntime):
    compare_between_iree_and_onnxruntime(
        model_url="https://github.com/onnx/models/raw/main/validated/vision/body_analysis/age_gender/models/gender_googlenet.onnx",
        artifacts_subdir=artifacts_subdir,
    )

I can iterate on further parameterization as you suggest here... maybe using pytest.mark.parametrize (docs here) or one of the more powerful options.
The ergonomics question is partially solved by

keeping the test runner code organized and documented
having the tests log what they are doing (iree-import-onnx -> iree-compile -> iree-run-module), since then users can run the tests and copy the log output for their own uses


Following update to Power BI, the option to show values with no data has been removed

[Enter feedback here]
There is no option to show items with no values in the new interface; at least, not that I can find.

Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

ID: 52f1d303-a1ee-766e-dac0-d68dad08080c
Version Independent ID: 053d3adb-90c2-03ce-87c7-47d441a1a79e
Content: Show items with no data in Power BI - Power BI
Content Source: powerbi-docs/create-reports/desktop-show-items-no-data.md
Service: powerbi
Sub-service: pbi-reports-dashboards
GitHub Login: @davidiseminger
Microsoft Alias: davidi

Thanks for your comment, @adam-pope-arup. This issue has been assigned to the doc author for review.

Thanks for your feedback, this feature is still operational. Since this may be a support request, you might find a solution in the Power BI Community, found at https://community.powerbi.com. If not, you can get support by going to https://support.powerbi.com.


Make stringification a parameter of the sorted_json helper functions

Move the 'quoted'ness of child values to be a function parameter instead of a local variable to avoid an issue where keys processed after a node_meta|meta|tags key were inadvertently quoted. This should allow it to track down the recursion as hopefully was intended.
The escape hatch for SRV weights should no longer be needed.
The base simple_generate() function has been simplified to jsonify either a string interpolation of int, float, and boolean or the actual value depending on the quoting request of the parent.
Fixes #526

@rtkennedy can you please rebase?

Rebased and added in the change to quote args' children per https://github.com/solarkennedy/puppet-consul/pull/548.

Hi,
it looks like you merged master into your branch instead of rebasing. Can you please do an actual rebase to have a clean git history.

I'll try, but I thought force pushes caused the PR to die.

in general it won't close the PR. there are edge cases, for example force pushing a branch that is equal to master. That's the only situation I'm aware of that will close the PR (and deleting the branch).

Hopefully that fixed it up, absent the commit message being less interesting than last time. I can amend it with the original text if that's useful.

Thanks @rtkennedy !
Is there any test case we can add to actually prove this fixes something?

I amended the commit message back to the original in case someone is looking through the logs outside of github. As for test cases, making a consul::service with tags and a port triggered the initial issue https://github.com/solarkennedy/puppet-consul/issues/526. But a better test might be a service with all the things, and running a 'consul validate' on the resulting config.

So uh, should I click the green button? :)

I think so? If there's something else for me to do let me know.


Chart is really small for some reason

I've tried messing around with the frame and sticking it in it's own view, but no change unfortunately

same here, any workaround ?


FIX-#1927: Fix performance issue related to sparse attribute access

Signed-off-by: Igoshev, Yaroslav<EMAIL_ADDRESS>
What do these changes do?

[x] commit message follows format outlined here
[x] passes flake8 modin
[x] passes black --check modin
[x] signed commit with git commit -s
[x] Resolves #1927
[x] tests added and passing

Time of hasattr for pandas.DataFrame.sparse = 0.00023566000163555145
Time of hasattr for modin.pandas.DataFrame.sparse = 3.1324103474617004e-05
 script
import pandas
import modin.pandas as pd

from timeit import default_timer as timer
import numpy as np

NCOLS = 2 ** 10
NROWS = 2 ** 10
RAND_LOW = 0
RAND_HIGH = 100
random_state = np.random.RandomState(seed=42)

def measure(df):
    start = timer()
    hasattr(df, "sparse")
    end = timer()
    return end - start

float_nan_data = {
    f"col{int((i - NCOLS /2)% NCOLS + 1)}": [
        x if (j % 4 == 0 and i > NCOLS // 2) or (j != i and i <= NCOLS // 2) else np.nan
        for j, x in enumerate(random_state.uniform(RAND_LOW, RAND_HIGH, size=(NROWS)))
   ]
   for i in range(NCOLS)
}

pdf = pandas.DataFrame(float_nan_data)
mdf = pd.DataFrame(float_nan_data)
p_time = np.min([measure(pdf) for _ in range(10)])
m_time = np.min([measure(mdf) for _ in range(10)])
print(f"pandas time = {p_time}")
print(f"modin time = {m_time}")

can we somehow add test_sparse_api_equality to pandas/test/test_api.py similary as test_dataframe_api_equality?

can we somehow add test_sparse_api_equality to pandas/test/test_api.py similary as test_dataframe_api_equality?

Probably, we could but I would prefer doing this after a structure of these classes becomes wider a bit and more similar because there are some differences related to their structure.

@YarShev was that made on purpose that Series sparse accessor missing npoints, density, fill_value and sp_values properties?

@YarShev was that made on purpose that Series sparse accessor missing npoints, density, fill_value and sp_values properties?

It is that we should work in below on. We will need to make an effort to implement full support for sparse API because pandas contains more complex logic for this. I think it is enough for the upcoming release.

It is that we should work in below on. We will need to make an effort to implement full support for sparse API because pandas contains more complex logic for this. I think it is enough to fix the issue for the upcoming release.

Should we file separate task(s) then?

It is that we should work in below on. We will need to make an effort to implement full support for sparse API because pandas contains more complex logic for this. I think it is enough to fix the issue for the upcoming release.

Well, then we're getting mismatching API with pandas. Can we just in case of hard-supported properties do something like this?
@property
def fill_value(self):
    return self._parent.to_pandas().sparse.fill_value

It is that we should work in below on. We will need to make an effort to implement full support for sparse API because pandas contains more complex logic for this. I think it is enough to fix the issue for the upcoming release.

Should we file separate task(s) then?

I think, yes.

It is that we should work in below on. We will need to make an effort to implement full support for sparse API because pandas contains more complex logic for this. I think it is enough to fix the issue for the upcoming release.

Well, then we're getting a mismatch with pandas API. Can we just in case of hard-supported properties do something like this for now?
@property
def fill_value(self):
    return self._parent.to_pandas().sparse.fill_value

Or raise not implemented error, if the solution above doesn't work

We can do them as defaulting to pandas for now.


Investigate [Kubernetes] Init Container Terminated With Error alerts

We're seeing a lot of [Kubernetes] Init Container Terminated With Error alerts from bc0192-dev namespace. Specifically, these are coming from CrunchyDB traction-database-dev Postgres cluster.
https://app.sysdigcloud.com/#/event-feed?last=1209600&scope=kube_namespace_label_name+in+("bc0192")+and+kube_namespace_label_environment+in+("dev")

Alerts are re-triggering even after manually resolving previous alerts from the same pod. This is without the pod changing.
The pods in OCP do not show evidence of Init containers terminating with errors.

The alert is specifically about the nss-wrapper-init container.

The state and reason of an init container can be seen using oc describe pod, for example oc -n bc0192-dev describe pod traction-database-dev-pg-tg79-0.  The current deployments are reporting State: Terminated and Reason: Completed, but these deployments are not the ones that triggered the alerts.

Alert was set to repeat every 10 minutes until resolved.  I turned that off to reduce the number of alerts.

The state and reason of an init container can be seen using oc describe pod, for example oc -n bc0192-dev describe pod traction-database-dev-pg-tg79-0. The current deployments are reporting State: Terminated and Reason: Completed, but these deployments are not the ones that triggered the alerts.

I could not find any history of pods that may have been replaced.
I've replaced the crunchy cluster earlier today, didn't see more of these alerts being triggered. So, hopefully the resource requests/limits changed from before have fixed it.

I see the value in having the metric, as it can aid in troubleshooting an issue with a pod. However, I do not understand the value of having an alert triggered from this query alone.
As per: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/#understanding-init-containers

If a Pod's init container fails, the kubelet repeatedly restarts that init container until it succeeds. However, if the Pod has a restartPolicy of Never, and an init container fails during startup of that Pod, Kubernetes treats the overall Pod as failed.

If the init container continues to fail the pod will have the status of: Init:CrashLoopBackOff
Do we really need to be alerted about Init Containers that terminate with errors?
Would it not be more valuable to be alerted if the pod itself is in CrashLoopBackOff, or if we see Init:CrashLoopBackOff status?

Hi @i5okie We already have an alert set-up when for  [Kubernetes] Pod Container Creating For a Long Time (Crashloopbackoff) status which triggered 43 times in past 20 days.

Hi @i5okie We already have an alert set-up when for [Kubernetes] Pod Container Creating For a Long Time (Crashloopbackoff) status which triggered 43 times in past 20 days.

Right, so my question is, if the issue with an initContainer will lead to the pod being in CrashLoopBackOff, resulting in an alert. do we still need to be alerted for the init container terminating with error?

Good point, If @WadeBarnes agrees, I can go ahead and delete the initContainer alert

Hello Rajpal & Dustin,
I took a deeper look at these alerts and they are indeed legitimate alerts.
Looking at your metrics dashboard and filtering by traction-database-dev deployment and do see quite a bit of activity that occurred starting around 2pm on Oct 10th. It does appear that this deployment ran into some resource limits which caused the pod/containers to restart.
Can you check the AGE of the pods within your bc0192-dev namespace and whether you see any restarts?
Thanks,
-Gilberto Rosario


Add first two sections: header and headline

I've added the header and headline section. It includes content, assets, and styling.

In addition to the HTML and CSS file, I've included the README file and MIT license.
These changes complete the "Portfolio: setup and mobile first" activity

Thank you! I made the corrections according to the observations of linters


Support numeric time index for Dask entityset

Previously time columns were always cast to datetime for Dask entities during DFS. This caused an issue with a numeric time index. This PR fixes that issue.
Fixes #991

The question above prompted a review of the test test_single_table_dask_entityset_cutoff_time_df which revealed that the entityset time indexes were never set in that test. Setting the time index caused the dask feature matrix to be different than the pandas feature matrix for the rows that had NaN values except for the pass through label and id value. This was due to a merge issue with the Dask implementation, which has been resolved with the latest commit fix dask test with cutoff times.


feat/Stage table for Pandas Dataframe

Is your feature request related to a problem? Please describe.
The tables extracted from source documents are represented as text and HTML tables. It's hard for users to transform and manipulate tables for specific rows, columns, and cells.
Describe the solution you'd like
Build a staging brick to load tables to Pandas Dataframe.
Describe alternatives you've considered
n/a
Additional context
Ref:

LlamaIndex Pandas Index
LangChain Pandas Dataframe

Pandas Dataframe can be useful for tabular data in CSV format too.
Ref: Feature Request from Slack community

duplicate #560


Changed rulesDirectory entry of tslint-microsoft-contrib to be peer...

...friendly. Fixes #30

Let's get this in :) Easy win for those of us who also use tslint-microsoft-contrib in our projects.

AppVeyor build is currently failing: https://ci.appveyor.com/project/Glavin001/tslint-clean-code/build/1.0.43#L332
Running "tslint:prod" (tslint) task
Warning: Could not find custom rule directory: C:\projects\tslint-clean-code\tslint-microsoft-contrib Use --force to continue.

And it appears Travis CI decided to skip testing:

Here's my PR: https://github.com/Glavin001/tslint-clean-code/pull/34
Please take a look and 👍 and/or comment! I will wait for some feedback before merging and publishing.

Merged https://github.com/Glavin001/tslint-clean-code/pull/34 instead.
Thank you for contributing and bringing this to my attention!


"Open in Windows Terminal" of WT Preview disappears if WT is not installed

Windows Terminal version (or Windows build number)
Windows 10.0.21354.0
Windows Terminal Preview v1.8.1032.0
Other Software
No response
Steps to reproduce
Uninstall Windows Terminal
Expected Behavior
Only "Open in Windows Terminal" of WT disappears
Actual Behavior
"Open in Windows Terminal" of both WT and WT Preview disappear

What versions of the Terminal did you have installed? 1.6...? 1.7...?
Have you rebooted since installing/uninstalling? explorer.exe is notoriously aggressive about caching context menu entries.

@zadjii-msft Rebooting doesn't help
Just install Windows Terminal can get them back, but I just want to see the one of WT Preview

Again, which version of Windows Terminal Preview, release, do you have installed? We just changed how those dialogs are registered, but the Store isn't always the best at rolling out updates. This might be a weird interaction between the 1.8 preview and the 1.7 stable, but I'll need some version numbers to be sure

Windows Terminal Preview v1.8.1032.0
(I added them to the issue description just now

I think I had Windows Terminal v1.7.1033.0 installed but I'm not sure since I can't install it from Microsoft Store now
Windows Terminal Preview installed by msixbundle from GitHub Release

Can confirm - clean install of Insider build 21364, Terminal Preview 1.8.1032.0.
If the non-preview version is not installed, the context menu entry does not appear. If I install it, both regular and preview entries appear.

I can repro this also.  I have 1.7.1091.0 and 1.8.1092.0 both installed using bundles from GitHub.  With this

uninstalling 1.7.1091 removes both context menu items.  Reinstalling 1.7.1091 brings both back.

I hit this problem the other day, both versions installed from the store, same Preview version as the original bug report, but I didn't notice the non-preview version number before I uninstalled it. This is Windows 10 20H2 (19042.928).
After a reboot or two after uninstalling the non-Preview version (and the last reboot was a system-crash, so know it was a clean boot), I happened to notice that my "Open in Windows Terminal" option is back, just for the Preview version (the only one I have installed), so I assume this symptom can be triggered by Explorer caching stuff. In that session I was also removing "X here" registry keys for things like Git Bash or MSYS (since I want to run those in Terminal, not conhost) and it was taking effect immediately, but I assume things registered by CLSID rather than a 'shell' registry key are more likely to be cached until reboot.
Not that useful for @Henry-ZHR, but for anyone else who finds this ticket with the same symptoms, check again after next system startup.

What can we do until then? Please provide a .reg file or PowerShell command to fix that!

My home-brewed solution works.  I have an unzip-only installation of WT. I use this this with similar entries in HKEY_CLASSES_ROOT\Directory\Background, HKEY_CLASSES_ROOT\Drive, and HKEY_CLASSES_ROOT\Folder.  I reckon you'll have to (at least) use the path to your preferred shell, change the "*cdd" command to something appropriate, and duplicate this for the other three keys.  If WT has a command line option for start-up directory, or you don't mind getting the default start-up shell and directory, you might be able to omit the shell and its command.  I don't know if that will work; I haven't tried it.
An entry in "...\Background" lets you start WT from the right-click context menu for the background in an Explorer window and on the desktop itself (which is a directory background).
Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Directory\shell\WT]
@="WTHere"

[HKEY_CLASSES_ROOT\Directory\shell\WT\command]
@="wt.exe nt d:\\tc28\\tcc.exe *cdd \"%V\""

I also have this, which makes life nice.  Without this you might (above) have to supply a path to windowsterminal.exe.
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\App Paths\wt.exe]
@="d:\\wt\\exe\\windowsterminal.exe"
"Path"=" d:\\wt\\exe"


Fullscreen

Subtitles are hidden on fullscreen.
To prevent this, you have to override/rewrite the fullscreen method,
so the div will be fullscreen and not only the video.

https://github.com/weizhenye/ASS/issues/2#issuecomment-271158016

So you have to use a library like video.js or overwrite the controls to solve this issue?

I think there is no way to support fullscreen from video. When clicking fullscreen button inside <video>, only <video> itself can be display on screen, any other elements is hidden.
You should write fullscreen button yourself, like many video libraries do.

Okay, thanks.


Can you provide complete docker command example

My Environment

ArangoDB Version:        3.6
Storage Engine:
Deployment Mode:
Deployment Strategy:
Configuration:
Infrastructure:
Operating System:
Total RAM in your machine:
Disks in use:
Used Package:

Component, Query & Data
Affected feature:

AQL query (if applicable):
AQL explain (if applicable):

Dataset:

Size of your Dataset on disk:

Replication Factor & Number of Shards (Cluster only):

Steps to reproduce

Problem:
can you provide complete docker command example？
like:
docker run -e ARANGO_ROOT_PASSWORD=xxx -p <IP_ADDRESS>:8530:8529 arangodb/arangodb arangod --server.endpoint tcp://<IP_ADDRESS>:8529
include: -v /mnt/test:/var/lib/arangodb3
I don't be familiar with docker.
thank you!
Expected result:

like:
-v /mnt/test:/var/lib/arangodb3
-v /mnt/test2:/var/lib/arangodb3-apps
is above right?

/etc/arangodb3
need config -v?
like:
-v /mnt/test3:/etc/arangodb3

i  wnat to work on this issue

we try to keep https://hub.docker.com/_/arangodb accurate with the basic documentation.
You can as well use the arangodb starter to launch docker containers: https://www.arangodb.com/docs/stable/deployment-cluster-using-the-starter.html#using-the-arangodb-starter-in-docker


Executing a meta-target just hangs

Using version 1.0, upon sake combo where combo is the name of a meta-target, it just seems to hang; if I removed the meta-target and just execute the individual tasks, it's fine:
> cat Sakefile
combo:
  help: combo
  uno:
    help: uno
    formula: echo "UNO"
  dos:
    help: dos
    formula: echo "DOS"

> sake -v combo
Call to check_integrity issued
Sakefile passes integrity test
Going to construct Graph
Adding 'dos'
Adding 'uno'
Nodes are built
Building connections
checking node dos for dependencies
checking node uno for dependencies
# hangs right here

I was using Python 2.7.14, pip installing master-sake to see this behavior. I can provide a Dockerfile to repro this if it isn't reproducible on your end.

Hmm, it also seems if you just run sake (without combo) it works, too

Fixed as of 6d4e726ad4932f
Thanks so much for bringing this to my attention :)

Let me know if you run into any problems with it. I'll be submitting to PyPI later today


Allow gift card codes to be created with a starting balance

The gift_card resource now accepts a current_balance parameter on gift card code creation:
https://api.foxycart.com/rels/gift_card
Let's get that added to the element to make things easier for folks creating multiple gift card codes with starting balances.

Fixed in a6ca793bff2231c21ec669653a97ca2509dd3d60


[iOS] Option to automatically generate icons and launch screens

3.x cherry-pick of https://github.com/godotengine/godot/pull/34140.
Note that generate_missing is still disabled by default, like in its master counterpart. I'm not sure whether we should enable it by default. cc @bruvzg

Hello!
I'm testing it with this PR's Github actions build...
Test Environment:
MacBook Pro 13 - 2018 Big Sur Version 11.4
Xcode 12.4 - Build version 12D4e
Is the idea of the feature to generate only-blank placeholder images or scale down from a certain input image?
If so, is there an expected min-size image to be used as an input for auto generation?
XCode Proj was generated with blank images

Then I tried to build it, but XCode got a parse error to open the project files.

Also, I've only marked Required Icons in Editor

Attached test project here.
godot-game-template.zip


Add OnlyOffice themes

@bphd could you provide some information.
Does OnlyOffice currently has a theming engine. Is there CLI to change these themes?

Currently OO has 3 default themes, not personalizable, it's in the parameters
This enhancement could permit automatic adaptation of the UI with schedule or with system theme
Later it will be OO job to ass live theme change to not necessitate restart anymore


Add changelog

Similar to what we now have in climaparameters and the Julia NEWS.md

Closed by #2773


GRIDSS support

Hello
I am looking at adding GRIDSS to parliament, but it appears that survivor does not support GRIDSS output. AFAIK, GRIDSS is fully compliant with VCFv4.3 but there are a number of issues I've encountered:

GRIDSS reports all events in BND notation (including DEL and DUP events)
GRIDSS reports supporting/reference fragment counts, not read counts. This approach gives more accurate VAFs as it allows soft clips to be incorporated via assembly, and prevents double-counting of fragments that support a variant as both a split read and a discordant read pair.
How does survivor define an INV event? An actual inversion involves two breakpoints. Are the two orientation treated as independent calls?
GRIDSS does not perform any genotyping
GRIDSS uses the CIPOS field to encode the exact breakpoint homology interval (as the spec-defined HOMSEQ and HOMLEN fields are insufficient)
GRIDSS reports single breakends.

Is there any chance SURVIVOR could be updated to handle GRIDSS VCFs?

Hi,
sorry for the late reply.
So SURVIVOR currently tries to translate a BND event either in an INV or in a TRA event internally. This is determined if the two breakpoints are on the same chr or not.
So only the first point of your list is basically relevant and also only if you require a type save merge.
This is because a DEL is for me not the same as a BND although I know that a BND can be everything according to the standard.
I dont want to open up this constraint to be type save especially since it is a parameter already.
I hope that makes sense.
Thanks
Fritz

@d-cameron - did you ever find a solution to this? I am trying to merge SVs from a population discovered via GRIDSS and would be helpful to use survivor

No I didn't get around to doing this. I did a lot of benchmarking (PMID: 31324872) and came to the conclusion that a) GRIDSS and manta (the ones that incorporate assembly) are the only tools really worth running, b) X of Y ensembles are pretty bad, and c) the value in ensemble calling is in the additional work that is done (e.g. parliament running SVTyper).
Good ensemble callers seem to effectively turn into (targetted) callers in their own right, so my time was better spent making GRIDSS better, and improving the downstream interpretation (e.g. see https://www.biorxiv.org/content/10.1101/781013v1 for our work on complex event interpretation)

If it's just GRIDSS calls from multiple samples, merging should be relatively straight-forward since you don't have to deal with the mess of merging from multiple callers each of which uses their own dialect. I use BioConductor StructuralVariantAnnotation package for this, although if you're dealing with hundreds or thousands of samples R would probably run out of memory when jointly processing full GRIDSS unfiltered call sets (since the full VCFs are ~1GB per sample)

Thanks @d-cameron . I think my only problem with GRIDSS is that I find it hard to interpret that all events are BND. It is true that this is standard confirm, but how do I distinguish if a gene is e.g. deleted or inverted? I never got around to see what information GRIDSS provides to distinguish that.
 You are right Manta does a good job, but the other SV caller also improves above that for certain SV that are missed initially.
I will read up your paper, but would be great if you can extend ob your points that ensembles methods are not worth the time. Does not seem to be the case for everyone else, but maybe we are overlooking something.
Thanks
Fritz

@d-cameron I know a population can be merged using structural variant annotation R, but it would be nice to have them as a VCF for a few reasons - to have the VAF calculations that you suggested for genotyping and to use the VCF format without downstream analysis that require a VCF format. Indeed I am working with 100s of samples, (that expect to be prohibitive to R memory) and that is why I'm hoping to merge them as VCF using an existing tools (like SURVIVOR). Do you have any recommendation? Can this actually be done with StructuralVariantAnnotation -  I have had problems trying to use the "breakpointGRangesToVCF" function?  I appreciate your input before I launch into writing some custom scripts.

@zlye are you just using GRIDSS? I believe the merge across samples should still work. Can you try a subset and let me know if not?
Please pay attention if SURVIVOR wrongly converts BND to INV.
Thanks
Fritz

but the other SV caller also improves above that for certain SV that are missed initially.

We also found you could improve performance with a simple X of Y ensemble. The reason that I'm so pessimistic about simple ensemble calling is that that a) all the best ensembles had both manta and GRIDSS, b) the top ensembles had too many callers (e.g 7 of 10, or 6 of 9), and c) the top ensembles were not stable - the 'best' for HG002 could be worse than GRIDSS alone for NA12878.

I think my only problem with GRIDSS is that I find it hard to interpret that all events are BND.

By design, it doesn't. Fundamentally, GRIDSS is a breakpoint caller and since all it does it detect breakpoints. My choice of BND notation was intentional as it makes it quite clear what the actual claim is. Calling a deletion from breakpoint support alone is misleading because you haven't actually checked that the segment is actually deleted - it could be part of a copy number neutral complex rearrangement. For pretty much every NGS caller (lumpy being an exception), DEL just means a an intra-chromosomal breakpoint in +- orientation.

how do I distinguish if a gene is e.g. deleted or inverted?

You can't and shouldn't do that by just iteratively looking individual breakpoints in isolation. Here is an example of a complex event that GRIDSS/PURPLE/LINX fully resolves (the outer line is the reconstructed derivative chromosome and the inner tracks indicate CN and LOH):

Want to know what happened to you gene of interest? You need to know everything that happens in that genomic region.
Fundamentally, the problem is due to a lack of separation of concerns. Primitive (breakpoint, CN) identification and event interpretation have been confounded and treated as the same thing when in fact they are two very different things. We're actually discussing this in the GA4GH file formats group and I'm currently drafting up some big changes to SV representation for VCFv4.4. If you have time, let me know what you think of the proposed changes (https://github.com/samtools/hts-specs/pull/465) as it'd be great to have eyeballs familar with the mess that is SV calling looking at it.

I will read up your paper, but would be great if you can extend ob your points that ensembles methods are not worth the time. Does not seem to be the case for everyone else, but maybe we are overlooking something.

One of the PhD students in our lab is doing ML ensemble calling and the getting really good results out of that. The thing is, she's also getting really good results with single-caller ML quality score recalibration. Ensemble-ness itself doesn't actually give you that much - it's all about running a sensitive caller and controlling your FP rate. I wouldn't be surprised if she was able to outperform all traditional ensembles with a single-caller ML qual recalibration using manta, GRIDSS, or pindel. Yes, you'll get slightly better performance by doing an ensemble on top of that, but you'll get most of the way there off a single caller.
A good example of why tuning callers to your data is the PCAWG consensus somatic call set vs somatic GRIDSS/PURPLE/LINX on the Hartwig metastatic cancer cohort. PCAWG spent a lot of time and effort on the ensemble meta-caller - we spent a lot of effort tuning our single caller pipeline for our cohort. Using unexplained CN transition as a proxy for sensitivity, we found that we missed 2% of somatic SVs, whereas PCAWG missed 20%. Subsetting to PCAWG clonal calls for a more fair comparison (we did deeper sequencing), they still missed 13%. Better algorithms beat ensemble calling every time. Why did we do so well? Well, when 7% of your calls are to repetative sequence (e.g. centromeres), ensemble calling doesn't help for those calls - you need a caller that reports single breakends (hence why GRIDSS2 does single breakend calling).
All that said, where ensemble calling does shine is in merging disparate callers. Take an SVMerge style approach and merging the results from a general purpose SV caller, a retrotransposon caller, and a microsatellite caller, a viral/novel insertion caller and an indel caller makes a lot of sense. Combining multiple callers that use the same evidence and perform the same steps gives you a much lower return on your investment. These callers will tend to have similar results, including sharing the same false positives (PMID: 31324872 Fig 3b). It's simpler to just run a good caller and spend your time identifying and filtering systematic false positives.

I get the distinct feeling that ensemble calling is popular amougst the large consortia for political reasons. The last time I attempting to convince a consortium I was involved in to switch pipelines I got so much pushback that I abandoned that attempt and it turned into just one more caller for their ensemble.

Nice writeup. I still value in ensemble calls for cases where you need an all purpose method and are not able to tweak parameter settings to your specific library prep / sample quality / sequencing method, but that's an aside. Are you planning to put some of those examples on a blog? Would be really good to have a writeup somewhere!

@d-cameron I know a population can be merged using structural variant annotation R, but it would be nice to have them as a VCF for a few reasons - to have the VAF calculations that you suggested for genotyping and to use the VCF format without downstream analysis that require a VCF format. Indeed I am working with 100s of samples, (that expect to be prohibitive to R memory) and that is why I'm hoping to merge them as VCF using an existing tools (like SURVIVOR).

Do you have any recommendation? Can this actually be done with StructuralVariantAnnotation - I have had problems trying to use the "breakpointGRangesToVCF" function? I appreciate your input before I launch into writing some custom scripts.

breakpointGRangesToVCF doesn't work (yet).
If I had your cohort, I'd do, GRIDSS joint multisample calling on any related samples then single-sample filtering to remove the low VAF noise and get the files to a managable size. From there you can either run SURVIVOR (assuming it runs happily), or use StructuralVariantAnnotation to calculate mutual overlap. You might get away with just renaming the shared variants to have the same name in all samples then something like vcfools to merge into a single VCF (disclaimer: do not know if this works - have not tried it myself). Depending on what you want to do, it might even be easiest not both with merging and just use the ID field for uniqness (same ID = same variant) in much the same manner dbSNP as unique variant identifiers.
VCF doesn't scale well to many samples. Since both # variants and # FORMAT columns scale with the number of sample, space used scales super-linearly (mostly full of lots of zeros for rare variants). It's something that's being looked at for VCF currently looking at.
SURIVIOR

Thanks for your insight - I am not totally sure what you meant when you
said:
"Depending on what you want to do, it might even be easiest not both with
merging and just use the ID field for uniqness (same ID = same variant) in
much the same manner dbSNP as unique variant identifiers."
On Sun, Mar 22, 2020 at 5:26 AM Daniel Cameron<EMAIL_ADDRESS>wrote:

@d-cameron
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_d-2Dcameron&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=163oVAfqYW8rLwmeRfPScQ&m=8wwrSkCzVHJAYdDY47d-scV0JK34rXouhcO6twfdg3Y&s=64Wk7SRcOcuITXMdh_BITRpmxpKO5aMFuu-dGaVEiPI&e=
I know a population can be merged using structural variant annotation R,
but it would be nice to have them as a VCF for a few reasons - to have the
VAF calculations that you suggested for genotyping and to use the VCF
format without downstream analysis that require a VCF format. Indeed I am
working with 100s of samples, (that expect to be prohibitive to R memory)
and that is why I'm hoping to merge them as VCF using an existing tools
(like SURVIVOR).
Do you have any recommendation? Can this actually be done with
StructuralVariantAnnotation - I have had problems trying to use the
"breakpointGRangesToVCF" function? I appreciate your input before I launch
into writing some custom scripts.
breakpointGRangesToVCF doesn't work (yet).
If I had your cohort, I'd do, GRIDSS joint multisample calling on any
related samples then single-sample filtering to remove the low VAF noise
and get the files to a managable size. From there you can either run
SURVIVOR (assuming it runs happily), or use StructuralVariantAnnotation to
calculate mutual overlap. You might get away with just renaming the shared
variants to have the same name in all samples then something like vcfools
to merge into a single VCF (disclaimer: do not know if this works - have
not tried it myself). Depending on what you want to do, it might even be
easiest not both with merging and just use the ID field for uniqness (same
ID = same variant) in much the same manner dbSNP as unique variant
identifiers.
VCF doesn't scale well to many samples. Since both # variants and # FORMAT
columns scale with the number of sample, space used scales super-linearly
(mostly full of lots of zeros for rare variants). It's something that's
being looked at for VCF currently looking at.
SURIVIOR
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_fritzsedlazeck_SURVIVOR_issues_92-23issuecomment-2D602170352&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=163oVAfqYW8rLwmeRfPScQ&m=8wwrSkCzVHJAYdDY47d-scV0JK34rXouhcO6twfdg3Y&s=Zb6VwasND_6pLSBOPuYNTRTt6_AF7tdjiHbneJazOLE&e=,
or unsubscribe
https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AKFUYU4DWETS2HY42W6SSUDRIXKWBANCNFSM4HXEWEUA&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=163oVAfqYW8rLwmeRfPScQ&m=8wwrSkCzVHJAYdDY47d-scV0JK34rXouhcO6twfdg3Y&s=P8Lsn5QLaX70alIiVB4gQBKqblFiHtIsw-Vj64ZqQvY&e=
.

--
Zoe Lye
P.h.D. candidate
12 Waverly Place
New York University
New York, NY 10003

Sorry just catching up. Thanks for the insights @d-cameron
@DEL vs. INV:  you can detect if a region is deleted or not. Just based on coverage. Manta etc. has that included. It is true that some of the larger variations get tricky...
@Ensamble caller: I think the real strength of the ensemble calling is maybe not sensitivity (I think you are focusing on that and many other callers (MetaSV)) but precision.  Not to use a union but a consensus. If two caller agree on the same SV its more likely to be correct. Exceptions are alignment artifacts, but that is always problematic.
Franky, I think you are quite deep in cancer calling these breakpoints. Which is fair and important. General SV calling also needs to detect SV that plays importance in population settings etc. You cannot just say there are no deletions and duplications. This is simply not true. Sorry if I misunderstood.
With VCF scaling: that is true. The same problem goes for SNV. There are some techniques but hard to implement for SVs (eg. gVCF).
@GRIDSS: when you say by default it does not let you detect del vs. bnd: Do you have a parameter?
Thanks
Fritz

If two caller agree on the same SV its more likely to be correct.

This issue with this approach is that similar callers make the same FP calls (https://www.nature.com/articles/s41467-019-11146-4/figures/3). The tail of TPs called by few callers is smaller than the head of FP calls called by many callers. That said, those results are NA12878 so if those FPs are actually TP omissions from the true set used in the benchmark, this will be less of a problem. Ensemble calling works well when your input caller call different calls. GRIDSS and manta are already close to the pareto-optimial curve for simple ensembles so the value of a simple ensemble over those callers is much less than for callers such as breakdancer, delly and lumpy.

when you say by default it does not let you detect del vs. bnd: Do you have a parameter?

GRIDSS reports all breakpoints in BND notation.

del vs. INV: you can detect if a region is deleted or not. Just based on coverage. Manta etc. has that included. It is true that some of the larger variations get tricky...

I was under the impression that it was only lumpy that actually looked at the depth of the deleted segment. There's nothing about coverage/read depth filtering in either than manta user manual or manuscript. It has a high-depth filter and reports coverage at both breakends, but it doesn't actually check a DEL has a CN loss or a DUP has a CN gain over the segment in question.

Franky, I think you are quite deep in cancer calling these breakpoints. Which is fair and important. General SV calling also needs to detect SV that plays importance in population settings etc. You cannot just say there are no deletions and duplications. This is simply not true. Sorry if I misunderstood.

Even in the cancer setting most variants are simple DEL or DUP. Assuming a +- breakpoint is a DEL, and a -+ breakpoint is a DUP is an assumption that is probably correct for somatic analysis and almost always correct for germline analysis. The reason GRIDSS reports in BND notation is because that's what the GRIDSS evidence supports. A DEL call is an implicit claim of both a +- breakpoint and a copy number loss. If a caller hasn't check both of these, it shouldn't make a DEL call.


feat(EC2): New EC2 AWS check (#852)

Adds check to find exposed secrets in EC2 Launch Templates. Currently, Prowler does not scan EC2 Launch Templates.
Context
The existing check autoscaling_find_secrets_ec2_launch_configuration only considers EC2 AutoScaling launch configurations, which are deprecated and can no longer be created. The replacement feature - Launch Templates - is currently not checked by Prowler. However, just like launch configurations, launch templates can also contain UserData that might include secrets.
Issue #852 raised this issue in 2021 but failed to result in relevant changes to Prowler. The issue was closed due to inactivity.
Description

Updates the EC2 service to include launch templates, including all versions.
Adds check to search for secrets in EC2 launch templates
Adds relevant tests for EC2 service and added check
Adds new check to relevant compliance benchmarks, where autoscaling_find_secrets_ec2_launch_configuration was included already.
Updates the pentesting.md tutorial document.

Fixes #852
License
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Hi @rieck-srlabs why did you close it?
Thanks!

@jfagoagas, had some trouble with flake8. Lets continue in the new PR (#4076)


Removed the console expression

It is considered a best practice to avoid using any of console's methods in JavaScript code that will run on the browser. Usually, console methods are only used for debugging and can leak internal info to the client.

Thanks for the refactoring, but I would be glad if you may let us know where are these deliverables as per your project.

@iamrajiv I would like to request you to create an issue for volunteers and org members to communicate with you like this

Hi, @Rohitesh-Kumar-Jain I have organized my work in this repository https://github.com/iamrajiv/GSoD-2021 do checkout. Also, refer to my repository proposal, not the blog because that was made suddenly. Also, after going through the repository of gRPC and OpenAPI I got to know some works are already done like CLA and DCO bots, etc so I have updated my proposal accordingly.
Apart from it, I will also create an issue that describes my project more concisely.

I have organized my work in this repository https://github.com/iamrajiv/GSoD-2021 do checkout.

I wasn't aware of that, but we didn't ask you to put so much effort into creating that repository.

Also, refer to my repository proposal, not the blog because that was made suddenly. Also, after going through the repository of gRPC and OpenAPI I got to know some works are already done like CLA and DCO bots, etc so I have updated my proposal accordingly.

For orgs reference you need to update the blog post, I have been taking reference from that blog post since the start of the program, you can't modify your proposal without letting the org know (making a PR to modify the proposal on wechaty).
Had you raised a PR for editing your proposal (blog post), we would have become aware of the changes, and may have had some discussion. You can't edit proposals on your own.

Apart from it, I will also create an issue that describes my project more concisely.

No need to describe your project, maintaining followups is enough (no need to maintain your repo aswell). I want to keep track of our conversations that's why asking for an issue. Communication like this (in PRs) or in mail gets lost with time, and is not open to the community.

@Rohitesh-Kumar-Jain I created the repository for my work record so that at last it will be easy to make a solid work report for my project. I would have created PR for the updated proposal but there were minor updations in the proposal and also PRs are not getting merge so I thought of creating at last in one PR with all changes like even I have to add my project links to other blog posts since my project was selected and decided at last and that time other blogs which describe project and proposals for GSoD 2021 are already merged.
For my pending works, I have done the changes just have to push and create the PRs I complete it by tonight. Also, for some PRs even I am in confusion whom to request for reviews.

PRs related to documentation should request a review from Simin.
PRs related to code refactoring like this would be reviewed by the community ig.

It is considered a best practice to avoid using any of console's methods in JavaScript code that will run on the browser.

As Huan said it is not running inside the browser, I am closing this PR


added legal pages as reusable components

Trello: https://trello.com/c/VgBHk5v8
We use the legal pages in both Gatsby (as the static legal pages) and in the main app (as part of the legal modal component). This was causing issues where there was not a single source of truth for the content. Case in point, when adding these to radiance and comparing the dates updated on the two repos, there was an instance where the information on the main app was updated and the gatsby one was not.
From here, we can use the following components in either repo:
<TermsOfService />
<PrivacyPolicy />
<Disclaimer />
<TelehealthConsent />
<ParentGuardianTelehealthConsent />
<LegalContainer />

The last component would be used for various sweepstakes / promotions terms pages so they can easily have the same styling as our other legal pages.

@ZeMunchkin What do you think of creating a separate repo for this and the guides config that should be shared between PocketDerm and Gatsby? It doesn't feel like this belongs in Radiance

@snags88 Yeah, Matt and I went back a forth a bit on this while pairing yesterday. I agree this doesn't 100% fit into radiance, and I could go either way. The reason we initially landed on putting into radiance is because both forms of content do have a reusable component aspect to them. The legal page's LegalContainer (which is sort of a stretch as far as arguments go), and the guide content has the GuideCard. GuideCard actually takes a couple different forms currently, so it would resemble our reusable components in radiance quite a bit more than LegalContainer.
But it definitely makes sense to just break out and make another repo for this. Any fun name ideas? :)

@ZeMunchkin makes sense. My main concern, especially for the legal components, is that this is open source so anyone can make PRs against it. I think merging is locked down to Curology Engineers, but just a precaution if we ever allow others outside of the organization to do things in this repo

@snags88 That's an interesting point that we didn't talk about... I guess I'm thinking about how it would look as a separate repo if that repo was not open-source. A published package was a nice way of having the content available at build time and ready to use in React components. It looks like NPM has a private registry, so that might be an option?: https://www.npmjs.com/products/enterprise
The other thing to consider is that Gatsby is SSR, so we have to ping an API at build time. And If we had a separate repo for content that we were making API calls to instead, we'd need to figure out the mechanics of doing that. It will introduce graphql into our codebase: https://www.gatsbyjs.org/docs/client-data-fetching/
The content we'd get from an API would have to be in JSON format (I believe?), so we'd still need to add the reusable components (LegalContainer and GuideCard) to radiance so that we can plug in the data we get from the API into those. We'd have to think a bit about how to format the legal content in a way that we could easily have it render in the components, since there's a lot of different html elements that are nested. And there's an argument that the content, how to parse it, and the component it that renders it are very tightly coupled and should be kept together.

Hmm interesting. It looks like we can ping the pocketderm api from gatsby so we can get the info from our servers if we need (https://www.gatsbyjs.org/docs/using-gatsby-without-graphql/)
As for the actual components, it makes sense to keep them in a centralized repository if the style is always going to be the same


Any Java samples?

I am trying to implement a custom QR like the ones provided in the examples (Instagram to be specific) code in Java. I have attempted to use the builder but was not successful.
can any one provide me with examples?
thanks.

These example are outdated, they use deprecated features.
There are no java examples in this repo but you can find something useful here

thanks. I will take a look!

thanks! I got something working!


Move autograph into a separate package and into a separate repo

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): any
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: n/a
TensorFlow installed from (source or binary): any
TensorFlow version (use command below): any
Python version: any
Bazel version (if compiling from source): any
GCC/Compiler version (if compiling from source): any
CUDA/cuDNN version: any
GPU model and memory: any
Exact command to reproduce: n/a

Describe the problem
I guess autograph should be moved into a separate package and repo. Because I guess it may be possible to add support of other computation graph frameworks there.

Indeed, and we plan to do that very soon. Due to time constraints, we've initially moved it inside TF core, and from there we'll extract the reusable parts into a standalone repo. We hope to have that set up by the end of this year.

@KOLANICH,
Sorry for the late response. The autograph moved to a new repo and it can be accessed as tf.autograph. Thank you.

Thank you, and everyone involved into it. :tada:

The autograph moved to a new repo and it can be accessed as tf.autograph. Thank you.

It hasn't been moved to a new repo:

This repository contains tests and example code

Yes, this is still an open issue (I think @chunduriv was referring to autograph being available in its own namespace). Mainly because we need someone to set up and maintain a separate repo - otherwise, significant parts of the source code (tensorflow/python/autograph/pyct) are already independent of TF and could be mover rather mechanically.

Commenting here for visibility. We have made a static copy of the autograph codebase for use in a framework which builds on autograph to convert Python functions into OpenQASM programs for quantum computations. We did not want to take a dependency on TensorFlow, so the quickest path forward was copying the codebase and making minor modifications to decouple it from TensorFlow. We would be very interested in consuming autograph as a standalone package, if it ever becomes available, and would be happy to help validate the work with our use case.
Here is our copy of the autograph codebase:
https://github.com/amazon-braket/amazon-braket-sdk-python-staging/tree/feature/autoqasm/src/braket/experimental/autoqasm/autograph
More details on how we are using it:
https://github.com/amazon-braket/amazon-braket-sdk-python-staging/blob/feature/autoqasm/src/braket/experimental/autoqasm/README.md

Hey all, having autograph available as a separate repo would also be super valuable/important from our end. We're in a similar position to @rmshaffer -- we're depending on Autograph in Catalyst, but don't want to have to depend on the full TensorFlow installation for users of our package.
Would it be possible to split autograph out -- without the TensorFlow dependency -- into its own PyPI package? We would be very happy to help out here :)

It's certainly possible to split it out - autograph was designed with that in mind. Besides some dependencies in the high level API that accumulated as technical debt and should be easy to remove, only the operators are TF-specific, so as long as you have replacement for those (which I suspect you want anyway), a split should definitely be doable.
I'm not sure if there are any concrete plans to do such a split, so I'd recommend taking the lead and doing it.

Hello TensorFlow maintainers, I'm currently tackling the separation of TensorFlow and AutoGraph as a contribution to TF.
I have a question regarding the preferred code organization, for which I could see the following options:

keep all code within the tensorflow repo, while separating out build & packaging processes etc.

Following this model, either of the below could still be taken as a follow up.

keep all code within the tensorflow repo (as above), but also make a mirror available in a separate repository

I believe this was the model followed by mlir-hlo, although that code has now moved from tensorflow to xla.

move everything to a different repository, for instance the existing autograph repository

I believe this is also the model followed by the tensorflow estimator package, which is also available as a separate pip package.

I think the last option makes most sense, at least in the long run, although a clean move would not be easy to achieve. It would probably be most practical to start with a fork, and continue improvements there, while in parallel configuring TF for an external dependency.

Hi,
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.

Hi @tilakrayal, thank you for checking in. I don't think the latest TensorFlow release is resolving this issue, which is about separately packaging and releasing one of the sub-components of TensorFlow.


test: lint error/warning tests

Test to validate error message
Test to validate warning message
Test to validate the linting error

Test coverage results :test_tube:
:white_circle: Total coverage has not changed
// Code coverage diff between base branch:release and head branch: test/lintTests

Status
File
% Stmts
% Branch
% Funcs
% Lines

:green_circle:
app/client/src/utils/autocomplete/TernServer.ts
50.71 (0.24)
39.13 (0.87)
36.21 (0)
55.35 (0.27)

I am holding off the review as there are few changes expected for linting ex warnings will be disabled


Flatbuffers may be interested in accepting this upstream

Would you consider incorporating this tool directly into flatbuffers itself? It seems several people are interested here: https://github.com/google/flatbuffers/issues/7.

I have given it some thought, but having a C schema compiler was a more important goal when the project started.
Personally I need to put a limit to how much additional work I put into this project as there are other pressing issues, but for anyone who wants to take it on:
There are several ways to go about an integration:

write the codegenerator in C++ for flatc and reuse the runtime parts.

rewrite the codegenerator to use binary schema and interface this with flatc.

add the libflatcc library as is and call directly from flatc cli.

alternative binary schema.

Was avoided initially so projects didn't have to rely on a C++11 compiler and retrofitting it is a lot of work, but possible. There is great risk of the flatcc and flatc code generators quickly becoming incompatible. So it will also be difficult to maintain - not really a good option.

was initially expected to be supported at some point. But I am not sure the binary schema has enough information. For example it doesn't order structs by dependency. That requires more work in code generation.

This is currently the only practical short term solution. Error messages would be different, but the flatc could pre-check the schema before passing it on. That would be very simple to do since all path logic is compatible. Any edge cases in compatibility would have to be sorted as they are discovered, but that is only a good thing. However, I am not sure that Googles flatc project would want to make a major special case for one language backend - and it would blow up the code size significantly. It could work as an optional build option that pulls source from FlatCC github and enables C output in the command line interface.

I am envisioning an alternative binary schema that includes hash tables for fast json parsing and which might also be a replacement for option 2) above. This requires a flatc backend to a new binary schema format that flatcc also uses internally. This would be the best solution and would work for other code generators in other languages as well. However, this is a long term objective and not something to expect any time soon and would require coordination with Googles FlatBuffers project. This would also clean up the flatcc codegenerator which currently works on obscure parser AST graphs.

But even with 4) there is very little benefit over simply passing the schema on to libflatcc - parsing text is after all not such a big deal in itself and I am not sure an alternative schema is worth the effort.

We'd be happy to have C support directly in the main project. And while I understand the value of an all-C toolchain, for maintainability anything going into the main project would have to use the C++ schema parser.
So 1) is preferable, and 2) would be acceptable, but I don't see the point, since reading from the binary schema is not necessarily easier than just reading the C++ parser state.
3) is a maintenance problem, in which case I'd rather just point people to your project if they want to use C.
4) I is unlikely to happen.

This is understandable.

is simpler not because it might be conceived as easier than the C++ interface, but because both schema frontend parsers can generate it such that the code generator can be shared and thus be maintained. The code generator would still be in C but use whatever frontend parser is attached. For MSVC as frontend target, a bit care is required.

As it is 2) is insufficient or at least suboptimal - but it doesn't have to be total rewrite to enhance the schema:
a) there need to be an order of structs - at least the order supplied in the original schema. The current listing is sorted for lookup by binary search. It can probably be worked around using forward struct declarations, but and additional vector in the schema with all symbols in original schema order would be useful - also for regenerating the text schema.
b) there need to be a list of include files in order of dependency - again - the order of includes in the schema is sufficient - without this either everything gets mashed together into a single file, or the user must manually add include statements for dependent schema.
c) there need to be a scope reference for each symbol. At can just be a dotted prefix in the name - FlatCC does this by default in the binary schema output, but can strip it.
d) I am not quite sure if dependent symbols are needed at code generation level, but if they are present there need to be a way to know if they are dependencies or something part of the current schema being generated.
With this covered, I think the code generator has enough information, just on top of my head.

a) The binary search order is important for speed in most reflection cases. I think the better approach is if the C generator wants to work from this data, that it does its own sort.
b) Yes, the schema is self contained, and this makes sense for its use case. Replicating how the C++ parser handles seems like a very fragile thing to me.
c) / d)  This is not data that most reflection uses care for, so.. can be done, but not ideal.
It seems like ensuring the binary schema holds all information the parser provides is more pain than it is worth. A better solution would be, if you want to keep sharing code between the two projects, is to put the bulk of your code generator in a .c file with an interface, then have specialized code call into that from both flatc and flatcc.

a) sorting is trivial because flatcc has sort builtin sort logic to do in-place unstable heap sort within the generated reader code, but as you say, it is not needed in the code generation step. What is needed is a guarantee that it is not sorted, then it would work.
I think you suggestion about a common interface is good. Some data structures will be needed and whether they are C arrays of structs, or another flatbuffer schema is not really important as long as all the issues a) .. d) are being addressed somehow.

After considering the interface structure required for an integration, and the many small details required, it does not seem worthwhile for the time being. I am in favor of an extended binary schema format that all parses can output to and all code generators can generate from, but this is an extended effort for everyone involved so I will let it rest for now and close the issue.


Not supported by Swagger 2.0: Multiple operations with path 'v1/api/Admin/' and method 'POST'.

I installed Swashbuckle v5.5.2 in a new Web.API project that is .Net 4.6.1.  With no controllers, it worked with no errors.  However, after I added a controller with a Get and a Put method, I get the following error in Swashbuckle:
500 : {
"Message": "An error has occurred.",
"ExceptionMessage": "Not supported by Swagger 2.0: Multiple operations with path 'v1/api/Admin/' and method 'POST'. See the config setting - "ResolveConflictingActions" for a potential workaround",
"ExceptionType": "System.NotSupportedException",
"StackTrace": " at Swashbuckle.Swagger.SwaggerGeneratorOptions.DefaultConflictingActionsResolver(IEnumerable1 apiDescriptions)\r\n at Swashbuckle.Swagger.SwaggerGenerator.CreatePathItem(IEnumerable1 apiDescriptions, SchemaRegistry schemaRegistry)\r\n at Swashbuckle.Swagger.SwaggerGenerator.<>c__DisplayClass7.b__4(IGrouping2 group)\r\n at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable1 source, Func2 keySelector, Func2 elementSelector, IEqualityComparer`1 comparer)\r\n at Swashbuckle.Swagger.SwaggerGenerator.GetSwagger(String rootUrl, String apiVersion)\r\n at Swashbuckle.Application.SwaggerDocsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at System.Net.Http.HttpMessageInvoker.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at System.Web.Http.Dispatcher.HttpRoutingDispatcher.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at System.Net.Http.DelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)\r\n at System.Web.Http.HttpServer.d__0.MoveNext()"
}
http://localhost:20923/swagger/docs/v1
The problem is, there is no Admin controller anywhere in the project.  The controller I added does not have any POST methods at all.

I found the controller.  It was coming from RavenDB DLLs.

@stricq  How did you filter the RavenDB DLls?  I am experiencing the same kind of issue.
Thanks, kdcllc

I uncommented the ResolveConflictingActions line in the config.


Ignores some Tailwind classes while accepting others

Namely, I wanted to use hover:bg-gray-200 (or any other bg color) class from Tailwind on the Card component to highlight the card mouse hovers over among many other cards, and it absolutely ignores it.
hover:cursor-pointer works just fine. I had to resort to force-injecting style via CSS file, and even then it only worked once I used !important tag:
.card-hover:hover {
background-image: #cacaca !important;
}
It still didn't work without the !important flag.

Could you put together a codesandbox example?
I am also using themes + tailwind and it is working great!

Could be helpful: #16

You can use Tailwind’s important modifier, e.g. hover:!bg-gray-200 to strong-hand any CSS.
That said—if you can't achieve the customisation that you need by tweaking the theme variables and just need custom styles on the components, you are probably better off creating a quick component yourself. It’s easier and more maintainable than fighting the built-in styles. The point of Radix Themes is to build apps faster by skipping a lot of styling and design choices. If you want different design choices for a particular component, creating your own version is always safer.


YT Broadcast query failed: Error: No filter selected. Expected one of: default, id, mine

I'm trying to use this plugin with companion on RPI but after configuring and enabling the plugin I get

Broadcast query failed: Error: No filter selected. Expected one of: default, id, mine

I'm using version 3.0.0 (3.0.0+6009-beta-874bd267)

In case anyone else faces this issue, I faced it also. I accidentally selected the wrong YouTube Channel when I authenticated my account through companion. The fix for me was reauthenticating and selecting the correct YouTube Channel.
I am not sure what went wrong with the initial YouTube account that I selected - perhaps it is because there were no currently or scheduled Live videos. Or Perhaps it is because that channel does not have livestreaming enabled.
Either way I looked around for a solution for this for ages and didn't find anything, so I wanted to write this in the hopes that it helps someone else who may have accidentally selected the wrong channel.

Do you still have the issue in the latest versions?

Ever since I migrated from brand account access to user permissions (see here) I have been unable to select my channel when regenerating the Authentication Key. It was either the migration of the brand account access or it was Googles change to the new sign in page.
Essentially I have a user<EMAIL_ADDRESS>which is a personal google account, and it has a YouTube channel. However it also has permissions to access a brand/business YouTube channel, and whenever I try and authenticate now days it defaults to select the personal channel rather than selecting the other YouTube channel, so I have not been able to use this plugin for some time. It did work a while ago though.


Fix duplicate Product entries in Android CommerceEvents

Summary
Fix the problem where Products (and Impression and Promotions) where being added twice when they aligned with the productAction/promotionAction/impression(1st)
This bug should be pretty obvious when you look at the fix; we were adding the 0th index item that aligned with the constructor, but then later looping through all the items and adding them individually. This meant that the 0th index item would be added twice.
The fix here was to remove the 0th item after we add it the first time, then proceed as normal

Moving this to the private repo


Update rates - finish pets

Update the finish pets to the more common methods of obtaining the pets.

These were coded as being the most common training methods, not the most common methods of farming pets, which are usually NOT the most common training methods. And most people just get pets from their regular training methods.

These were coded as being the most common training methods, not the most common methods of farming pets, which are usually NOT the most common training methods. And most people just get pets from their regular training methods.

Think it's rare people chop magic trees or fish monkfish anymore really but understandable I just made the pr as it was suggested to update them as they're outdated really.

Mind runes and volcanic mine?

most people do not get the pets from regular training methods, you dont even get close to the rates by having 99 in a skill. So post 99 you would go for the farming pet methods,.
These rates are taken from the osrs petcord (a server showing the quickest way to get each pet) and excluding any alt or tick manipulation methods. See mind runes and VM below.

maybe we could add a method: option? or something similar
/finish input:Heron pet method:Karambwans
for example

Like I said though, these were coded as being the "meta way to hunt pets", they're coded as being the most recognizable, common places that people get them.


Add test server mindev command

Summary
This adds the ability for mindev to spawn an all-in-one minder test server.
This server has noop authn/z and uses an embedded postgres database to operate.
This is meant for integration testing and is not for production, hence why it was
added in mindev.
Change Type
Mark the type of change your PR introduces:

[ ] Bug fix (resolves an issue without affecting existing features)
[x] Feature (adds new functionality without breaking changes)
[ ] Breaking change (may impact existing functionalities or require documentation updates)
[ ] Documentation (updates or additions to documentation)
[ ] Refactoring or test improvements (no bug fixes or new functionality)

Testing
Outline how the changes were tested, including steps to reproduce and any relevant configurations.
Attach screenshots if helpful.
Review Checklist:

[x] Reviewed my own code for quality and clarity.
[ ] Added comments to complex or tricky code sections.
[ ] Updated any affected documentation.
[ ] Included tests that validate the fix or feature.
[ ] Checked that related changes are merged.

coverage: 39.755% (-0.04%) from 39.796%
when pulling dd1200c4e07147ce659f64382f0c6c1a582ab44b on all-in-one
into 5518d12e40fdd3120dd4eac143a4782c17cde862 on main.


the result is terrible

Having run the code here. The result is terrible even with the exampled image.
Have you checked the result

The function to load the image was removed from scipy. I replaced it by imread() from imageio. The reconstrucion of the example image test.png should be identical to the image up to translation. I also added an example to demonstrate the importance of oversampling.


Produce more than ~1000 records per second?

Hi,
I am trying to test what throughput I can expect with my currently available hardware and configured numbers of brokers / partitions / replication factor, etc.
However, using this monitor, it seems like I can only lower "produce.record.delay.ms" to 1, which obviously means the most that I am able to produce/consume is 1000 records / second.
Is there any way of improving this?  I tried changing the value of "produce.record.delay.ms" to 0, and the entire thing stops working (all values 0.0).  Can I spin up one more than one producer, for example?
Thank you for your help.

@rjharvey kafka-monitor is intended to be used for monitoring availability and functionality of the kafka service, and therefore should not need to send more than 1000 messages a second.
If you want to stress test the maximum throughput of your kafka cluster, kafka-monitor is probably not the right tool for you even if produce.record.delay.ms can be set to 0. Can you try using bin/kafka-producer-perf-test.sh in the Apache Kafka's repository to stress test your cluster?

@lindong28 Ok, thank you very much.


Add ability to put custom default value instead of now()

issue https://github.com/ng2-ui/ng2-datetime-picker/issues/63


Status: no errors with WiFi off

First report works but then spam provides empty report

Upon turning off hotspot, error flickers onto screen
Maybe make distinct until change again or denounce the emit by a bit


Easee: resume charge is requested, while charger is waiting for authentication

Describe the bug
evcc continuously sends resume commands to the charger, although the charger is not ready and awaiting authentication via RFID/App. This leads to superfluous API calls on every interval. This is in so far unfortunate, as the charger opMode is known and we can easily prevent this.
Steps to reproduce

Enable authentication on the charger
Plugin in car
Do not authenticate via App or RFID tag
evcc continously sends resume and settings requests that have no effect

Configuration details
N/A

Log details
not needed, can reproduce myself

What type of operating system are you running?
Docker container
Version
0.118.1

@andig please assign to me


Migrate some IGNITION_* macros

🦟 Bug fix
Part of https://github.com/gazebo-tooling/release-tools/issues/698
Summary
The IGNITION_* macros from the autogenerated Export.hh have been migrated to support GZ_* as well as of https://github.com/gazebosim/gz-cmake/pull/249, so the GZ_UTILS_VISIBLE macro is now being used (https://github.com/gazebosim/gz-utils/commit/97a1cfa12fc7e2ebf9c0c4142736531779929706). This also tick-tocks most of the macros in config.hh.in (notably not the *_BUILD_TYPE_* macros, which are problematic) and migrates to GZ_UTILS_VERSION_NAMESPACE where relevant.
Checklist

[X] Signed all commits for DCO
[ ] Added tests
[ ] Updated documentation (as needed)
[ ] Updated migration guide (as needed)
[ ] Consider updating Python bindings (if the library has them)
[X] codecheck passed (See contributing)
[X] All tests passed (See test coverage)
[ ] While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

LGTM, I guess there's no way to trigger a warning when the IGNITION_ versions are used, right? It's more of a slient-tick and then tock, but I guess that's what we can do.

once you mentioned that, it occurred to me that we could move the IGNITION_* macros to ignition/utils/config.hh, so I did so in https://github.com/gazebosim/gz-utils/pull/62/commits/06f7b943b063181dd5603e45b1d8040c16b0db52
how does that look?


MEI-1st cheat sheet

Perhaps not the best place to report it, but the MEI 1st webpage:
http://music-encoding.org/support/tutorials/mei-1st/exploring-the-mei-header
Contains a link to a PDF:
http://music-encoding.org/wp-content/uploads/2015/03/CheatSheetHeader.pdf
The schematic is great; however, the use of the asterisk symbol is poorly chosen: The asterisk typically means "0 or more"(i.e., optional) in XML DTD, while the footnote on the PDF says it means "required":

It would be better to choose a different symbol for representing a mandatory element.  Perhaps a dagger (†) instead of an asterisk? — as your file will die if it does not include those elements.
It would be even more useful if it were slightly more technical and gave the standard schema symbols used for each element: * for 0 or more, + for one or more, ? for optional, nothing for mandatory.  But that might get too complex when considering larger-scale patterns of elements.

Kristina Richts is the keeper of the MEI-1st tutorial.  I'll email her to see if/when she can address this.

OK, I will bug her if I notice problems in the tutorial :-)  I will be reading the header tutorial today...

As MEI 1st in no longer in placeI will close this issue and mark it as wontfix.

As MEI 1st in no longer in placeI will close this issue and mark it as wontfix.