Introduce Partitioned aggregation mode

Is your feature request related to a problem or challenge?
Currently, we often use two modes in aggregations:Partial + FinalPartitioned. FinalPartitioned requires the input to be hash-repartioned, so a RepartionExec is added in between.
In certain cases, like when only aggregating on a single column, it is faster to skip the Partial aggregation and directly perform the aggregation on hash-partitioned input, as doing the Partial + RepartionExec + FinalPartitioned  will be more work than doing the aggregation in one step (RepartionExec + Partitioned).
Reasoning: RepartionExec (hash) is itself faster than Partial (although it doesn't reduce the output) and is necessary for FinalPartitioned.
Describe the solution you'd like

Introduce Partitioned aggregation mode that requires input to be partitioned on the groupby-keys.
Utilize it in certain cases, like aggregations utilizing only one/few columns (consider queries such asSELECT COUNT(DISTINCT a) FROM T)

Describe alternatives you've considered
No response
Additional context
No response

cc @mingmwang  who I think was considering something similar recently

Working on this one

Probably https://github.com/apache/arrow-datafusion/issues/6937 makes more sense to do than doing this statically, closing  this issue

I'm not sure for TPC-H queries whether this will help or not, but for some TPC-DS queries, this is quite useful.

I think at least q17 has a high cardinality grouping that it might help for (in the subquery), for what it is worth

Yes that is the idea

I found out the Single aggregation mode which already does what we want to do (do aggregation in one go), so there is no need to create

I did some experiments skipping the Partial based on heuristic (e.g. for tables up to a number of columns), but this gets mixed results:

┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃ fast_gby_hash ┃ aggregate_partition_mode ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │      194.12ms │                 194.38ms │     no change │
│ QQuery 2     │       36.59ms │                  30.54ms │ +1.20x faster │
│ QQuery 3     │       44.99ms │                  45.69ms │     no change │
│ QQuery 4     │       37.34ms │                  37.88ms │     no change │
│ QQuery 5     │       91.69ms │                  90.49ms │     no change │
│ QQuery 6     │       10.27ms │                  10.25ms │     no change │
│ QQuery 7     │      193.05ms │                 190.58ms │     no change │
│ QQuery 8     │       69.37ms │                  69.39ms │     no change │
│ QQuery 9     │      132.29ms │                 132.95ms │     no change │
│ QQuery 10    │       91.51ms │                  90.86ms │     no change │
│ QQuery 11    │       40.53ms │                  39.73ms │     no change │
│ QQuery 12    │       67.70ms │                  66.71ms │     no change │
│ QQuery 13    │      130.96ms │                 132.62ms │     no change │
│ QQuery 14    │       11.87ms │                  12.10ms │     no change │
│ QQuery 15    │       14.80ms │                  19.92ms │  1.35x slower │
│ QQuery 16    │       37.79ms │                  37.07ms │     no change │
│ QQuery 17    │      210.67ms │                 209.54ms │     no change │
│ QQuery 18    │      315.60ms │                 381.18ms │  1.21x slower │
│ QQuery 19    │       57.40ms │                  57.59ms │     no change │
│ QQuery 20    │       70.88ms │                  58.71ms │ +1.21x faster │
│ QQuery 21    │      248.35ms │                 252.92ms │     no change │
│ QQuery 22    │       28.11ms │                  27.87ms │     no change │
└──────────────┴───────────────┴──────────────────────────┴───────────────┘

My hope is better for https://github.com/apache/arrow-datafusion/issues/6937 which I think might be similar to the "adaptive partial aggregation" of snowflake / teradata?

Based on the experiment I think the only way to do this reliable enough during planning is to have cardinality statistics on the group by columns, so the cardinality of the aggregation can be estimated.

Based on the experiment I think the only way to do this reliable enough during planning is to have cardinality statistics on the group by columns, so the cardinality of the aggregation can be estimated.

I think it is for this reason that most people argue for doing the adaptation at runtime (not plan time) because the statistics always have edge cases (like data skew) that can result in poor plans without any degree of predictability


Add external network white list for port mapping

Depends on https://github.com/hyperhq/hyperstart/pull/114 and https://github.com/hyperhq/runv/pull/246.
Example pod
{
    "containers": [
        {
            "image": "nginx",
            "name": "test",
            "ports": [
                {
                    "containerPort": 80,
                    "hostPort": 8080,
                    "protocol": "tcp"
                },
                {
                    "containerPort": 443,
                    "protocol": "tcp"
                }
            ]
        },
        {
            "image": "insaniti/alpine_iptables",
            "command": [
                "/bin/sh"
            ]
        }
    ],
    "hostname": "test",
    "portmappingWhiteLists": {
        "internalNetworks": [
            "<IP_ADDRESS>/24"
        ],
        "externalNetworks": [
            "<IP_ADDRESS>/24"
        ]
    },
    "dns": [
        "<IP_ADDRESS>"
    ]
}

Rules for this Pod is
# Generated by iptables-save v1.4.21 on Fri Jun  3 07:17:28 2016
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:hyperstart-PREROUTING - [0:0]
-A PREROUTING -j hyperstart-PREROUTING
-A hyperstart-PREROUTING -s <IP_ADDRESS>/24 -p tcp -m tcp --dport 8080 -j REDIRECT --to-ports 80
-A hyperstart-PREROUTING -j RETURN
COMMIT
# Completed on Fri Jun  3 07:17:28 2016
# Generated by iptables-save v1.4.21 on Fri Jun  3 07:17:28 2016
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:hyperstart-INPUT - [0:0]
-A INPUT -j hyperstart-INPUT
-A hyperstart-INPUT -s <IP_ADDRESS>/24 -p tcp -m tcp --dport 443 -j ACCEPT
-A hyperstart-INPUT -s <IP_ADDRESS>/24 -p tcp -m tcp --dport 80 -j ACCEPT
-A hyperstart-INPUT -s <IP_ADDRESS>/24 -p tcp -m tcp --dport 80 -j ACCEPT
-A hyperstart-INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A hyperstart-INPUT -p icmp -j ACCEPT
-A hyperstart-INPUT -i lo -j ACCEPT
-A hyperstart-INPUT -j DROP
COMMIT
# Completed on Fri Jun  3 07:17:28 2016

let's merge and test it.


Similar to where the main function should be？

I put the code in the OnStart function，thank

I'm not supporting this repository anymore.
Additionally, when you create an issue, please provide a good description for it. What the issue is, how to reproduce it, what the effects are and such.


java: add restore endpoint

After restore text message:

After restore text message:

Currently, it's difficult for me to determine why you received this error because I'm running this example in containers and the restore endpoint is functioning properly. I've shared my sketches of docker files for this example, so you might want to try using them to test the restore endpoint and potentially identify the cause of the error you're experiencing.
Below, I've attached the output of the storage directory. I created multiple versions, restored one, and then created another one.

Show output

$ pwd
/usr/local/tomcat/webapps/ROOT

$ find app_data | sort
app_data
app_data/<IP_ADDRESS>
app_data/<IP_ADDRESS>/new.docx
app_data/<IP_ADDRESS>/new.docx-hist
app_data/<IP_ADDRESS>/new.docx-hist/1
app_data/<IP_ADDRESS>/new.docx-hist/1/changes.json
app_data/<IP_ADDRESS>/new.docx-hist/1/diff.zip
app_data/<IP_ADDRESS>/new.docx-hist/1/key.txt
app_data/<IP_ADDRESS>/new.docx-hist/1/prev.docx
app_data/<IP_ADDRESS>/new.docx-hist/2
app_data/<IP_ADDRESS>/new.docx-hist/2/changes.json
app_data/<IP_ADDRESS>/new.docx-hist/2/diff.zip
app_data/<IP_ADDRESS>/new.docx-hist/2/key.txt
app_data/<IP_ADDRESS>/new.docx-hist/2/prev.docx
app_data/<IP_ADDRESS>/new.docx-hist/3
app_data/<IP_ADDRESS>/new.docx-hist/3/changes.json
app_data/<IP_ADDRESS>/new.docx-hist/3/key.txt
app_data/<IP_ADDRESS>/new.docx-hist/3/prev.docx
app_data/<IP_ADDRESS>/new.docx-hist/4
app_data/<IP_ADDRESS>/new.docx-hist/4/changes.json
app_data/<IP_ADDRESS>/new.docx-hist/4/diff.zip
app_data/<IP_ADDRESS>/new.docx-hist/4/key.txt
app_data/<IP_ADDRESS>/new.docx-hist/4/prev.docx
app_data/<IP_ADDRESS>/new.docx-hist/createdInfo.json

Fixed file creation for Windows.

@LinneyS, is ready to merge.


Random playlist menu should not use percentages

Keep percentages behind the scenes, but present integers to the user

closed by pull request #146


Doesn't seem to accept API key

Signed up for an API key here: https://opendata.metlink.org.nz/dashboard
Added the home assistant integration, paste API key, no matter what I try it won't accept it:
"The api key provided is not valid. Check you have subscribed to the Metlink Open Data API."
Only thing is I have signed up the the Metlink Open Data API.

Oops I see a closed issue describing the exact problem I was having.

Works now, figured it out by looking at closed issues.


alders, fuchsine

This pull request was generated by the 'mq' tool
[test]
flake rate: 0.1
logical conflict every: 100
sleep for: 300s
close stale after: 4 hours
[pullrequest]
requests per hour: 30
deps=[a,f]

/trunk merge


[ceres] Add feature schur

This small PR adds a default feature schur to control the SCHUR_SPECIALIZATIONS option in Ceres. It remains on by default.
As described briefly in the documentation, SCHUR_SPECIALIZATIONS enables some template specializations which improve runtime performance of the SPARSE_SCHUR solver, but it requires a large number of explicit instantiations and generates large object files—large enough to require /bigobj in MSVC. This can cause issues with some compilers particularly in constrained environments.

[x] Changes comply with the maintainer guide.
[ ] SHA512s are updated for each updated download.
[ ] The "supports" clause reflects platforms that may be fixed by this new version.
[ ] Any fixed CI baseline entries are removed from that file.
[ ] Any patches that are no longer applied are deleted from the port's directory.
[x] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[x] Only one version is added to each modified port's versions file.

If you are ready to merge, please click "ready for review". This feature test passed.

I don't really think this is a feature, it behaves more like an alternative. Notably, you can't ever add constraints that features are off. At a minimum, the sense should be flipped. That is, add a non-default feature no-schur-specializations. That way you aren't broken when there are ports that don't say default-features: false.
@ras0219-msft points out that the triplet variable VCPKG_CMAKE_CONFIGURE_OPTIONS can already do this. If you do if (PORT STREQUAL "ceres") set(VCPKG_CMAKE_CONFIGURE_OPTIONS "-DSCHUR_SPECIALIZATIONS=OFF") endif() you get this without any port modifications.


How can I filter ALL telemetry within request scope?

Using telemetry processor I can filter my requests according to certain attributes, for example query string parameters or request path, which is really useful. However if I don't care about this path, what I really would like to do is ignore all telemetry for this request path, not just requests.
I would be useful if I could get to an instance of RequestTelemetry from ITelemetry.Context.OperationContext. Then in my telemetry processor I could do something like this:
if (IsIgnoredPath(item.Context.Operation.ParentRequest.Uri))
{
 return;
}
else
{
 this.Next.Process(item);
}

Is there any way I can do it today?

I guess this question is related to Web SDK rather than Core (since in Core depending on how you create telemetry item you would have your own method of sharing it with the rest of your code)
For Web SDK

If you are interested in a Path. That would ne your HttpContext.Current.Request.Path that you are interested in
If there are some properties in RequestTelemetry you should use HttpContext extension method: System.Web.HttpContextExtension.GetRequestTelemetry


Fixes for all of Jo's comments

Note that although this commit includes changing update to change I'm not
sure how to test this with the current test infrastructure.

@stefanpenner @joliss this should resolve all the issues noted on #34.
A couple of things

Removing the guard against duplicate files will not cause the tests to fail until stefanpenner/fs-tree-diff#11 is in and released.
I haven't tested the change command, which previously used the wrong name update.  Is there a straightforward way of testing that with the current merge-trees testing infrastructure?

Removing the guard against duplicate files will not cause the tests to fail until stefanpenner/fs-tree-diff#11 is in and released.

it's merged, release at will.

I haven't tested the change command, which previously used the wrong name update. Is there a straightforward way of testing that with the current merge-trees testing infrastructure?

this is a good @joliss question

it's merged, release at will.

released

@joliss does this PR look like it addresses your concerns for #34?

Yup, thanks! Looks pretty good, just haven't gotten around to reviewing. Will do soon.

Well it rethrows, so the normal error logging should cover us I believe

@stefanpenner good point

@hjdivad we could add extra debug logging. Let me do that..

I haven't tested the change command, which previously used the wrong name update. Is there a straightforward way of testing that with the current merge-trees testing infrastructure?

I'd use the fixture.Builder from broccoli-fixture to build, change files, and then build again.
This might be useful to test - other than that I think this is good-ish to merge.

I think my bigger worry about the way this is implemented right now - and I assume you share this - is that a bunch of the generic patch-application logic still lives in merge-trees. This makes it hard to reason about the correctness, and hard to test as well. If this logic lived in fs-tree-diff, it would be easier to test exhaustively: We could generate a set of n sample tree states (like currentTree objects) containing files, linked dirs and real dirs with different names and capitalizations, and then we could test that for all $n$ * $n$ possible state transitions, the patching algorithm arrives at the correct output.
I'm pretty sure that something like this is what we eventually want to arrive at, but maybe you don't have the time for this right now, so we can also merge this PR now and then evolve it later.

@joliss this logic is unique to this plugin, each other consumer has different implementation here.

Merging this into the make-things-faster branch. Discussion can continue there


[REF] Fixes Cammoun et al., 2012 Freesurfer files

Previous surface .annot files for the Cammoun 2012 parcellation were out-of-sync with the volumetric images. This fixes them and addresses a whole bunch of other changes that were resolved in the processing of that fix.

Codecov Report

Merging #31 into master will increase coverage by 0.21%.
The diff coverage is 7.14%.

@@            Coverage Diff             @@
##           master      #31      +/-   ##
==========================================
+ Coverage   49.55%   49.77%   +0.21%
==========================================
  Files          17       17
  Lines         890      870      -20
==========================================
- Hits          441      433       -8
+ Misses        449      437      -12

Impacted Files
Coverage Δ

netneurotools/datasets/utils.py
87.5% <ø> (ø)
:arrow_up:

netneurotools/tests/test_datasets.py
98% <ø> (ø)
:arrow_up:

netneurotools/freesurfer.py
20% <0%> (+3.11%)
:arrow_up:

netneurotools/plotting.py
7.82% <0%> (-2.98%)
:arrow_down:

netneurotools/datasets/datasets.py
98.9% <100%> (ø)
:arrow_up:

netneurotools/stats.py
46% <0%> (-3.46%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 335b99f...8d34c51. Read the comment docs.


Using custom speech model using C++ (How-to question)

I have deployed a custom speech model and am accessing it via C++, but it does not seem to be working.

(The custom speech model has product names, like "ecocollect" for instance, but these are being returned in the transcript as "eco collect" which makes me think it is not detecting according to the model).

I have the endpoint id from when I deployed the model, and in my C++ code I am referencing it like this:
auto speechConfig  = SpeechConfig::FromSubscription(subscriptionKey, region);
speechConfig->SetEndpointId(endpointId);

Is this correct, or is there something more or different I need to do?

That seems correct to me, but I have not used custom models much.
The public sample for this topic can be found at https://github.com/Azure-Samples/cognitive-services-speech-sdk/blob/19fe4407c427b161ec628eae5f4c6540f8a7982d/samples/cpp/windows/console/samples/speech_recognition_samples.cpp#L216-L228
I suspect the model just isn't doing what you expect it to.
I will ask about the best ways to test your model to make sure it will behave the way you expect, and get back to you.

great thanks!

@davehorton
What is your end goal with the custom speech model?
Do you only need to recognize a half dozen product and company names more accurately, or do you need to do something more complex like a large set of terms like medical terms and phrases?
Here is some information that might help.
If you just need to improve recognition of a few product names, you might be able to keep using the general topic public speech endpoint, and add a PhraseListGrammar before you start recognition.  You can find more information on PhraseListGrammars here, and it is cheaper and easier to use than creating a custom speech recognition model and deploying it on your own private endpoint.  We have a public sample for PhraseListGrammars here.
If you do need a full custom recognition model, try testing your deployed models with the Speech Command line tool
If you run this SPX command line "spx help csr examples", you will give see some examples to create a Custom Speech Recognition(CSR) endpoint, and how to to test it's recognition output from command line.  I would suggest using SPX to test your current model to ensure it generates the basic output you expect, and then move on to integrate that model/endpoint into your C++ application.


Attempting to use event dispatcher.

I am just exploring this concept. It could be used to improve the stability of merge_to_main workflow.

Thinking about this better... I think I'm going to go for nightly docs builds


Filtering on a nested collection's field, with surrounding parentheses and exactly 2 passed filter values for the field, generates an incorrect query

Version
2.8.1
Details
I'm filtering on a nested collection, mapped like so:
_gridifyMapper = new GridifyMapper<Model>(true);
_gridifyMapper.AddMap("AppId", a => a.Applications.Select(app => app.AppId));

If I pass it a filter such as appId=1,appId=2, I get something like this (which I think is correct):
WHERE (EXISTS (
        SELECT 1
        FROM [UserApplications] AS [d0]
        WHERE ([u].[UserId] = [d0].[UserId]) AND ([d0].[AppId] = @__Value_2)) AND EXISTS (
        SELECT 1
        FROM [UserApplications] AS [d1]
        WHERE ([u].[UserId] = [d1].[UserId]) AND ([d1].[AppId] = @__Value_3))

However, if I surround the filter string with parentheses, it instead becomes this (which seems incorrect):
WHERE (EXISTS (
        SELECT 1
        FROM [UserApplications] AS [d0]
        WHERE ([u].[UserId] = [d0].[UserId]) AND (([d0].[AppId] = @__Value_2) AND ([d0].[AppId] = @__Value_3))

If I then add a third appId in the filter string, still surrounded with parentheses, it generates the correct query, again.
Seems like a bug to me?
I'm using EF Core 6, if that makes any difference.
Steps to reproduce

Create a mapper and map a field to a nested collection.
Query with a filter string such as (field=value1,field=value2) (parentheses are important, and it has to be exactly 2 values passed (1 or (3 or more) will generate the correct query).

Hi @sebdm,
This is not a bug, but rather a design choice; in this case, your parenthesis is redundant.
when you don't use any parenthesis by default you get something like the below query:
var query = list.Where(q => q.Any(w=> w. field == value)  && q.Any(w=> w. field2 == value2)  );

but sometimes we want both conditions inside the Any function. like:
var query = list.Where(q => q.Any(w=> w. field == value && w. field2 == value2) );

So we can use a pair of parenthesis to group two conditions in one expression. (field=value1,field=value2). if you want to have separate expressions with parenthesis you have to add another set of parenthesis around the query. (field=value1),(field=value2)
Let me know if you solved the problem or not.

First of all, thanks for your quick response (and for a great library :))
These do what I want:
field=value1,field=value2
(field=value1),(field=value2)
These don't:
(field=value1,field=value2)
((field=value1),(field=value2))
So basically this is limiting me a bit when it comes to some edge cases.
Maybe it would be more clear with a different operator for "IN", e g:
field IN value1,value2
Or perhaps even cleaner if I didn't need to map the nested field to the top-level at all, e g:
child.field IN value1,value2
Or for multiple where's on the same IN:
child.field as F IN f>lower,f<upper
These are just ideas off the top of my head. I realize that implementing this in expression trees is far from trivial (not my expertise anyway :/)
I can, for now, in this particular case, just remove the parentheses altogether, but I can see other potential cases when that wouldn't be a viable solution due to this ambiguity.

Maybe it would be more clear with a different operator for "IN", e g:
field IN value1,value2

I forget this one. This is a good idea to somehow support parenthesis only for values ( filed=(value1,value2) ) but I didn't do that because this way you have to deal with a lot of escape characters.
but with the custom operators feature, you can implement your own IN if you need it.


(fix, typescript): remove node:stream import to play nicely with webpack

Updates the FormDataWrapper to use import { Readable } from "stream"; as is recommended in their README

When you go from 1 variation of a test definition -> 2, seed doesn't hande deleting source code. I manually deleted one i found which is what caused the large number of deleted lines.


Problem with token magic fx

The "token magic fx" effect is activated on the token from which the effect starts, even if it is disabled on itself.
It is also activated if "ignore self" is checked

This is standard behaviour im afriad. The "ignore self" option only prevents mechanical effects of the Active Effect. To prevent this you can create the effect on the actor themself which will not trigger the TMFX effect


Fix the position of pagination

ref: https://github.com/bm-sms/daimon_news-layout/pull/12

Looks good :+1:


music titles not showing up ingame

As stated, the titles of custom music are not there, all the slots are empty. It works fine besides that. Happend after newest Acropolis Update

Is that due to the new XMSBT feature? does it happen even if that's the only mod enabled?

thanks for the response.
I dont know if it´s due to that (im relativly new to smash modding and have no idea what this XMSBT feature is). And yes, it does happen when its the only mod enabled.

Same issue here :(

Even as we near the end of 2024, this is still an issue. This tool hasn't been updated in two and a half years and I'm not sure if anyone is even looking at these issues.


Keltner Channels improvements and volume indicators isolation

Keltner Channel:

Fixing bug on Keltner channel high band calculation. Based on: https://github.com/bukosabino/ta/issues/126
Included percentage band and width band indicators for Keltner Channel.
Included 2 ways to calculate the centerline of Keltner Channel: original version (SMA of typical price) vs modern version (EMA of close)

Volume indicators isolation: Isolate all the indicators using the volume input feature to get all together using add_volume_ta wrapper function. Based on: https://github.com/bukosabino/ta/issues/129

Hi @bukosabino ,
is it planned not to expose the ov parameters to the methods keltner_channel_xxxx? I see this param can only be set when instantiate KelnerChannel class and cannot be passed in when calling a single method.
Thanks in advance for your attention.

Great @fettunch ,
I will add this option asap.
Thanks for your advice!

Hi @fettunch ,
This fix is included in the last library version (0.5.17).
Update to use it:
pip install -U ta


Uninformative error message when fstar.exe isn't in PATH

make -C test
../krml -tmpdir out  -add-include '"testlib.h"' -I . testlib.c -no-prefix Hoisting -o Hoisting.exe Hoisting.fst
KreMLin will drive F*. Here's what we found:
KreMLin called via: ../krml
readlink is: greadlink
the Kremlin executable is: [...]/kremlin/_build/src/Kremlin.native
KreMLin home is: [...]/kremlin
Process.Exit.Error(_)
Fatal error: exception Process.Exit.Error(_)

The error message should explain what's the problem and the solution.

That should be fixed; also, kremlin now uses FSTAR_HOME first, if present.


Move most things out of ra_ide_api_light

This moves everything except structure out of ra_ide_api_light. So this PR and #1019 finish up #1009, whichever is merged last should probably remove the ra_ide_api_light crate.
Also, LocalEdit was removed since it wasn't used any more.

bors r+
Thanks!


Update Imap.php

Added expunge method for the Imap class. Because this is an imap library it may be usefulll to support Expunge too.

this change broke the SMTP


Fix typo

@JorgeMartinezG this is a really straightforward type fix, this PR is pointing at develop should it get pointed to master since it is currently the most up-to-date?


chain azurite_workaround when in any one crate

This addresses one of the two issues raised in #1252 1252

Addressing the other issue will simplify this moving forward.


Mostly using pip3 now, adapt install instructions

Most pip3s are just pip now. Not sure we need to add this.

I wouldn't know. Following the instructions (Mac 10.14.6 with Xcode 11.2.1 installed) there is no pip. Sounds to me like the instructions only work if you have other things installed already.

With Python 2 support ending, most Pythons will be Python 3 in the future. So pip and pip3 will be the same. So no need to clutter our docs and increase mental load for readers. Thanks for the idea anyways!


Fix the redirect for routes that require login

I noticed that if you are logged out and try a route which requires login, the redirect was failing.

Good catch


Improve documentation

The section about update method was hard to understand , so I rewrote it.

Thanks a lot


feat(core): Allow useContext in same elem as provider

Overview
What is it?

[x] Feature / enhancement
[ ] Bug
[ ] Docs / tests / types / typos

Description
This allows useContext() within the same component that already ran useContextProvider().
Use cases and why
This is useful for terseness, code reuse, and it eliminates an element of surprise.
Checklist:

[x] My code follows the developer guidelines of this project
[x] I have performed a self-review of my own code
[ ] I have made corresponding changes to the documentation
[x] Added new tests to cover the fix / functionality

So this turns out to be a breaking change for that one e2e test: It was reading the context after the useContextProvider hook and expecting it to hold the old context.
IMHO this is incorrect usage and in any case undocumented behavior, so it doesn't warrant a major version change?

Thank you. ( I think the change to test is fine.)


feat: Add Admin User Statistics

Is it a ?

[ ] Bug Report

[x] Feature Request

[ ] Chore

Description
There should be an option to view user admin statistics in the app.
Parent
#618
Would you like to work on the issue?
Yes

@iamareebjamal I'm thinking to making a overflow menu in dashboard toolbar from which user can select to view admin statistics. The reason behind not using the same UI as event statistics is that dashboard is showing details related to a particular event whereas admin stats contains details for the user and his/her collective events. So Admin Statistics fragment after completion will contain Admin User, Mail, Event, Session stats. Is this fine?

@shubham-padia @iamareebjamal  Are the Admin Statistics for the server side admin or for this app?

@Masquerade0097 It can be delayed but is on scope of the app, however not immediately required

@iamareebjamal Ok, so should I implement it now?

Let's save this for later

@iamareebjamal Can I take up this issue and start working on it?

No, this is still not important. There are other issues which need attention


Fire emails on triggers

This is an advanced feature. We will implement this last.

Depends on #59.


GradientTape issue

thanks in advance for support

System information

Windows 10 OS, NO GPU
TensorFlow backend
TensorFlow version:  2.1.0
Keras version: 2.3.1
Python version:  3.6.10

Issue:
I am trying to run customized training on a model updating only a subset of the weights.

Code:
I made the sample code only to show the issue. variable names are functionality is meaningless.
run_ExampleC.zip

Error message:

Traceback (most recent call last):
File "run_ExampleC.py", line 101, in
run()
File "run_ExampleC.py", line 97, in run
controller.push(input_nd, output_nd, output_1d)
File "run_ExampleC.py", line 68, in push
self.train_function(input_nd[np.newaxis, :])
File "run_ExampleC.py", line 54, in train_function
grads = tape.gradient(learn_loss, self.model1.trainable_weights)
File "C:\Users\sorgil\AppData\Local\Continuum\anaconda3\envs\p36_tf\lib\site-packages\tensorflow_core\python\eager\backprop.py", line
1029, in gradient
unconnected_gradients=unconnected_gradients)
File "C:\Users\sorgil\AppData\Local\Continuum\anaconda3\envs\p36_tf\lib\site-packages\tensorflow_core\python\eager\imperative_grad.py"
, line 77, in imperative_grad
compat.as_str(unconnected_gradients.value))
File "C:\Users\sorgil\AppData\Local\Continuum\anaconda3\envs\p36_tf\lib\site-packages\tensorflow_core\python\eager\backprop.py", line
141, in _gradient_function
return grad_fn(mock_op, *out_grads)
File "C:\Users\sorgil\AppData\Local\Continuum\anaconda3\envs\p36_tf\lib\site-packages\tensorflow_core\python\ops\array_grad.py", line
229, in _ConcatGradV2
op, grad, start_value_index=0, end_value_index=-1, dim_index=-1)
File "C:\Users\sorgil\AppData\Local\Continuum\anaconda3\envs\p36_tf\lib\site-packages\tensorflow_core\python\ops\array_grad.py", line
118, in _ConcatGradHelper
concat_dim._numpy().item(0) % input_values[0]._rank())  # pylint: disable=protected-access
AttributeError: 'Tensor' object has no attribute '_numpy'

I think this is caused by Keras expecting a computational graph instead of the default eager mode of TensorFlow 2.0. Keras 2.3 does not support eager mode. It is recommended using tensorflow.keras in that case (https://github.com/keras-team/keras/releases).
Once I change the imports statement in your code from import keras to  from tensorflow import keras this error is solved. Note, however, that this change also requires fixing the arguments passed to Model: instead of input and output, it expects inputs and outputs.

@lsorgi ,
Can you please take a look at this links for similar error.It helps.Link1,Link2.Thanks


[BUG]: rdcman: unexpected switch: /accepteula

Prerequisites

[X] I have written a descriptive issue title.
[X] I have searched all issues/PRs to ensure it has not already been reported or fixed.
[X] I have verified that I am using the latest version of Scoop and corresponding bucket.

Package Name
rdcman
Expected/Current Behaviour
When I attempt to start rdcman after installing from scoop I get the following:

Steps to Reproduce
N/A

Possible Solution
N/A
Scoop and Buckets Version
❯ scoop --version
Current Scoop version:
v0.3.1 - Released at 2022-11-15

'extras' bucket:
49b46fcb7 (HEAD -> master, origin/master, origin/HEAD) linphone: Update to version 5.1.0

'main' bucket:
57cbfa46f (HEAD -> master, origin/master, origin/HEAD) terraform-provider-ibm: Update to version 1.56.1

'sysinternals' bucket:
db7ddd5 (HEAD -> main, origin/main, origin/HEAD) zoomit: Update to version 7.1

### Scoop Config

```console
❯ scoop config

last_update                       scoop_branch scoop_repo
-----------                       ------------ ----------
2023-08-21T08:59:15.0994514-04:00 master       https://github.com/ScoopInstaller/Scoop

PowerShell Version
N/A

Additional Softwares
No response

Fixed by 062afe4


Support Refaster templates which use lambdas

This still doesn't allow a template to directly return a lambda, because the JavaTemplate doesn't capture the lambda's type in any way, but at least a template can still use a lambda (e.g. as a method argument).
The actual fix is to make sure that variable references aren't printed as fully qualified names with the owning method as qualifier.

This change brings the totals of rules from ErroProne Support not supported to
@Matches is currently not supported=1
@NotMatches is currently not supported=2
Lambdas are currently not supported=2
Method references are currently not supported=3
If statements are currently not supported=8
@Placeholder is currently not supported=15
@Repeated is currently not supported=21
Generic type parameters are currently not supported=519

I doubt this change will have moved the needle by a lot. Using lambdas as a method argument in a Refaster template is probably fairly uncommon. Also, I don't even think we had a detection for this, only the detection for a template returning a lambda, which unfortunately still isn't supported. We should try to overhaul JavaTemplate so that we can support more Refaster templates.


Support byass 2fa?

This tool capture cookies sessions in google?

i do not know what you mean honestly. if you just want to observe cookies in http requests and responses, you can do that by modifying the message handler module and implementing extra parts parsing cookies.

This looks a multipost spam. close.


Deletes empty groups

@Marco-Lindsay Is this a feature we want? I have found it kinda annoying...

@Marco-Lindsay and I decide this was a feature, not a bug,


CAMEL-11882: ServiceDefinition.metadata not passed to

RibbonServiceLoadBalancer
https://issues.apache.org/jira/browse/CAMEL-11882

merged, do you mind closing the PR ?

@lburgazzoli you are faster than your shadow!
Merged as 0ad3f62402805ec0ae1a15a9cac210c7b8096e2e


The method 'write' was called on null

I made a simple example and i fail to write a state on a button press.
Did i do something silly?
flutter --version
Flutter 2.1.0-12.2.pre • channel beta • https://github.com/flutter/flutter.git
Framework • revision 5bedb7b1d5 (10 days ago) • 2021-03-17 17:06:30 -0700
Engine • revision 711ab3fda0
Tools • Dart 2.13.0 (build 2.13.0-116.0.dev)

I use binder version ^0.4.0
And i used the following environment (default from flutter create):
environment:
  sdk: ">=2.7.0 <3.0.0"

Example:
import 'package:flutter/material.dart';
import 'package:binder/binder.dart';

final testRef = StateRef(100);

final testLogicViewRef = LogicRef((scope) => TestLogic(scope));

class TestLogic with Logic {
  TestLogic(Scope scope);

  @override
  Scope scope;

  void writeState(int val) {
    write(testRef, val);
  }
}

void main() {
  runApp(BinderScope(child: MyApp()));
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: MyHomePage(),
    );
  }
}

class MyHomePage extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text('Binder test'),
      ),
      body: Center(
        child: Column(
          children: [
            Text(context.watch(testRef).toString()),
            TextButton(
              child: Text('Test'),
              onPressed: () {
                context.use(testLogicViewRef).writeState(999);
              },
            ),
          ],
        ),
      ),
    );
  }
}

I get the following error:
Restarted application in 327ms.

════════ Exception caught by gesture ═══════════════════════════════════════════
The following NoSuchMethodError was thrown while handling a gesture:
The method 'write' was called on null.
Receiver: null
Tried calling: write<int>(Instance of 'StateRef<int>', 999, null)

When the exception was thrown, this was the stack
#0      Object.noSuchMethod (dart:core-patch/object_patch.dart:54:5)
#1      Logic.write
#2      TestLogic.writeState
#3      MyHomePage.build.<anonymous closure>
#4      _InkResponseState._handleTap
...
Handler: "onTap"
Recognizer: TapGestureRecognizer#e2765
    debugOwner: GestureDetector
    state: possible
    won arena
    finalPosition: Offset(192.0, 106.0)
    finalLocalPosition: Offset(36.5, 7.0)
    button: 1
    sent tap down
════════════════════════════════════════════════════════════════════════════════

Oh I see, the scope is not affected. Replace TestLogic(Scope scope); with  TestLogic(this.scope); and it should work.
If you're using vscode, you can add this extension  and use the lref snippet. It will correctly create the Logic.

Ah. My bad. I actually had installed you plugin (which is great). But i did not use it that time.

Ah. My bad. I actually had installed you plugin (which is great). But i did not use it that time.

No worries 😉


Add Molecule job for copy-containers

It will help to test the copy-container golang binary build process.
Depends-On: https://github.com/openstack-k8s-operators/ci-framework/pull/999
As a pull request owner and reviewers, I checked that:

[ ] Appropriate testing is done and actually running
[ ] Appropriate documentation exists and/or is up-to-date:

[ ] README in the role
[ ] Content of the docs/source is reflecting the changes

recheck

@raukadah @arxcruz An entry needs to be added into the config file [1] as were failing here [2]
[1] https://github.com/openstack-k8s-operators/ci-framework/blob/be22d6787a3db72a8b1d691a059b990c7781e4ca/roles/copy_container/files/copy-quay/config.yaml#L5
[2] https://github.com/openstack-k8s-operators/ci-framework/blob/be22d6787a3db72a8b1d691a059b990c7781e4ca/roles/copy_container/files/copy-quay/main.go#L86

@raukadah thanks for taking this point I mentioned the past day day during the sync

@raukadah @arxcruz An entry needs to be added into the config file [1] as were failing here [2]
[1] https://github.com/openstack-k8s-operators/ci-framework/blob/be22d6787a3db72a8b1d691a059b990c7781e4ca/roles/copy_container/files/copy-quay/config.yaml#L5-L18 [2] https://github.com/openstack-k8s-operators/ci-framework/blob/be22d6787a3db72a8b1d691a059b990c7781e4ca/roles/copy_container/files/copy-quay/main.go#L86-100

Done


Support of case (in)sensitivity for queries

IMHO there is no true solution for this. Couch Slacker supports regular expressions, or some kind of standardization for data is needed.

IMHO there is no true solution for this. Couch Slacker supports regular expressions, or some kind of standardization for data is needed.


add agrc (branch: py3) as a submodule

Do you want to pull the ags file out of agrc and commit it with this repository?

We still maintain that repo for now, so I think that a submodule is best since it has to use the py3 branch for now. Unless there's a reason that you have for doing it?


SFDX: Specify Package Ancestor Value to "NONE" Invalid

I am unable to set ancestorVersion or ancestorId in my sfdx-project.json under packageDirectories to the value of "NONE" (with quotes)
The only valid option is NONE (without quotes), though that is invalid JSON
The information and example stated here says to use quotes

SFDX - DEV 2GP - Config Ancestors

Our old workaround is to not include the  ancestorVersion or ancestorId attributes if there is none to specify.
This no longer works because we are forced to include that value in Spring '22 (v54), if we do not we get the following error

Below is a screenshot of the ancestorId with "NONE" (with quotes) value not working

Below is a screenshot of the ancestorVersion with "NONE" value not working

Below is a screenshot of the information around ancestorVersion && ancestorId (v54)

Below is a screenshot of the ancestorId with NONE (without quotes) value working to make scratch org, though invalid JSON and breaks build

Parse error we get for using NONE

Is there a bug within SFDX?
Is there some documentation that needs to be updating?
If you could shed some clarification that would be great, thank you

@v-goyal-sfdc

@dlouvton @v-goyal-sfdc Any update on this? Is this a known issue? Do you plan on fixing this in upcoming release? Thanks :)

To confirm this is also currently an issue in Salesforce Summer '22 (v55 API)
To confirm this is also currently an issue with 7.152.0 of SFDX

@estrudels Have you tried using the flag --skipancestorcheck when doing package version create. Meanwhile, I'll check if this with packaging team.


Bump go-winio and refactor Windows snapshotters

This PR bumps the version of go-winio to the latest commit and refactors the lcow and windows snapshotters to use go-winio's utility functions for checking the filesystem type. This means other folks like myself can write more windows-based snapshotters without copy/pasting this code everywhere.
It seems there is one breaking change in the EnableCallback func in https://github.com/microsoft/go-winio/commit/a2a7cd6b7d9f8b079b910d0a8c21a1e9968df090#diff-affd4dec288883c3d72b7e20004dfae5 - hence the changes to etwCallback.
This PR closes #3362
@kevpar / @jterry75 / @jhowardmsft PTAL.
Signed-off-by: Eric Hotinger<EMAIL_ADDRESS>

Codecov Report

Merging #3385 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3385   +/-   ##
=======================================
  Coverage   45.05%   45.05%
=======================================
  Files         113      113
  Lines       12563    12563
=======================================
  Hits         5660     5660
  Misses       6049     6049
  Partials      854      854

Flag
Coverage Δ

#linux
48.98% <ø> (ø)
:arrow_up:

#windows
40.27% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6617def...9fda4a5. Read the comment docs.

LGTM


Fix Possible Mistype

I am going to go ahead and merge this without testing the course since it is a minor text update. Thank you, @AliabbasMerchant!


Show splash window until bundled extensions have been loaded

Change app start up so that startup waits until the bundled extensions have been loaded and enabled before splash window is hidden and main window is shown.
This is because some of the extensions want to render UI elements and they should be ready when the main screen is shown. Less flicker and better UX.
Signed-off-by: Juho Heikka<EMAIL_ADDRESS>

I don't know if I agree that this is the best course of action. Would it not be possible instead to render a "dummy" component that is roughly the correct size and shape but has a loading animation on top (similar to what some websites do and what I did for the cluster sidebar icon)?
That would therefore mean that the app still loads just as fast and we wouldn't have this additional loopback dependency.

I don't know if I agree that this is the best course of action. Would it not be possible instead to render a "dummy" component that is roughly the correct size and shape but has a loading animation on top (similar to what some websites do and what I did for the cluster sidebar icon)?
That would therefore mean that the app still loads just as fast and we wouldn't have this additional loopback dependency.

The dummy component wouldn't be rendered until the extension has loaded. Anyway a bigger problem that dummy component wouldn't solve is that the Spaces extension wants to show a fullscreen modal that should be visible already when main window is shown and not after a while (like it does now).

The dummy component wouldn't be rendered until the extension has loaded. Anyway a bigger problem that dummy component wouldn't solve is that the Spaces extension wants to show a fullscreen modal that should be visible already when main window is shown and not after a while (like it does now).

Fair enough, if this is for a fullscreen modal then I don't really see any other way to do it. I wonder if it needs to be one but I don't really know what you are trying to achieve.

Had a unsigned commit. Fixed.

Rebased to fix conflicts

While testing I noticed that bundled extensions are still loading after main window is opened. Pod menu is an example:
bundled.extensions.wait.mov

Interesting. I think it might be because extensions are loaded in 3 places, main and root-frame init on startup and cluster-frame init https://github.com/lensapp/lens/blob/f8fa33ec8cde5dcc5b56f77262b6ccb233f13115/src/renderer/cluster-frame.tsx#L105-L115
I think these menus are loaded in the third phase. This PR is currently waiting in the root-frame phase.
Also this approach that I take requires the extension to override LensExtension::onActivate-method to do it's initial loading which many of the standard bundled extensions don't so they return immediately on the startup.


Our keys in Picker stories don't always match the value

🐛 Bug Report

<EMAIL_ADDRESS>The story should be easy to test, and so the keys should at least resemble their value.

This issue can be closed

@khoad thanks for the heads up!


InternalOAuthError: Failed to obtain access token

Now getting this error, friends had to change oauth so it works, but I'm not sure what to do:
Error
InternalOAuthError: Failed to obtain access token
    at Strategy.OAuth2Strategy._createOAuthError (/home/runner/markplek/node_modules/passport-oauth2/lib/strategy.js:408:17)
    at /home/runner/markplek/node_modules/passport-oauth2/lib/strategy.js:175:45
    at /home/runner/markplek/node_modules/oauth/lib/oauth2.js:191:18
    at passBackControl (/home/runner/markplek/node_modules/oauth/lib/oauth2.js:132:9)
    at IncomingMessage.<anonymous> (/home/runner/markplek/node_modules/oauth/lib/oauth2.js:157:7)
    at IncomingMessage.emit (events.js:326:22)
    at IncomingMessage.EventEmitter.emit (domain.js:483:12)
    at endReadableNT (_stream_readable.js:1241:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)

@ItsMrReals can you share your implementation?
Also, I fixed some bugs in a fork you can use here. This package isn't being maintained any longer.

@ItsMrReals can you share your implementation?
Also, I fixed some bugs in a fork you can use here. This package isn't being maintained any longer.

Looks like this was a discord issue, it is fixed now.


Sinks should include the Timestamp

Each event in summingbird has a timestamp. We should consider exposing that on sink or merge. Sink, almost certainly, since it can be reimported to another flow.

This bug is only in storm now, I think. At least it is not in scalding.


Update MD file on new branch 2017-31-16_01-12-30

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

split/updateTestRef.md
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


7/15 Patch Releases

Summary:
Task covering 7/15 Patch releases work.
These patch releases have security fixes (to be announced later in a GitHub issue and at release time in release notes).
List of required releases:
To release as soon as able:
v1.16.13-rc1+k3s1 (just for typical testing)
v1.17.9-rc1+k3s1 (RC needed for KDM testing in Rancher)
v1.18.6-rc1+k3s1 (RC needed for KDM testing in Rancher)
To release once have approval from QA:
v1.16.13+k3s1
v1.17.9+k3s1
v1.18.6+k3s1
Prep work:

[x] EIO team notified of 7/15 releases
[x] PM and QA review changes and understand testing efforts
[x] Create GH issue in rancher/k3s explaining security fixies (MUST wait for embargo to lift before issue creation) https://github.com/rancher/k3s/issues/2027
[x] Prep PR for rancher/k3s (update stable channel) https://github.com/rancher/k3s/pull/2022
[x] FORK rancher/kontainer-driver-metadata (and bump v1.17 and v1.18 RC versions for QA to test KDM) data.json
[x] Prep PR for rancher/kontainer-driver-metadata (bump v1.17 and v1.18 release versions) https://github.com/rancher/kontainer-driver-metadata/pull/317
[x] Release Notes prepared

Vendor and release work:

[ ] Vendor in new patch versions into the appropriate branches and release
PM needs to provide release notes

Post-Release work:

[ ] Merge in channel server change (to mark v1.18.6+k3s1 as stable)
[ ] Merge in rancher/kontainter-driver-metadata change (so new v1.17 and v1.18 patch versions are shown in Rancher UI)
[ ] Close the rancher/k3s security issue

We are experiencing some issues and are planning to release tomorrow, 7/16/20.


Example "BasicUsage" does not compile

Arduino IDE Version 1.8.3
TCA9548A.h Version 1.1.1
It does not compile, but as it is an example, it should?!

Need a bit more information.
Can you copy the error message(s), and the MCU you are using?
Also - do you mean IDE version 1.8.13?

Arduino Nano, Arduino Uno.
Errormessages:
Arduino: 1.8.13 (Linux), Board: "Arduino Nano, ATmega328P (Old Bootloader)"
/home/to/sciebo/TT_Arduino/libraries/TCA9548A/src/TCA9548A.cpp: In member function 'uint8_t TCA9548A::read()':
/home/to/sciebo/TT_Arduino/libraries/TCA9548A/src/TCA9548A.cpp:84:73: error: call of overloaded 'requestFrom(uint16_t, uint8_t, bool)' is ambiguous
this->myWire->requestFrom((uint16_t)this->_address, (uint8_t)1, true);
^
In file included from /home/to/sciebo/TT_Arduino/libraries/TCA9548A/src/TCA9548A.h:13:0,
from /home/to/sciebo/TT_Arduino/libraries/TCA9548A/src/TCA9548A.cpp:9:
/home/to/Downloads/arduino-1.8.13/hardware/arduino/avr/libraries/Wire/src/Wire.h:66:13: note: candidate: uint8_t TwoWire::requestFrom(uint8_t, uint8_t, uint8_t)
uint8_t requestFrom(uint8_t, uint8_t, uint8_t);
^~~~~~~~~~~
/home/to/Downloads/arduino-1.8.13/hardware/arduino/avr/libraries/Wire/src/Wire.h:69:13: note: candidate: uint8_t TwoWire::requestFrom(int, int, int)
uint8_t requestFrom(int, int, int);
^~~~~~~~~~~
exit status 1
Fehler beim Kompilieren für das Board Arduino Nano.
Dieser Bericht wäre detaillierter, wenn die Option
"Ausführliche Ausgabe während der Kompilierung"
in Datei -> Voreinstellungen aktiviert wäre.

Cheers,
This is an issue fixed last month.  I've been having trouble getting the library manager to update the code - will try again.
If you replace your local library files TCA9548A.h and TCA9548A.cpp , with their equivalents in the src directory here, it should fix the issue.
Essentially it's due to changes in the Wire library which vary between MCUs and their respective Arduino cores - It's not sure which function to call.

Ok, thanks for your help and solution :-)

No worries - hopefully, it, will update automatically in future :)
I will close this issue for now. Just re-open/comment if no luck.


docs: store magento2 credentials in the global config, for #5932, for #6380

The Issue

#5932

The documentation says https://ddev.readthedocs.io/en/stable/users/quickstart/#magento

Note that you can install the Adobe/Magento composer credentials in your global ~/.ddev/homeadditions/.composer/auth.json and never have to find them again.

But the current quickstart doesn't check composer's global configuration for this auth, resulting in a situation where you have to add these credentials each time for each project.
How This PR Solves The Issue
Stores magento2 credentials in the global config ~/.ddev/homeadditions/.composer/auth.json
This is one-time action. If you run it again, it will simply check if your credentials are correct.
Manual Testing Instructions
Review at https://ddev--6442.org.readthedocs.build/en/6442/users/quickstart/#magento
Try it out.
Automated Testing Overview

Release/Deployment Notes

Just a note that... the config could also be in project .ddev/homeadditions/.composer. So many things.

Just a note that... the config could also be in project .ddev/homeadditions/.composer. So many things.

Fixed by adding another check.

For this I added a very long bash line 🙂
It stores magento2 credentials in the global config ~/.ddev/homeadditions/.composer/auth.json (and respects $XDG_CONFIG_HOME)
This is one-time action. If you run it again, it will simply check if your credentials are correct.


Utility for calculating fisher expectation

Write a utility for automatically calculating expectation of posterior uncertainties from a Fisher analysis. See A.2 in https://arxiv.org/pdf/1511.08752.pdf

In case it is useful, there is a package known as gwbench for computing uncertainties using Fisher formalism. Here is the methods paper

Also Jan Harms maintains gwfish: https://arxiv.org/abs/2205.02499


Error running new Upload API Indicator

Cudeso, any help on this would be appreciated
I ran python3 script.py
script.py:1: DeprecationWarning:
As our baseline system is the latest Ubuntu LTS, and Ubuntu LTS 22.04 has Python 3.10 available,
we will officially deprecate python versions below 3.10 on January 1st 2024.
Please update your codebase.
from pymisp import ExpandedPyMISP
Unknown error: the response is not in JSON.
Something is broken server-side, please send us everything that follows (careful with the auth key):
Request headers:
{'User-Agent': 'PyMISP 2.4.172 - Python 3.8', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'CAKEPHP=e2r4gou0h5q5ra55g8tfo66nq2edmoc8', 'Content-Length': '381', 'content-type': 'application/json'}
Request body:
{"returnFormat": "stix2", "page": 2, "limit": 50, "tags": ["workflow:state="complete""], "withAttachments": 0, "metadata": 0, "published": 1, "enforceWarninglist": 0, "to_ids": 1, "includeEventUuid": 0, "includeEventTags": 0, "sgReferenceOnly": 0, "includeContext": 0, "headerless": 0, "includeSightings": 0, "includeDecayScore": 0, "includeCorrelations": 0, "excludeDecayed": 0}
Response (if any):
{"name":"An Internal Error Has Occurred.","message":"An Internal Error Has Occurred.","url":"/events/restSearch"}
----------------CLEAR existing_indicators_hash---------------------------

This is "expected", in the sense that for now it's not possible to get the number of pages (result of paginated search) that are left in the MISP search. Hence the only way to get all the results is querying MISP until it errors-out.
When querying for JSON or others it doesn't throw an error, but the MISP-STIX is passed on to another Python script and when requesting for an in-existing page this Python script returns an error.
See: https://github.com/cudeso/misp2sentinel/tree/upload_indicators_api#upload-indicators-api-misp-to-stix-and-errors-an-internal-error-has-occurred
and https://github.com/MISP/misp-stix/issues/44

Thank you for responding quickly.
Those links seem more like explanations for the problem rather than solutions?
the first link makes reference to settings relevant to Upload indicators API which aren't actually in the config.py script for example, ms_api_version = "2022-07-01"
I've added that to my script, but also wondering what else isn't in the script in the repo?
Should I assume that it isn't possible to get MISP data into Sentinel using this script for Upload Indicator API currently? Is there an alternative

Looking more closely at your first input, the line of code "CLEAR existing_indicators_hash" is no longer in the Upload Indicators branch > https://github.com/cudeso/misp2sentinel/blob/upload_indicators_api/RequestManager.py
It does exist in the main branch though. I think you're branch is not set to "upload_indicators_api".
Check with git branch. Switch branches with git checkout upload_indicators_api
The uploading of indicators to Sentinel with the Upload Indicator API definitely works (setups with between 1k and 45k indicators syncing).

Yes my branch was set to main branch, now switched to upload_indicator_api by first doing git stash
I noticed in the upload indicators branch there's no config.py only config.py.default?
So I got rid of everything cloned the upload_indicator_api branch, and i keep getting the config.py version from the main, I then update it's contents with that of config.py.default.
So every time I clone, I get the main branch, not sure if others have the same issue.
Ran python3 script.py and the error is what I think you were expecting above, but It's not clear what the solution is:
python3 script.py
script.py:1: DeprecationWarning:
As our baseline system is the latest Ubuntu LTS, and Ubuntu LTS 22.04 has Python 3.10 available,
we will officially deprecate python versions below 3.10 on January 1st 2024.
Please update your codebase.
from pymisp import *
Unknown error: the response is not in JSON.
Something is broken server-side, please send us everything that follows (careful with the auth key):
Request headers:
{'User-Agent': 'PyMISP 2.4.172 - Python 3.8', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'CAKEPHP=m2ogf6p7c079bm25vo98a97tivpic4sn', 'Content-Length': '381', 'content-type': 'application/json'}
Request body:
{"returnFormat": "stix2", "page": 2, "limit": 50, "tags": ["workflow:state="complete""], "withAttachments": 0, "metadata": 0, "published": 1, "enforceWarninglist": 1, "to_ids": 1, "includeEventUuid": 0, "includeEventTags": 1, "sgReferenceOnly": 0, "includeContext": 0, "headerless": 0, "includeSightings": 0, "includeDecayScore": 0, "includeCorrelations": 0, "excludeDecayed": 0}
Response (if any):
{"name":"An Internal Error Has Occurred.","message":"An Internal Error Has Occurred.","url":"/events/restSearch"}
You can ignore 'An Internal Error Has Occurred' message thrown by MISP.
The MISP STIX export does not return the number of page results. We query for results until we get an error.
Also see: https://github.com/MISP/misp-stix/issues/44

How do you use misp-stix directly instead of restsearch?
Which one of your script need to be configured for that to happen
Or do I need to develop something
Are there nay other steps required not describes under README.md
To get indicators into Sentinel all I'm doing is python3 script.py
There's something wrong with cloning this repo, files not in upload_indicator_api get pulled, I've had this issue multiple times, can you try replicating the intrusions in your document I see others are running into the same issue.

This is my most recent error - my RequestObject.py is the right one, although that wasn't clear at first
python3 script.py
script.py:1: DeprecationWarning:
As our baseline system is the latest Ubuntu LTS, and Ubuntu LTS 22.04 has Python 3.10 available,
we will officially deprecate python versions below 3.10 on January 1st 2024.
Please update your codebase.
from pymisp import *
Traceback (most recent call last):
File "script.py", line 6, in
from RequestObject import RequestObject, RequestObject_Event, RequestObject_Indicator
File "/home/SOM-Admin/sentinel/misp2sentinel/RequestObject.py", line 5, in
from stix2.base import STIXJSONEncoder
ModuleNotFoundError: No module named 'stix2'

Hello. Can you check with the latest commit to the branch upload_indicators_api? This should fix the Exception error.
There's an update for the documentation and for the script. This scrip requires misp-stix (it's part of the requirements file). You can also manually add it to your Python virtual environment with pip install misp-stix.

downloading misp-stix removed the error message now I'm left with:
python3 script.py
script.py:1: DeprecationWarning:
As our baseline system is the latest Ubuntu LTS, and Ubuntu LTS 22.04 has Python 3.10 available,
we will officially deprecate python versions below 3.10 on January 1st 2024.
Please update your codebase.
from pymisp import *

Logs :
2023-07-19 14:13:55,121 - misp2sentinel - INFO - Start MISP2Sentinel
2023-07-19 14:13:55,121 - misp2sentinel - INFO - Fetching and parsing data from MISP ...
2023-07-19 14:13:55,121 - misp2sentinel - INFO - Using Microsoft Upload Indicator API
2023-07-19 14:13:55,658 - misp2sentinel - INFO - Received MISP events page 1 with 5 events
2023-07-19 14:13:56,005 - misp2sentinel - INFO - Received MISP events page 2 with 5 events
2023-07-19 14:13:56,186 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara
2023-07-19 14:13:56,186 - misp2sentinel - ERROR - Unable to process indicator
2023-07-19 14:13:56,187 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara
2023-07-19 14:13:56,187 - misp2sentinel - ERROR - Unable to process indicator
2023-07-19 14:13:56,187 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara
2023-07-19 14:13:56,187 - misp2sentinel - ERROR - Unable to process indicator
2023-07-19 14:13:56,570 - misp2sentinel - INFO - Received MISP events page 3 with 5 events
2023-07-19 14:13:56,981 - misp2sentinel - INFO - Received MISP events page 4 with 5 events
2023-07-19 14:13:57,002 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara
2023-07-19 14:13:57,002 - misp2sentinel - ERROR - Unable to process indicator
2023-07-19 14:13:57,002 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara
2023-07-19 14:13:57,002 - misp2sentinel - ERROR - Unable to process indicator
2023-07-19 14:13:57,336 - misp2sentinel - INFO - Received MISP events page 5 with 1 events
2023-07-19 14:13:57,503 - misp2sentinel - INFO - Received 392 indicators in MISP
2023-07-19 14:13:58,668 - misp2sentinel - INFO - Start uploading indicators
2023-07-19 14:14:01,280 - misp2sentinel - ERROR - Error when submitting indicators. {"error":{"code":"UnauthorizedAccess","message":"The Object ID [8a0499df-82bd-4d34-a124-d7463e3e27c8] does not have required permission to perform this action on the workspace [de98fe3d-3da5-41dc-846e-5d5c4fd9079a].","details":[],"target":null,"additionalInfo":null}}
2023-07-19 14:14:01,280 - misp2sentinel - INFO - Finished uploading indicators
2023-07-19 14:14:01,281 - misp2sentinel - INFO - End MISP2Sentinel

{"code":"UnauthorizedAccess","message":"The Object ID [8a0499df-82bd-4d34-a124-d7463e3e27c8] does not have required permission to perform this action on the workspace [de98fe3d-3da5-41dc-846e-5d5c4fd9079a].","details":[],"target":null,"additionalInfo":null}}: your Azure App does not have the necessary permissions to access the workspace.
You can check the required permissions here https://github.com/cudeso/misp2sentinel/tree/upload_indicators_api#installation and here https://learn.microsoft.com/en-us/azure/sentinel/connect-threat-intelligence-upload-api#register-an-azure-ad-application

Thanks for pointing that out can you remove the azure details from your comment, thank you, I'll try that now

I provided the new config script the same details as what i used for Graph API.
With the new Upload API it also needs Sentinel Contributor role given to it which i didn't provide and now have and finally logs have arrived.
Thank you very much for your hard work and taking time to respond!

It doesn't change screen and start to tally indicators sent to the API like it used to, so at first it doesn't seem as though its sent, had to check the logs, but confirmation is there

Errors still show in the logs not sure if this will cause an issue later or if they are expected:
2023-07-19 16:32:10,679 - misp2sentinel - INFO - Start MISP2Sentinel
2023-07-19 16:32:10,679 - misp2sentinel - INFO - Fetching and parsing data from MISP ...
2023-07-19 16:32:10,679 - misp2sentinel - INFO - Using Microsoft Upload Indicator API
2023-07-19 16:32:11,219 - misp2sentinel - INFO - Received MISP events page 1 with 5 events
2023-07-19 16:32:11,564 - misp2sentinel - INFO - Received MISP events page 2 with 5 events
2023-07-19 16:32:11,744 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara
2023-07-19 16:32:11,744 - misp2sentinel - ERROR - Unable to process indicator
2023-07-19 16:32:11,745 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara
2023-07-19 16:32:11,745 - misp2sentinel - ERROR - Unable to process indicator
2023-07-19 16:32:11,745 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara
2023-07-19 16:32:11,745 - misp2sentinel - ERROR - Unable to process indicator
2023-07-19 16:32:12,034 - misp2sentinel - INFO - Received MISP events page 3 with 5 events
2023-07-19 16:32:12,527 - misp2sentinel - INFO - Received MISP events page 4 with 5 events
2023-07-19 16:32:12,548 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara
2023-07-19 16:32:12,548 - misp2sentinel - ERROR - Unable to process indicator
2023-07-19 16:32:12,548 - misp2sentinel - ERROR - Ignoring non STIX pattern type yara
2023-07-19 16:32:12,548 - misp2sentinel - ERROR - Unable to process indicator
2023-07-19 16:32:12,793 - misp2sentinel - INFO - Received MISP events page 5 with 1 events
2023-07-19 16:32:12,959 - misp2sentinel - INFO - Received 392 indicators in MISP
2023-07-19 16:32:14,087 - misp2sentinel - INFO - Start uploading indicators
2023-07-19 16:32:18,820 - misp2sentinel - INFO - Indicators sent - request number: 0 / indicators: 100 / remaining: 292
2023-07-19 16:32:21,761 - misp2sentinel - INFO - Indicators sent - request number: 1 / indicators: 100 / remaining: 192
2023-07-19 16:32:22,283 - misp2sentinel - INFO - Indicators sent - request number: 2 / indicators: 100 / remaining: 92
2023-07-19 16:32:26,936 - misp2sentinel - INFO - Indicators sent - request number: 3 / indicators: 92 / remaining: 0
2023-07-19 16:32:26,937 - misp2sentinel - INFO - Finished uploading indicators
2023-07-19 16:32:26,937 - misp2sentinel - INFO - End MISP2Sentinel

Hello. Thank you, and good to hear everything worked out.
The errors are due to the fact that only those indicators that can be translated to STIX are synchronised. YARA (and Sigma) indicators cannot be imported in Sentinel, mostly because Sentinel does not accept them -yet-. > https://github.com/cudeso/misp2sentinel/tree/upload_indicators_api#ignored-types


feat(ingest): powerbi # add powerbi workspaces to containers

Conditionally add PowerBI workspace as a container in ingestion, the implementation then adds all of the entities ingested from the workspace to the container.
The workspace name is not delivered outside from the ingestion phase, so it's not possible to achieve similar behavior with transformation steps.
Configuration defaults to not extracting containers.
Checklist

[x] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[x] Links to related issues (if applicable)
[x] Tests for the changes have been added/updated (if applicable)
[x] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
[x] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@mohdsiddique Pinging for heads up, I've fixed your suggestions. The review may continue.

Oops - looks like there are conflicts. @mohdsiddique can help you sort any out in Slack before merging :)

Fixed, all of the conflicts were from my other PR.

Waiting on CI to do final merge. Cheers!


Installtion process-python command cannot be found, can be rectified.

In the installation process of plone for training an error occurs -python command cannot be found so we can either specifically mention in the docs to edit the makefile and change python -m venv venv  to python3 -m venv venv or directly make changes in the repository from where the code is getting cloned on to our system.

@Tishasoumya-02 I think the correct thing to do here is to change in the repository:
https://github.com/collective/training_buildout/blob/d4c780da1013da285f21cf01d39b51bb5f8a5932/Makefile#L31

I agree with @wesleybl. @Tishasoumya-02 would you be so kind as to create a new issue in that repository, and reference this issue? And better yet, please create a PR to make that change? Thank you!

Sure @wesleybl @stevepiercy .


Able to enter float values in GROTH field

Description
Currently it's possible to enter float values into the field "GROTH" in the "Send" screen.
We need to only allow integer values there, since GROTH is already a minimal coin unit.

checked


samples: support blackpill_f411ce and nucleo_f746zg boards

Both boards provide ethernet interface, so extend sample code with
enabling of DHCPv4 client.
Add hal_st and hal_stm32 west modules, so that those boards are built
correctly when this repo serves as west manifest repository.
Co-authored-by: Marcin Niestroj<EMAIL_ADDRESS>Based on https://github.com/golioth/zephyr-sdk/pull/51 with following changes:

fix coding style
include dhcpv4 header before using dhcpv4 api
check if CONFIG_NET_DHCPV4 is enabled before starting DHCPv4 client
do not use #ifdef around #include

@mniestroj I'd like to only support the Nucleo-H745ZI-Q at the moment.
Can you keep the work to add the other targets on a separate branch somewhere and enable the nucleo?
I'll take it from there and do some testing.

@0Grit see #76

We are not going to add those boards to the "officially supported" targets, so closing PR for now.


[Bug] Child age should accept 0 as value

This denotes children < 1 year old.

382317e7 closes this bug.


ng e2e does not serve assets in 9.0.0-next.9

🐞 Bug report
Command (mark with an x)

- [ ] new
- [ ] build
- [ ] serve
- [ ] test
- [x] e2e
- [ ] generate
- [ ] add
- [ ] update
- [ ] lint
- [ ] xi18n
- [ ] run
- [ ] config
- [ ] help
- [ ] version
- [ ] doc

Is this a regression?
Yes, the previous version in which this bug was not present was: 9.0.0-next.8
Description
The assets are not served in e2e tests, leading to errors in the console and failing e2e task.
🔬 Minimal Reproduction
This is fairly straightforward to repro:
npx<EMAIL_ADDRESS>new e2e-assets --defaults
# update app.component.html to add something from src/assets like an image
ng e2e

I put together a small repro: https://github.com/cexbrayat/e2e-assets
🔥 Exception or Error

1) workspace-project App should display welcome message
  - Expected [ Entry({ level: SEVERE, message: 'http://localhost:4200/assets/angular-logo.png - Failed to load resource: the server responded with a status of 404 (Not Found)', timestamp:<PHONE_NUMBER>657, type: '' }), Entry({ level: SEVERE, message: 'http://localhost:4200/favicon.ico - Failed to load resource: the server responded with a status of 404 (Not Found)', timestamp:<PHONE_NUMBER>671, type: '' }) ] not to contain .

🌍 Your Environment

Angular CLI: 9.0.0-next.9
Node: 12.7.0
OS: darwin x64
Angular: 9.0.0-next.10
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.900.0-next.9
@angular-devkit/build-angular     0.900.0-next.9
@angular-devkit/build-optimizer   0.900.0-next.9
@angular-devkit/build-webpack     0.900.0-next.9
@angular-devkit/core              9.0.0-next.9
@angular-devkit/schematics        9.0.0-next.9
@angular/cli                      9.0.0-next.9
@ngtools/webpack                  9.0.0-next.9
@schematics/angular               9.0.0-next.9
@schematics/update                0.900.0-next.9
rxjs                              6.5.3
typescript                        3.5.3
webpack                           4.41.0

Anything else relevant?
Reverting to 9.0.0-next.8 removes the error.

I realize there is also an issue when running ng build --prod, but it’s maybe the same. In my repro, I have an image in src/assets. If the image is in src/assets/images, then running ng build --prod throws: Unable to copy assets: ENOTSUP: operation not supported on socket, copyfile '/Users/.../e2e-assets/src/assets/images' -> '/Users/.../e2e-assets/dist/e2e-assets/assets/images'.


Update deps

Purpose
Closes #370
Updates dependencies. In particualar:
↑ ClimaAtmos v0.15.4 ⇒ v0.15.5

Content

Solution implemented:

[x] Updated Manifest files
[x] Updated ClimaAtmos CLI options

Review checklist
I have:

followed the codebase contribution guide: https://clima.github.io/ClimateMachine.jl/latest/Contributing/
followed the style guide: https://clima.github.io/ClimateMachine.jl/latest/DevDocs/CodeStyle/
followed the documentation policy: https://github.com/CliMA/policies/wiki/Documentation-Policy
checked that this PR does not duplicate an open PR.

In the Content, I have included

relevant unit tests, and integration tests,
appropriate docstrings on all functions, structs, and modules, and included relevant documentation.

[x] I have read and checked the items on the review checklist.

CI passes. Ready for review. Thanks

Thanks. I reviewed the artifacts produced and are identical to main. I'm going to merge this.

bors r+


chore: Lints and linting workflow using Ruff

- What I did

Ran Ruff against src files and edited to eliminate infractions
Added a workflow that will ensure continued compliance by running Ruff against the main.py and lib/*.py on an ongoing basis

- How I did it

Manual fixes (and testing)
Workflow based on markdown lints in at_protocol

- How to verify it
Workflow will run against this PR
- Description for the changelog
chore: Lints and linting workflow using Ruff

Converted to draft until I can get lints workflow passing


Fix for https://github.com/aim42/htmlSanityCheck/issues/282

hints:

build.gradle/com.gradle.build-scan has been moved to settings.gradle (https://docs.gradle.com/enterprise/gradle-plugin/#gradle_6_x_and_later).
gradle wrapper updated to current 6.8.1
configuration names updated (testCompile -> testImplementation)
feature preview 'STABLE_PUBLISHING' is not supported anymore in current gradle version
java primitives must not be marked as @Optional

unchanged:

doctoolchain/build.gradle: configuration names should get updated, too.

as commented in the issue, they have been failing testcases locally befor making the changes - and the tests still fail after my changes. at least running 'gradlew --warning-mode all' doesnt show any warnings / deprecations anymore.

as there were failing testcases on my machine (windows 10) i checked the tests: there were problems dealing with file separators.
i tested the fix on a windows10 machine and virtualbox/ubuntu and committed it.
probably i should have open a new issue/branch/pull request for it. sorry ;-(

as there were failing testcases on my machine (windows 10) i checked the tests: there were problems dealing with file separators.
i tested the fix on a windows10 machine and virtualbox/ubuntu and committed it.
probably i should have open a new issue/branch/pull request for it. sorry ;-(

@rdmueller I'm wondering what hold this PR from being merged? I'm in the process of upgrading a project to Gradle 7.0 and a new release of Html Sanity Check plugin that include this fix would be really appreciated :slightly_smiling_face:

sorry - it's my fault... shame on me for not following up

@amottier - for your info:
gernotstarke@iMac htmlSanityCheck % gradle publishPlugins

> Configure project :
after processResources config, version= 1.1.4

> Task :publishPlugins
Publishing plugin org.aim42.htmlSanityCheck version 1.1.4
Thank you. Your new plugin org.aim42.htmlSanityCheck has been submitted for approval by Gradle engineers. The request should be processed within the next few days, at which point you will be contacted via email.
Publishing artifact build/libs/htmlSanityCheck-1.1.4.jar
Publishing artifact build/libs/htmlSanityCheck-1.1.4.jar
Publishing artifact build/publications/pluginMaven/pom-default.xml
Publishing artifact build/publications/pluginMaven/module.json
Activating plugin org.aim42.htmlSanityCheck version 1.1.4

BUILD SUCCESSFUL in 10s
14 actionable tasks: 3 executed, 11 up-to-date
gernotstarke@iMac htmlSanityCheck %


SEO for https://jamesmfriedman.github.io/rmwc/

Hi James, I wanna point you to the fact that whatever search you do in Google related to RMWC, the site never shows up, only the GitHub files. You might wanna do something about that / check your github pages config.

I have limited experience with SEO. Is this something you can help with? I know setting proper page titles would be a good start.

Allright, after looking into it I see that you use CRA. Now that's a SEO problem since it is not really crawlable by default. What you need is Server Side Rendering (SSR) OR a static site generator to generate the site. The second option is impossible at the moment, since I do not know any static site generator that plays nice with Storybook. That leaves us with the first option, I found an example here: https://medium.com/bucharestjs/upgrading-a-create-react-app-project-to-a-ssr-code-splitting-setup-9da57df2040a
I do have experience with Static site generators, but not with CRA & SSR. So I don't think I can help you out here...

The good news is Google started crawling SPAs a few years back, so I think it might be as simple as fixing the page titles, adding in descriptions, a sitemap, and some additional content.

yeah, that's indeed what Google says. In practice it's not indexed properly in most cases. That's why tools like prerender still exist.

Ah, good call. I'll start there :)

Also, it's really too bad because typing react material design doesn't bring out rmwc although it's really, apart from material ui which is really on another path, the best project !

Thanks! I’ll look into this more this week.

Does material-components-web support server side rendering?

Yes, It’s a feature of RMWC that has decent test coverage. Haven’t personally used it but I know people are.


Broken link: "supported languages" under heading "Code highlighting"

The "supported languages" link points to the following:
http://highlightjs.readthedocs.io/en/latest/css-classes-reference.html#language-names-and-aliases
That page does not produce a list of languages, but rather some CSS class reference.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: fa622358-f5bb-1f5c-177a-081a37b6acef
Version Independent ID: 72f7fd16-a822-66c8-3a34-8905645ce5f2
Content: Markdown syntax for files, widgets, and wikis in Azure DevOps - Azure DevOps
Content Source: docs/project/wiki/markdown-guidance.md
Product: devops
Technology: devops-collab
GitHub Login: @chcomley
Microsoft Alias: chcomley

@bradleyscollins, I've just published a replacement for this link, which should be live sometime today. In the meantime, please go here: https://github.com/highlightjs/highlight.js/tree/9.10.0/src/languages. Thanks so much for reaching out!

Thanks a ton!


retweets from other

Does anyone know why they can't crawl the tweets that users retweet?

I think I have an answer for this


[Feature] Progressive upgrade if versions skipped

Checklist

[X] I have already read docs/Tips.md, but my feature is not implemented.
[X] My suggested feature was not mentioned by others, and it is not a duplicate feature.

Summary
The differential packages of HoYoverse games (in my case, Genshin Impact) upgrade the game by modifying the files in the previous version. This approach is optimal for most players as it reduces the size of the data downloaded.
However, for players skipping (one or more) versions, the current approach may not be optimal in terms of disk usage. Starward and the official launcher (IIRC) download all the differential packages at once but still apply them one by one due to their technical nature. During an iteration, differential packages not used in this iteration are redundant, yet still span up disk spaces, resulting in extra disk usage.
Solution or Design
Updates can be handled step-by-step: download the differential package of the version after the installed one, apply its patch, delete the package, and then repeat these steps for the next version. This will be useful for players having a disk space shortage, especially if the available space is a few GBs larger than the required size.

Your suggestion is very good, but in fact it is not feasible. The latest version is 4.6, and the versions that can be updated through differential packages are 4.4 and 4.5. If a player's local game version is 4.3, I have no way to get the differential packages from 4.3 to 4.4 or 4.5.


Unable to create code version to development

Thank you for building and maintaining this!!
I've got the webDav and OCAPI data settings using client id "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" and set up the open api credentials in Jenkins to the same. The build however errors out with a 411 failed to create code version -see clip of console out below. It is successful when I build to my sandbox and in the API Explorer.
I can also use a client key created for Staging and Dev, w/ webdav and OCAPI settings that was made for develop and that throws the same error in the console below. It is also successful from the API Explorer and Postman.
-----Console output ----------
[+] Creating new code version
 - development-<redacted>-realm-corp.demandware.net (b26_20210325_dev)
Deleting 1 temporary files
ERROR: Failed to create the new code version! 411 - Length Required!
Response=
<TITLE>Bad Request</TITLE>

Bad Request

Your browser sent a request that this server could not understand.

Reference #7.4d012417.1616693934.35d99813

We have few builds being done on development instances as well and didn't noticed this error.
Last build was just done about 50 minutes ago and it worked:
[+] Creating new code version
 - development-eu01-something.demandware.net (b181_20210326_development)
 + Ok

Are you connecting directly to the SFCC development instance? or trough some kind of proxy? Maybe the proxy is removing the Content-Length header? As the 411 http error is related to this header missing: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/411
I just did a quick test and pointed the plugin to attempt to make the deployment to Ngrok endpoint instead of SFCC endpoint so I can capture the request and here it is bellow:
PUT /s/-/dw/data/v20_10/code_versions/b1_20210326_dev?client_id=abc HTTP/1.1
Host: 5b548fceff41.ngrok.io
User-Agent: Jenkins (OSF Builder Suite For Salesforce Commerce Cloud)
Content-Length: 0
Accept-Encoding: gzip
Authorization: Bearer abcd

The Content-Length: 0 is set and sent in the request.
So maybe there's that HTTP proxy that I was asking about above that your requests are going trough and that proxy is removing this header?

Ngrok is a good idea!
I believe we do go through a proxy and will check in with our Network people to see if they have any info. It's odd though since the same Jenkins instance also deploys to our Staging instance without issue which is why I did not go there first.

From what I know development has the eCDN (Cloudflare) on top, while staging does not. So maybe this error is triggered by Cloudflare (on dev - WAF?) while on Staging since you connect directly to SFCC instance they are not that strict (no WAF) and allow the request without the header as well.

As it turns out I was wrong and we do not go through a proxy and our network does not mess with any headers. I've confirmed that development is the issue because we can create code versions on Staging (w/2FA) and Sandboxes (using client key and 'aaa...').
So if this is an issue on Cloudflare then how do I resolve it?

Actually I CAN successfully hit the PUT code_versions/ endpoint using Postman! Is there a better way to view the headers than ngrok?

Sorry to be a pain but I found that ngrok reports Content-Length:0 whether it is there or not. I sent a request using Postman with Content-Length unchecked. Ngrok reported that the Content-Length: 0 is there. Note the same Postman token header.

Sorry to be a pain

No worries, I'm happy to help troubleshoot this.

I found that ngrok reports Content-Length:0 whether it is there or not.

Hmm, then I'll have to think of a different way to test the headers. Any idea what else to use besides Ngrok? Ngrok was my go to tool for debugging http requests.
Can you try to make same request with curl from same server where Jenkins is running? You can even do it from Jenkins. Create new job and run sh command in it.
Strange that we don't have this issue on development instances that we use. If I could reproduce this it would be easier to debug.

Also are you on latest version of the plugin? Or an older one?
Latest one is 1.1.4

I've pushed a new version https://github.com/jenkinsci/osf-builder-suite-for-sfcc-deploy-plugin/commit/04db8c621af715b536083edd6277b4f02caf133d#diff-3093699ec1a4cd4ac4a4aac05dcbfe44c662fe1aa7855e646c13f91823426af2R696
where I'm setting an empty body "" for the request. Maybe this helps forcing the Apache HTTP Client (the library I use for HTTP requests) to set the Content-Length header.
Can you try with this one and see if it works?

Hey I am using your latest on my laptop to test. Now getting:
Deleting 1 temporary files
ERROR: Failed to create the new code version! 415 - Unsupported Media Type!
Response={"_v":"19.8","fault":{"arguments":{"contentType":"text/plain; charset=UTF-8"},"type":"UnsupportedContentTypeException","message":"The media type 'text/plain; charset=UTF-8' in the Content-Type header isn't supported. Note: Only the UTF-8 charset is supported."}}

I tried to add the Content-Length header manually and that seemed to work. I can put in a PR.

Merged and pushed a new version v1.1.6
Thank you.

Thank you we are using this for Development and Staging now!


Creating an invalid tx when trying to mint too large amount of tokens

Describe the bug
Trying to mint too many tokens produces and invalid transaction (basically minting<PHONE_NUMBER>232534000000 here for example)
This results in ogmios failing to evaluate/submit the transaction with a very obscure error message.
To Reproduce
TBD
Logs
ogmios.errors.ResponseError: Ogmios responded with error: {'jsonrpc': '2.0', 'method': 'evaluateTransaction', 'error': {'code': -32602, 'message': "Invalid transaction; It looks like the given transaction wasn't well-formed. Note that I try to decode the transaction in every possible era and it was malformed in ALL eras. Yet, I can't pinpoint the exact issue for I do not know in which era / format you intended the transaction to be. The 'data' field, therefore, contains errors for each era.", 'data': {'babbage': "invalid or incomplete value of type 'Transaction': expected word", 'conway': "invalid or incomplete value of type 'Transaction': expected word", 'alonzo': "invalid or incomplete value of type 'Transaction': expected list len or indef", 'mary': "invalid or incomplete value of type 'Transaction': Size mismatch when decoding Object / Array. Expected 4, but found 3.", 'allegra': "invalid or incomplete value of type 'Transaction': Size mismatch when decoding Object / Array. Expected 4, but found 3.", 'shelley': "invalid or incomplete value of type 'Transaction': Size mismatch when decoding Object / Array. Expected 4, but found 3."}}, 'id': None}

Expected behavior
Ogmios could report that the integer is breaking/incorrectly serialized. PyCardano should abort the building or raise an error/warning this.
Environment and software version (please complete the following information):

OS: Ubuntu 22
PyCardano Version: 0.10.0

Additional context
Looping in @KtorZ for ogmios, but the error is really with PyCardano here.

Indeed, the mint value must fit into a word which is a u64 or u32 depending on OS arch, but most likely u64 (so 18_446_744_073_709_551_615 max).

Note that: this is as good as it gets when it comes to deserialization errors here. I cannot get anything better without re-writing the CBOR parsers which I am slightly reluctant to at the moment 😅 ...

So it should be possible to just check that mint is smaller than sys.maxsize.

From the ledger spec, it looks like mint field should be int64, and uint should be used in all other cases, which doesn't have an upper bound (at least I couldn't find its definition in the spec). Maybe we should only limit mint field to int64. What do you think?

Yes that was the exact thought. There is no indication that any other fields need to be restricted.


problem with trigger on monthly basis

The following script doesn't trigger (while it should trigger ever minute of this month). I'm currently on a Windows 7 box, not yet checked on another OS.
var CronJob = require('cron').CronJob;
var job = new CronJob({
  cronTime: '* * * 5 *',
  onTick: function() {
    console.log('triggered')
  },
  start: false
});
job.start();

The months are specified from 0-11 not 1-12. So, if you still see the issue using * * * 4 * then the module is misbehaving.

Thanks @ncb000gt that was an easy fix!
However, this is not in line with the crontab specifications where months are specified from 1 to 12. See e.g. http://unixhelp.ed.ac.uk/CGI/man-cgi?crontab+5. Is there a reason for this divergence?


Add audio/general/text_to_speech project setting to enable/disable TTS.

Adds setting to completely disable TTS support.
The issue seems to be Linux specific, but for consistency, the setting is applied on all systems.
Fixes https://github.com/godotengine/godot/issues/77124

On principle the default should be false imo.
Users will not expect text to speech to be using resources and CPU when they have never requested it - slowing down their game or eating battery unnecessarily. We don't for example, render shadow maps just in case user might want shadows.
If this is to be used for editor accessibility (I don't know if it is currently), we could consider a separate editor setting. Also useful would be a runtime command line argument to override these settings (in case end user wants to disable text to speech).
Another approach once the project setting is working, as @bruvzg suggested previously, is simply allow deferring speech startup until a speech command is issued, and then the default could perhaps be changed to on, if there was no cost. But in any case having it switch-offable is eminently desirable.

On principle the default should be false imo.

I would set it to false, but this will break existing projects with TTS (but I do not think there are many, so might be ok for 4.1).

Users will not expect text to speech to be using resources and CPU when they have never requested it - slowing down their game or eating battery unnecessarily. We don't for example, render shadow maps just in case user might want shadows.

It seems to Linux specific, and might be specific to speech dispatcher or Pulse Audio version (I do not get more the 1.5% of usage).

If this is to be used for editor accessibility (I don't know if it is currently), we could consider a separate editor setting.

It's not used for anything in the editor, and not a good way to implement accessibility (I'm working on a proper screen reader support right now - https://github.com/godotengine/godot/pull/76829).

is simply allow deferring speech startup until a speech command is issued

Not sure if it's worth overcomplicating it.

I would set it to false, but this will break existing projects with TTS (but I do not think there are many, so might be ok for 4.1).

Surely all it will take is those few users flipping a project setting when they next export the project? Provided this is included in the release notes, this seems unlikely to cause many problems (it should be fairly obvious if a feature they are using is no longer working).
There's no change to e.g. code needed. This seems imo by far the lesser of two evils, if the alternative is making everyone else pay for this oversight, in perpetuity.

Well, error messages directly say what to do, so I guess it's OK. Changed default to false.

Looks good to me. The commit message should be amended like I just did with the PR title.
Also, I think @lawnjelly's concern also includes the CPU usage of the editor, which this wouldn't address as it's only for the project settings so project runtime. So this might also need an editor setting to disable editor TTS.
It could be disabled by default until we add accessibility features using it (in which case it should be enabled by default IMO).

@bruvzg Please amend the commit message, it still incorrectly refers to the setting name. Should be audio/general/text_to_speech.

Thanks!

Would need a 3.x backport if we want to make it opt-in for 3.6 too.


Dynamic Framework project

I created an Xcode project that contains an iOS dynamic framework target for SWHttpTrafficRecorder. It also contains a framework unit test target, but there were no tests to migrate to this target.
The demo project was updated to link the framework.
I moved the source files into a directory to match the convention of Xcode projects.
Framework targets could be added for watchOS and tvOS, but I didn't try their compatibility.
A framework target for OSX wasn't added as the library uses some iOS SDK stuff for file writing, but with some modifications it should work and can be added.
This also means Carthage can build this project.
I have signed the CLA.

Thanks for cleaning things up and add the Carthage support. It looks great overall. Will spend some time to look further to see whether the change for "Added Xcode project for framework target with unit tests" will be necessary.

The framework target is required for Carthage, and for users that wish to add an Xcode project as a sub-project in their project.


handle plugin buffer response

Changes

Now we support return Buffer; from plugins
Resolves issue brought up in https://github.com/pikapkg/snowpack/discussions/945

Testing

We don't have good testing for third party plugins at the moment, hmm

I've followed @jaredLunde's "Option 1" instructions here: https://github.com/pikapkg/snowpack/discussions/945#discussioncomment-58257
I've run yarn dev and then the local snowpack from this pull request side-by-side, but I don't see any differences. I was expecting to see a different image size? What am I looking for?

@gr2m I updated the plugin to return {result: Buffer} instead of just a binary string so neither option will be broken anymore. Previously the image file was corrupted.

I have reverted this change directly in node_modules/snowpack-plugin-resize-images/dist/main/index.js. I saw the image being corrupted. But it was still corrupted when I run snowpack dev from this pull request.
Did you check if this pull request fixes the problem for you?

The old plugin returned a binary string. It will still be corrupted w/ this PR because this doesn't attempt to handle binary strings, it just fixes a bug where you couldn't return just a buffer. You had to return an object {result: Buffer}. @stramel covered the issues around handling binary strings in the discussion.

We don't have good testing for third party plugins at the moment, hmm

@FredKSchott could we create a testcase with a minimal plugin local to the test?

@gr2m I'd rather not implement a new test suite as a part of this PR, but I definitely agree that we're overdue. I actually think that testing would be a great place for you to dig into to set up these things for the long term.