Prototype Data Quality Feature

Is your feature request related to a problem? Please describe.
When creating pipelines, data issues can silently wreak havoc; your code didn't change but the data did and now things are wonky...
To combat that, there are projects like pandera that allow you to annotate functions with expectations, and at runtime, have those expectations checked and appropriately exposed.
Hamilton, should have some support for runtime data quality checks, so that we can not only support clean code bases, but also clean data as well.
Describe the solution you'd like
We should prototype an approach where there is:

A way to set expectations on the output of a function, what the data should like.
Use those expectations either right after function execution, or on conclusion of a Hamilton DAG, or some other way.
A way to specify what should happen when an expectation is not met -- e.g. log warnings, surface warnings, or stop execution.
Thinking of a way to bootstrap these expectations from a dataset -- so that users can update/change expectations easily as time goes on.

Directionally https://pandera.readthedocs.io/en/stable/ seems like a good first approach to try, i.e. via decorators.
Describe alternatives you've considered
This is something that as the prototype is being built out, we're thinking about alternatives considered too.
Additional context
Some ideas on approaches:

https://pandera.readthedocs.io/en/stable/
https://github.com/awslabs/deequ
https://greatexpectations.io/

One idea for implementation -- decorate functions with something that will inspect the output.
@check_output({'type': float, 'range': (0.0, 10000.0)})
def SOME_IMPORTANT_OUTPUT(input1: pd.Series, input2: pd.Series) -> pd.Series:
   """Does some complex logic"""

Some thoughts about implementation:

When/how do we want checks to fail? This could be configurable -- should that be at the local or global level?
We could make this an additional output node -- E.G. for every node outputted by that function, we add a check on it... We have the decorator abstraction to make this really easy.
This could be (a) a runtime error or (b) an output in the DAG itself. We'd want some way of knowing which nodes are outputs to the DAG, see tags for something that could help mark them correctly. If this is a runtime error, then we need to figure out what the node is like (kind of a side-effect?).

I think we should first prototype this as a decorator that appends a pass-through node that runs the check, or just runs the check on its own. Then we can think about running more configurations of different failure modes. But putting the fatality of it in the decorator itself might make a lot of sense. Allowing for thresholds, etc... might be nice as well.
@check_output({'type': float, 'range': (0.0, 10000.0), 'max_empty': 0.0, }, fatal=True)
def SOME_IMPORTANT_OUTPUT(input1: pd.Series, input2: pd.Series) -> pd.Series:
   """Does some complex logic"""

H/T @jimdowling

Jim Dowling 23 minutes ago
For me, data validation can be group into data validation rules that apply at the row level or dataframe level. Assuming we have a pipeline that is computing a dataframe that we want to write somewhere. Then the pipeline will run periodically and you also need to be able to do something about a failed rule - exclude a row, log a row for debugging, fail the write of the whole dataframe, etc.
Then there is data monitoring, which is a whole different problem - like feature monitoring.
Stefan Krawczyk 4 minutes ago
oh yep.
I think excluding/omitting rows would be a little tough — that would be better modeled as a separate transform step. Otherwise logging a warning, or halting execution would be quite do-able and configurable I’m thinking.
Stefan Krawczyk 2 minutes ago
To me data monitoring is just a bit of a layer on top, where it’s auto-adjusting these “rules” — and likely out of scope of Hamilton itself — because Hamilton doesn’t control materialization and wouldn’t know about previous runs by itself.
Jim Dowling 1 minute ago
Agreed on both points

Rough sketch of data quality. Going to implement it as a prototype then send it to a few folks for feedback who've expressed interest. Will tag and hopefully start a discussion!

One thought: handling dataframe/dict of output. Do we specify things on a per column/field basis (at the top level, or within each kwarg)? Or some other way? or both?

One thought: handling dataframe/dict of output. Do we specify things on a per column/field basis (at the top level, or within each kwarg)? Or some other way? or both?

Yeah I just had that same though. I've designed for series, but we should have something for dataframes/dicts. You could imagine something like:
@check_output(column="col_1", max_val=10)
@check_output(column="col_2", max_val=20)
def important_df_function() -> pd.DataFrame:
    ...

Or:
@check_output(col_1={'max_val' : 20}, col_1={'max_val' : 20})
def important_df_function() -> pd.DataFrame:
    ...

With the decorator framework the second is easier but its a little uglier IMO.

hmm - riffing some more ideas here:
Same as above but using dict():
@check_output(col_1=dict(max_val=20), col_2=dict(max_val=20))
def important_df_function() -> pd.DataFrame:
    ...

or
Splitting each kwarg by column:
@check_output(max_val={"col_1": 20, "col_2": 20})
def important_df_function() -> pd.DataFrame:
    ...

I personally like this option the most from a visual esthetics perspective (would change "column" to perhaps "name"):
@check_output(column="col_1", max_val=10)
@check_output(column="col_2", max_val=20)
def important_df_function() -> pd.DataFrame:
    ...

But perhaps this is where we maybe do something like Pandera and one would define some object and instead do:
schema=... # some object type? or a dict of column to dicts? or?

@check_output(schema=schema)
def important_df_function() -> pd.DataFrame:
    ...

@chrisaddy would love for you to branch off of #115 and see what approach would work best for dataframes:
Option 1 - individual checkout outputs for each column
@check_output(name="col_1", max_val=10)
@check_output(name="col_2", max_val=20)
def important_df_function() -> pd.DataFrame:

Option 2 - come up with a Hamilton schema was analogous to Pandera, like in your example -- https://github.com/stitchfix/hamilton/issues/123
schema=... # some object type? or a dict of column to dicts? or?

@check_output(schema=schema)
def important_df_function() -> pd.DataFrame:

Option 3 - integrate pandera as a custom validator -- like in your example https://github.com/stitchfix/hamilton/issues/123


Write shortest possible code: python how to trim string (python)

Please write shortest code example for this question:
python how to trim string in python
How to do it:

Go to python codes
Create new file (named in underscore case, should contain key words from title) with md extension (markdown file).
Propose new file with following content (please use all three blocks if possible - title, code itself and explanations list):

# python how to trim string

```python
code part1 part2 part3 ...
```

- part1 - explain code part 1
- part2 - explain code part 2
- ...

More advanced template for examples and linked solutions. More docs here.

How to trim string


Match code-generation

This was implemented, but I think it would be nice to support:
match some_var {
    4, 5 -> ; // 4 or 5?? maybe with or (||)?
    _ -> ; // default
}

Also the match needs to break by default, unless return/etc is specified?

Should use ,

I'm not a big fan of separators in this case. You could un-ambiguously allow either a single statement or a block on the right hand of a ->, e.g.
match x {
    4, 5 -> print("4 or 5")
    6 -> {
        print("this is not 4");
        print("and it's also not 5")
    }
}

@muscar I don't see what you mean. I think that maybe the -> should be removed when there's a block; as for the arrow in general, I think it makes more sense to show a single statement. (If that's what you're getting at).

@felixangell I misread @MovingtoMars suggestion, I thought he meant using , as the match case separator, and I suggested the syntax in my previous comment.

@muscar Ohh, okay :)

Sorry for not being clear. I just think , is best for multiple expressions, mainly because 3 || 6 technically evaluates to true.

Yeah, it should perform multiple expressions. So:
match x {
    4, 5 -> print("4 or 5");
}

Would be x == 4 || x == 5

Yep, looks great.

Yeah looks nice :+1:

In fact, it might be better to generate as an if/elseif block? So then we can support nice things like doing:
x := "some_str";

match x {
    "another_str" -> printf("wow!\n");
}

Thoughts?

Yeah, I agree, if / else would allow for more use cases. :+1:

@felixangell +1

+1
On Fri, May 15, 2015 at 8:03 PM MovingtoMars<EMAIL_ADDRESS>wrote:

@felixangell https://github.com/felixangell +1
—
Reply to this email directly or view it on GitHub
https://github.com/felixangell/alloy/issues/290#issuecomment-102494186.

Hmm, making the codegen use if-else statements it basically impossible with the current C codegen. This is because a match statement requires the expression to be evalated only once, while using ifs would evaluate it each time.

@MovingtoMars We'll save this issue for LLVM codegen then I suppose?

Guess so.


sys.excepthook error when using beets in bash script

When I am using beets in a bash script and do a redirection of the input, like
beet import $foo < /dev/tty
I get a sys.excepthook error after the input which is causing the script to hang. I don't know where this error is coming from, is there maybe a workaround to just redirect it?
Btw: beets version is 1.3.13, bash version 4.3.30 on Ubuntu 15.04.

I just checked again, it doesn't depend on redirecting the input from tty, the sys hook error keeps coming up:
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr
That's where the script is hanging.

To clarify: this happens every time you invoke beets? Only with the import command? Only in a bash script? Every time it appears in a bash script, or only after certain other commands? Or only at the end of a bash script? Does it happen with other shells, or just bash? Can you paste the full script, and how you're invoking it?

Thanks for the reply, i'll try asap, so far it only occured with the import command on bash, even in minimal bash scripts.

Ok, I tried a simple minimal script, and when using </dev/tty to parse the input from bash to beets, the script hangs after the import. I ran it in beets' debug mode with -v, and the last line I get is "Sending event: cli_exit". The bash script doesn't move on from there. When removing the </dev/tty, beets tells me "error: stdin stream ended while input required" because the input is not being parsed properly. Any ideas?

Ok, I just realized that the sys.excepthook error sometimes appears randomly, even when beets seems to have finished a certain command already. I the CLI, a new prompt appears, and out of nowhere, the error is printed. Really hard to reproduce it. All I can say is that beets successfully finished the commands.

I gave it another try, just figured out that it doesn't depend on the input parsing, even when I call beets with '/usr/local/bin/beet -c $BEETS_CONFIGFILE -v import -q -P -C -W "$ALBUM"' it doesn't work. The last line printed from the debug log is "Sending event: cli_exit", then the bash script hangs.

https://github.com/sampsyo/beets/issues/1507


Generate ADI ModelDesign format from NodeSet2 format

Scope

[x] possibility to import NodeSet file
[x] possibility to convert NodeSet to ModelDesign
[ ] validation against the schema and compliance with the specification (Part 3 and Part 5) of the NodeSet file

Contributes to #8

validation failed - DI is required.


Feature Request: EditorBrowsable.Advanced

Issue Description
UnityEngine has a deprecated public class iPhoneSettings, which always show before if for me. It lives under UnityEngine namespace so I can't avoid importing it.

Since very few class names start with lowercase letters, I wonder if I could control the order of this specific example. Or just exclude classes with lowercase letters from suggestions.
(Also I would like to confirm this behaviour is by-design: this plugin will always list iClass before if when typing on a newline?)
Steps to Reproduce
Open any Unity project repo, create a new script, import UnityEngine namespace, and type.
Expected Behavior
Prefer to list if over iPhoneSettings as it's more likely to be used (given context).
Actual Behavior
iPhoneSettings always listed before if.
Logs
N/A
OmniSharp log
N/A
C# log
N/A
Environment information
VSCode version: 1.33.1
C# Extension: 1.18.0
Mono Information
        OmniSharp using global mono :5.18.1
Dotnet Information
.NET Core SDK (reflecting any global.json):
 Version:   2.2.105
 Commit:    7cecb35b92
Runtime Environment:
OS Name:     Mac OS X
OS Version:  10.14
OS Platform: Darwin
RID:         osx.10.14-x64
Base Path:   /usr/local/share/dotnet/sdk/2.2.105/
Host (useful for support):
Version: 2.2.3
Commit:  6b8ad509b6
.NET Core SDKs installed:
2.1.4 [/usr/local/share/dotnet/sdk]
2.2.105 [/usr/local/share/dotnet/sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.2.3 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.2.3 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.0.5 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 2.2.3 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

Visual Studio Code Extensions

Extension
Author
Version

applescript
idleberg
0.14.2

csharp
ms-vscode
1.18.0

mel
sator-imaging
0.1.2

python
ms-python
2019.3.6139

shader
slevesque
1.1.4

shaderlabvscodefree
amlovey
1.1.3

unity-debug
Unity
2.7.2

vscode-jade-snippets
mrmlnc
1.0.1

The best option is for Unity to decorate these APIs with the EditorBrowsable attribute.
I don't see the C# extension solving this problem.

@richlander I could request it, but there is a problem:
https://docs.microsoft.com/en-us/dotnet/api/system.componentmodel.editorbrowsablestate?view=netframework-4.8
Here is no Deprecated key for EditorBrowsableState, meaning there is no way to tell IntelliSense "we want to deprioritize this class in suggestion".
The Advanced key not well supported and doesn't change display order neither, I believe: https://github.com/dotnet/roslyn/issues/19940

@bitinn @richlander As always, the situation is nuanced:

Roslyn's completion list implementation would theoretically allow the OmniSharp to request/hide "Advanced" members
C# extension/OmniSharp don't currently expose a way for you to control what level you want
Unity doesn't decorate members

With regards to preferring if over iPhoneSettings: VS Code extensions don't have any control over how the list is filtered as you type. We provide the full list of items up front VS Code makes all decisions about what order to choose things.
The best we could do would be for all of the bullet points above to get implemented, in which case iPhoneSettings would always be hidden. However, there's no way for us to use EditorBrowsable data to drive how matches are selected as you type.
I'll keep this open in our backlog as a feature request.

@rchande Thx, I have opened a fix report to Unity, hopefully they will eventually add the EditorBrowsableState.Never to this class.
https://github.com/Unity-Technologies/UnityCsReference/blob/2018.3/Runtime/Export/iOS/iOS.deprecated.cs#L96

This is a very annoying issue I have to admit. Really hope something can be done.

I'd like to chime in to say this impacts me as well.
Much of the time if I quickly type "int" and spacebar it changes to "iPhoneSettings". Over a long period of time this had been a rather large drain on my productivity.

workaround: "editor.snippetSuggestions": "top"

workaround: "editor.snippetSuggestions": "top"

This works. However I'm also getting a full-blown foreach now instead of just the keyword :/
This issue is annoying as fruits :(

Would it be too inconvenient/hard/clumsy/expensive to keep a mini database of the amount of times each word was used and suggest first the most used one that matches what's being typed?

I am frustrated by this one as well. Especially IPhoneSettings often appearing when I want to type if or int. Would it be possible to always prefer such basic language keywords over imported classes?

Closing as the fix for this issue is outside the C# extension.


Confirm payment with hash with indifferent access

Fixes the problem of confirming the resource with Rails 'params'.
The change is to convert the params argument of confirm_resource method to a Hash, so that the rest of the method works correctly with Rails HashWithIndifferentAccess.
I looked into symbolising the keys of a HashWithIndifferentAccess but keys can't be a symbol so that wasn't a good option.
ActiveSupport is now a development dependency, is not required for general use of the gem.
I also added a spec for rejecting the params inside confirm_resource as it was missing.

I have refactored the spec for params rejection, should be clearer now


winreg.h/advapi32.dll: State that RegGetValue accepts null subkey name

Like the older SHRegGetValue function on which it is based, RegGetValue allows you to specify a null reference or empty string for the lpSubkey parameter, which means to query the base key referred to by the hkey parameter.
This change adds that piece of information to the RegGetValue documentation pages.

@metathinker : Thanks for your contribution! The author(s) have been notified to review your proposed change.


Cursor won't jump at the end of line

If I write following command:
systemctl lis<cursor>(t-unit-files)

, then after pressing right arrow I will get following behavior:
systemctl list<cursor>-unit-files

, but expected behavior is following:
systemctl list-unit-files<cursor>

This bug occurs only in 0.6.2-1. Version 0.6.1-1 works as expected.

I got the same problem.
The issue was introduced in commit 676aebd by setting the cursor to max_cursor_pos instead of #BUFFER.
- CURSOR=${#BUFFER}
+ CURSOR=${max_cursor_pos}

where max_cursor_pos is equal to the length of the buffer before the suggestion is appended.
Hence the cursor stays where it is.
The problem also exists when using vi-keybindings.

Thanks guys, major oversight on my part. Will get this fixed soon. And will get a test written up for this so it should never happen again.

Ok please try branch fixes/move-cursor-on-accept (PR #453). It should fix this issue and maintain the fixed behavior for #302 as well.

I also encountered the same problem, please fix it as soon as possible.:rose:

fixes/move-cursor-on-accept works for me. The cursor jumps to the end and dl works as intended.

fixes/move-cursor-on-accept works for me. The cursor jumps to the end and dl works as intended.

I second that.

@ericfreese Thanks for resolving this so quickly


Benchmark against chemprop

Are there any experiments that benchmark MAT against chemprop? Have the folks over at Ardigen already tried this comparison?

Thanks for the interest, and the pointer! We are planning on adding the comparison. I will leave this issue open for the time being, and post here a comment when we finish the experiments.


[MRG] ENH: Improve connectivity data structure

Closes #298. The current change reduces the size of net.connectivity (default network) from 55 MB to 10 MB. I think this could be further reduced by changing how src-target pairs are enumerated. Specifically:
{'gid_pairs': (src_gid1, [target_gid1, target_gid2, ...]), (src_gid2, [target_gid1, target_gid2, ...]), ...}

Also I think this will be a good opportunity to discuss better high level API's. For sparse connectivity, I think the simplest implementation would be adding a probability argument to net.add_connection() that defaults to 1.0, and randomly removes connections passed via gid_pairs.
The previous API allowed for passing a single src_gid, and a list of target_gids. I can certainly restore that ability with helper function, but my feeling is it won't be that useful in the grander scheme of high level functions.

Codecov Report

Merging #313 (9004baa) into master (00ebb0f) will decrease coverage by 0.12%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #313      +/-   ##
==========================================
- Coverage   90.69%   90.57%   -0.13%
==========================================
  Files          14       14
  Lines        2365     2366       +1
==========================================
- Hits         2145     2143       -2
- Misses        220      223       +3

Impacted Files
Coverage Δ

hnn_core/network.py
94.48% <100.00%> (ø)

hnn_core/network_builder.py
91.92% <100.00%> (+0.03%)
:arrow_up:

hnn_core/parallel_backends.py
82.70% <0.00%> (-0.87%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 00ebb0f...9004baa. Read the comment docs.

Agree about adding a probability argument to add_connection !

Finished a "first draft" of a more high level API. src_gids and target_gids can be specified as an int, list, range, or even str! In those cases, net.add_connection(src_gids, target_gids) will produce  an all to all connectivity pattern.
If you want to specify a separate set of target_gids for each src, the first option is to use multiple calls to net.add_connection. Alternatively, you can construct target_gids as a list of lists, where each inner-list corresponds to a unique set of targets for each src_gid.
Also as a nice bonus, I got the size of net.connectivity of the default network down to 3.5 MB.

On Wed 14 Apr 2021 at 14:40, Nick Tolley @.***> wrote:

@.**** commented on this pull request.
In hnn_core/network.py
https://github.com/jonescompneurolab/hnn-core/pull/313#discussion_r613497803
:

       Targets can be uniquely specified for each src_gid by passing a

       list of lists (must contain one element for each src_gid).

That sounds good to me! Are you wanting to enforce the same src_type and
target_type for each connection?

I would enforce this for now. Too much flexibility is not good API. What I
really want is to be able to type print(net.connectivity) and get some
sense of the connections without having to scroll through a ton of text.

It is technically possible to have a mix of cell classes at the moment, it
just doesn't show up in the default instantiation.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/jonescompneurolab/hnn-core/pull/313#discussion_r613497803,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADY6FIUIMV2J2CI3VDWFUGLTIXOQZANCNFSM42RYSZDA
.

--
Sent from my iPhone

@jasmainak how do you like this representation? Printing net.connectivity produces:
[L2_pyramidal -> L2_pyramidal
 Cell counts: 100 srcs; 100 targets
 loc: 'proximal'; receptor: 'nmda'
 weight: 0.0005; delay: 1.0; lamtha: 3.0
  ,
 L2_pyramidal -> L2_pyramidal
 Cell counts: 100 srcs; 100 targets
 loc: 'proximal'; receptor: 'ampa'
 weight: 0.0005; delay: 1.0; lamtha: 3.0
, ...]

yeah that's fantastic!!

Pretty happy with the new data structure! Ready for any final reviews before merging. I think the next PR should be adding the probability argument to net.add_connection.

Thanks @ntolley ! we're making progress 🍻 🥳

I definitely agree with getting the conn dictionaries to match. Working on this PR made me realize is might be worth converting net.connectivity itself to a dict, where the indexing requires specification of all parameters except for gids. For example:
{('L2_basket', 'L2_pyramidal', 'soma', 'gabab'): {'gid_pairs': ...,, 'weight': 0.0005, 'lamtha': 3.0}}

This will have two benefits, we will force net.connectivity to only have the canonical 16 connection classes (the repr can be copied for documentation of valid classes). Any additional connections will come from the drives. All functions that modify connections can index straight to the appropriate connection and operate on gid_pairs. This additionally gets around the aforementioned problem of "redundant" connections.
I think this change will be pretty straight forward now that this PR has laid down the structure of gid_pairs. It's just a matter of moving things around.

I need to ponder over this ...
we should aim to keep the data structures relatively simple, for the benefit of future contributors and our future selves. I have never seen dictionaries being indexed by tuples. Maybe we could have a concatenated string like:
{'L2_basket -> L2_pyramidal (soma/gabab)': {'gid_pairs': ..., }}

then you can index the connections using something similar to Params dict:
net.connectivity['L2_basket -> *']

not entirely sure. I would say let's prioritize the visualization and probability parameters.


ERROR: Failed building wheel for torch-scatter

📚 Installation
Following the docs, installing torch-geometric worked fine. The rest of the packages not so much. I can't get to much out of the error log. Microsoft Visual Studio is up to date. Something seems to be wrong with the cl.exe compiler.
Error_log.txt

Environment

OS: Windows 10
Python version: 3.8.3
PyTorch version: 1.8.1+cpu
CUDA/cuDNN version: none
GCC version: 6.3.0
How did you try to install PyTorch Geometric and its extensions (wheel, source):

From the docs: https://pytorch-geometric.readthedocs.io/en/latest/notes/installation.html
pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.8.1+cpu.html

Any other relevant information:

Checklist

[x ] I followed the installation guide.
[~ ] I cannot find my error message in the FAQ.
[ ] I set up CUDA correctly and can compile CUDA code via nvcc.
[ ] I do have multiple CUDA versions on my machine.

Additional context
I am using Anaconda and tried both the conda installation and using the pip commands. None of them worked. My graphics card sadly doesnt support CUDA so I have to run it on my CPU. This seems to cause the problems. I am out of ideas. Any help would be really appreciated.
Greetings
Gerrit

Sorry for the misconfusion. The correct link is
pip install torch-scatter -f https://pytorch-geometric.com/whl/torch-1.8.0+cpu.html

(even for PyTorch 1.8.0).

Thanks a lot, totally worked that way. Should have properly read the docs where you mention that the pip command with 1.8.0 is for both pytorch 1.8.0 and 1.8.1.

Great :)


net/http: improve performance for parsePostForm

My main concern is using ioutil.ReadAll everytime the io.Reader is read in parsePostForm since it will then trigger bytes.makeSlice and allocate new space each and every time the function is called. Very similar to my open PR to @bradfitz's memcache library (https://github.com/bradfitz/gomemcache/pull/45)
Instead I think using a sync.Pool of *bytes.Buffer would be a fairly better solution.
I've implemented the solution and added a benchmark to show you the performance improvements.
First of, here's the benchmark:
func BenchmarkParsePostForm(b *testing.B) {
	b.ReportAllocs()

	// Create bodies
	bodies := [][]byte{
		make([]byte, 100),
		make([]byte, 1000),
		make([]byte, 10000),
		make([]byte, 100000),
		make([]byte, 1000000),
		make([]byte, 10000000),
	}

	// Run
	b.RunParallel(func(pb *testing.PB) {
		for pb.Next() {
			// Create request
			r, e := NewRequest("POST", "/test", bytes.NewReader(bodies[rand.Intn(len(bodies))]))
			if e != nil {
				b.Error(e.Error())
			}

			// Add content type
			r.Header.Add("Content-Type", "application/x-www-form-urlencoded")

			// Parse post form
			e = r.ParseForm()
			if e != nil {
				b.Error(e.Error())
			}
		}
	})
}

Now here's the first run of the benchmark before the improvement:
$ go test ./src/net/http -v -run=^$ -bench=BenchmarkParsePostForm -benchtime=10s -parallel=4 -memprofile=prof.mem
BenchmarkParsePostForm-4	    2000	  11205322 ns/op	 7821503 B/op	      25 allocs/op
PASS
ok  	net/http	32.253s

$ go tool pprof --alloc_space http.test prof.mem Entering interactive mode (type "help" for commands)
(pprof) top
23140.16MB of 23187.37MB total (99.80%)
Dropped 11 nodes (cum <= 115.94MB)
      flat  flat%   sum%        cum   cum%
17720.08MB 76.42% 76.42% 17720.08MB 76.42%  bytes.makeSlice
 5418.08MB 23.37% 99.79% 23141.66MB 99.80%  net/http.parsePostForm
       2MB 0.0086% 99.80% 17722.08MB 76.43%  io/ioutil.readAll
	 0     0% 99.80% 17720.08MB 76.42%  bytes.(*Buffer).ReadFrom
	 0     0% 99.80% 17722.08MB 76.43%  io/ioutil.ReadAll
	 0     0% 99.80% 23141.66MB 99.80%  net/http.(*Request).ParseForm
	 0     0% 99.80% 23144.16MB 99.81%  net/http_test.BenchmarkParsePostForm.func1
	 0     0% 99.80% 23187.37MB   100%  runtime.goexit
	 0     0% 99.80% 23144.16MB 99.81%  testing.(*B).RunParallel.func1

(pprof) list parsePostForm
Total: 22.64GB
ROUTINE ======================== net/http.parsePostForm in /home/asticode/projects/go/go/src/net/http/request.go
    5.29GB    22.60GB (flat, cum) 99.80% of Total
	 .          .    884:	// RFC 2616, section 7.2.1 - empty type
	 .          .    885:	//   SHOULD be treated as application/octet-stream
	 .          .    886:	if ct == "" {
	 .          .    887:		ct = "application/octet-stream"
	 .          .    888:	}
	 .   512.02kB    889:	ct, _, err = mime.ParseMediaType(ct)
	 .          .    890:	switch {
	 .          .    891:	case ct == "application/x-www-form-urlencoded":
	 .          .    892:		var reader io.Reader = r.Body
	 .          .    893:		maxFormSize := int64(1<<63 - 1)
	 .          .    894:		if _, ok := r.Body.(*maxBytesReader); !ok {
	 .          .    895:			maxFormSize = int64(10 << 20) // 10 MB is a lot of text.
	 .          .    896:			reader = io.LimitReader(r.Body, maxFormSize+1)
	 .          .    897:		}
	 .    17.31GB    898:		b, e := ioutil.ReadAll(reader)
	 .          .    899:		if e != nil {
	 .          .    900:			if err == nil {
	 .          .    901:				err = e
	 .          .    902:			}
	 .          .    903:			break
	 .          .    904:		}
	 .          .    905:		if int64(len(b)) > maxFormSize {
	 .          .    906:			err = errors.New("http: POST too large")
	 .          .    907:			return
	 .          .    908:		}
    5.29GB     5.29GB    909:		vs, e = url.ParseQuery(string(b))
	 .          .    910:		if err == nil {
	 .          .    911:			err = e
	 .          .    912:		}
	 .          .    913:	case ct == "multipart/form-data":
	 .          .    914:		// handled by ParseMultipartForm (which is calling us, or should be)

You can see that ioutil.ReadAll(reader) at line 898 is eating 17.31GB.
Now here's the result of the benchmark after the improvement I suggest:
$ go test ./src/net/http -v -run=^$ -bench=BenchmarkParsePostForm -benchtime=10s -parallel=4 -memprofile=prof.mem
BenchmarkParsePostForm-4	    2000	   7830166 ns/op	 1874354 B/op	      18 allocs/op
PASS
ok  	net/http	17.011s

$ go tool pprof --alloc_space http.test prof.mem Entering interactive mode (type "help" for commands)
(pprof) top
4030.01MB of 4033.51MB total (99.91%)
Dropped 7 nodes (cum <= 20.17MB)
      flat  flat%   sum%        cum   cum%
 3638.12MB 90.20% 90.20%  3998.51MB 99.13%  net/http.parsePostForm
  359.40MB  8.91% 99.11%   359.40MB  8.91%  bytes.makeSlice
   31.99MB  0.79% 99.90%    31.99MB  0.79%  net/http_test.BenchmarkParsePostForm
    0.50MB 0.012% 99.91%  4001.52MB 99.21%  net/http_test.BenchmarkParsePostForm.func1
	 0     0% 99.91%   359.40MB  8.91%  bytes.(*Buffer).ReadFrom
	 0     0% 99.91%  3999.01MB 99.14%  net/http.(*Request).ParseForm
	 0     0% 99.91%  4033.51MB   100%  runtime.goexit
	 0     0% 99.91%  4001.52MB 99.21%  testing.(*B).RunParallel.func1
	 0     0% 99.91%    31.99MB  0.79%  testing.(*B).launch
	 0     0% 99.91%    31.99MB  0.79%  testing.(*B).runN

(pprof) list parsePostForm
Total: 3.94GB
ROUTINE ======================== net/http.parsePostForm in /home/asticode/projects/go/go/src/net/http/request.go
    3.55GB     3.90GB (flat, cum) 99.13% of Total
	 .          .    895:	// RFC 2616, section 7.2.1 - empty type
	 .          .    896:	//   SHOULD be treated as application/octet-stream
	 .          .    897:	if ct == "" {
	 .          .    898:		ct = "application/octet-stream"
	 .          .    899:	}
	 .   512.05kB    900:	ct, _, err = mime.ParseMediaType(ct)
	 .          .    901:	switch {
	 .          .    902:	case ct == "application/x-www-form-urlencoded":
	 .          .    903:		var reader io.Reader = r.Body
	 .          .    904:		maxFormSize := int64(1<<63 - 1)
	 .          .    905:		if _, ok := r.Body.(*maxBytesReader); !ok {
	 .          .    906:			maxFormSize = int64(10 << 20) // 10 MB is a lot of text.
	 .          .    907:			reader = io.LimitReader(r.Body, maxFormSize+1)
	 .          .    908:		}
	 .          .    909:		buf := bufferPool.Get().(*bytes.Buffer)
	 .          .    910:		defer freeBuffer(buf)
	 .   359.40MB    911:		_, e := buf.ReadFrom(reader)
	 .          .    912:		if e != nil {
	 .          .    913:			if err == nil {
	 .          .    914:				err = e
	 .          .    915:			}
	 .          .    916:			break
	 .          .    917:		}
	 .          .    918:		if int64(buf.Len()) > maxFormSize {
	 .          .    919:			err = errors.New("http: POST too large")
	 .          .    920:			return
	 .          .    921:		}
    3.55GB     3.55GB    922:		vs, e = url.ParseQuery(buf.String())
	 .          .    923:		if err == nil {
	 .          .    924:			err = e
	 .          .    925:		}
	 .          .    926:	case ct == "multipart/form-data":
	 .          .    927:	// handled by ParseMultipartForm (which is calling us, or should be)

The very same command, this time at line 911, now eats only 359.40MB for the same number of requests.
If you need any extra information please let me know.
If you think this performance proposal is sound I'll mail the change for review using codereview.
Cheers
Quentin

You've showed that it allocates less, but what about execution times? Is it faster? Is it slower? How much faster/slower?

Actually, instead of a sync.Pool I'd prefer to parse the io.Reader directory into a url.Values instead of doing the conversion from []byte to string on the whole input.
(In the future, it might be nice to not even allocate for the common map keys, since most http.Handlers receiving POSTs will be getting the same map keys repeatedly. But there's no great place to stash that cache, since we don't know which handler the request is for (yet). We might know that later, if #14660 happens, since we could associate a Handler with a Request via context keys... but that's just thinking ahead and not needed for addressing this garbage)

You mean for instance adding a url.ParseQueryFromReader method that takes an io.Reader and returns a url.Values by reading byte by byte in the reader and parsing the body ?

Without adding any new public API, though.

OK so to avoid adding a public API I've added a parsePostFormURLEncoded method in the http package such as :
var (
	byteAmp = []byte("&")
	byteEqual = []byte("=")
	byteSemiColon = []byte(";")
)

func parsePostFormURLEncoded(reader io.Reader, vs url.Values, maxFormSize int64) (err error) {
	var buf bytes.Buffer
	var key string
	var value string
	var i int64
	b := make([]byte, 1)
	for {
		if i >= maxFormSize {
			err = errors.New("http: POST too large")
			return
		}
		n, e := reader.Read(b)
		if e != nil && e.Error() != io.EOF.Error() {
			err = e
			return
		}
		i += int64(n)
		if bytes.Equal(b, byteEqual) {
			key, e = url.QueryUnescape(buf.String())
			if e != nil {
				err = e
				return
			}
			buf.Reset()
		} else if bytes.Equal(b, byteAmp) || bytes.Equal(b, byteSemiColon) || n == 0 {
			value, e = url.QueryUnescape(buf.String())
			if e != nil {
				err = e
				return
			}
			vs[key] = append(vs[key], value)
			buf.Reset()
		} else {
			buf.Write(b)
		}
		if n == 0 {
			return
		}
	}
	return
}

Is that what you had in mind ?
Because running the same benchmark gives me worse results than the last 2 benchmarks :
$ go test ./src/net/http -v -run=^$ -bench=BenchmarkParsePostForm -benchtime=10s -parallel=4 -memprofile=prof.mem
BenchmarkParsePostForm-4	     500	  30509320 ns/op	 7506321 B/op	      29 allocs/op
PASS
ok  	net/http	26.665s

Just by looking at this result we can see that it's 3 times slower and allocates as much as the original implementation (since this time it only does 500 repetitions vs 2000 last time)
$ go tool pprof --alloc_space http.test prof.mem Entering interactive mode (type "help" for commands)
(pprof) top
6091.52MB of 6093.02MB total (  100%)
Dropped 2 nodes (cum <= 30.47MB)
Showing top 10 nodes out of 12 (cum >= 6050.36MB)
      flat  flat%   sum%        cum   cum%
 4644.73MB 76.23% 76.23%  4644.73MB 76.23%  bytes.makeSlice
 1403.63MB 23.04% 99.27%  6048.36MB 99.27%  net/http.parsePostFormURLEncoded
   42.66MB   0.7%   100%    42.66MB   0.7%  net/http_test.BenchmarkParsePostForm
    0.50MB 0.0082%   100%  6048.86MB 99.28%  net/http.parsePostForm
	 0     0%   100%  4644.73MB 76.23%  bytes.(*Buffer).Write
	 0     0%   100%  4644.73MB 76.23%  bytes.(*Buffer).grow
	 0     0%   100%  6048.86MB 99.28%  net/http.(*Request).ParseForm
	 0     0%   100%  6050.36MB 99.30%  net/http_test.BenchmarkParsePostForm.func1
	 0     0%   100%  6093.02MB   100%  runtime.goexit
	 0     0%   100%  6050.36MB 99.30%  testing.(*B).RunParallel.func1
(pprof) list parsePostFormURLEncoded
Total: 5.95GB
ROUTINE ======================== net/http.parsePostFormURLEncoded in /home/asticode/projects/go/go/src/net/http/request.go
    1.37GB     5.91GB (flat, cum) 99.27% of Total
	 .          .    940:				err = e
	 .          .    941:				return
	 .          .    942:			}
	 .          .    943:			buf.Reset()
	 .          .    944:		} else if bytes.Equal(b, byteAmp) || bytes.Equal(b, byteSemiColon) || n == 0 {
    1.37GB     1.37GB    945:			value, e = url.QueryUnescape(buf.String())
	 .          .    946:			if e != nil {
	 .          .    947:				err = e
	 .          .    948:				return
	 .          .    949:			}
	 .          .    950:			vs[key] = append(vs[key], value)
	 .          .    951:			buf.Reset()
	 .          .    952:		} else {
	 .     4.54GB    953:			buf.Write(b)
	 .          .    954:		}
	 .          .    955:		if n == 0 {
	 .          .    956:			return
	 .          .    957:		}
	 .          .    958:	}

Is there any particular reason you don't want to go the sync.Pool way ?
If you still do, is there anything I've misinterpreted from your guidelines that would worsen the benchmark ?

If you want me to do a code review, please send a change in Gerrit. See https://golang.org/doc/contribute.html
Otherwise I can look into this later, writing a fast version. There are a number of things not fast about your code.

Just sent the change in Gerrit. Looking forward for your code review :)
Cheers

@odeke-em, are you interested in taking this over now that the tree is open?
Or @asticode, if you're still interested you could pick up your CL and finish it.

@bradfitz I'll try to pick it up this week-end
@odeke-em let me know if you're working on this as well
Cheers

Thank you for the assignment @bradfitz am very happy to help out with this.
Alright, @asticode we can work on your CL, I am already one of the reviewers on it so I'll be updated when there is movement on it, thanks.

@odeke-em FYI I've just pushed my changes with the benchmark included in the commit message


ValuesOnCorrectTypes validation rule

Pull Request Test Coverage Report for Build 737

146 of 149 (97.99%) changed or added relevant lines in 11 files are covered.
2 unchanged lines in 2 files lost coverage.
Overall coverage increased (+2.2%) to 81.15%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/Error/GraphQLException.php
6
7
85.71%

src/Language/Visitor/SpecificKindVisitor.php
4
6
66.67%

Files with Coverage Reduction
New Missed Lines
%

src/Provider/ScalarTypesProvider.php
1
71.19%

src/Type/definition.php
1
42.42%

Totals

Change from base Build 733:
2.2%

Covered Lines:
3840

Relevant Lines:
4732

💛  - Coveralls

Pull Request Test Coverage Report for Build 737

146 of 149 (97.99%) changed or added relevant lines in 11 files are covered.
2 unchanged lines in 2 files lost coverage.
Overall coverage increased (+2.2%) to 81.15%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/Error/GraphQLException.php
6
7
85.71%

src/Language/Visitor/SpecificKindVisitor.php
4
6
66.67%

Files with Coverage Reduction
New Missed Lines
%

src/Provider/ScalarTypesProvider.php
1
71.19%

src/Type/definition.php
1
42.42%

Totals

Change from base Build 733:
2.2%

Covered Lines:
3840

Relevant Lines:
4732

💛  - Coveralls


Create a branch off of 1.4.1, so PRs can be made against that version

I was debugging https://github.com/slackhq/hubot-slack/issues/203 and the fix ended up being in node-slack-client. I have a branch off of that, but I can't make a PR for it because the only branch is master.
To create, it'd be like:

git checkout 1.4.1
git checkout -b 1.4.x
git push origin 1.4.x

Actually, it'd be better to make it 1.x rather 1.4.x, assuming we are using semantic versioning.

https://github.com/slackhq/node-slack-client/tree/1.5.0 is now a thing
I'd be interested in chatting about what kind of upgrade path to future versions of the node-client you're thinking of. There's been a lot of work on handling reconnection logic since 1.4.1 and there'll be some additional features around that coming shortly as well.

https://github.com/slackhq/node-slack-client/tree/1.5.0 is now a thing

Thanks. I was actually thinking to literally call it 1.x so if there's any other changes, that can effectively be the master of the 1.x line, so PRs can be made from that. That way, it's a little more future proof than just this one change, specifically so we don't have to make more branches named like that or have it be confusing if we keep making changes against it. Also: I think it would conflict with a 1.5.0 tag whenever that happens.

I'd be interested in chatting about what kind of upgrade path to future versions of the node-client you're thinking of. There's been a lot of work on handling reconnection logic since 1.4.1 and there'll be some additional features around that coming shortly as wel

I'd ask you the same thing since you've been working on this :grin:  If you are using semantic version, 2.0 is a major release, and does allow for breaking API changes. However, that doesn't necessarily make it easy for people to upgrade to it.
I'd start with upgrading an existing application using 1.x to 2.x, and just see what breaks. hubot-slack would be a good one, but maybe it's too complicated to try for the first pass. I think that would highlight a bunch of changes. If there are a lot of things breaking, it'd be worth asking backwards compatability should be added to the 2.x branch.
If there are specific things that are definitely changing in 2.x, it's worth adding deprecation messages in 1.x. If it's functions or events being rename, you can keep the old one there, log the deprecation, and then pass it onto the new thing. If it's function parameters changing (ie adding more parameters), you can check the function arity to conditionally do an old behavior and log the warning, vs doing the new behavior. By doing this, something using the older version of the library would get hints on how to upgrade.

1.x is up at https://github.com/slackhq/node-slack-client/tree/1.x. I'll write up a response to the other questions this weekend.

I would be interested in seeing the other responses.  :) Trying to gauge if I use a work around or wait for #183

Is this issue resolved?

This particular issue, ie having a 1.x branch is resolved, see https://github.com/slackhq/node-slack-sdk/tree/1.x


GitHub Workflows: Upgrade to checkout v4.1.4

Upgrade to actions/checkout v4.1.4 in our GitHub workflows. I noticed we were using inconsistent versions of it (v2, v3, v4), so why not upgrade to the latest.

Yes, please.


ISPN-14195+ISPN-14203 Hit count efficiency contributions

https://issues.redhat.com/browse/ISPN-14195
https://issues.redhat.com/browse/ISPN-14203

Thanks @wburns. I applied the changes here.

Thank you @wburns for the further review.
I did the changes.
About the comment I would like to keep 10,000 as default hit count accuracy limit.

Merged, thanks

Thank you for merging it!


[issue-126] Update default transaction timeouts in FlinkPravegaWriter to sync with Pravega defaults

Signed-off-by: Vijay Srinivasaraghavan<EMAIL_ADDRESS>Change log description
This PR addresses the issue https://github.com/pravega/flink-connectors/issues/126

The transaction timeouts are updated to use the maximum defaults of Pravega

Purpose of the change

To avoid flink app from failing due to larger transaction timeouts

What the code does

configures default timeouts when the value is not supplied

How to verify it
./gradlew clean build should pass

Here's the two options that I favor:

Rename withTxnTimeout to withTxnLeaseRenewalPeriod, default value of 30 seconds:

    /**
     * Sets the transaction lease renewal period.
     *
     * When the writer mode is set to {@code EXACTLY_ONCE}, transactions are used to persist events to the Pravega stream.  The transaction interval corresponds to the Flink checkpoint interval.   Throughout that interval, the transaction is kept alive with a lease that is periodically renewed.    This configuration setting sets the lease renewal period.  The default value is 30 seconds.
     *
     * @param period the lease renewal period
     */
    public B withTxnLeaseRenewalPeriod(Time period)

Keep the API as-is, change the default value to 8 hours, and divide the value by 1000 when configuring the underlying Pravega client (since it works in terms of lease renewal period).

My suggestion is to merge this PR and create a separate issue post EAP since any change to the API will impact other dependencies.

Can this be merged?


Update npm package configs

Alaska Airlines Pull Request
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes: #146
Summary:
Please summarize the scope of the changes you have submitted, what the intent of the work is and anything that describes the before/after state of the project.
Changes to be committed:
modified:   .npmignore
modified:   package.json
This PR updates the building process of the element and produces the following packaged deliverable. it should be noted that the same 56 files are in the package, but simply the location was changed. There is no performance increase or package size change with this update.
/datepicker/package
├── CHANGELOG.md
├── LICENSE
├── README.md
├── demo
|  ├── api.js
|  ├── api.md
|  ├── api.min.js
|  ├── demo.md
|  ├── index.js
|  ├── index.md
|  └── index.min.js
├── dist
|  ├── auro-datepicker__bundled.js
|  ├── src
|  |  ├── auro-calendar-cell.d.ts
|  |  ├── auro-calendar-cell.d.ts.map
|  |  ├── auro-calendar-month.d.ts
|  |  ├── auro-calendar-month.d.ts.map
|  |  ├── auro-calendar.d.ts
|  |  ├── auro-calendar.d.ts.map
|  |  ├── auro-datepicker.d.ts
|  |  ├── auro-datepicker.d.ts.map
|  |  ├── style-auro-calendar-cell-css.d.ts
|  |  ├── style-auro-calendar-cell-css.d.ts.map
|  |  ├── style-auro-calendar-css.d.ts
|  |  ├── style-auro-calendar-css.d.ts.map
|  |  ├── style-auro-calendar-month-css.d.ts
|  |  ├── style-auro-calendar-month-css.d.ts.map
|  |  ├── style-css.d.ts
|  |  └── style-css.d.ts.map
|  └── vendor
|     └── wc-range-datepicker
|        ├── day.d.ts
|        ├── day.d.ts.map
|        ├── range-date-picker-cell.d.ts
|        ├── range-date-picker-cell.d.ts.map
|        ├── range-datepicker-calendar.d.ts
|        ├── range-datepicker-calendar.d.ts.map
|        ├── range-datepicker.d.ts
|        └── range-datepicker.d.ts.map
├── index.js
├── package.json
├── packageScripts
|  └── postinstall.mjs
├── src
|  ├── auro-calendar-cell.js
|  ├── auro-calendar-month.js
|  ├── auro-calendar.js
|  ├── auro-datepicker.js
|  ├── style-auro-calendar-cell-css.js
|  ├── style-auro-calendar-css.js
|  ├── style-auro-calendar-month-css.js
|  └── style-css.js
└── vendor
   └── wc-range-datepicker
      ├── day.js
      ├── day.js.map
      ├── range-date-picker-cell.js
      ├── range-date-picker-cell.js.map
      ├── range-datepicker-calendar.js
      ├── range-datepicker-calendar.js.map
      ├── range-datepicker.js
      └── range-datepicker.js.map

directory: 9 file: 54

Type of change:
Please delete options that are not relevant.

[ ] New capability
[x] Revision of an existing capability
[ ] Infrastructure change (automation, etc.)
[x] Other (please elaborate)

Checklist:

[x] My update follows the CONTRIBUTING guidelines of this project
[x] I have performed a self-review of my own update

By submitting this Pull Request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Pull Requests will be evaluated by their quality of update and whether it is consistent with the goals and values of this project. Any submission is to be considered a conversation between the submitter and the maintainers of this project and may require changes to your submission.
Thank you for your submission!
-- Auro Design System Team

:tada: This PR is included in version 2.0.7 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Nightly: Increase timeouts

Jenkins Nightly test failed due timeout in the following builds:
https://jenkins.cilium.io/job/Cilium-Master-Nightly-Tests-All/143/console
https://jenkins.cilium.io/job/Cilium-Master-Nightly-Tests-All/142/console
Signed-off-by: Eloy Coto<EMAIL_ADDRESS>

test-me-please


gh-786 - Updated try-catch blocks to consistently use final keyword.

Codecov Report

Merging #787 into develop-0.7.0 will not change coverage.
The diff coverage is 0.85%.

@@               Coverage Diff                @@
##             develop-0.7.0     #787   +/-   ##
================================================
  Coverage            58.56%   58.56%
  Complexity            2697     2697
================================================
  Files                  432      432
  Lines                12498    12498
  Branches              1427     1427
================================================
  Hits                  7320     7320
  Misses                4603     4603
  Partials               575      575

Impacted Files
Coverage Δ
Complexity Δ

...named/operation/handler/NamedOperationHandler.java
6.66% <0%> (ø)
2 <0> (ø)
:arrow_down:

...on/hdfs/mapper/SampleDataForSplitPointsMapper.java
0% <0%> (ø)
0 <0> (ø)
:arrow_down:

...ulostore/operation/handler/GetElementsHandler.java
60% <0%> (ø)
3 <0> (ø)
:arrow_down:

...bitmap/serialisation/utils/RoaringBitmapUtils.java
68.18% <0%> (ø)
6 <0> (ø)
:arrow_down:

.../gchq/gaffer/data/elementdefinition/view/View.java
69.73% <0%> (ø)
22 <0> (ø)
:arrow_down:

...e/retriever/impl/AccumuloAllElementsRetriever.java
0% <0%> (ø)
0 <0> (ø)
:arrow_down:

...n/java/uk/gov/gchq/gaffer/store/schema/Schema.java
71.51% <0%> (ø)
22 <0> (ø)
:arrow_down:

...v/gchq/gaffer/accumulostore/MockAccumuloStore.java
71.42% <0%> (ø)
7 <0> (ø)
:arrow_down:

...tore/key/core/impl/CoreKeyBloomFilterIterator.java
65.62% <0%> (ø)
7 <0> (ø)
:arrow_down:

...v/gchq/gaffer/accumulostore/utils/IngestUtils.java
51.47% <0%> (ø)
9 <0> (ø)
:arrow_down:

... and 60 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ca6aad...3e85f13. Read the comment docs.


ColumnChart Adding Labels Above Each Bar

Is there a way to add a label above each bar in a ColumnChart / BarChart? I am trying to display the vAxis value above each bar similar to if you hover over it, but for it to always be there. I'll add what I am trying to replicate.

@Isaiaher I am trying to do the same thing with a ColumnChart.  Were you able to do this?  Thanks.

@Isaiaher You can use the { role: 'annotations } with columns labels
You can find it here https://developers-dot-devsite-v2-prod.appspot.com/chart/interactive/docs/gallery/columnchart


This change is

I don't want to do it now, probably we need to use metaclass or other mechanism to merge decorators when creating views in viewset


One-time news (notifications)

As development of markdown-notes continues and new features are added, we should think of some mechanism of notifying the old users about new features. The list of features is kept in the Getting started note, but that is generated only one time, when the user is registered, so they are not aware about any new features introduced after their registration (unless they follow this repo, which is very unlikely).
So I would propose one-time news, which are displayed to user only once after login or opening the editor (in case they have active Remember me login feature). Actually I would force them to confirm reading it, in the way of hitting closing cross, which would than send notification to server and the the news would be marked for them as read and never displayed again.
The news could be added either through admin section or through migrations.

Implemented and merged in #50 with bit of design tweaking in #51


fix(1468): console error in tab component

By creating this pull request you agree to the terms in CONTRIBUTING.md.
https://github.com/Infineon/.github/blob/master/CONTRIBUTING.md
--- DO NOT DELETE ANYTHING ABOVE THIS LINE ---
Description
A console error would appear in react app in tab component because of useRef, fixed it by using useState hook.
Additionally, implemented array-index-out-of-bound check in tabs code component.
Related Issue
#1468

📦 Published PR as canary version: 25.7.1--canary.1532.973c0cf5060dd8d68b01c0ee1c7c1cae3363135c.0

:sparkles: Test out this PR locally via:
npm install<EMAIL_ADDRESS># or
yarn add<EMAIL_ADDRESS>

Can be merged, after the merge conflicts are resolved :)


Python 3 compatibility

Switch unicode checks to six.text_type
Switch str checks to builtins.bytes checks
Modify tests to work with python 3
Run travis with both python 2.7 and 3.5

@cfstout

Cool stuff. Appreciate the PR. Feel free to merge down 👍

@joeslice do you have write access or can you comment someone who does? 🔑

I'm not sure I understand what you mean by merge down? I don't have permissions to merge.

@cfstout @joeslice Is it possible for this to be reviewed and merged? I need it for a project that I'm working on.

@wesleybalvanz-wf I'll get someone who has write access to merge today.

LGTM, build passes with changes.


Added a description to be able to use text extraction.

It took a while until the reading of the PDF content worked. This is to help others save time.

Thank you!


BaseDataStore delete and deleteEntries test

We will create a BaseDataStoreTest and expect the delete and deleteEntries methods to be fully tested.  This requires an understanding of expected behavior for these methods.  Queries and write/ingest will be covered in other issues.

relates to #1002 and #1004

No longer relevant


Package Creation, Step by Step

What this PR does / why we need it
There have been a few questions in the Slack channels asking about creating a package. I decided to make a document that takes a user through creating a package, step by step.
Details for the Release Notes (PLEASE PROVIDE)

package creation, step by step

Which issue(s) this PR fixes

Fixes: #
Describe testing done for PR
none - this is just a doc.
Special notes for your reviewer

This might be better as a guide instead of a blog.


Update useNow.ts

Refactor useNow hook to accurately update as the time changes instead of every 60 seconds

Hey, thanks for taking the time to review. Yeah I understand but in my case I need to show the current time and now was not updating precisely enough. Just thought I would make the PR just in case. Thanks for the great library!

Thanks. If the PR provides a customization option for the interval, it would be nice to merge.


Add Double Split Complementary Color Harmony

The Harmonies plugin seems to be missing the double split complementary color harmony.
I think it will be a great addition and should be simple to add as it is just a [-30, 0, 30, 150, 210] rotation array.

Hi! Thanks for the issue!
@EricRovell could I ask to add this one to the harmonies plugin?

@omgovich Yes, I will do it :)
Thank you, @lbragile

Closed via #69

Available in v2.6. Thanks, guys!


setCookie example

It would be very helpful if there were a simple setCookie() example.
Does this command accept an array in the following format?  e.g.
...
chromy.setCookie([
  {
    "domain": ".www.mydomain.com",
    "expirationDate":<PHONE_NUMBER>,
    "hostOnly": false,
    "httpOnly": false,
    "name": "abc",
    "path": "/",
    "sameSite": "no_restriction",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "renderMode=vanilla",
    "id": 112
  },
  {
    "domain": ".www.mydomain.com",
    "expirationDate":<PHONE_NUMBER>,
    "hostOnly": false,
    "httpOnly": false,
    "name": "xyz",
    "path": "/",
    "sameSite": "no_restriction",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "featureFlag=foo",
    "id": 112
  }
])
...

Currently setCookie() does not support an array.
But I think supporting array is very helpful.
I'll support it in next version and add example of cookie.

As we wait for the update to include cookies inside of an array, is it possible to set multiple cookies at all? For example:
const cookiesArray = [{...cookie1}, {...cookie2}, ... ];

cookiesArray.forEach(cookie => {
    chromy = chromy.setCookie(cookie);
  });

return chromy;

Will this successfully add all the cookies in the array to chromy?

@shanemcgraw If chromy variable in that code represents a chain object -- return value of chromy.chain() -- that code has no problem.

Thanks :) Just a bit confused as to what you mean by 'chain object'... would you mind posting a quick example here?

Also, is this the syntax for a chrome cookie for testing locally?
  .setCookie({
        "domain": ".",
        "url": "/",
        "expirationDate":<PHONE_NUMBER>,
        "hostOnly": false,
        "httpOnly": false,
        "name": "test",
        "path": "/",
        "sameSite": "no_restriction",
        "secure": false,
        "session": false,
        "storeId": "0",
        "value": "ABCDE",
        "id": 1000
      })

@shanemcgraw
const chromy = new Chromy()
const chainObject = chromy.chain()
// chainObject is chain object.

@garris
most simple example is here:
.setCookie({url: 'http://localhost/', name: 'name1', value: 'val1'})

Chromy#setCookie() is a wrapper of this:
https://chromedevtools.github.io/devtools-protocol/tot/Network/#method-setCookie

ok. thanks. WIll keep working on it.

Ok got the cookie working in my example.  thanks.

Here is an example of setting a cookie from an array of cookies...
const URL = 'https://<IP_ADDRESS>:3001/featureTests/';
const COOKIES = [
  {
    "url": "https://<IP_ADDRESS>:3001/",
    "expirationDate":<PHONE_NUMBER>,
    "hostOnly": false,
    "httpOnly": false,
    "name": "test2",
    "sameSite": "no_restriction",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "XYZ",
    "id": 1000
  },
  {
    "url": "https://<IP_ADDRESS>:3001/",
    "expirationDate":<PHONE_NUMBER>,
    "hostOnly": false,
    "httpOnly": false,
    "name": "test",
    "sameSite": "no_restriction",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "ABCDE",
    "id": 1001
  },
  {
    "url": "https://<IP_ADDRESS>:3001/",
    "expirationDate":<PHONE_NUMBER>,
    "hostOnly": false,
    "httpOnly": false,
    "name": "test3",
    "sameSite": "no_restriction",
    "secure": false,
    "session": false,
    "storeId": "0",
    "value": "FOO",
    "id": 1001
  }

]

let chromy = new Chromy({chromeFlags: ['--window-size=1280,800'], visible: true});

var chromyChain = chromy.chain();

chromyChain
  .console(function (text, consoleObj) {
    if (console[consoleObj.level]) {
      console[consoleObj.level]((consoleObj.level).toUpperCase() + ' > ', text);
    }
  })

COOKIES.forEach((cookie) => {
   chromyChain = chromyChain.setCookie(cookie);
})

chromyChain
  .goto(URL)
  .wait('._READY')
  .screenshotMultipleSelectors(
    ["body", "h1", "h2"],
    handlescreenshots,
    {useQuerySelectorAll: true}
  )
  .end()
  .then(_ => {
    chromy.close()
    console.log('success');
  })
  .catch(e => {
    chromy.close()
    console.log('error caught > ' + e);
  });

function handlescreenshots(error, png, index, selectors, sub){
  console.log('>>>',error, selectors[index], sub);
  fs.writeFileSync(`./screens/${selectors[index]}_${sub}.png`, png);
}

and also, FWIW, here is how it is done in backstopjs...
https://github.com/garris/backstop-feature-tests/blob/master/backstop_data/engine_scripts/onBefore.js


Add README to describe the STAC domain model

We need to generate some text for the README that describes the domain model of the STAC at a high level. Some components to be defined are:

SpatioTemporal Asset Catalog (STAC): This is the general name of the catalog system specified in this repository.
Item: An Item is contained in a STAC, and is described by a OGC SimpleFeature (expressible as GeoJson) that contains the footprint and timestamps of one or more linked Items or Assets.
Asset:  An Asset represents the final thing a Catalog can point to, such as a GeoTiff representing imagery, a laz file representing some lidar information, a metadata file describing other assets, etc.

We should potentially also include the concept for a STAC that was introduced by the Static Catalog working group in the Boulder Sprint, namely the idea that a STAC JSON representation can point to other STACs, creating a tree of crawlable, hierarchically grouped STACs.

Working with https://github.com/radiantearth/stac-spec/blob/feature/add-flat-file-spec/spec/json/README.md. @notthatbreezy will iterate on this over implementing a full NAIP example, and work on writing up the open api spec.


feat: added player#teleportWithFlags for relative teleportation

I have no idea the use case for this, more info required.

I have no idea the use case for this, more info required.

The flags let you teleport the client with relative values rather than absolute values.
They also let you preserve the clients' velocity/look angles which let you smoothly teleport a player without forcing their yaw/pitch to be a specific way or stopping their momentum.
An excellent use case for this would be an endless staircase which teleports you back to the bottom without being noticeable because your yaw/pitch was never forcefully set nor momentum lost.

Thanks for the info. Some comments:

I think this API should just be an overload to the normal teleport function, not its own function (just a rename, but it would be better if there was not duplication, ie teleport can just call relative teleport with no flags set)
I would prefer if the flags were just int constants OR-ed together instead of enum varargs (with some extras for X|Y|Z and YAW|PITCH perhaps)

We can modify this code to simply work with the teleport function instead
Original I did have them as int constants, however from an API standpoint I couldn't figure out an intuitive place to put the constants for people to find (any ideas?). Other places use Enum flags like item metadata as well

Thanks for the info. Some comments:

I think this API should just be an overload to the normal teleport function, not its own function (just a rename, but it would be better if there was not duplication, ie teleport can just call relative teleport with no flags set)
I would prefer if the flags were just int constants OR-ed together instead of enum varargs (with some extras for X|Y|Z and YAW|PITCH perhaps)

Not a fan of int flags. Neither future-proof nor intuitive. varargs are better suited for an api imo.

Alright im convinced. It should not be declared halfway though the Player class however. Should be on its own preferably or at the top maybe.

@mworzala
Big changes here

Refactored to simply be a part of the entity's teleport method
Removed the includeSelf argument from synchronizePosition (we no longer need it) and replaced with Player#synchronizePositionAfterTeleport
Moved the enum to its own file
Added a way to get common flag cases in the RelativeFlag enum to reduce code bloat when including a bunch of flags
Tidied up the teleport function (also no longer need to explicitly send the head packet because synchronize does that now)


Url encode the request before calling STS backend

Our backend expect the request fields to be url encoded.
@markdroth Please take a look and let me know if you have any questions. Thanks!

Commits squashed. Please trigger the tests if there is no other issue. Thanks!

Hey @markdroth , is it safe to merge this or should we re-trigger the tests?

Known issue: #25095

Known issue: #25095


Mythril has a problem at EtherCoin contract

hi, I test the EtherCoin contract with Mythril, but it takes me an error.
//SPDX-License-Identifier: MIT

pragma solidity ^0.7.0;
contract EtherCoin{
    EtherPool public etherpool;
    mapping (address=>uint) Balances;
    function hasFunds(address user) public view returns(bool){
        if(Balances[user]>0){return true;}else{return false;}
    }
    function creator(address payable user) internal{
        etherpool=(new EtherPool){value:Balances[user]}({_ethercoinAddress:address(this),user:user});
        Balances[user] = 0; //set balance to zero
    }
    function deposit() external payable{
        Balances[msg.sender]+=msg.value;
    }
    function withdrawAll() external {
        creator(payable(msg.sender));
    }

}
contract EtherPool{
    EtherCoin public ethercoin;
    constructor(address _ethercoinAddress,address payable user) payable{
        if(user!=address(0)){
            ethercoin=EtherCoin(_ethercoinAddress);
            if(ethercoin.hasFunds(user)){
                (bool success,)=user.call{value:msg.value}("");
                require(success,"revert");
            }
        }
    }
}

Error:
mythril.interfaces.cli [ERROR]: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/mythril/interfaces/cli.py", line 961, in parse_args_and_execute
    execute_command(
  File "/usr/local/lib/python3.8/dist-packages/mythril/interfaces/cli.py", line 871, in execute_command
    report = analyzer.fire_lasers(
  File "/usr/local/lib/python3.8/dist-packages/mythril/mythril/mythril_analyzer.py", line 180, in fire_lasers
    issue.add_code_info(contract)
  File "/usr/local/lib/python3.8/dist-packages/mythril/analysis/report.py", line 151, in add_code_info
    codeinfo = contract.get_source_info(
  File "/usr/local/lib/python3.8/dist-packages/mythril/solidity/soliditycontract.py", line 202, in get_source_info
    file_index = mappings[index].solidity_file_idx
IndexError: list index out of range

I use solc-select use 0.7.3
thanks.

another problem is Mythril is not working on Rubixi and Parity wallet contract.
I test other smart contracts from the SWC repo and it works well.

@BuggerBag what version of mythril are you using?

hi, here is:
Mythril version v0.22.32


Run|Debug hover actions.

This hover actions work exactly like corresponding lenses.

@vsrs could you remove @mention from the commit message? They create pings when folks run rebases in their forks :)

They create pings when folks run rebases in their forks :)

Like this :)

@vsrs could you remove @mention from the commit message? They create pings when folks run rebases in their forks :)

Ups, sorry. Sure, I'l remove it.

I assume
bors r+
?

Yes. Didn't notice that all checks were already completed.

Yes. Didn't notice that all checks were already completed.

You don't need to wait until the checks are completed. It's totally fine to r+ immediately after the push, the PR won't be merged if it fails the checks. Generally, the earlier you r+, the better.

Generally, the earlier you r+, the better.
OK. I'll remember that next time.


Enabled exclude-dir to allow specifying multiple directories

I needed to specify more than one directory to ignore.  The following commit allows specifying multiple directories using either commas or semicolons as separators.  It provides the expected results locally.

I agree with the handling of the multiple exclude dirs. It is nice to have that option. The suppression of the lib includes however I do not really see the value of.  Having them in the output file should not hinder compilation. I can understand the output file is 'cleaner' without them, but there is no real value of having a clean output file. I'd prefer to keep this utility as simple as possible...


Checking on EET compatibility

Hello Endarire,
I am putting together a ridiculously huge EET installation together, essentially every quest, NPC, item, spell, and kit mod plus a healthy dose of tweaks and tactics mods.
I wanted to include CoM_Solaufein's Undying mod into my installation, but there was some uncertainty with its EET compatibility. What has your experience been with Undying in an EET environment?
Here is my Project Infinity installation sequence so far, but I am still trying to perfect it. Do you have an installation sequence or WeiDU log for a epic EET installation you could share?
P&P'sPreposterousEETInstallSequence.csv

To my present understanding, The Undying mod version I include on this repository has been updated with the EET auto converter (meaning EET compatibility) and the Big World Fix Pack fixes.  As of this writing, it doesn't try to fix any bugs or expand the scope beyond this.
As for other matters, let's talk more on Discord about this since that's easier for me to handle.  My Discord ID is Endarire.
Thankee!

Thanks for the update! I sent you a Discord request.

What is your Discord ID?


How to use with angular 5

how do i use following code in angular 5:
$('table').stickyTableHeaders({scrollableArea: $('.scrollable-area')});

I'm looking a way to use with Angular 2+ as well, you have any luck @sushmit-oodles ???


c7n-mailer Fixed LDAP lookup in Assumed Roles #1350

Wondering what the status of this is.  I don't see it in master or latest.
jtroberts83 committed on Jun 29
commit 2e06be6c87ff6b41ed74894294e5e910bf79810c
contains needed code in email_delivery.py:
        if identity['type'] == 'AssumedRole':
            assume_role_msg = 'AssumedRole Found: %s' % identity['arn']
            self.logger.debug(assume_role_msg)
            user_id = identity['principalId'].split(':', 1)[-1]
            return user_id

@JCarey23 it looks like the original PR was closed by @jtroberts83 and was not merged.
https://github.com/capitalone/cloud-custodian/pull/1350


Array idioms

Hi, am wondering what the current idioms are for Vue dat array handling in vue-typescript?
i.e:

adding an element to an array?

removing an element from an array?
changing (updating) an element in an array?
removing all elements from an arry?

Thanks
David

I'm not 100% sure what you're refering to,.
If you're talking about this, it works exactly the same way and in standard vuejs. In fact, the vue typings even add autocompletion for $set and $remove.
I personally use push() and splice() as well as standard array modifications (if it is an array of objects, else i use $set). To clear an array i guess this works: this.array = []
For this, i guess whatever you would do in Javascript Vue is valid in vue-typescript
Hopefully i didn't completely miss your point.

Seems like these work on both standard Vue arrays, and Vuex arrays changed by mutations:
    // 1) remove all:

    arr = []

    // 2) remove element by ID: apparently filter is efficient:

    arr =  arr.filter(o => o[id_name] != id)

    // 3) upsert element, and have its properties update the GUI too:

    const existing = _.find(arr, id_name, id)
        if( existing )
           _.assign(existing, child)
       else
          arr.push(child)

Adding this purely as a comment for other searchers...
The following syntax works for Vue 2 and Typescript.
 constructor( private people: Administrator[]) {
           ... snip ....
 Replace( index: number, person: Administrator ) {
    Vue.set( this.people, index, person);
}


Multiple calendar month view

Hi. Has anyone forked this so it can show multiple months at the same time? Like: https://jqueryui.com/resources/demos/datepicker/multiple-calendars.html
Thanks

Duplicate of #1154


Localization support for messages reported as part of callback in the ProblemHandler

I want to localize the messages reported as part of the problem handler.
Is localization support available in justify ?

Hello @maheshcamath
Thank you for contacting me.
All of the messages for reporting the problems are defined in the properties file messages_<locale>.properties in the package org.leadpony.justify.internal .
If your application is running under classpath, you can put the localized properties file in the package named as org.leadpony.justify.internal directly in your jar file and Justify will pick it up.
Meanwhile, If your application is running under module path, unfortunately there is no way to provide the localized properties file for Justify of the current version.
Please see also #34

Thanks a lot.


Don't count "users in afk" (Feature-Request)

as option:

don’t count users in “away" - afk”-channel, as displayed in "the headphones-logo".

as far as i see, the json/xml does not reveal the name of the AFK-channel. in this case the string has to be definable in the setup.

Fixed by #9


More...

Added .nvmrc file
Fixed the morning sun which was not displaying

Removed the npm start because it is misleading, you have to run two separate terminals


[Bug]: support for newer ESLint versions

What happened?
On newer eslint versions strip-ansi and text-table where dropped from it's dependencies but rules_lint depends on it implicitly by requiring them within it's stylish formatter.
Version
Development (host) and target OS/architectures: linux/arm64
Output of bazel --version: bazel 7.4.0
Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file: 1.0.3
Language(s) and/or frameworks involved: eslint
How to reproduce
Try to use eslint in version 9.12 or newer.

https://github.com/eslint/eslint/releases/tag/v9.12.0
> refactor: replace strip-ansi with native module

https://github.com/eslint/eslint/releases/tag/v9.15.0
> perf: optimize text-table by replacing regex with trimEnd

Any other information?
I would suggest adding the dependencies into the rules_lint. IMHO it's not a good idea to have a script which has dependency requirements but does not provide them. This would further break as eslint evolves. The formatter should be treated as standalone formatter despite its origin. This would help in further diverging from the original implementation in case this is required.

I've gone ahead and created one possible solution for newer eslint versions, which goes down the route of explicitly stating the dependencies of the formatter. I'm not sure if this is the desired solution for the repository. More than happy to update the PR in order to align with the project.


Update CHANGELOG.md

gradle (warpper) is even updated to 4.3 - isn't it ?

True - the thing is that there is and will be a lot of changes for release 4 so  that's why I haven't updated it.


please add Notifiers to Telegram

@andruwa13 Telegram is awesome, isn't it :) wish it was more popular in USA

Telegram is popular in Europe too. You have added WeChat. But did not add a telegram. Telegram is much more popular

Possible temporary workaround https://github.com/KostyaEsmukov/smtp_to_telegram


Add margin/padding to webview hosting extension readme

Testing #8215
It would look a lot better

Not critical - assigning to Joao and July so he can decide.

Actually the fix is easy and would make it look considerbly better -> june and me

I agree, this should go in for June.

Argh I hate these build issues...


Fixed small typo.

Hi Guillaume,
Just a small typo :)
Thanks for pulling.
Emin

Thank you very much!


APP-1341 Fix error handling on 2-way dialogs

Return error code provided by JIRA during the first step of OAuth dance

Codecov Report

Merging #127 into dev will decrease coverage by 0.07%.
The diff coverage is 33.33%.

@@            Coverage Diff             @@
##              dev     #127      +/-   ##
==========================================
- Coverage   88.16%   88.09%   -0.08%
==========================================
  Files         133      133
  Lines        2383     2385       +2
  Branches      276      276
==========================================
  Hits         2101     2101
- Misses        167      169       +2
  Partials      115      115

Impacted Files
Coverage Δ

...gration/authorization/oauth/v1/OAuth1Provider.java
95.23% <0%> (-4.77%)
:arrow_down:

...orization/oauth/v1/OAuth1HttpRequestException.java
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9f9397f...fc83a48. Read the comment docs.


Update outdated dependencies

I've tried to create sensible commits, so it's probably easier to read this PR one commit at a time.
Switch spies/stubs to Sinon.JS
Jasmine's spies are not very elegant, and they've not become better in Jasmine 2.
The log spamming the console during test runs is a testament to that. It might be that I've missed some minor detail, or that it just wasn't documented that well
I would suggest that we switch to using Sinon.JS instead. It's battle hardened and is well documented.
I've not created any commits for this yet, as I don't want to waste effort if it's not a welcome change.
Intermittent test failure
On my machines, there's something weird going on with testHelpers.js and Jasmine's spies.
>> Error: require: 'undefined' is not an object (evaluating 'realLogMethod.calls.count') at
>> _SpecRunner.html:21
>> .grunt/grunt-contrib-jasmine/require.js:12 v
>> .grunt/grunt-contrib-jasmine/require.js:19
>> .grunt/grunt-contrib-jasmine/require.js:23
>> .grunt/grunt-contrib-jasmine/require.js:8
>> .grunt/grunt-contrib-jasmine/require.js:23
>> .grunt/grunt-contrib-jasmine/require.js:7 y
>> .grunt/grunt-contrib-jasmine/require.js:23
>> .grunt/grunt-contrib-jasmine/require.js:19
>> .grunt/grunt-contrib-jasmine/require.js:23
>> .grunt/grunt-contrib-jasmine/require.js:17
>> .grunt/grunt-contrib-jasmine/require.js:14 D
>> .grunt/grunt-contrib-jasmine/require.js:28
>> .grunt/grunt-contrib-jasmine/require.js:29

This happens on about 50% of all test runs.
This could either be from the updated grunt-template-jasmine-requirejs or just a bug in Jasmine's spies.
I would suggest that we add a method/property to loglevel itself, that we can query for the currently set level, that would obviate the toBeAtLevel test helper entirely.

@pimterry do you have any comments for this pull request?
Would you be ok with switching to Sinon.JS for stubs and spies?
Would you be ok with a method/property on loglevel that indicates the current status set?

Sorry, yes, I will be taking a look at this shortly. That does all sound fine in principle, I just need to find a chunk of time for a proper review here.

No worries. I'll add more commits to use Sinon.JS and expose a way of querying logevel for it's current level.

Right, a couple of small comments there that we should solve first. On your other questions:

I'm very happy to switch to Sinon, if you're happy to do that, that sounds great.

I would rather not add a method to loglevel to get the current level, if possible (sorry if my previous comment was a bit ambiguous; I meant that the code generally looked fine). I don't think querying the current level is ever something that any real clients of the library should do in any sensible use case, so I'd like to avoid adding that just for the tests.


Marauder Update

are there any plans to update the marauder app? the app is outdated with some good features like Scripts, Signal Monitor, etc

yes, after sending the last ones I will focus on the app.


AWS S3 Object Lock

Hello,
We are attempting to use Opendal to push backups to an AWS S3 bucket with object lock enabled. Currently this is throwing errors like:

"Content-MD5 OR x-amz-checksum- HTTP header is required for Put Part requests with Object Lock parameters"

I would assume this is a simple change, and I would be happy to try and implement it. Correct me If I'm wrong.

I would assume this is a simple change, and I would be happy to try and implement it.

Yes and thanks in advance! I think we can add a new config called checksum_algorithm: Option<String>. It's by default to be empty and could be md5, crc32 or sha1. I believe we can start with crc32 support first (since it's fastest!).

Thanks for getting back to me, I'll take a look at it 👍
One more question: I think the Header name for the checksum should also be configurable, how would you go about that?

I think the Header name for the checksum should also be configurable, how would you go about that?

We don't need to expose this to user:

checksum_algorithm=None: did nothing
checksum_algorithm=Some("md5"): calculate md5 and set CONTENT-MD5.
checksum_algorithm=Some("crc32c"): calculate crc32c and set x-amz-checksum-crc32c

Very good point, i implemented it in the above PR. I'm very new to contributing in Open source projects, so apologgies if I made any silly mistakes.
I have another question as well:
I'm working on implementing this for v0.45.x which has the AsyncBody Struct instead of Buffer as it would take a bit work to update our codebase to work with main.
Is it possible to submit a pull request for this as well, as in for a possible 0.45.2 release?

I'm very new to contributing in Open source projects, so apologgies if I made any silly mistakes.

Thank you in advance. Please feel comfortable contributing.

Is it possible to submit a pull request for this as well, as in for a possible 0.45.2 release?

OpenDAL works on main branch only and won't do cherry-pick. We don't have a plan to release 0.4.52. Please go on main branch directly. Thanks!

This should have been supported.


fix(chat): removes technical text when renaming a [replay] conversation (Issue #2425)

Description:
removes technical text when renaming a conversation
Issues:

Issue #2425

Checklist:

[x] the pull request name complies with Conventional Commits
[x] the pull request name starts with fix(<scope>):, feat(<scope>):, feature(<scope>):, chore(<scope>):, hotfix(<scope>): or e2e(<scope>):. If contains breaking changes then the pull request name must start with fix(<scope>)!:, feat(<scope>)!:, feature(<scope>)!:, chore(<scope>)!:, hotfix(<scope>)!: or e2e(<scope>)!: where <scope> is name of affected project: chat, chat-e2e, overlay, shared, sandbox-overlay, etc.
[x] the pull request name ends with (Issue #<TICKET_ID>) (comma-separated list of issues)
[x] I confirm that do not share any confidential information like API keys or any other secrets and private URLs

/deploy-review


Change HTTP details

Moved one block and extended the description. Removed explicit reference to GET & POST

You're welcome! Nameko is definitely my favorite import right now :)


Bug fix

@ajbalogh kindly hold on from merging this until we gather more data on failures. This change may cause change in some test expectations.
Last successful pipeline on master: https://github.com/open-traffic-generator/ixnetwork/actions/runs/408243579

Hi @ashutshkumr  - that was by accident - I will leave the merge to you :)

All test cases are passing with this fix as well as handling original  issue #108 . Please review


Convert federated tests into test suite.

Add specialization for learning to rank.

After #8822, only integer label is accepted for ndcg and pairwise, and only binary is accepted for map. This PR assigns binary label for LTR, along with which, the tests are converted into gtest parametrized tests.
Extracted from https://github.com/dmlc/xgboost/pull/8822 .

cc @rongou


Problem with well-formedness check and composition of rewrites CT

Compose the two rewrites
f = RewriteN {
    dimension: 1,
    cones: [
        Cone {
            index: 0,
            internal: ConeN {
                source: [
                    Cospan {
                        forward: Rewrite0(0:0, 1:1, Label([(1, [Boundary(Source)])])),
                        backward: Rewrite0(0:0, 1:1, Label([(1, [Boundary(Target)])]))
                    },
                    Cospan {
                        forward: Rewrite0(0:0, 1:1, Label([(1, [Boundary(Source)])])),
                        backward: Rewrite0(0:0, 1:1, Label([(1, [Boundary(Target)])]))
                    }
                ],
                target: Cospan {
                    forward: Rewrite0(0:0, 2:2, Label([(2, [Interior(Singular(0)), Boundary(Source)])])),
                    backward: Rewrite0(0:0, 2:2, Label([(2, [Interior(Singular(0)), Boundary(Target)])]))
                },
                regular_slices: [Rewrite0(0:0, 2:2, Label([(2, [Boundary(Source), Interior(Regular(1))])]))],
                singular_slices: [
                    Rewrite0(1:1, 2:2, Label([(2, [Boundary(Source), Interior(Singular(0))])])),
                    Rewrite0(1:1, 2:2, Label([(2, [Boundary(Source), Interior(Singular(1))])]))
                ]
            }
        }
    ]
}

g = RewriteN {
    dimension: 1,
    cones: [
        Cone {
            index: 0,
            internal: ConeN {
                source: [
                    Cospan {
                        forward: Rewrite0(0:0, 2:2, Label([(2, [Interior(Singular(0)), Boundary(Source)])])),
                        backward: Rewrite0(0:0, 2:2, Label([(2, [Interior(Singular(0)), Boundary(Target)])]))
                    },
                    Cospan {
                        forward: Rewrite0(0:0, 1:1, Label([(1, [Boundary(Source)])])),
                        backward: Rewrite0(0:0, 1:1, Label([(1, [Boundary(Target)])]))
                    }
                ],
                target: Cospan {
                    forward: Rewrite0(0:0, 3:3, Label([(3, [Interior(Singular(0)), Boundary(Source), Boundary(Source)])])),
                    backward: Rewrite0(0:0, 3:3, Label([(3, [Interior(Singular(0)), Boundary(Source), Boundary(Target)])]))
                },
                regular_slices: [Rewrite0(0:0, 3:3, Label([(3, [Boundary(Source), Interior(Singular(0)), Interior(Regular(1))])]))],
                singular_slices: [
                    Rewrite0(2:2, 3:3, Label([(3, [Boundary(Source), Interior(Singular(0)), Interior(Singular(0))])])),
                    Rewrite0(1:1, 3:3, Label([(3, [Boundary(Source), Interior(Singular(0)), Interior(Singular(1))])]))
                ]
            }
        }
    ]
}

Actual result:
actual = RewriteN {
    dimension: 1,
    cones: [
        Cone {
            index: 0,
            internal: ConeN {
                source: [
                    Cospan {
                        forward: Rewrite0(0:0, 1:1, Label([(1, [Boundary(Source)])])),
                        backward: Rewrite0(0:0, 1:1, Label([(1, [Boundary(Target)])]))
                    },
                    Cospan {
                        forward: Rewrite0(0:0, 1:1, Label([(1, [Boundary(Source)])])),
                        backward: Rewrite0(0:0, 1:1, Label([(1, [Boundary(Target)])]))
                    },
                    Cospan {
                        forward: Rewrite0(0:0, 1:1, Label([(1, [Boundary(Source)])])),
                        backward: Rewrite0(0:0, 1:1, Label([(1, [Boundary(Target)])]))
                    }
                ],
                target: Cospan {
                    forward: Rewrite0(0:0, 3:3, Label([(3, [Interior(Singular(0)), Boundary(Source), Boundary(Source)])])),
                    backward: Rewrite0(0:0, 3:3, Label([(3, [Interior(Singular(0)), Boundary(Source), Boundary(Target)])]))
                },
                regular_slices: [
                    Rewrite0(0:0, 3:3, Label([(2, [Boundary(Source), Interior(Regular(1))]), (3, [Boundary(Source), Interior(Singular(0)), Interior(Singular(0))])])),
                    Rewrite0(1:1, 3:3, Label([(3, [Boundary(Source), Interior(Singular(0)), Interior(Singular(1))])]))
                ],
                singular_slices: [
                    Rewrite0(1:1, 3:3, Label([(2, [Boundary(Source), Interior(Singular(0))]), (3, [Boundary(Source), Interior(Singular(0)), Interior(Singular(0))])])),
                    Rewrite0(1:1, 3:3, Label([(2, [Boundary(Source), Interior(Singular(1))]), (3, [Boundary(Source), Interior(Singular(0)), Interior(Singular(0))])])),
                    Rewrite0(1:1, 3:3, Label([(3, [Boundary(Source), Interior(Singular(0)), Interior(Singular(1))])]))
                ]
            }
        }
    ]
}

Expected result (ignoring labels):
expected = RewriteN {
    dimension: 1,
    cones: [
        Cone {
            index: 0,
            internal: ConeN {
                source: [
                    Cospan {
                        forward: Rewrite0(0:0, 1:1),
                        backward: Rewrite0(0:0, 1:1)
                    },
                    Cospan {
                        forward: Rewrite0(0:0, 1:1),
                        backward: Rewrite0(0:0, 1:1)
                    },
                    Cospan {
                        forward: Rewrite0(0:0, 1:1),
                        backward: Rewrite0(0:0, 1:1)
                    }
                ],
                target: Cospan {
                    forward: Rewrite0(0:0, 3:3),
                    backward: Rewrite0(0:0, 3:3)
                },
                regular_slices: [
                    Rewrite0(0:0, 3:3),
                    Rewrite0(0:0, 3:3)
                ],
                singular_slices: [
                    Rewrite0(1:1, 3:3),
                    Rewrite0(1:1, 3:3),
                    Rewrite0(1:1, 3:3)
                ]
            }
        }
    ]
}

@NickHu Rewrite composition is definitely wrong, it computes the wrong regular slices. See the example above (the second regular slice is wrong). I will try to debug it, but honestly I don't really understand how it works very well.


PUB-735: Build Subscriptions URN Search Screen

Before creating a pull request make sure that:

[ ] commit messages are meaningful and follow good commit message guidelines
[ ] README and other documentation has been updated / added (if needed)
[ X] tests have been updated / new tests has been added (if needed)

Please remove this line and everything above and fill the following sections:

JIRA link
https://tools.hmcts.net/jira/browse/PUB-735
Change description
Added urn search page and result page
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[ ] No

Looks good :)
My only thought is that this currently won't work in staging / dev environments as we don't have a subscriptions URN endpoint on business rules (or soon to be data management).
@johnoneill46 - Are we OK with a separate ticket being created to link this up to the backend (once the other changes are done), or do we feel the backend bit also needs to be done as part of this?

Fine to be done as separate ticket PUB-735 is just for the actual UI screen - logic can be done separately

Closing PR as there is another pending one including this work (#128 )
Re-open if necessary


Import empty Anchore policies without error

This results in objects such as checks and images not being a dictionary when parsing a empty policy.
Signed-off-by: Cosmin Cojocar<EMAIL_ADDRESS>This template is for your information. Please clear everything when submitting your pull request.
Note: DefectDojo is now on Python3.6 and Django 2.2.x. Please submit your pull requests to the 'dev' branch as the 'legacy-python2.7' branch is only for bug fixes. Any new features submitted to the legacy branch will be ignored and closed.
When submitting a pull request, please make sure you have completed the following checklist:

[ ] Give a meaningful name to your PR, as it may end up being used in the release notes.
[ ] Your code is flake8 compliant.
[ ] Your code is python 3.6 compliant (specific python >3.6 syntax is currently not accepted).
[ ] If this is a new feature and not a bug fix, you've included the proper documentation in the ReadTheDocs documentation folder. https://github.com/DefectDojo/Documentation/tree/master/docs or provide feature documentation in the PR.
[ ] Model changes must include the necessary migrations in the dojo/db_migrations folder.
[ ] Add applicable tests to the unit tests.
[ ] Add the proper label to categorize your PR.

Moderators: Labels currently accepted for PRs:

Import Scans (for new scanners/importers)
enhancement
performance
feature
bugfix
maintenance (a.k.a chores)
dependencies
New Migration (when the PR introduces a DB migration)
settings_changes (when the PR introduces changes or new settings in settings.dist.py)

Contributors: Git Tips
Rebase on dev branch
If the dev branch has changed since you started working on it, please rebase your work after the current dev.
On your working branch mybranch:
git rebase dev mybranch

In case of conflict:
 git mergetool
 git rebase --continue

When everything's fine on your local branch, force push to your myOrigin remote:
git push myOrigin --force-with-lease

To cancel everything:
git rebase --abort

Squashing commits
git rebase -i origin/dev

Replace pick by fixup on the commits you want squashed out
Replace pick by reword on the first commit if you want to change the commit message
Save the file and quit your editor

Force push to your myOrigin remote:
git push myOrigin --force-with-lease

I wonder if this is enough. As a user using the API to upload a file with 10 findings. And after the import returns succesfully, there are only 8 findings in defect dojo. Where did the other 2 go? What Error/AttributeError was in the file? I think it was one of the complaints that there sometimes is a mismatch between the number of findings in the report and ending up in defect dojo. might go to far for this PR to refactor the result/report after import, but what could be helpful for now is to additionally create a notification (alert) in the category 'other':

It might be that some Evaluation are not properly formatted, but I don't believe that the Anchore's API returns invalid JSON when downloading the policies evaluation file. We have upload many times in DefectDojo the policy evaluation result, and I haven't notice that something is missing. It might be also the severity level, for instance when is set to medium all the evaluations which below that level are not uploaded.

but what could be helpful for now is to additionally create a notification (alert) in the category 'other':

I don't quite understand how is this notification related to the anchore policy import. Do you mean instead of logging a warning message, or in addition to the logging message to add also a notification when an AttributeError exception is encountered.

Yeah I meant additionally having an alert generated so that if the user goes into DD and see that he is missing 2 findings he might be able to see why in the alerts. But maybe that's something for another PR to improve the way import errors are handled/reported.


bug: openrpc.json has output line in file, breaking node-rpc-docs.celestia.org

I cloned the old version of RPC docs to evaluate which one to use (either using a component like is in the docs now or improving site @distractedm1nd made) and noticed that the openrpc.json spec that is referenced on the open-rpc docs has extra output in the json file:
--> Generating OpenRPC spec
{
    "openrpc": "1.2.6",

@jcstein this was removed in the branch openrpc-spec with last set of commits:
https://github.com/celestiaorg/celestia-node/commit/9e43c637c6d3fd4d9b5bcdb051a1b3110cac3b30#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52L153
@vgonkivs @distractedm1nd what is the plan with this branch?

hey @ramin - it looks like it's still there, no? https://github.com/celestiaorg/celestia-node/blob/openrpc-spec/openrpc.json
this pr resolves it https://github.com/celestiaorg/celestia-node/pull/2816
i don't think we need this anymore though as until they are stored with releases on main https://github.com/celestiaorg/celestia-node/issues/2611 , i will be manually generating them

@jcstein merged to the feature branch. Yes we removed but didn't commit a fixed openrpc.json, sorry!

all good! thank you!

Should we close this one https://github.com/celestiaorg/celestia-node/issues/2611 ?

i don't think so. i don't see on the release or in main?


Default mode on config.json and service files is too permissive

I noticed that when provisioning configs (config.json or services via consul::service) the default file permissions are used. Since these files will likely have tokens in them, I was thinking the permissions should be more restrictive.
Thoughts on using something like mode => 0600? I'd be happy to open a PR.

600 sounds too restrictive, even for me, and would be inflexible to other who are integrating the tool with other tools. We are setting the group, so I would accept a PR for an explicit 660, and if people need to read it, they can add their users to the consul group of their choosing.
It is using the default mode for whatever puppet has for your system. Is your default mode 600?
https://docs.puppetlabs.com/puppet/latest/reference/lang_defaults.html
File {
  mode => '0600',
}

Or use an alternative method of overriding permissions of a file in puppet without modifying the code?
https://docs.puppetlabs.com/puppet/latest/reference/lang_classes.html#overriding-resource-attributes

I think we can have both by creating a class param for mode and defaulting it to 0660.
Also, we need to do this for any other configuration that has tokens; service.pp, checks.pp and watch.pp.


Rabble AI Destination Documentation

Proposed changes
This is the initial commit for Rabble AI Destination Documentation
Merge timing
ASAP once approved
Related issues (optional)

just one change for me

Went ahead and committed this so we can deploy today.


Updated JSP/CLO to output prop names (something changed in API)

@dwertheimer did Eduard ever respond to your question about API change that broke?

No @mikeerickson but I withdrew the question, because I found a workaround and I figured it was a goose chase to figure out what changed it.