Disconnect / stop listening

I'm confused how to manage beacon within the iOS app lifecycle events (e.g. going to background, foreground, suspend etc.). I have created a Beacon.Client instance, told it to connect, and I registered a function to listen for incoming events. How do I disconnect / stop listening when the app goes into background and reconnect when it's reopened? I can't see any public disconnect / stop listening function, only the ability to remove a peer entirely

As of v2.0.0 there are now 4 methods in Beacon.Client to manage the connection: Beacon.Client#connect, Beacon.Client#disconnect, Beacon.Client#pause and Beacon.Client#resume.
Calling Beacon.Client#pause will suspend all ongoing and scheduled network calls as well as all registered listeners, BUT it'll keep the listeners in memory, so later everything can be resumed with a call to Beacon.Client#resume.
Stopping the connection with Beacon.Client#disconnect cancels all calls and removes all listeners, resulting in an almost irreversible state. To bring the connection back, it needs to be restarted with Beacon.Client#connect and Beacon.Client.listenOnRequest.
I hope this covers what you needed 😄
If you encounter any issues with those methods or have a question/request for something related to them, please open a new issue. I'm closing this one.

@jsamol thanks for the quick turn around on this. Be a little while for I have a cycle to test it, but looks like exactly what I need. Thanks


Add Nop Instruction and Pass

It is a good option for a debugger to use implicit sequence points by adding NOPs to the assembly. A pass could then remove them for a release buid (see https://learn.microsoft.com/en-us/archive/blogs/rmbyers/debuggingmodes-ignoresymbolstoresequencepoints)

Added in latest release.


[APIM-3.2.0][Workflows] - Missing ApplicationDeletion, SubscriptionDeletion and SubscriptionUpdate workflows

Describe your problem(s)
In wso2am-3.2.0, ApplicationDeletion, SubscriptionDeletion, and SubscriptionUpdate workflows are not implemented to be worked as OOTB features just like other workflows in the product.
Describe your solution
Just as ApplicaitonCreation, SubscriptionCreation, and APIStateChange workflows were implemented, the above-mentioned missing workflows also should be implemented in a way to work as OOTB features of the product.

These are available in the latest APIM versions now


Complete example

[Enter feedback here]
Hi, thanks for the documentation of that feature. It would be helpful to have a complete example of a sample pipeline because for me it is not clear at what part of the pipeline the code-snipped should be used.
best regards

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 86285f72-9e28-da97-59bb-c29eb60f627d
Version Independent ID: 18d5a591-a7d3-c261-6bff-8808ae433f54
Content: Configure pipeline triggers - Azure Pipelines
Content Source: docs/pipelines/process/pipeline-triggers.md
Product: devops
Technology: devops-cicd-process
GitHub Login: @ashkir
Microsoft Alias: ashkir

The current author's alias cannot be found.


fix(headless): allowed empty string for query pipeline

https://coveord.atlassian.net/browse/KIT-2225
I'm not sure of the exact reason why, but changing the page in Storybook or changing any option caused the whole search interface to crash with an error that said search: value is an empty string..
My guess is that my recent changes to the atomic-search-interface (specifically adding mutable: true to pipeline and setting this.pipeline = this.engine!.state.pipeline; after initialization) caused Stencil to add pipeline="" to the search interface, and a piece of logic in Storybook that's responsible for cloning the interface and replacing it with the clone, caused the search interface to try to initialize the engine with an empty string as a pipeline.
When I added a check that excludes the pipeline from the engine's initial configuration when the pipeline is an empty string, the error would stop happening.
I decided to allow empty strings when configuring the pipeline, since:

the pipeline's default value is an empty string. See: https://github.com/coveo/ui-kit/blob/51fa84ec66072d7974fec66106fa1f73aba79531/packages/headless/src/features/pipeline/pipeline-state.ts#L1
the pipeline's setPipeline action allows empty strings. See: https://github.com/coveo/ui-kit/blob/51fa84ec66072d7974fec66106fa1f73aba79531/packages/headless/src/features/pipeline/pipeline-actions.ts#L5-L10

However, the searchHub's setSearchHub action also allows empty strings, despite its default parameter instead being "default". See: https://github.com/coveo/ui-kit/blob/e91f6597baf1d661f761d4045fd08664ba6a3c61/packages/headless/src/features/search-hub/search-hub-actions.ts#L5-L10 and https://github.com/coveo/ui-kit/blob/e91f6597baf1d661f761d4045fd08664ba6a3c61/packages/headless/src/features/search-hub/search-hub-state.ts#L1

If not configured, is an empty string sent to the API before & with this fix? If it differs it could be problematic.

If not configured, is an empty string sent to the API before & with this fix? If it differs it could be problematic.

Nope, since the request is built with this: https://github.com/coveo/ui-kit/blob/37bd2a8cdabacc920fc3356a1db5891c93fd55a8/packages/headless/src/features/search-and-folding/search-and-folding-request.ts#L51-L53


Add cell matching support

Description
What is this PR

[ ] Bug fix
[x] Addition of a new feature
[ ] Other

While working on the pytorch changes, I wanted to have a way of comparing how my changes affect cell detection. Because the filtering in pytorch is potentially slightly different. So I needed a way to compare two sets of cells for distance.
This PR adds support for this. It uses the Hungarian method to compute the best paring between two groups of cells such that the euclidean distance within a pair across all pairs is minimized, I really hope someone already implemented it in the repos. I searched but didn't find anything 😅
The algorithm is of time complexity O(n3). Which seemingly is the fastest algorithm around for this problem. I adapted the code directly from Wikipedia.
References
None
How has this PR been tested?
I tested it matching two sets of 250k cells, one from the main branch and one my working changes and it took about 1.5 hours.
I also added tests.
Is this a breaking change?
No.
Does this PR require an update to the documentation?
No.
Checklist:

[x] The code has been tested locally
[x] Tests have been added to cover all new functionality (unit & integration)
[x] The documentation has been updated to reflect any changes
[x] The code has been formatted with pre-commit

For the actual example to see how it works. I compared my changes to main. The matching was able to pair 245952 cells, with 8426 left unpaired due to unequal number of cells found between the two sets.
Looking at the histogram of distance between all the pairs we get this image, which is pretty good IMO:

I also visualized the histogram in x, y, and z of the 8426 unmatched cells to see if something stands out but nothing stood out:

Thanks a lot @matham - I had a more simplistic version of something similar in a local script (no global optimisation or guaranteed 1-1 pairing, just brute-force finding minimum distance for each point in any point in the other) that I could/should have shared - apologies!
I am just wondering whether it's worth at some point replacing the core logic of match_cells with scipy.optimize.linear_sum_assigment, to reduce our long-term maintenance burden (I can't confidently say I understand each step well enough)? I've opened an issue for this, and appreciate your thoughts if you are willing to spend time.
I am satisfied the implementation matches the Wikipedia one AFAICT, and testing it on local data seems to give reasonable outputs (see below), so I'll approve.
I think this would benefit from some negative tests (checking e.g. that expected errors are thrown for invalid input) as sanity checks, and also to keep the coverage 90%+ (something @K-Meech has worked really hard to achieve) - I can work on this (https://github.com/brainglobe/brainglobe-utils/issues/76).

Should we document this in a separate part of the website (somewhere under development)? Otherwise I fear it will be forgotten the next time it's needed?

Is this the best repo for this? This won't be used by any tools directly, and it adds a numba dependency. Ideally we'd have a dev tools repo, but considering how much time we've spent reducing the number of repos we have, on balance I think this is the correct place.

At today's developer meeting we decided that

this might be useful for users as well, e.g. for validating a manual cell detection with cellfinder
in the interest of simplicity we'll keep this here and write a quick example script to document its existence and use.

I'll track this in https://github.com/brainglobe/brainglobe.github.io/issues/187

We should use this to compare:
* Old cellfinder (pre numba)

* Previous release (pre @matham's optimisations)

* Main (with @matham's optimisations)

* Future (@matham's torch implementation)

I've started doing some comparisons today, but will finish them tomorrow with a clearer mind :)

I've compared cell candidate detection results before (old_cells) and after (new_cells) https://github.com/brainglobe/cellfinder/pull/398. I've run match_cells(old_cells, new_cells, threshold=5) which throws out matches that are more than 5 pixels apart.
I ran on cells with z coordinate in slices 1000-1050 (somewhere in the middle) of a cFOS stained brain.
The "border" consists of slices 1000-1005 and 1045-1050 (I differentiate here because it's possible these cells would have a close match e.g. in slice 999 or 1051).
The result:
There are 96835 old cells
There are 97033 new cells
Matching cells: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 96835/96835 [08:41<00:00, 185.66cells/s]
There are 6042 old cells without a good match in new cells
There are 6240 new cells without a good match in old cells
88954/90793 of good (<5) matches are exact (distance==0).
There are 6042 old cells with no good match in new cells, 4646 of which are not on the border
There are 6240 new cells with no good match in old cells, 4814 of which are not on the border

The distribution of good matches with distance > 0 is:

So far, so good.
But then I ran the script below:
from pathlib import Path
from brainglobe_utils.cells.cells import match_cells
from brainglobe_utils.IO.cells import get_cells, save_cells

if __name__ == "__main__":
    pre_optimisation_cells = get_cells(str(Path("/media/ceph-neuroinformatics/.../cells.xml")))
    post_optimisation_cells = get_cells(str(Path("/media/ceph-neuroinformatics/.../other_cells.xml")))

    # only consider cells in slices 1000 - 1050
    z_min, z_max = 1000, 1050
    old_cells = [cell for cell in pre_optimisation_cells if cell.z > z_min and cell.z < z_max]
    new_cells = [cell for cell in post_optimisation_cells if cell.z > z_min and cell.z < z_max]
    print(f"There are {len(old_cells)} old cells")
    print(f"There are {len(new_cells)} new cells")

    # minimize total euclidean distance for a 1-1 pairing between old and new.
    # pairings with distance > 5 are considered "bad"
    good_threshold = 5
    no_match_in_new, matching, no_match_in_old  = match_cells(old_cells, new_cells, threshold=good_threshold)

    print(f"There are {len(no_match_in_new)} old cells without a good match in new cells")
    print(f"There are {len(no_match_in_old)} new cells without a good match in old cells")

    old_cells_with_no_match_in_new = [old_cells[i] for i in no_match_in_new]
    new_cells_with_no_match_in_old = [new_cells[i] for i in no_match_in_old]

    save_cells(new_cells_with_no_match_in_old, str(Path.home()/"no_match_in_old_2.xml"))
    save_cells(old_cells_with_no_match_in_new, str(Path.home()/"no_match_in_new_2.xml"))

to visualise "bad" matches.
Visualising the bad matches in napari shows that most of the yellow points (no_match_in_old) overlap perfectly (orange_points) with an equivalent red point (no_match_in_old):

I've probabably done something extremely silly in the script, or misunderstood how this should work, but I'm not comfortable merging this until I know what's going on :thinking: - pointers/thoughts about how to interpret this appreciated!

You code seems correct. Can you share the cells xmls so I can look at it? Perhaps there's something wrong with the threshold computation. But I also wonder if the axis were somehow flipped during visualization.

Also, here's the scipy code for this - it's quite similar.

Can you share the cells xmls so I can look at it?

Slight adaptation of the script above (avoiding the cell filtering by z coordinate)
from pathlib import Path
import napari
from brainglobe_utils.cells.cells import match_cells, to_numpy_pos
from brainglobe_utils.IO.cells import get_cells, save_cells

if __name__ == "__main__":
    old_cells = get_cells(str(Path.home()/"cells-z-1000-1050.xml"))
    new_cells = get_cells(str(Path.home()/"other-cells-z-1000-1050.xml"))

    good_threshold = 5
    no_match_in_new, matching, no_match_in_old  = match_cells(old_cells, new_cells, threshold=good_threshold)

    print(f"There are {len(no_match_in_new)} old cells without a good match in new cells")
    print(f"There are {len(no_match_in_old)} new cells without a good match in old cells")

    old_cells_with_no_match_in_new = [old_cells[i] for i in no_match_in_new]
    new_cells_with_no_match_in_old = [new_cells[i] for i in no_match_in_old]

    save_cells(new_cells_with_no_match_in_old, str(Path.home()/"no_match_in_old-z-1000-1050.xml"))
    save_cells(old_cells_with_no_match_in_new, str(Path.home()/"no_match_in_new-z-1000-1050.xml"))

    viewer = napari.Viewer()

    viewer.add_points(to_numpy_pos(new_cells_with_no_match_in_old), edge_color="yellow", face_color="yellow", symbol="ring", opacity=0.5, name="no match in old")
    viewer.add_points(to_numpy_pos(old_cells_with_no_match_in_new), edge_color="red", face_color="red", symbol="ring", opacity=0.5, name="no match in new")
    napari.run()

Should be reproducible with attached files saved to Path.home() and extension changed to .xml
cells-z-1000-1050.txt
other-cells-z-1000-1050.txt

I think I figured out the issue and it has to do with us finding the global minimum and then removing those above the threshold. This results in the following problem:
Say you have 4 points on a line. Set 1 is {0, 12}, set 2 is {10, 22}. Without threshold, the matching is (0, 10), (12, 22). Because that results in the minimum global cost of 20 If we were then to apply the threshold of 5 after this matching, we would have no matching as both pairs is above 5. Then, looking at the unmatched cells we'd have a potential pair of (10, 12), but it's not in the match.
This is exactly the issue you noticed. Because the pairs of cells that are right on top of each other but not in match, each are actually matched with other cells to minimize the global cost. Then, removing those above threshold, results in the unmatched sets having cells that are right on top of each other.
This can be tested with the following code:
cells1 = [
    Cell((0, 0, 0), Cell.UNKNOWN),
    Cell((12, 0, 0), Cell.UNKNOWN),
]
cells2 = [
    Cell((10, 0, 0), Cell.UNKNOWN),
    Cell((22, 0, 0), Cell.UNKNOWN),
]

missing_c1, good_matches, missing_c2 = match_cells(cells1, cells2, threshold=np.inf)
assert not missing_c1
assert not missing_c2
assert good_matches == [[0, 0], [1, 1]]

missing_c1, good_matches, missing_c2 = match_cells(cells1, cells2, threshold=5)
# before the fixes the following assert is correct
# assert missing_c1 == [0, 1]
# assert missing_c2 == [0, 1]
# assert not good_matches
# with final fixes this is correct
assert missing_c1 == [0, ]
assert missing_c2 == [1, ]
assert good_matches == [[1, 0]]

I changed it to account for the threshold during the matching so this issue doesn't occur. I haven't yet tested it on my whole brain, to be sure, but will do soon.

Also, I added tests so there's full coverage. But it needs to be run with NUMBA_DISABLE_JIT=1. I'm not quite sure how to do it in the CI because you definitely want tests with and without it disabled to catch any issues. So I'm not changing the runner to test it with it disabled.

So after more looking, there are multiple things going on, some which is not a bug, but one that was a bug...
There was a bug where a variable was not properly reset. This was fixed in the first commit after your last comment. When translating vector<bool> in_Z(W + 1); from the original algo, I didn't realize that c++ will initialize the vector to zero (I assume it will). But in the source from where wiki gets the algo, it clearly does this reset to False at every iteration (vector<char> used (m+1, false);).

Sadly, with this fix, the algorithm became a lot slower, in particular when using a small threshold (see below some reasons behind this). Which is quite in line with it being O(N3). So, I also added an optional initial step (ON by default), that extracts all zero distance pairs before running the optimization step.
And, after that I also added the option to use scipy as the optimization backend. This is only practical when we do the zero-distance extraction, because scipy cannot handle the full dataset due to memory requirements. However, with scipy, we cannot use the threshold during the matching because scipy doesn't take it as an option, so we can only use it after.
And with that, I compared the matching with and without threshold and with and without scipy. I found that without threshold, the total cost of the matched pairs were the same - which is good as that is what we are minimizing. Although the matched pairs were not identical, which makes sense.
These images, on your two cell sets show the result:

Besides all this, there are complexities with this algorithm that may result in some of what you saw. This algorithm is perhaps not precisely what we want. This algorithm minimizes the total global cost. We don't quite care about the global cost, we want to minimize all the local costs, even if it results in larger global costs.
Some examples how this may appear in the algorithm:

Consider 2 sets of points in 1d: (0, 10), (10, 20). If we get match pairs (0, 10), (10, 20) vs. (0, 20), (10, 10), both will have global cost of 20 so a global matching may select either. But for us, with a local matching, we'd prefer the latter match.
Using a threshold kinda accomplishes a more local minimization, as points further away are all considered equally far.
But, scipy doesn't support a threshold and in my implementation, using a threshold, especially smallish, makes it much much slower. I speculate it's because much further away points are not as easily eliminated as they are all equal and so it has to iterate the full set more often (I verified that # iterations increases).

You observed that the number of bad matches (i.e. outside the threshold) increased with the previous changes. I also observed this in my dataset. And this is because (besides the bug, potentially), I changed the cost function from sum(square(distance[i])) to sqrt(sum(square(distance[i]))), so I could use the threshold during the matching. I didn't take the sqrt before as an optimization.
But, why would that change the matching? Shouldn't the matching stay the same? The reason (I think) is because we're doing a global optimization, not local. And the sqrt scaling is non-linear. So without taking the sqrt, everything gets pushed further apart by power of 2.
So, assuming most points are close to each other in the data set. The algorithm, to minimize the global distance, has to work really hard to keep as much of them paired as close to each as possible (i.e. doing a better local matching). So it can afford e.g. to not pair points that are e.g. 1 voxel apart, if that pair can help those 2 apart. Because for the latter the cost is 4 vs. just 1!
On the other hand, now that I am taking the sqrt, the algorithm has less incentive to do focus as much on the local minimization. Because comparing things a distance 1 vs 2 apart is less significant and can be made up with other better matches of things further apart. And so maybe there are more pairs slightly further apart, and when applying the threshold all of a sudden we have more points outside the threshold.
This is handwavy, but if it wasn't the case (that the globalness of the algorithm results in this), changing the cost to a sqrt shouldn't change the matching.

For comparison, here's how the previous histogram looks like if we don't take the sqrt, using our implementation. You can see, using the threshold it is the same. But if we don't use the threshold, if you look at e.g. the second bar from left, not doing sqrt results in a lot more matches that are closer to each other. Which seems to show my above analysis was correct.

So, I'm not sure what you think about the code now. But my sense is that first extracting all the zero-distance pairs, whatever remains, we care the most about the overall histogram. Because that gives us a sense of how bad the changes in e.g. cellfinder was. And while we would prefer a local optimization algorithm (that e.g. always prioritizes zero-distance pairs) rather than global, I don't know of such an algorithm. So perhaps a the global optimization algorithm is good enough?

The following are points to consider about the API

The threshold, as part of the optimization stage, only works for our implementation, not scipy. And perhaps scipy would never support it. Should I remove our support for it? Should I make it optional so the threshold can be taken into account only after the matching like scipy? Does that make the API too complicated?
Leave whether to take the sqrt a parameter? So we can tune the matching tightness?
I warned in the docs that using a small threshold makes my implementation a lot slower. And that the scipy backend requires a lot of memory and so it should be used with zero-distant points extraction. But, does all this make the API too complicated? Is this getting out of hand? 😅

Thanks for the review!
I'm with you on the API and I can make the changes (remove scipy, keep sqrt).
About adding the cells you shared as a test to compare scipy to us. Each xml file is 12MB. Is that (not) too big to add to the repo? Because everyone cloning will have to download it.
I was kinda running into this issue on the pytorch branch because I added images to make sure the 2d/3d filtering is similar, which easily added a few MBs. And I also have 3 25MB images from a very bright brain that is super helpful for regression testing the full process and helped me figure out where there wasn't parity. But that definitely seemed too big to include in the repo. I'm not sure if you have encountered/considered this before!?

too big to include in the repo. I'm not sure if you have encountered/considered this before!?

Good point. I will update https://github.com/brainglobe/cellfinder/issues/282 with our long-term plan for this (and other BrainGlobe repositories). Short version: we already have too much data in this repo that should get moved to the GIN platform, downloaded with pooch and cached on CI.
I'll put the XML files on GIN now, and happy to do the same with your images if you could temporarily share them via Dropbox/onedrive/google drive/...?

should get moved to the GIN platform, downloaded with pooch

I've made a sketch PR for the regression test on the XML files to your fork @matham, with how I imagine this working (which are now on GIN)

I can make the changes (remove scipy, keep sqrt).

Thank you!! :tada:

About adding the cells you shared as a test to compare scipy to us.

I can add the test itself too, if you like?

Ok, removed scipy as an option and filled in the scipy test. Thanks for the pooch PR, I've never used it so that was helpful!

I can add the test itself too, if you like?

Adding tests that pass is the most fun part 😁

https://github.com/matham/brainglobe-utils/pull/2 may help add more of the tests discussed above. The data you shared with us is on GIN now :)

https://github.com/matham/brainglobe-utils/pull/2 may help add more of the tests discussed above. The data you shared with us is on GIN now :)

Given that we now have the discussed regression test, and this data is meant for wider use than just cell matching, I think I can merge this now. Thanks very much, @matham ! :tada:


circular dependency

Hi,
I came across this circular dependency:
boilerplates needs version-query -> https://github.com/mbdevpl/python-boilerplates/blob/v1.1.1/requirements.txt
and
version-query needs boilerplates  -> https://github.com/mbdevpl/version-query/blob/v1.5.5/requirements.txt
while building tvb-ext-bucket from source.

I wonder if there's a way to resolve this at all


Write documentation on deployments

User Story
As a Reportstream engineer I want documentation on deployments
Description/Use Case
Write a document on the deployment process, keeping in mind to leave out any security information. Could perhaps just reference the document currently used for deployments.
Risks/Impacts/Considerations
Dev Notes
If https://app.zenhub.com/workspaces/platform-6182b02547c1130010f459db/issues/gh/cdcgov/prime-reportstream/10208 is completed, use the existing structure, otherwise put the doc in docs/features.
Acceptance Criteria

[ ] Define content that should be in documentation and review with team before writing documentation
[ ] A document exists on deployments

Hey team! Please add your planning poker estimate with Zenhub @jack-h-wang @JessicaWNava @JFU-NAVA-PBC @luis-pabon-tf @mkalish @thetaurean

@mxk0 @pstonebu this doesn't feel like it should live with Platform in particular. All engineers are on the hook for deployments, and engineering writ large should be managing this doc. In fact it feels like this should just be a continuous effort, and done each time a deployment is done. "Are the docs up to date? If not, update them." This of course assumes that the current docs are up to date, and if they're not, why not?
cc: @bishoyayoub @avnieldravid

I agree with Mo above and also we currently have a deployment doc that is linked in the deployment channel is slack. Closing.


How to delete a directory in bucket

I'd like to ask about how to delete a exist directory/folder in bucket by gcloud-ruby.
  require "gcloud"

  gcloud = Gcloud.new
  storage = gcloud.storage

  bucket = storage.bucket "my-bucket"
  file = bucket.file "path/to/my-folder"
  file.delete

Is right?
Thank you and best regards,
Bui The Hien

I tested in rails console with above command but I have error
Gcloud::Storage::ApiError: Not Found
	from /home/hienbt/.rvm/gems/ruby-2.3.0/gems/gcloud-0.10.0/lib/gcloud/storage/bucket.rb:467:in `file'
	from (irb):8
	from /home/hienbt/.rvm/gems/ruby-2.3.0/gems/railties-4.2.6/lib/rails/commands/console.rb:110:in `start'
	from /home/hienbt/.rvm/gems/ruby-2.3.0/gems/railties-4.2.6/lib/rails/commands/console.rb:9:in `start'
	from /home/hienbt/.rvm/gems/ruby-2.3.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:68:in `console'
	from /home/hienbt/.rvm/gems/ruby-2.3.0/gems/railties-4.2.6/lib/rails/commands/commands_tasks.rb:39:in `run_command!'
	from /home/hienbt/.rvm/gems/ruby-2.3.0/gems/railties-4.2.6/lib/rails/commands.rb:17:in `<top (required)>'
	from /mnt/DATA/Work/VIENASite/tmp/DeThiMoi2/bin/rails:9:in `require'
	from /mnt/DATA/Work/VIENASite/tmp/DeThiMoi2/bin/rails:9:in `<top (required)>'
	from -e:1:in `load'
	from -e:1:in `<main>'

So, could you please help me how to delete a folder/directory in Cloud Storage.
Thank you so much.

Hi @buithehein1991, to delete a file you will need to specify the exact file name. See File#delete
require "gcloud"

gcloud = Gcloud.new
storage = gcloud.storage

bucket = storage.bucket "my-bucket"

file = bucket.file "path/to/my-file.ext"
file.delete

If you want to delete all files that match a directory, you will first have to select the files, and then delete them.
require "gcloud"

gcloud = Gcloud.new
storage = gcloud.storage

bucket = storage.bucket "my-bucket"

files = bucket.files prefix: "path/to/my-folder"
files.map &:delete

This is because there are no directories in Google Cloud Storage, only files. Using prefix and delimiter we can query the files as if there were directories.

@buithehien1991 Does this answer your question? If so, can this issue be closed?


community[patch]: bugfix: Unstructured empty text

This issue was raised here https://github.com/langchain-ai/langchainjs/issues/2144
It was fixed int he old langchain module and still is fixed here https://github.com/langchain-ai/langchainjs/blob/main/langchain/src/document_loaders/fs/unstructured.ts#L328
The community version, however, doesn't have this check.

This also adds the missing field https://github.com/Unstructured-IO/unstructured-api?tab=readme-ov-file#chunking-elements

Thank you!


KeepAliveSuite flake

I saw this on a JS run, but at a glance I have no reason to believe it's JS specific.
org.http4s.headers.KeepAliveSuite:
failing seed for header.header does not match arbitrary name is hJyBbXICp5OwOcoOzsaBHYTzIF9y5zo3pHoSs3oX0gO=
==> X org.http4s.headers.KeepAliveSuite.Keep-Alive: header.header does not match arbitrary name 0.09s munit.FailException: Failing seed: UjCaqEjILdSnoSjRhoICQxtFpTAoxvYsTBQfsngpQhG=
You can reproduce this failure by adding the following override to your suite:

  override val scalaCheckInitialSeed = "UjCaqEjILdSnoSjRhoICQxtFpTAoxvYsTBQfsngpQhG="

Exception raised on property evaluation.
> Exception: org.http4s.ParseFailure: Invalid Keep-Alive header: Reserved token of list List(token, max) was found in the extensions.

Another, from a JVM run. Did something change recently affecting this?
==> X org.http4s.headers.KeepAliveSuite.Keep-Alive: header.parse(a.value) == right(a)"  0.072s munit.FailException: Failing seed: uspBgDvsXmf7d1xthWKB6ys_YsaM_2sY5SwH_N4U0DM=
You can reproduce this failure by adding the following override to your suite:

  override val scalaCheckInitialSeed = "uspBgDvsXmf7d1xthWKB6ys_YsaM_2sY5SwH_N4U0DM="

Exception raised on property evaluation.
> Exception: org.http4s.ParseFailure: Invalid Keep-Alive header: Reserved token of list List(token, max) was found in the extensions.
    at munit.ScalaCheckSuite.propToTry(ScalaCheckSuite.scala:98)

I'm not that sure about this variant, but could it be caused by the ScalaCheck update?

That's what I thought too, but I think I reported this before merging https://github.com/http4s/http4s/pull/6686. Should have noted the failing hash, let me see if I can find it.

I found it. The original failure was in https://github.com/http4s/http4s/commit/13e5826b13b76cff5990b53c5862658176cd4c56 which was still using scalacheck v1.16.0.

@armanbilge @danicheg after a quick check looks like a possible fix could be in the PR. Would that be acceptable?

@FrancescoSerra it would be great 👍🏻


how about script class？

this class Description is A class stored as a resource.   it is like unitys scriptable object that can be serialized？

it is like unitys scriptable object that can be serialized

Well, yeah scripts are like any other resource.
As your sentence is not really clear, I guess this is a question about the Script class. If you have such question please ask first on the other community channel. Guthub is to submit bug or ask for new features. :)


Cython

Description
This PR converts the functionality of GluonNLP that previously supported optional numba JIT to Cython. The main motivation is to allow EmbeddingCenterContextBatchify to operate on strings.
Checklist
Essentials

[ ] Changes are complete (i.e. I finished coding on this PR)
[X] All changes have test coverage
[X] Code is well-documented

Changes

[X] Replace numba with Cython

Comments

This removes the runtime dependency on numba to a compile time dependency cython. While numba was formally optional, the execution speed without numba was unusable for the optimized code paths.
Thanks to Cython language_level=3str, the .pyx can be written using Python 3 syntax and features, with Cython taking care of the bindings to Py2

TODO

[x] Sphinx documentation https://opendreamkit.org/2017/06/09/CythonSphinx/
[x] Code coverage https://github.com/pytest-dev/pytest-cov/issues/163 https://cython.readthedocs.io/en/latest/src/tutorial/profiling_tutorial.html https://github.com/cython/cython/issues/1461

Job PR-389/10 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-389/10/index.html

Job PR-389/11 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-389/11/index.html

Job PR-389/12 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-389/12/index.html

@szha FYI, after the split in serial / non-serial test execution, only the code coverage of serial tests was uploaded. Fixed by 527b23d

Job PR-389/18 is complete.
Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-389/18/index.html


add support for electrum server

Currently, the only supported explorer is esplora. It would be useful to also be able to use an electrum server (e.g. electrs) as was the case in RGB v0.10.
electrs requires less resources than esplora and current RGB developers and users already rely on an electrum server, so switching to esplora requires quite some work and is therefore not ideal.
I know esplora was the fastest solution but I think having electrum server support is still worth it. @dr-orlovsky can you please add electrum server support?

This would require rewriting rust-electrum crate to use BP instead of rust-bitcoin, which is a non-trivial and would require about a week of work just on this task. esplora crate was much smaller and that was another reason to use it for v0.11

I'm not sure a rewrite of the rust-electrum crate is necessary, I think we can use raw methods and convert the server responses to BP structures directly in the implementation of the Resolver.
Is it ok for you @dr-orlovsky if I try to implement this myself and propose a PR roughly by the end of the next week?

@zoedberg the approach is fine. BTW we do not need to put any changes into anything other than cli - the Runtime should work with any resolver

Closing as this is now done.


Loading resources failed with proxy params

With this proxy params:
spooky = new Spooky({
    child: {
        'transport': 'http',
        'proxy': 'proxy.domain.com:8010',
        'proxy-auth': 'user:password'
    },
    casper: {
        logLevel: 'debug',
        verbose: true
    }
}, /* ... */);

I try multiple domains and works.
I try with the command line directly:
casperjs --proxy=proxy.domain.com:8010 --proxy-auth=user:password proxy.js
and works.
I try differents urls and fail all the time.

Note that:
The proxy don't use the ip but the domain name; as in the example above.
The proxy-aut username contains a dash -.

Any idea?

I don't know why the speed of access website is very slow after use proxy, the speed is fast in browser.


💡 [REQUEST] - Hover effect to heading

[x] I am from GSSoC23.
[x] I would like to be assigned this task.

fill [x] to check the boxes.

Section
Description

Concise Description
Write a concise and appropriate description of the feature request

Solution Proposed
Describe the solution you would want

Alternative Solutions
Describe any alternative solutions you can think of

Additional Context
Write any other contexts related to the feature request

Screenshots
[Provide screenshots here]

Code of Conduct
By submitting this issue, you agree to follow our Code of Conduct.

[x] I follow the Contributing Guidelines of this project.
[x] I have checked the current issues for duplicate problems.
[x] I have read the readme of this repository.

In this on hovering the resource gallery there is a slider effect or we can also add glowing effect accordingly which suits more good in this case. For example case I have provided you with the template below:
Screencast from 15-07-23 02:48:40 AM IST.webm


[WIP Do Not Merge] fix: input icon background color and position

Open PR for now.sh link.

@mikemai2awesome @remydenton is this ready to go or is something holding us back on this? This branch does say WIP so I wanted to double-check!

@sghoweri Not ready. Only ready when I remove WIP from title and request a review.


Typo in additionalReportLables

Description
Related issues

Close #1103

Checklist

[x] I've read the guidelines for contributing to this repository.
[ ] I've added tests that prove my fix is effective or that my feature works.
[ ] I've updated the documentation with the relevant information (if needed).
[ ] I've added usage information (if the PR introduces new options)
[ ] I've included a "before" and "after" example to the description (if the PR is a user interface change).

@bgoareguer thanks for catching it up , lgtm 🚀


change th to th:visible

change th to option, i using th:visible .col-name
only change to $(el).filter(':visible').find(options.headerSelector).each(function() {
i don't know how to pull

You can simplify click on the edit button for that file. That will automatically fork this repository into your account.
When you are ready you can click on create pull request


Timestamp decode/encode

Hi, I wanted to suggest a feature, but why not implement it right away.
I'm a Scala developer and I often deal with timestamps encoded as epoch milliseconds, which are hard to read and work with.
This PR adds 2 actions under Encode/Decode that convert between milliseconds and ISO timestamp (java.time.Instant). The input parsing is lenient and can understand (the absence of) timezones (defaults to UTC) or date without time (assumes start of day). See tests for examples.
It doesn't cover:

error reporting - any exception will ignore the action
UNIX seconds - I have no use for this ¯\(ツ)/¯

I love your plugin, it has been invaluable to me, I would be very glad for this to be included in the official distribution.
Let me know if you need me to change anything or have other suggestions.

Nice.
I would maybe change the action text to Instant to Epoch Milliseconds and Epoch Milliseconds to Instant so that it is clearer what it does.
Depending on your use case, you could also benefit from making an intention... https://jetbrains.org/intellij/sdk/docs/reference_guide/custom_language_support/code_inspections_and_intentions.html

I agree about the label, it wasn't even clear what encode / decode meant - whether it should be the other way around. Your suggestion makes sense, I think it's clearer.
My use case is mostly just converting milliseconds to timestamp when manually debugging / investigating issues, so it isn't even constrained to a particular language. That makes this action almost a perfect solution.


The specified type member 'Name' is not supported in LINQ to Entities

Hi,
When I use AutoMapper， I  need to convert string to Uri that because automapper's projection don't support the converter , so I have to use projectusing as below according to automapper's documentation
Mapper.CreateMap<EFEntities.RelyingParties, Models.RelyingParty>() .ProjectUsing(source => new Models.RelyingParty { realm = new Uri(realm)});
when running "Model.Rows.Count()" will throw exception
The specified type member 'Name' is not supported in LINQ to Entities,
I have tested , when I use formember,everything is ok ,but cannot use the projectusing.
Thanks in advance

Grid doesn't do anything special with the query, if you are filtering name in your view, it will generate the same LINQ expression as written by hand. However mapping queries with AutoMapper can be tricky sometimes, but most of the time they can be fixed by inspecting generated query or replacing faulty properties. Try filtering out or running count on your created query in your controller and see if you don't get the exception.
public ActionResult Index()
{
    var relyingParties = GetRelyingParties().AsQueryable();
    var parties = relyingParties.Where(party => party.Name == "Test").Count();
    var relyingPartiesList = relyingParties.Where(party => party.Name == "Test").ToList();

    return View(relyingParties);
}

You are right，That's my fault。
another thing ,I found the message " Obsoleting Project.To, creating all the overloads for ProjectTo" in automapper.
maybe you should modify your code in Select.cs


New clang-scanbuild warnings

Fix scanbuild warnings found after the latest merges

-v ? :)

@akallabeth when we got some calm minute it's finally time to add clang to the pull request builder (again) \o/


Anyone knows something about Yunbi

I know it's not an appropriate to put this kind of issue on this page but sorry, I have no option.
I've been having communication issue with yunbi's customer service - They don't respond to any tickets everyone sends.
I tried so hard to find the company's address or phone number to contact but there was nothing.
The only thing I found is that Peatio and Yunbi have same parent company - Bitfundpe
So If someone knows anything- address phone number ETC- about Peation, Bitfundpe, Yunbi
Please share what you got.

@temi71 Hotline:<PHONE_NUMBER>


jsonschema>=4.2.1 incompatible with altair 4.1 transform_fold()

See: https://github.com/altair-viz/altair/issues/2525
The best fix would be not to require jsonschema>=4.2.1 in datapane, because it leads to performance problems in other areas of altair (see: https://github.com/altair-viz/altair/issues/2501)
Could we release this constraint in datapane?

fixed and released as 0.13.1 - thanks for raising the issue! 👍

Thanks for the quick fix!


Add setting for file extensions to check

Side note: we should get some integrated tests up

This looks pretty good to me. Happy to merge, or did you want to add anything else first?

Let's get this in now


Add Pasargad driver to docs

Check the "Implemented" and "Unit Tests" boxes for the driver in the docs - Supported Drivers
Link the PR description next to the driver as a note for people to know it probably doesn't work on prod as it is right now

Is it OK if I assign this one to you? @Keivan-sf

@alitnk  Sure, I was about to pull this one anyway


Presenting the ScyllaDB Helm Chart

Adding ScyllaDB to the google charts incubator so that the community can
benefit and contribute.

What this PR does / why we need it:
Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #
Special notes for your reviewer:

Regarding the ci/circleci: lint-charts test: Do I need to clean trailing spaces on my repository and submit a new PR or is there an easy way to fix it?
When I go to details on cla/linuxfoundation after login with my corporate e-mail I get the message

You are already authorized to contribute code to this project through your membership with CNCF - ScyllaDB.

/assign @sameersbn


DocsArchive-<IP_ADDRESS>Auto archive content

Please don't merge this PR until target archive repo PR https://github.com/OPS-E2E-PPE/docs-yml-archive-non-versioning-archive/pull/436 is merged into live branch.
Auto archive content to https://github.com/opstest2/docs-yml-archive-non-versioning-archive.git

Docs Build status updates of commit 19116cd:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

.openpublishing.redirection.json
:white_check_mark:Succeeded

docs/TOC.yml
:white_check_mark:Succeeded
View

docs/to-be-archive/level1/article3-1-1.md
:white_check_mark:Succeeded
View

docs/to-be-archive/level1/article3-1-2.md
:white_check_mark:Succeeded
View

docs/to-be-archive/level1/article3-1-3.yml
:white_check_mark:Succeeded
View

docs/to-be-archive/level2/article3-2-1.md
:white_check_mark:Succeeded
View

docs/to-be-archive/level2/article3-2-2.md
:white_check_mark:Succeeded
View

docs/to-be-archive/level2/article3-2-3.yml
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Protobuf scalar default values and ByteString detail

Fixes https://github.com/dotnet/AspNetCore.Docs/issues/19760
@jskeet

Performance warning: this copy copies byte[] how many times?

ByteString.FromBytes and BytesString.ToByteArray are an allocation plus copy.
For reading values, Memory/Span properties exist. For creating values there is discussion about adding an unsafe API - https://github.com/protocolbuffers/protobuf/pull/7645
If it does get added I don't think it is appropriate to talk about in this doc. The performance guide would be a better place.


Can't install tlint on mac

Here's the error:
Problem 1
- Conclusion: don't install tightenco/tlint 3.0.4
- Conclusion: don't install tightenco/tlint 3.0.3
- Conclusion: don't install tightenco/tlint 3.0.2
- Conclusion: remove illuminate/container v5.8.31
- Installation request for tightenco/tlint ^3.0 -> satisfiable by tightenco/tlint[3.0.0, 3.0.2, 3.0.3, 3.0.4].
- Conclusion: don't install illuminate/container v5.8.31
- tightenco/tlint 3.0.0 requires illuminate/view ^6.0 -> satisfiable by illuminate/view[v6.0.0, v6.0.1, v6.0.2, v6.0.3, v6.0.4, v6.1.0, v6.2.0, v6.3.0, v6.4.1, v6.5.0].
- illuminate/view v6.0.0 requires illuminate/container ^6.0 -> satisfiable by illuminate/container[v6.0.0, v6.0.1, v6.0.2, v6.0.3, v6.0.4, v6.1.0, v6.2.0, v6.3.0, v6.4.1, v6.5.0].
- illuminate/view v6.0.1 requires illuminate/container ^6.0 -> satisfiable by illuminate/container[v6.0.0, v6.0.1, v6.0.2, v6.0.3, v6.0.4, v6.1.0, v6.2.0, v6.3.0, v6.4.1, v6.5.0].
- illuminate/view v6.0.2 requires illuminate/container ^6.0 -> satisfiable by illuminate/container[v6.0.0, v6.0.1, v6.0.2, v6.0.3, v6.0.4, v6.1.0, v6.2.0, v6.3.0, v6.4.1, v6.5.0].
- illuminate/view v6.0.3 requires illuminate/container ^6.0 -> satisfiable by illuminate/container[v6.0.0, v6.0.1, v6.0.2, v6.0.3, v6.0.4, v6.1.0, v6.2.0, v6.3.0, v6.4.1, v6.5.0].
- illuminate/view v6.0.4 requires illuminate/container ^6.0 -> satisfiable by illuminate/container[v6.0.0, v6.0.1, v6.0.2, v6.0.3, v6.0.4, v6.1.0, v6.2.0, v6.3.0, v6.4.1, v6.5.0].
- illuminate/view v6.1.0 requires illuminate/container ^6.0 -> satisfiable by illuminate/container[v6.0.0, v6.0.1, v6.0.2, v6.0.3, v6.0.4, v6.1.0, v6.2.0, v6.3.0, v6.4.1, v6.5.0].
- illuminate/view v6.2.0 requires illuminate/container ^6.0 -> satisfiable by illuminate/container[v6.0.0, v6.0.1, v6.0.2, v6.0.3, v6.0.4, v6.1.0, v6.2.0, v6.3.0, v6.4.1, v6.5.0].
- illuminate/view v6.3.0 requires illuminate/container ^6.0 -> satisfiable by illuminate/container[v6.0.0, v6.0.1, v6.0.2, v6.0.3, v6.0.4, v6.1.0, v6.2.0, v6.3.0, v6.4.1, v6.5.0].
- illuminate/view v6.4.1 requires illuminate/container ^6.0 -> satisfiable by illuminate/container[v6.0.0, v6.0.1, v6.0.2, v6.0.3, v6.0.4, v6.1.0, v6.2.0, v6.3.0, v6.4.1, v6.5.0].
- illuminate/view v6.5.0 requires illuminate/container ^6.0 -> satisfiable by illuminate/container[v6.0.0, v6.0.1, v6.0.2, v6.0.3, v6.0.4, v6.1.0, v6.2.0, v6.3.0, v6.4.1, v6.5.0].
- Can only install one of: illuminate/container[v6.0.0, v5.8.31].
- Can only install one of: illuminate/container[v6.0.1, v5.8.31].
- Can only install one of: illuminate/container[v6.0.2, v5.8.31].
- Can only install one of: illuminate/container[v6.0.3, v5.8.31].
- Can only install one of: illuminate/container[v6.0.4, v5.8.31].
- Can only install one of: illuminate/container[v6.1.0, v5.8.31].
- Can only install one of: illuminate/container[v6.2.0, v5.8.31].
- Can only install one of: illuminate/container[v6.3.0, v5.8.31].
- Can only install one of: illuminate/container[v6.4.1, v5.8.31].
- Can only install one of: illuminate/container[v6.5.0, v5.8.31].
- Installation request for illuminate/container (locked at v5.8.31) -> satisfiable by illuminate/container[v5.8.31]
Composer version 1.9.0 2019-08-02 20:55:32
PHP 7.3.8

Looks like you have some other globally installed package that uses illuminate/container v5.x (Laravel 5.x), and are trying to install the latest version of tlint globally (Which is updated to Laravel 6.x). You can try to install tlint as a project dependency instead (composer require tightenco/tlint for a Laravel 6 project instead of composer global require tightenco/tlint) OR upgrade your other global packages that may rely on illuminate/container (likely valet or laravel-installer)

We should be able to bundle the blade compiler into tlint to avoid this common issue (especially with the updated release cadence), looking into this.

Hi Logan! Is this linter only for Laravel6?

@devramguiao No, the linter is not only for Laravel 6, I have published a new version that should ease the dependency requirements you were running into (v3.0.5), could you try upgrading?


https://github.com/MAMIDITEJA32/Hack-Web3Conf/edit/main/postman/MAMIDITEJA32

@aslams2020 https://github.com/GSSoC24/Postman-Challenge/pull/4292
https://github.com/GSSoC24/Hack-Web3Conf/pull/717


fix: DEV-3692: Comment edited session state

Hi @Travis1282!
This PR was created in response to a PRs in upstream repo:

https://github.com/heartexlabs/label-studio-frontend/pull/954

We don't have comments in opensource


Elections fail on Python 2.6

This appears to be a python-etcd incompatibility.  The handle() function that is supposed to turn error responses into the EtcdKeyNotFound error that we're expecting, contains this line:
msg = '{} : {}'.format(message, cause)

Which fails on python 2.6, raising ValueError('zero length field name in format').  That gets translated into a generic EtcdException.
Not sure how much we care about 2.6 support but I think it's broken right now.

We don't care about this until Python 2.6 support comes back on the table.


Document federation requirements

This is a follow-up to  #1463.
It would be good to document specific federation requirements for remote networks, i.e. that fed.brid.gy must be federated with in addition to bsky.brid.gy or web.brid.gy and (or?) individual custom domains of bridge accounts.
The explanation should mention 'limited federation mode' to make it easier to discover for Mastodon admins.

Yes! This would be part of a bigger "I'm an instance admin, what do I need to know?" section that I want to write. Will do.

Other things to note here:

Opt in
Only fully public profiles, posts, and other data are bridged. unlisted/quiet public not included.
Multiple subdomains: bsky, web, fed, ap, more in the future

Some web site actors will be on their own domains, eg<EMAIL_ADDRESS>

Authorized fetch by instance actor https://fed.brid.gy/fed.brid.gy
Spam filters, other troubleshooting info in https://fed.brid.gy/docs#troubleshooting
DMs
blocks are translated, both directions
reports (Flags) are translated to Bluesky reports
bot accounts are Service, bridged users are Person
all accounts are normal AP accounts, subject to normal moderation tools, etc


[ENH] - Bump to latest Node.js

Feature description
We are currently using Node 18
Value and/or benefit
Node is already at version 22, so we're two versions behind.
Anything else?
No response

We are not using 18.x, we have "node": ">=18.0.0" in our package.json which, depending on when one last updated/created their env should be on the latest point release of node (for example right now should be on 22.x)
I think the action here would be to bump the minimum version to 22 based on the Node EOL calendar https://endoflife.date/nodejs (technically we could go to 20 but everything seems to work on 22)

Left a comment in #415  we can address this in that PR directly or do a stack of PRs


about stride of resa

您好，
我阅读了代码，发现四次迭代的stride分别为2, 4, 8, 16， 并未实现论文中列举的1,2,4,8....，比较好奇为什么不去实现stride=1呢？
另外，我理解是以stride=2为例，x0位置 = x0+x2的特征信息，x1位置 = x1 + x3 ....，其实本质是特征图的错位相加。由于是对整个特征图(c,h,w)做的conv，似乎没有真正意义上的: x2n->.....->x8->x6->x4->x2->x0这样类似与SCNN的信息传递，似乎没有一个特征信息从下到上一直传递（类似预resnet跳跃链接），这是不是一个方向需要多次迭代的原因呢？或者可以解释一下为什么一个方向需要多次迭代吗？
非常期待您的解答，谢谢！

Thanks for your interest.

About Stride.  In the paper, it's a more general way to define the stride. In our setting, stride = 2,4,8,16.  We can also use stride = 1,2,4,8,16, but inference may be slower.  The experiment result is also similar.  The reason why we use different strides is to help network gather global information like ASPP. The stride is similar to dilation. The information may be similiar between adjacent featuremap slice (i.e, stride = 1), thus it may not help network gather more information.
About message passing. Addition is one message passing method.  The skip connection in ResNet also uses the addition.
About iteration. In our paper, we have analyzed the iteration and how it gather the information. Please see the Figure 3 in our paper and the related analysis.

感谢你的解答！


integrate simRank algorithm

I was planning to integrate simRank algorithm within this. My original thought is:

export data into hdfs
run simRank algo via spark
import the the scores back into neo4j (aka. binded with relationships between nodes)

I'm not positive if that's the right way to go and where to start. Any suggestions? That would be much appreciated.
/cc @kbastani

Hey, thanks for your node. Would love to discuss it, can you drop me an email to michael at neo4j.com

@jexp Looking forward to your reply:)


Modulo encoding

Adds Modulo/Circular Encoding for the inputs of Performance RNN.

I signed it!
I'm a Googler.

I signed it!


How to remove excess div 'rich-text' from template?

I dont want use the div with that name. How to edit default render template for RichTextField?

Hi @Sibiryakanton - please use the Wagtail tag on Stack Overflow for support queries.
(This is an existing unresolved issue - see #1214)


watcher instead threaded

!test

Refer to this link for build results (access rights to CI server needed):
http://jenkins.2gis.local/job/vmmaster.staging.regression/283/
Test PASSed.

!test

Refer to this link for build results (access rights to CI server needed):
http://jenkins.2gis.local/job/vmmaster.staging.regression/284/
Test PASSed.

Пока не требуется.


[#2021] move the rest-client from cli module to common module

What changes were proposed in this pull request?

Move the rest-client from cli module to common module.
Some other small optimizations.

Why are the changes needed?
Fix: #2021
Does this PR introduce any user-facing change?
No.
How was this patch tested?
CI

@xianjingfeng I suppose to abstract a new module server-common by  https://github.com/apache/incubator-uniffle/pull/1884 aimed to reduce the client fat jar size.
If we move rest-client from cli to common, the uniffle client fat jar would be greater than before although the shaded rss spark client never use the rest-client?
So we can move the rest-client to server-common, and all server-side modules can use server-common as dependency, and rss-client can avoid introduce the extra unnecessary dependencies.

@xianjingfeng I suppose to abstract a new module server-common by #1884 aimed to reduce the client fat jar size.
If we move rest-client from cli to common, the uniffle client fat jar would be greater than before although the shaded rss spark client never use the rest-client?
So we can move the rest-client to server-common, and all server-side modules can use server-common as dependency, and rss-client can avoid introduce the extra unnecessary dependencies.

Agree. But when will #1884 be ready?

Codecov Report
Attention: Patch coverage is 42.85714% with 4 lines in your changes missing coverage. Please review.

Project coverage is 52.79%. Comparing base (5ddcc28) to head (84201fb).
Report is 34 commits behind head on master.

Files
Patch %
Lines

...pache/uniffle/common/util/http/RestClientImpl.java
0.00%
3 Missing :warning:

...he/uniffle/common/util/http/HttpClientFactory.java
0.00%
1 Missing :warning:

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #2022      +/-   ##
============================================
+ Coverage     52.77%   52.79%   +0.02%
- Complexity     2498     2982     +484
============================================
  Files           398      456      +58
  Lines         18135    24003    +5868
  Branches       1660     2236     +576
============================================
+ Hits           9570    12672    +3102
- Misses         7981    10502    +2521
- Partials        584      829     +245

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

One high level question: why does the coordinator module need the rest client?

For #1941. The shuffle server will provide a http interface for obtaining application information. The coordinator needs to obtain this data from each shuffle servers and then aggregate them.

Why RPC calls cannot be used? I think it will more efficient and doesn't rely on the http interface.

We need obtain this data from a single shuffle server sometimes, so we need provide a http interface in the shuffle server side. I think it is not necessary provide another RPC interface. The call frequency of this interface is not high.

Like that the shuffle servers could report all its running applications back to coordinator via rpc calls? Just like how the heartbeat is made.

The amount of the data is a bit large. The data of an application would be as follow.
public class ApplicationMetricsVo {
  private long totalDataSize;
  private long inMemoryDataSize;
  private long onLocalFileDataSize;
  private long onHadoopDataSize;
  private Map<Integer, Map<Integer, Long>> partitionDataSizes;
  private boolean existHugePartition;
}

@xianjingfeng I suppose to abstract a new module server-common by #1884 aimed to reduce the client fat jar size.
If we move rest-client from cli to common, the uniffle client fat jar would be greater than before although the shaded rss spark client never use the rest-client?
So we can move the rest-client to server-common, and all server-side modules can use server-common as dependency, and rss-client can avoid introduce the extra unnecessary dependencies.

Agree. But when will #1884 be ready?

@xianjingfeng I just refactor my PR and keep only create a server-common module, please take a look.

Like that the shuffle servers could report all its running applications back to coordinator via rpc calls? Just like how the heartbeat is made.

The amount of the data is a bit large. The data of an application would be as follow.
public class ApplicationMetricsVo {
  private long totalDataSize;
  private long inMemoryDataSize;
  private long onLocalFileDataSize;
  private long onHadoopDataSize;
  private Map<Integer, Map<Integer, Long>> partitionDataSizes;
  private boolean existHugePartition;
}

Hmmm, looks pretty compact to me. Except the partition data sizes, I don't think this info has much different than the heartbeat message.
If you haven't finished all the related code, I would suggest to implement a new ReportRunningApplications service point in coordinator instead, which I think would be less code.
The shuffle server could still export its internal metrics/running applications via the Web interface.

If this method is used, each shuffle server needs to report hundreds of application information and tens of thousands of partition information.This may cause performance issues for the coordinator.
In general, we only need to check the information of a particular application, not all of them.

In general, we only need to check the information of a particular application, not all of them.

Could you elaborate a bit more on this? Based on your umbrella issue(#1941), I think we still need all the application infos to detect abnormal apps.

Scenario 1:
When we find that the cluster load is high, we need to know whether the shuffle data volume of some applications is too large. In this scenario, we need the brief information about all applications, such as the total shuffle data volume.
Scenario 2:
When we find that an application is very slow, we need to quickly determine whether the application has data skew. In this scenario, we need to know the detailed information of this application, including the amount of data in each partition.
In general, we only query this data when something unusual happens, and there is no need to report the data to the coordinator regularly.

I see, it's kind of throw away messages and for diagnostic purpose. I'm neutral to this change then. However I still think it would be better to report application info back and that can cover all the use cases you have described. The additional overhead might be negligible compared to other infos.

Scenario 1: When we find that the cluster load is high, we need to know whether the shuffle data volume of some applications is too large. In this scenario, we need the brief information about all applications, such as the total shuffle data volume.
Scenario 2: When we find that an application is very slow, we need to quickly determine whether the application has data skew. In this scenario, we need to know the detailed information of this application, including the amount of data in each partition.
In general, we only query this data when something unusual happens, and there is no need to report the data to the coordinator regularly.

What's your opinions? @jerqi @zuston @rickyma @maobaolong

@xianjingfeng Thanks for invite me to join this discussion. In fact we have done some work on displaying the application detailed information in coordinator.

Application buffer(size/count) flush(size/filecount)
Application exit reason

We also want to show the application spark-conf and find some lineage from the spark application and the upstream scheduled task, base on this, we can get effect and benefit after a spark job using Uniffle.
We also want to show more detailed include each stage write/read throughput cost and timeline.
All of these, we want to store and persist so that we can search the history information, maybe this can help us on troubleshooting.

After all of above, we want to store and persist into storage, so that we can search the history information, maybe this can help us in daily troubleshooting.

When will this be done?

Do you think we have to know each partition information? If we only report topN partition information to coordinator, do you think it can be enough and easy? If so, we can add some extra information in the heartbeat request of Driver->Coordinator and Server->Coordinator, the Coordinator can have a detailed Application information.

Ok for me. And when will this be done?

@lwllvyb Would you like to show some progress about the work about collect and show application detailed in coordinator and dashboard?

@xianjingfeng I suppose to abstract a new module server-common by #1884 aimed to reduce the client fat jar size.
If we move rest-client from cli to common, the uniffle client fat jar would be greater than before although the shaded rss spark client never use the rest-client?
So we can move the rest-client to server-common, and all server-side modules can use server-common as dependency, and rss-client can avoid introduce the extra unnecessary dependencies.

Agree. But when will #1884 be ready?

@xianjingfeng Glad to share that the #1884 has been merged.

@xianjingfeng Glad to share that the #1884 has been merged.

Get it

This PR is no longer needed. Let's wait for @lwllvyb

Recntly, I was working on how to display more information on the dashboard UI. More details about the application will be displayed in the Application Page.（I am new to Vue.js, can somebody help beautify the UI?）
I add some columns in the Application Page: PartitionNum | writeMemorySize | flushedLocalFileNum | flushedLocalTotalSize | flushedHadoopFileNum | flushedHadoopTotalSize | writeTotalSize

I will create a new PR.


[DO NOT MERGE] testing travis

I think it needs the pin-version fix. On it.

I think it needs the pin-version fix.

Yup. Should be all set now.


Update free-programming-books-langs.md

added links on C programming
What does this PR do?
Add resource(s) | Remove resource(s) | Add info | Improve repo
For resources
Description
Why is this valuable (or not)?
How do we know it's really free?
For book lists, is it a book? For course lists, is it a course? etc.
Checklist:

[x] Read our contributing guidelines.
[x] Search for duplicates.
[x] Include author(s) and platform where appropriate.
[x] Put lists in alphabetical order, correct spacing.
[x] Add needed indications (PDF, access notes, under construction).
[x] Used an informative name for this pull request.

Follow-up

Check the status of GitHub Actions and resolve any reported warnings!

Please look at other goalkicker links and follow the syntax

thank you @jarek-pacocha Happy New Year!

thank you @jarek-pacocha Happy New Year!

Glad I could help, Happy New Year! 😉


Remove abronan as a maintainer

Farewell 👋
Signed-off-by: Alexandre Beslic<EMAIL_ADDRESS>

Current coverage is 55.02% (diff: 100%)

Merging #1435 into master will increase coverage by 0.08%

@@             master      #1435   diff @@
==========================================
  Files            82         82
  Lines         12901      12901
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits           7088       7099    +11
+ Misses         4826       4821     -5
+ Partials        987        981     -6

Powered by Codecov. Last update 9028bcb...e5853a6

Sincerely sorry to see you go. It's a big loss for the project.

😢

Thanks and good luck everyone. Have fun for the next steps.

👎 :(


android build broken

There is a problem because of #1378. After https://github.com/graphhopper/graphhopper/commit/04514f24ba479538dc6a0de8c072fd108eb1a557 the Android apk fails to build.
There are two new dependencies jaxws-api and javax.ws.rs-api in android now and IMO this results in problems. With:
cd android/app
mvn package android:deploy

you will get:
...
[INFO] --- android-maven-plugin:4.5.0:dex (default-dex) @ graphhopper-android ---
[INFO] Convert classes to Dex : /home/peter/Documents/quell/graphhopper/android/app/target/classes.dex
[INFO]
[INFO] PARSE ERROR:
[INFO] unsupported class file version 53.0
[INFO] ...while parsing META-INF/versions/9/javax/xml/bind/ModuleUtil.class
[INFO] trouble processing "javax/xml/soap/AttachmentPart.class":
....

It seems that these dependencies are compiled with JDK 9 which is not possible when using Android at the moment. Android does not even support all 8 features.
btw: I wonder what this openjdk Android port is about.

https://firebase.google.com/docs/test-lab/
:-)

Or maybe a Travis build would be enough for starters:
https://docs.travis-ci.com/user/languages/android/

Travis has problems with mixing recent JDKs and Android: https://github.com/travis-ci/travis-ci/issues/8513 (it looks only closed - it is not fixed!)
But as we have our own magic to fetch openjdk10 etc we could probably try language:android ...

Compiles now - thanks! Will try to resurrect our Android deployment on a dedicated server and see if this is also runnable on Android.
As it now gets a bit crowded module-wise :) ... would you mind to add a (different) description and name to these modules and can you roughly explain why we need two api modules?

This is what we were just talking about, the JSON representation going in its own module.

From a quick look there are javax dependencies also in client-hc and in its transitive web-api module. So if using client-hc on Android, could fail again like @karussell reported?

Argh yes, but it doesn't really need to be there, will fix tomorrow.


Validation loss increases after a certain epoch.

Hi, I realized that my validation loss is increasing over time after a certain epoch while the training loss is converging. It seems to me that the network is overfitting. I have tried disabling random cropping, and introduced augmentations to mitigate this issue, however the problem seems to remain. I'm fairly new to neural network training, any suggestion is much appreciated!

Here is my opt file:
==> cudnn version: 7104
==> Cmd:
['main.py', 'circledet', '--exp_id', 'Apple5', '--arch', 'hourglass', '--batch_size', '4', '--master_batch', '4', '--lr', '2.5e-4', '--dataset', 'apple2', '--load_model', '/home/lmga-titanx/CircleNet/src/lib/models/ctdet_coco_hg.pth', '--num_epochs', '1000', '--input_w', '128', '--input_h', '128', '--K', '1', '--not_rand_crop', '--rotate', '1']
==> Opt:
  K: 1
  aggr_weight: 0.0
  agnostic_ex: False
  arch: hourglass
  aug_ddd: 0.5
  aug_rot: 0
  batch_size: 4
  cat_spec_wh: False
  center_thresh: 0.1
  chunk_sizes: [4]
  data_dir: /home/lmga-titanx/CircleNet/src/lib/../../data
  dataset: apple2
  debug: 0
  debug_dir: /home/lmga-titanx/CircleNet/src/lib/../../exp/circledet/Apple5/debug
  debugger_theme: white
  demo:
  demo_dir:
  dense_hp: False
  dense_wh: False
  dep_weight: 1
  dim_weight: 1
  down_ratio: 4
  eval_oracle_dep: False
  eval_oracle_hm: False
  eval_oracle_hmhp: False
  eval_oracle_hp_offset: False
  eval_oracle_kps: False
  eval_oracle_offset: False
  eval_oracle_wh: False
  exp_dir: /home/lmga-titanx/CircleNet/src/lib/../../exp/circledet
  exp_id: Apple5
  ez_guassian_radius: False
  filter_boarder: False
  fix_res: True
  flip: 0.5
  flip_test: False
  gpus: [0]
  gpus_str: 0
  head_conv: 64
  heads: {'hm': 1, 'cl': 1, 'reg': 2}
  hide_data_time: False
  hm_hp: True
  hm_hp_weight: 1
  hm_weight: 1
  hp_weight: 1
  input_h: 128
  input_res: 128
  input_w: 128
  keep_res: False
  kitti_split: 3dop
  load_model: /home/lmga-titanx/CircleNet/src/lib/models/ctdet_coco_hg.pth
  lr: 0.00025
  lr_step: [90, 120]
  lv: 2
  mask_focal_loss: False
  master_batch_size: 4
  mean: [[[0.40789655 0.44719303 0.47026116]]]
  metric: loss
  mse_loss: False
  nms: False
  no_color_aug: False
  norm_wh: False
  not_cuda_benchmark: False
  not_hm_hp: False
  not_prefetch_test: False
  not_rand_crop: True
  not_reg_bbox: False
  not_reg_hp_offset: False
  not_reg_offset: False
  num_classes: 1
  num_epochs: 1000
  num_iters: -1
  num_stacks: 2
  num_workers: 4
  off_weight: 1
  ontestdata: False
  output_h: 32
  output_res: 32
  output_w: 32
  pad: 127
  peak_thresh: 0.2
  print_iter: 0
  rect_mask: False
  reg_bbox: True
  reg_hp_offset: True
  reg_loss: l1
  reg_offset: True
  resume: False
  root_dir: /home/lmga-titanx/CircleNet/src/lib/../..
  rot_weight: 1
  rotate: 1.0
  rotate_reproduce: 0
  save_all: False
  save_dir: /home/lmga-titanx/CircleNet/src/lib/../../exp/circledet/Apple5
  scale: 0.4
  scores_thresh: 0.1
  seed: 317
  shift: 0.1
  std: [[[0.2886383  0.27408165 0.27809834]]]
  task: circledet
  test: False
  test_scales: [1.0]
  test_val_name:
  trainval: False
  val_intervals: 1
  vis_thresh: 0.2
  wh_weight: 0.1```

No problem. This is a common issue with convolutional neural networks. This article covers some things you can do.
I would encourage you to use random cropping in addition to whatever augmentation you have. You can also add regularization to the loss function.

Also, I would make sure that you're initializing your weights from pre-trained models. See the bottom of this page

@EthanHNguyen Thank you so much for your feedback! In regards to adding regularization, is there an option in opts that allows me to do so, or do I have to add new layers into the existing code?
I believe that I should've initialized the weights from pre-trained model by having load_model path/to/ctdet_coco_hg.pth , where ctdet_coco_hg.pth was downloaded from this page. Please let me know if I'm missing something. Thank you!

@EthanHNguyen Thank you so much for your feedback! In regards to adding regularization, is there an option in opts that allows me to do so, or do I have to add new layers into the existing code?

No, there isn't a built-in option. In our experiments, we mainly used image augmentation and early stopping as regularization. We simply chose the model that performed best on validation.
I would add regularization to the loss function here.

I believe that I should've initialized the weights from pre-trained model by having load_model path/to/ctdet_coco_hg.pth , where ctdet_coco_hg.pth was downloaded from this page. Please let me know if I'm missing something. Thank you!

Yes, that's right.


(bugfix) tagless components error part2

Mimic pre-glimmer2 behavior of attributeBindings
fixes #19

@runspired - this worked in 4 or so use cases where other hacks did not. For example, attribute bindings within ember-async-image now work upon deferred loading (as one example https://github.com/runspired/ember-async-image/blob/daf5e88ecc3aafcce9e32394f4848132628e07f0/addon/components/async-image.js#L26)
worth noting I did not mean to re-write the whole mixin - at all - I think there is an upstream bug in Ember TBH and I was about to give up but upon reading the source code saw a line that made me think I could hack around this by modifying the prototype of the mixin from within the constructor - and bam, all edge cases started working - I deleted the cruft but did not back out all the and return the file back to what you are likely more familiar with - let me know if thats an issue and I will

CI:

actually dont merge this - i think i can just take that magic line and paste it into master branch here after dinner - master has some good code i dont want to lose - just fyi

@eriktrom wasn't going to merge cause it was a big enough clobber I was going to need to work through it for a while :P a simpler PR is very welcome :)

yeah i know :) it was after too much hacking and not enough sleep late in the day haha


Support Ruby 3.3.4

There is an issue with the net-pop gem in Ruby 3.3.3, causing dependency errors during the build process. The problem has been resolved in Ruby 3.3.4, which includes necessary fixes for the net-protocol gem dependency.
Updating the DevContainer images to support Ruby 3.3.4 will address these issues.
Steps to Reproduce:

Set up a DevContainer with Ruby 3.3.3.
Include the net-pop gem in your Gemfile.
Run bundle install.

Expected Behavior:

Successful installation of all gems without dependency errors.

Actual Behavior:

Dependency errors due to missing net-protocol gem.

Solution
Update the DevContainer images to support Ruby 3.3.4. This update will ensure compatibility and resolve existing dependency issues related to the net-pop gem.
References:
https://github.com/ruby/net-pop/issues/26
https://bugs.ruby-lang.org/issues/20581
https://github.com/ruby/ruby/releases/tag/v3_3_4

Hi 👋
Thanks for reporting this issue. I am releasing a dev tag for the Ruby image, post that I'll create a prod release by end of this week.

Deployed https://github.com/devcontainers/images/pull/1126/files#diff-97a3d17cd8818d53d1610a0fdecf71a6d20e719eaeaa50c898fbc8ea98d66bafR42

In the meanwhile, you can pin to an older image version (1.1.6)

Thank you for the suggestion to pin to an older image version (1.1.6). Unfortunately, I cannot adopt this approach due to a requirement from Heroku.
Heroku mandates specifying the Ruby version in the Gemfile to ensure the correct version is installed. Consequently, my Gemfile specifies Ruby 3.3.4, leading to the following error during the build process when the DevContainer image has an older Ruby version:
 => ERROR [app dev_container_auto_added_stage_label 10/10] RUN bundle config set --global without "production"     && bundle install     && yarn install     && yarn cache clean     && chown -R vscode:vsc  0.3s
------
 > [app dev_container_auto_added_stage_label 10/10] RUN bundle config set --global without "production"     && bundle install     && yarn install     && yarn cache clean     && chown -R vscode:vscode /app /usr/local/rvm:
0.298 Your Ruby version is 3.3.3, but your Gemfile specified 3.3.4

This mismatch causes the build to fail, as the base image is still on Ruby 3.3.3, while the Gemfile requires Ruby 3.3.4.
Could you provide guidance on how to proceed with ensuring the DevContainer image is updated to include Ruby 3.3.4? This update is essential to align with the required Ruby version specified in my Gemfile and to maintain compatibility with Heroku.
Thank you!


Report checkpoints

Would be great to be able to store a report in some way that user can access it later and use as a reference. An example use case would be for somebody who needs to reason about changes made to their system over time in a written report or something like that, but they can also simply share checkpoint URLs in a team or something like that, it would also have more context then flat SVG export (which is useful for other purposes, of course).

(somewhat related to #988)


Can you make this work inside a callout?

It's a really convenient plugin for refactoring. However, there can also be headings inside callouts. Can this plugin be made to work with those headings too? Thank you!

No plans for implementation due to limited circumstances.


docs: toContain supports Sets as well

Codecov Report

Merging #8131 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #8131   +/-   ##
=======================================
  Coverage   62.43%   62.43%
=======================================
  Files         264      264
  Lines       10368    10368
  Branches     2515     2514    -1
=======================================
  Hits         6473     6473
  Misses       3318     3318
  Partials      577      577

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5bad349...a91d283. Read the comment docs.

Thanks! Maybe we should even extend it to "Arrays and other iterables"? Relevant matcher code.
@pedrottimark Do you know if iterables in general have only been supported since a specific version? I think they already were before your recent toContain refactor. If they've been supported in previous versions, we should also update the versioned docs.

From #2104 I think? So yeah, all versioned docs should be updated :)


⚠ Samsung has degraded performance

In d0bb1b7, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2694 ms

Resolved: Samsung performance has improved in 3eaa1be.


Make an icon in message header to see is "Reading confirmation" requested currently or not

Expected behaviour
User may see current status of "Reading confirmation" request. Is it set to the currently edited message or not. The best behaviour is to make some icon and show it in a header in right corner (where usually star is shown in view mode; but here in edit mode).
Actual behaviour
User cannot understand is "Reading confirmation" request is setuped or not.
Steps to reproduce

Just try to create message and setup and/or remove "Reading confirmation".

Environment
K-9 Mail version: 5.0.10
Android version: 6.0.1
Account type (IMAP, POP3, WebDAV/Exchange): POP3

I believe "Read confirmation" is not supported at all in K-9. I think it's implemented as a Flag. But I'm not sure.

So K-9 doesn’t send anything? That’s what I’ve been wondering lately, seeing mail in Thunderbird with “Reading confirmation” while nothing was happening on K-9.

When the sender sends an email with request for read receipt, Thunderbird shows a popup to send receipt

While we don't have any similar option in the K-9, We can implement a dialog for messages with that request. When the user accepts that dialog, a new message will compose with default text that tells user that the message has been received or read.

Messages with these headers have read receipt request. So we can identify these message with them:
"Disposition-Notification-To": "sender"
"X-Confirm-Reading-To": "sender"
"Return-Receipt-To": "sender"

I can make a pull request for this feature if you are OK. @cketti

Showing a strip of information with buttons that can be ignored and showing a dialog that interrupts the workflow are two very different user experiences. We definitely don't want to do the second. I'll have to talk to our designers to help come up with a mobile-friendly user interface for this feature.
If you want to, you can start by creating a pull request adding some code that will generate a message disposition notification (see RFC 8098. However, this code shouldn't be used in the app yet.


Contributors listed in package.json

generate all add contributors to package.json and CONTRIBUTORS. This adds a lot of bloat to package.json. Is there an option to only add them to the CONTRIBUTORS file?

cc @JustinBeckwith

this can be done by changing https://github.com/GoogleCloudPlatform/nodejs-repo-tools/blob/master/src/build_packs/nodejs.js#L157 to delete json.contributors.


News on update?

Love the work, this has got me to diamond with practicing and understanding cards better, but any news on the new update?

https://github.com/pixeltris/YgoMaster/issues/130


Integrate continuous integration / delivery

Currently there is no continuous correctness checking or mechanisms to
push updated artifacts to GitHub automatically. These tasks are manual,
repetitious and easy to be skipped.
This commit integrates CI that automates this tooling.

Blocked by https://discourse.drone.io/t/drone-not-seeing-nor-syncing-forked-repo/5846/11


Fix cargo clippy

Since we have the cargo clippy in the CI, we should fix all the warnings as well. While we are at it, we should also add the script to run these fmt and clippy checks locally, so it is easier for developers.

Didn't sync the main branch correctly. Seems all the Clippy warning is fixed in #176


capnp imports in JS

This is basically a continuation of https://github.com/jdiaz5513/capnp-ts/issues/75 - at this point, generating the TS files for foo.capnp referenced in the issue works but generating JS files fails with
foo.capnp.ts:9:21 Cannot find module './bar.capnp'.
foo.capnp.ts:9:21 Cannot find module './bar.capnp'.
Error: CAPNPC-TS008 Failed to transpile emitted schema source code; see above for error messages.
    at transpileAll (/usr/local/lib/node_modules/capnpc-js/lib/index.js:71:15)

Is there another way to make this work or a JS implementation oversight?

This is a bit tricky - the typescript compiler requires all imports to be valid so that means bar.capnp.ts needs to be emitted before foo.capnp.js is transpiled.
You can get around this by either compiling all the schema files at once (capnpc -ojs src/capnp/*.capnp) or by compiling bar.capnp first.
noEmitOnError: false might help here, leaving this issue open to remind me to try that.

Compiling bar.capnp first and then compiling foo.capnp gets me:
foo.capnp.ts:9:21 Could not find a declaration file for module './bar.capnp'. '/Users/stas/test/bar.capnp.js' implicitly has an 'any' type.
foo.capnp.ts:9:21 Could not find a declaration file for module './bar.capnp'. '/Users/stas/test/bar.capnp.js' implicitly has an 'any' type.

Is there something else that needs to be done to make that work?
Compiling all the schema files at once seems to work though, thanks!


PR to trigger tests and verify rhel-8 branch

The tests have all passed so I am happy with the status of the rhel-8-3 branch


Maximizing Chatterino creates a border on adjacent monitors

Describe the bug
check #1718 #1428 #1734
A white border appears on adjacent monitors when maximizing the main window on one monitor.
Not sure why all got closed, even if the problem still exists.
To reproduce
Toggle fullscreen mode in Chatterino.
Screenshots
check the issues
Chatterino version
Chatterino 2.2.2-fix (commit a82f57f94)
(no selfbuild)
Operating system
Windows 10 Pro 2004 (OS Build 19041.630)
Additional information
I read about a problem with QT < 5.14. The install uses 5.12.9 (maybe that's the problem?)
Chatterino About: Built with Qt 5.12.9, Windows SDK, MSVC 192729112

Fresh Windows 10 install, I also have this issue

Chatterino version 2.3.0 (commit  c65a893 , problem still exists.
Chatterino About: Built with Qt 5.14.0, Windows SDK, MSVC 191627043

also having this issue, Chatterino 2.3.1 (commit cf33566fd)

After looking into it, it seems this may be an issue with Windows, and I couldn't find any potential fix that can be implemented.
However, I may have found a workaround for this, we might want to instruct users to do it somehow until another fix is known.
The workaround:
Right click chatterino.exe -> Properties -> Compatibility -> Enable "Run this program in compatibility mode for:" -> Select Windows 8 in the drop-down menu.

After applying this the app seems to maximize with no white bar on the second monitor (on f605221).

This actually fixed it for me as well. Great Job!

Can confirm this is a Windows bug as Firefox has been showing the same behaviour for me. Probably worth closing :+1:

Yes, this issue should have the blocked label added and issue closed.


How to set global a=msid-semantic attribute?

I want to set a=msid-semantic: WMS 5kAAIomHNlen3snVvaSMUzojvdRtLpKBO8o3 into the global section of the SDP.
Doing this does not work:
var sdp = {};
// [...]

sdp['msid-semantic'] = 'WMS 5kAAIomHNlen3snVvaSMUzojvdRtLpKBO8o3';

sdpTransform.write(sdp);

The generated SDP string does not contain global 'msid-semantic' attribute.
Is there a way to add custom (or non natively supported by this library) global attributes into a SDP?

msid-semantic parses into an object with keys ['semantic', 'token'] if you look at the grammar. So you need to give it an object with those values at least.

Opps, I searched the sources for msid-semantic rather than msidSemantic :)
Thanks. However a proper documentation with the supported attributes (and their syntax) would be nice.


Shuffle around the wiggle crates

This commit reorganizes the wiggle crates slightly by performing the
following transforms:

The crates/wiggle crate, previously named wiggle, was moved to
crates/wiggle/crates/macro and is renamed to wiggle-macro.

The crates/wiggle/crates/runtime crate, previously named
wiggle-runtime, was moved to crates/wiggle and is renamed to
wiggle.

The new wiggle crate depends on wiggle-macro and reexports the macro.

The goal here is that consumers only deal with the wiggle crate
itself. No more crates depend on wiggle-runtime and all dependencies
are entirely on just the wiggle crate.

Thanks! Do we want to keep the crates subdir inside wiggle any longer? We don't have that indirection in wasi-common. This may be a good opportunity to get rid of it.

FWIW, it looks like the paths to the test witx files need to updated for the tests to pass.


refactor: scheduler register task

Description

Refactor scheduler register task.

Related Issue

Motivation and Context

Screenshots (if appropriate):
Types of changes

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] Documentation Update (if none of the other choices apply)

Checklist:

[x] Code compiles correctly.
[x] My code follows the code style of this project.
[ ] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

Codecov Report

Merging #924 (6a2af5b) into main (869f3d1) will increase coverage by 0.02%.
The diff coverage is 54.54%.

@@            Coverage Diff             @@
##             main     #924      +/-   ##
==========================================
+ Coverage   43.07%   43.10%   +0.02%
==========================================
  Files         106      106
  Lines        8273     8272       -1
==========================================
+ Hits         3564     3566       +2
+ Misses       4249     4248       -1
+ Partials      460      458       -2

Flag
Coverage Δ

Object-compatibility-e2etests
?

e2etests
?

unittests
43.10% <54.54%> (+0.02%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

scheduler/supervisor/task.go
67.15% <0.00%> (-2.55%)
:arrow_down:

internal/idgen/task_id.go
100.00% <100.00%> (ø)

pkg/util/net/urlutils/url_utils.go
66.66% <100.00%> (+11.11%)
:arrow_up:

client/daemon/daemon.go
5.57% <0.00%> (-0.02%)
:arrow_down:

client/daemon/peer/peertask_base.go
46.69% <0.00%> (ø)

pkg/source/source_client.go
1.88% <0.00%> (+0.01%)
:arrow_up:

cdn/supervisor/progress/progress.go
97.01% <0.00%> (+5.97%)
:arrow_up:


Not working INVALID_PARAMETER

Hi
I have tested it on vulnerable Windows 7 and Windows 2008 none of them worked:
SMB1 session setup allocate nonpaged pool success
SMB1 session setup allocate nonpaged pool success
good response status: INVALID_PARAMETER
done
no shell back in msf

Hey @hal9k2 would you mind running the eternalblue_checker.py against the machines your testing and sharing the output?
The output that you're seeing is standard and not indicative of any errors (although I do think it should be modified and I will look into it), but if you're not getting a shell back then something is obviously wrong. Windows7 and Windows Server 2008 are the two versions I have tested on the most and had the most success.
Like I stated above send me the output of the checker, and I'll look into this further.

Hi, thanks for the answer. So below resutls of checker:
root@kali:~/Dropbox/Tools/exploits/AutoBlue-MS17-010# python eternalblue_checker.py 10.XX.XX.XX
Target OS: Windows Server 2008 R2 Standard 7601 Service Pack 1
The target is not patched
=== Testing named pipes ===
spoolss: STATUS_ACCESS_DENIED
samr: Ok (64 bit)
netlogon: Ok (64 bit)
lsarpc: Ok (64 bit)
browser: STATUS_ACCESS_DENIED
root@kali:~/Dropbox/Tools/exploits/AutoBlue-MS17-010# python eternalblue_checker.py 10.XX.XX.XX
Target OS: Windows 7 Ultimate N 7600
The target is not patched
=== Testing named pipes ===
spoolss: STATUS_ACCESS_DENIED
samr: STATUS_ACCESS_DENIED
netlogon: STATUS_ACCESS_DENIED
lsarpc: STATUS_ACCESS_DENIED
browser: STATUS_ACCESS_DENIED
Nmap smb-vuln for Windows 7
Host script results:
| smb-vuln-ms17-010:
|   VULNERABLE:
|   Remote Code Execution vulnerability in Microsoft SMBv1 servers (ms17-010)
|     State: VULNERABLE
|     IDs:  CVE:CVE-2017-0143
|     Risk factor: HIGH
|       A critical remote code execution vulnerability exists in Microsoft SMBv1
|        servers (ms17-010).
|
|     Disclosure date: 2017-03-14
|     References:
|       https://blogs.technet.microsoft.com/msrc/2017/05/12/customer-guidance-for-wannacrypt-attacks/
|       https://technet.microsoft.com/en-us/library/security/ms17-010.aspx
|_      https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-0143
And for Windows 2008
Host script results:
| smb-vuln-ms17-010:
|   VULNERABLE:
|   Remote Code Execution vulnerability in Microsoft SMBv1 servers (ms17-010)
|     State: VULNERABLE
|     IDs:  CVE:CVE-2017-0143
|     Risk factor: HIGH
|       A critical remote code execution vulnerability exists in Microsoft SMBv1
|        servers (ms17-010).
|
|     Disclosure date: 2017-03-14
|     References:
|       https://technet.microsoft.com/en-us/library/security/ms17-010.aspx
|       https://blogs.technet.microsoft.com/msrc/2017/05/12/customer-guidance-for-wannacrypt-attacks/
|_      https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-0143
I used shell created with help of shell_prep.sh
My command was like:
python eternalblue_exploit7.py 10.XX.XX.XX sc_all.bin

Perfect, thanks for your help. You seem to be doing everything correctly as far as tool usage is concerned. I’ll be able to use the version information to look into this and see if perhaps there are some version specific issues that are unaccounted for.
Thank you for bringing this issue to my attention hopefully I’ll resolve it ASAP! If you happen to determine and solve the problem in the mean time, feel free to contribute your solution in a pull request related to this issue.

It happened to me as well, while I was trying to exploit without Metasploit, the HacktheBox retired machine named Blue - preparing for the OSCP :-)
At first it was due because I used the wrong IP as listener (my eth0 interface instead of the vpn interface), but then it still happened.
What I noticed is that the metasploit module, that never fails, doesn't always succeed in the same way.
Sometimes it needs a couple of tries (that does automatically), and it tries of different values of what in eternalblue_exploit7.py is called GroomConn.
It turns out that there is a 4th parameter in the python script, that if you do not specify, defaults to 13.
The metasploit module starts with 13 and then tries 17 (at least for 'Blue'...)
The technical details of this are far beyond my current level, but I ran the script with 13 (and 17) as 4th parameter and I got a shell.
 Similarly, using the metasploit exploit, the first time it succeeds with 13. If I exit meterpreter and try again, 13 fails and it has to try again with 17, then it succeeds.
As I said, I'm not sure what this means. The exploit looks pretty reliable but I don't know if an explanation to this behavior is simple or something that needs a deep understanding of how the exploit and the shellcode work.
Hope this helps.

@halfluke Thanks for your feedback!
You are right, in order to get success with this exploit you may have to try running it more than once and sometimes even with variable Groom Connections, even with the metasploit module.
Groom Connections are connections you are opening up to try and chain together kernel pool memory so that you can write to the buffer from a desired location. It gives the exploit some on the fly consistency in attempting to write to the buffer. I will update the docs to include this parameter, it was not originally included in the instructions/documentation because I assumed most people were aware of the optional feature from the original version that this code is forked from.
As far as stability, this exploit is actually not very stable or consistent at all. Varying success is expected. It will always work on a vulnerable system, but like you said sometimes it takes a few tries, so the frequency of success is variable and the exploit is also very unstable and is known to crash the exploitable service and sometimes even crash the box itself, so keep that in mind when applying this in the real world on pen-tests and even on the OSCP as it does have the potential to knock over a box. I personally am still researching the more stable variants on this exploit like eternalromance.
If you would like some more technical knowledge in regards to this exploit there is a fantastic write up paper by RiskSense that I recommend:
https://www.risksense.com/_api/filesystem/466/EternalBlue_RiskSense-Exploit-Analysis-and-Port-to-Microsoft-Windows-10_v1_2.pdf
As far as this issue is concerned I will hopefully get around to testing it very soon just in case it is legitimate, but to anyone else having the same problem please try the advice from @halfluke and re-run the exploit more than once.

Thanks for your reply. I will read the pdf!
I've just fixed my previous post where I messed up some GroomConn values :)
Perhaps it looked reliable to me because it always works on 'Blue'!
Thanks again.

After some research, I have been able to determine this is not a bug with these scripts, but more of a stability issue with the exploit as discussed above. If you run into this problem in the future, try adjusting groom connections or retry the exploit. Keep in mind that retrying the exploit without success could indicate the service has crashed and is no longer exploitable.
If there are further optimizations around this anyone would like to contribute feel free, otherwise there is no general issues with the code itself as far as I can tell.

The RiskSense link looks like its old here is the replacement one @3ndG4me if interested :)
https://risksense.com/wp-content/uploads/2018/05/White-Paper_Eternal-Blue.pdf


fix raw3-1 bringup

removed prosilica cams from raw3-1 startup

CR: OK

We definitely need urdf and launch checks for travis...!!!

I've readded group tag. Please review

All good now!
Created a new issue for launch file checks....#374

CR: OK


Install gnu bash in local setup

How to categorize this PR?

/area dev-productivity
/kind enhancement
What this PR does / why we need it:
#8766 has changed a bash script to use a feature for capitalizing words which is available in bash >=4.0, however it is failing on MacOS with the apple provided bash like
▶ make verify
hack/tools/logcheck
> Check
Executing golangci-lint
Executing gofmt/goimports
All checks successful
> Check Imports
...
> Check hack/tools/logcheck
> Check Helm charts
Checking for chart symlink errors
Checking whether all charts can be rendered
All checks successful
> Checking if license header is present in all required files
All files have license headers, nothing to be done.
hack/check-skaffold-deps.sh: line 21: > ${operation^} Skaffold Dependencies: bad substitution
make: *** [check] Error 1

In my case with MacOS Sonoma 14.1.1, the apple provided bash version GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin23), while the bash installed with brew is of version GNU bash, version 5.2.21(1)-release (x86_64-apple-darwin23.0.0) and after it is installed, make verify no longer fails with the bad substitution error.
/cc @timebertt
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Release note:

/retest-required


Deploy to AWS S3 / CloudFront / Route 53

Deploy the frontend Vue on AWS S3. However, S3 only serves HTTP requests, not HTTPS. HTTPS is needed for the authentication auth0 service to run.
Create a cloud front distribution with HTTPS point it to index.html.
Create an ACM certificate and an A-record on Route 53 to point to a domain

https://cloudgenetics.github.io/doc/#/frontend/aws


test add 2 files (bot test)

fasdfasdfa
Backport 9dc3f61c is needed to enable the blah1 feature
Backports b4d189ac..9dc3f61c are needed for performance for blah 2 feature
More edit after drok approved.

[ ]  Document the backports as errata in the pull request. They should have been documented before, but later is better than never.


Documentation tweak

Hi, Sean,
Could you add a little something to the documentation where we update libs.js? This is the only example you give where BOTH lines in the example code should be added, not just the bottom line. If you could make that a little more obvious for me, I would really appreciate it!
Thanks!
-Robyn

Should be resolved now.


Minor docs clarifications

Add some minor clarifications around labels-dependent L4 and where CIDR policies apply.

test-me-please


feat: more visible unplayable tracks (Snippet)

I always found it hard to easily see unplayable tracks in large playlists so I made this :D

FINALLY :/