Proposal to archive solid/solid-spec

It appears to be that the solid/solid-spec repository still attracts enagement and causing confusion. In order to direct the energy to solid/specification and other relevant Solid repositories, I suggest to archive solid/solid-spec. Archiving the repository will continue to keep all information public while prohibits updates eg files, issues.
If agreed, how to proceed:

Transfer useful issues to solid/specification or elsewhere.
Update its README to indicate the freeze/archival.
?
Archive

Including admins and editors for review so there is no oversight or surprises.

👍 agreed on all counts

Archiving also shuts off communication though; another option might be to have issue templates that force people to make choices before they submit anything.

Doesn't shut off entirely though. Header of the README needs to be simplified to guide the visitor to solid/specification.
The intention is not to have any issues in solid/solid-spec but to have it in solid/specification, so I'm not sure how a template would help with that other to say "stop. do it over there instead".
I still prefer archival. We don't need to dance around it anymore.

Yeah, with a good in indication in README, why not. No objections from me.

I also agree that we should do this, though I'd like to make sure that we get the README text right before hitting the archive button to avoid any confusion.

It appears to be that the solid/solid-spec repository still attracts enagement and causing confusion. In order to direct the energy to solid/specification and other relevant Solid repositories, I suggest to archive solid/solid-spec. Archiving the repository will continue to keep all information public while prohibits updates eg files, issues.
If agreed, how to proceed:

Transfer useful issues to solid/specification or elsewhere.
Update its README to indicate the freeze/archival.
?
Archive

Including admins and editors for review so there is no oversight or surprises.

Situation:
We have two repos related to the same topic i.e., Solid Specs.
I would encourage leaving the "original draft" version of the spec as is while simply focusing on the new "official specification" going forward.
If the old spec generates traffic, so be it. There's no obligation to respond to issues and comments i.e., it can fade away if it has too or act as a natural reminder about various aspects of the project.
Conclusion:
Let the nature of a repo just take its course. Traffic will go wherever it needs to go, over time :)

Beyond solid/solid-spec's historical role, there is no particular reason for it to receive any traffic. It is preferable to direct all energy towards solid/specification. Leaving solid/solid-spec subtracts from the potential. It also sends the wrong message eg. a contributor may get the impression that the spec work is inactive, or worse, their input is not valued due to lack of attention. Leaving it open is a recurring cost that we can easily avoid.

Beyond solid/solid-spec's historical role, there is no particular reason for it to receive any traffic. It is preferable to direct all energy towards solid/specification. Keeping solid/solid-spec active subtracts from the potential. It also sends the wrong message eg. a contributor may get the impression that the spec work is inactive, or worse, their input is not valued due to lack of attention. It has become a recurring cost that we can easily cut off.

The thing about community projects is that transparency has to be paramount, at all times.
If you don't have an obligation to act on an issue in the specs repo, it won't you cost anything in terms of time etc. In addition, anyone could courteously point an issue placed in the spec repo to the specifications repo etc..
Fundamentally, I advice transparency and caution when dealing with community projects. These issues aren't easy due to the fundamental nature of individuality i.e., one person can never really speak effectively for a community of individuals endowed with a cocktail of intentions :)

Archiving merely turns the repository into read-only. We are having this discussion in the open with exact steps outlined. Is there a particular criteria that you're thinking of that's not met in the proposal or how we are approaching it?
Even if I/we don't respond to an issue in solid/solid-spec, it costs me/us. That's input we'd rather have in solid/specification than not have. In the event someone responds, that's two additional steps; redirecting and re-creating issue. Three steps we can do without.
I'm all with you on transparency but I don't understand - and really I want to! - how leaving the old repo as is maintains transparency or how archiving it possibly goes against transparency.

There are quite a few open issues in solid/solid-spec
What should happen with them?

@akuckartz , as I've mentioned in the proposal:

If agreed, how to proceed:

Transfer useful issues to solid/specification or elsewhere.

Archiving merely turns the repository into read-only. We are having this discussion in the open with exact steps outlined. Is there a particular criteria that you're thinking of that's not met in the proposal or how we are approaching it?

My concern is about the consensus protocol. How many consenters or dissenters determine the final outcome here?

Even if I/we don't respond to an issue in solid/solid-spec, it costs me/us. That's input we'd rather have in solid/specification than not have. In the event someone responds, that's two additional steps; redirecting and re-creating issue. Three steps we can do without.

How does it cost you?
Who do you refer to when using the term "us" ?
Note, I am speaking from the perspective of the Solid Project as a Community effort and its natural evolution over time.
You shouldn't be maintaining two specs. And I am not asking you to do that. I am suggesting you maintain one spec (i.e., the current version) while leaving the draft variant to its own destiny, so to speak.

I'm all with you on transparency but I don't understand - and really I want to! - how leaving the old repo as is maintains transparency or how archiving it possibly goes against transparency.

If you archive the original draft spec you are sort of closing it down. What happens to those who may interpret such actions differently, based on their world views?
When I refer to "transparency" it is always about removing any cause for misinterpretation in a composite world of individuals. This is why the consensus protocol and what it means to the community at large needs to be crystal clear.
I hope I've clarified my position a little better? If not, you can ask on :)

I suggest yes for each issue in solid/spec either create a carried over on in solid/specification, or declare that it has been overtaken by events, either made irrelevant, or met, by the move itself, etc.
Yes and make it very clear  the action has moved to solid/specification in the README and readers should follow it there.

It appears to be that the solid/solid-spec repository still attracts enagement and causing confusion. In order to direct the energy to solid/specification and other relevant Solid repositories, I suggest to archive solid/solid-spec. Archiving the repository will continue to keep all information public while prohibits updates eg files, issues.

Okay, but can't you see that prohibition of updates (e.g., files and issues) is the issue of dissent right now.
How do these issues get resolved in a transparent manner, bearing in mind the composite nature of the community?
Actions should never be taken that some might perceive as a select minority dictating to the majority. This is basically what I mean by transparency.  None of us have control over the minds of other individuals and their world-views, which I why I personally suggest erring on the side of caution by being as flexible as possible i.e, don't kill the baby as per "Wisdom of Solomon" analogy.
The old repo isn't a perceived inconvenience to some. For instance, It is actually seen as a point of important historic reconciliation. Thus, why MUST the repo be archived bearing in mind these conflicting sentiments?
This issues are hard, but we shouldn't let them inflict damage on the broader project.
Let's try our best to see things both ways, for the greater good etc..

Like @RubenVerborgh, I am baffled by this argument.
Archiving the repo does not erase the historic reconciliation. All the issues are still there.
It is absolutely unacceptable to continue causing confusion for new developers, by presenting two similarly-named spec repos.

Archiving the repo does not erase the historic reconciliation. All the issues are still there.
It is absolutely unacceptable to continue causing confusion for new developers, by presenting two similarly-named spec repos.

Put differently, repo archiving "prohibits updates eg files, issue" . That's the issue that has triggered dissent.
If folks can no longer post issues or generally discuss items associated with their preferred repo-of-reference it could be perceived as "silencing their voice" and blurring historic reconciliation.
I hope the fundamental issue here is clear i.e., "prohibiting updates eg files, issues" regarding the draft repo is the focal point of dissent.
If that doesn't work for a segment of the community it shouldn't be imposed by what may be perceived as a select few.
This is why I continue to refer to the "wisdom of Solomon" anecdote i.e., keep the baby alive, above all else. We don't need a fragmented community since everyone loses when that happens.

I hope the fundamental issue here is clear i.e., "prohibiting updates eg files, issues" regarding the draft repo is the focal point of dissent.

Okay, BUT:

On the one hand, there is an agreement that 0.7 should and will not be updated.
On the other hand, we still invite issues and updates.

In other words, whatever input people give in the 0.7 repo will likely be lost anyway. Take this from someone who had to engage in very lengthy discussions, just to get obvious bugs in 0.7 fixed. It's a disproportionate effort, compared to solid/specification, where things seem to be more open.
So whether archived or not, solid/spec is in a state where it does not and cannot move anyway. The only thing that's being proposed in this issue, is to make that more obvious, precisely because we do want to listen to people. We're just giving them one clear place to listen to them, and the proposal here is also to indicate that.
That said, I'm afraid we've lost more time already discussing whether or not it should be archived, rather than just moving/copying an issue over every time someone posts something.
To which I perhaps want to re-iterate my earlier proposal https://github.com/solid/process/issues/198#issuecomment-624064887. although at this point, I don't feel strongly about anything anymore. I just want the spec to move with the least amount of distractions. If keeping solid/spec around means that there are fewer distractions, then so be it.
In any case, we want to make a big note in the solid/spec README about where to find what. Let's use that PR as an example to see how hard it is to get things done in that repo.

In any case, we want to make a big note in the solid/spec README about where to find what.

:+1:

Let's use that PR as an example to see how hard it is to get things done in that repo.

:+1:

I consider this discourse in response to a proposal that is publicly accessible to be transparent communication.
solid/solid-spec and solid/specifications are both under the same organisation and community that follows the Solid process to its best abilities and intentions. We are all part of the Solid community.
I originally wanted to stop new issues from opening and preferred to keep existing issues open and visible. GitHub only allows turning issues off from view. Archiving seemed liked the next best option. It also reduces energy and information fragmentation.

I originally wanted to stop new issues from opening and preferred to keep existing issues open and visible.

Yes, that was your intention. The problem is that in a community project of this nature those intentions could lead to unintended effects (in the worldview of others), hence the need to play as safely as possible.
As I've said, these issues are complex due to the fundamental nature of both individuality and community. Moving a diverse community in concert is hard++

IMO if repo is not meant to accept Pull Requests it better stays archived. When it comes to issues, moderators can transfer what they find to new repository. We could also have 'solid-spec migration requests' ongoing issue in new repository, and we would comment in all the issues in old repository that one can request transfer of issue in that dedicated issue.

The thing about community projects is that transparency has to be paramount, at all times.

I think I should have qualified my point a little better i.e., I should have said "transparency and flexibility" have to be paramount.
@csarven has been transparent i.e., his actions and intentions are in public view. Thus, in his case, I am requesting additional flexibility to complement demonstrated transparency re my "Wisdom of Solomon" anecdote :)

I strongly believe that explicitly archiving the solid-spec is the clearest implementation of what @timbl said:

Yes and make it very clear the action has moved to solid/specification in the README and readers should follow it there.

The 0.7 spec is frozen, per the request of the very part of the community that is currently fighting against archiving. No more alterations to it are possible. Continuing to confuse developers by having two seemingly-conflicted spec repositories, is irresponsible.
Why are we still arguing about this?

Why are we still arguing about this?

Because this is a community that is composite in nature i.e., a collection of individuals. People regularly perceive the same thing differently, so a little bit of transparency and flexibility goes a long way to preserving harmony.
We are all in this together, so protecting the community by seeking harmony should always be paramount, IMHO.
This is much easier said than done, as I've alluded to repeatedly.


Remove the metion of the old fetch command

Closes: 283

Thanks for the heads up on the "closes" syntax. :-)

Added an issue to track the other issue here:
https://github.com/exercism/kotlin/issues/286


Support hostpath volumes

This adds support for hostpath volumes, for SUSE/scf#1345 to mount the external /sys/fs/cgroup in diego-cell.  It also accepts (but does not implement) emptyDir volumes, sort of on the way to #340 (if we decide to implement this way, rather than empty-dir: true as suggested).
This is more for discussion than merging; for one thing, I'm not sure hostpath would always be available.

Rebased, and added the helm value thing (defaulting true). Also tests, because I am not confident enough in myself to not have it.  I was unable to find anything to make that detection automatic.


Most kanji marked as "learned"

I'm on level 3.
It marks most kanji as "learned" when they are not.
Here is my read-only api token to test with: c2e9b969-7d24-4064-90da-600363adc17c
Just go to easy nhk news to test.

I see, I will look into it right away.
Can you tell me if you have reseted your account ever? Just to understand if it might be causing the issue.

Ok, I fixed it! It won't be updated just yet though, only on the next release. It will be soon, probably in a week or so.

Np. Thanks for the quick response!
On Sun, Oct 3, 2021, 1:38 PM Diogo Correia @.***> wrote:

Ok, I fixed it! It won't be updated just yet though, only on the next
release. It will be soon, probably in a week or so.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/digas99/wanikani-kanji-highlighter/issues/2#issuecomment-932994364,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACGJ6ZR7UEMGPMQFQ5WOTLUFCIH5ANCNFSM5FG3VWNA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.


Add input as argument: python explain.py input.txt

Add input file as an argument to make it easy to run on other files:
python explain.py MobyDick.txt will produce MobyDick_Explained.txt.
And add those two files.

Thank you @hugovk for adding that feature!


Inaccessible host error from underlying AWS SDK library

🐞 Describe the bug
When using the action as described in the docs with the property Digital Ocean tokens, we are getting the following error:
  with:
    access_key: ***
    secret_key: ***
    space_name: ***
    space_region: ***
    source: storybook_build
Error: UnknownEndpoint: Inaccessible host: `***.***.digitaloceanspaces.com' at port `undefined'. This service may not be available in the `us-east-1' region.
Error: Inaccessible host: `***.***.digitaloceanspaces.com' at port `undefined'. This service may not be available in the `us-east-1' region.

Looking at the source code for this action, it's seems like this is an error bubbling up from the underlying aws-sdk library.
Can you provide any guidance for this? We've confirmed our secrets are correctly being loaded into our Github Action.
📚 To Reproduce
Set up a Digital Ocean Space per the Readme for this action. Go to deploy new data to the DO Space, and you should get this error.
💡 Expected behavior
Our updated build should be deployed to our Digital Ocean Space.
🖼 Screenshots

Here is our Github Workflow:
name: Deploy Storybook
on:
  push:
    branches:
      - main
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - uses: actions/setup-node@v2
        with:
          node-version: "16"
      - run: yarn
      - run:  NODE_ENV=production yarn exec nx run storybook:build-storybook --outputPath=storybook_build
      - uses: BetaHuhn/do-spaces-action@v2
        with:
          access_key: ${{ secrets.DO_SPACES_ACCESS_KEY}}
          secret_key: ${{ secrets.DO_SPACES_SECRET_KEY }}
          space_name: ${{ secrets.DO_SPACES_SPACE_NAME }}
          space_region: ${{ secrets.DO_SPACES_REGION_NAME }}
          source: storybook_build

⚙ Environment

OS: tried on ubuntu-20.04 and ubuntu-latest (22.04)
Action version: tried with @v2 and v2.0.43

📋 Additional context
This doesn't seem to be a problem on another repository that is using the same action. We tried different versions of ubuntu and the action to see if it was a caching issue, but it did not seem to resolve the issue. Any help that you can provide with this issue would be really appreciated!
Note, we have also confirmed that our Github secrets are being loaded into the action's environment and are correct.

⚠ This issue has not seen any activity in the past 2 months so I'm marking it as stale. I'll close it if it doesn't see any activity in the coming week.

I am currently experiencing thesame. Were you able to find a fix @niklilland  ?


Change position of close button to upper left or upper right menu?

Hi,
I would prefer, which is more or less common for all kind of navigations today to place the Close button on the upper left or upper right corner of the menu, and also switch it to a Cross (X), and have "Close" as a label.
What do you think ?
regards

You can always pass a custom HTML element as labelClose and style it per your needs.
Example:
var closeButton = document.createElement('div');
closeButton.className = 'nav-close-button';
closeButton.title = 'Close';

var Nav = new hcOffcanvasNav('#main-nav', {
    labelClose: closeButton
});

Or with jQuery:
$('#main-nav').hcOffcanvasNav({
    labelClose: $('<div class="nav-close-button" title="Close">')[0]
});

Thx I know that I can customize it even in CSS, but would be better to have it as standard in this library I think.
Regards

I'll think about this. Maybe if you could provide some screenshots examples for me to consider.


Mixed usage of n_samples_overall in get_normalized_expression

In TOTALVI.get_normalized_expression, MULTIVI.get_normalized_expression, and ContrastiveVI.get_normalized_expression, n_samples_overall just downsamples indices, while in RNASeqMixin.get_normalized_expression, it's used for the number of MC inference samples. This might be confusing for users who expect the argument to mean and do the same thing across methods.

@PierreBoyeau  Currently this means that these other function don't use importance weighting? I assume it would make sense to add it to these models as well?

I think we can improve the naming as well (but that's styling and I would stick to n_samples_overall for number of mc_samples and n_samples to number of samples for a single input cell in a forward pass.


Converting image to pencil sketch using python

Aim
Converting image to pencil sketch
Details

Capturing live images from webcam.
Or Uploading image to the webpage.
Implementation of script using flask

I am GSSOC'21 participant and would like to contribute this script.

@vaishnavirshah make it a script application where you take the path of the image as input, then convert the image into sketch form and then save that output image

@kaustubhgupta Hi, I did not quite understand your suggestion, I have implemented both taking images from webcam and converting to sketch as well as taking image path and converting to sketch on the webpage.
Should I not use flask? And implement both these things as a menu driven script?

@vaishnavirshah Yes, port those features to the menu-driven script as we are not looking for end-to-end applications


fix(autoware_multi_object_tracker): output from screen to both

Description
Set output from screen to both to improve the debugging efficiency of Autoware in general.
With this change, the output from node will not only be displayed in screen but also to the log file.
See https://github.com/ros2/launch/blob/986ea832cec49c18bee4e29ee4df37fe3844fa90/launch/launch/logging/init.py#L397-L487
Related links
TIER IV INTERNAL THREAD
How was this PR tested?
Not tested
Notes for reviewers
None
Interface changes
None.
Effects on system behavior
Although there will be a slight increase in computational load due to the additional log, I would say that the risk is minimal since the node's output does not occupy terminal at this moment. But just in case, I would like to make this change for several nodes chosen from perception component which tends to have some issues pretty often.

Please wait for the merge as it is under discussion here.
https://github.com/autowarefoundation/autoware.universe/pull/8411

Please wait for the merge as it is under discussion here.
https://github.com/autowarefoundation/autoware.universe/pull/8411


Update infer_cli.py

Save generated chunks in the output_dir folder; it helps with the multi-generation process.

was saving to different path then the final one, better same path; and better named with same prefix as final one with postfix {chunk_num}_{chunk_gen_text} e.g.
not accurate annotation e.g. mkdir but annotated as save file
better introduce with a flag to control whether save chunks

🫡


unable to resolve module expo from project/main.js

I create a project in EXPO. I run with iOS simulator. App is working. And when I run below command, i got this error:
npm install react-native-simple-modal --save
unable to resolve module expo from project/main.js
module does not exist in the module map or in these directories
I checked that almost all my node_modules in project file are gone after run npm install command. I think that`s why I got that error. After that I tried with other modules to install with npm, and result is same.

this seems to be a problem with expo and/or npm and unrelated to this library.


peer1.conf not created

Hey,
I love the concept of this but for some reason the server is not creating peer1.conf but only wg0.conf. Any ideas how to resolve this?
Best,
Daniel

If you create a new folder with a docker-compose.yml file in it:
services:
  wireguard-tunnel-server:
    image: ghcr.io/digitallyrefined/docker-wireguard-tunnel:v3
    container_name: wireguard-tunnel-server
    environment:
      # Update to your domain
      - DOMAIN=wireguard-tunnel.example.com
      # Number of peers to auto generate config for
      - PEERS=1
      # Services to expose format (comma-separated)
      # SERVICES=peer-id:peer-container-name:peer-container-port:expose-port-as
      - SERVICES=peer1:nginx:80:8080,peer1:nginx-demo:80:8081
    cap_add:
      - NET_ADMIN
    volumes:
      - ./config:/etc/wireguard
    restart: unless-stopped
    ports:
      - '51820:51820/udp'
      - 8080:8080
      - 8081:8081

Then run docker compose up -d this should create a new config folder with a newly generated peer1.conf file in it.
If not, please post any error messages from either the docker compose up -d or docker compose logs -f commands.

I figured it out. Say, any chance to somehow modify the configuration in order for the container to be in the VPN completely? Currently the container is exposed via the wireguard server, however, the docker container client side is still trying to connect elsewhere via it's real IP.

👍
If you don't want to use or make the servers IP address available to the public internet, a private mesh network could be used instead. For example Tailscale, Twingate, ZeroTier and even NordVPN offer a flat mesh internal network using private IP address on the 10.x.x.x/100.x.x.x subnets. Join 2 machines to the same mesh network and use the servers private hostname or IP address.


I got minimum stability issues

Hi,
When i try to download the files  from composer i got the minimum stability issues. Please refer this attached screenshots

Hi, sorry my delay.
Please, use:
composer require lab123/aws-sns:dev-master


Invalid order when sorting by fields that are not the primary.

I'm doing a sort by popular/trending/etc so ids are not ordered, results are incorrect.

Need a custom order field

Sorting must be done on a custom incremented 'rank' column.
IE: we want to sort by popular, trending and latest. Popular was based on 'votes', trending based on 'votes per last month' and latest from 'created_at'.
We can't sort by these methods because the 'last item id' is random.
In order to sort this way we must create a new column (for each type of sort) that is incremented. This will ensure our results are paginated based on the sort.
If anyone else has a solution we're eager to hear it. So far this will work for us, and pagination will remain fast.


Develop Server Status Manager

Description
The Server Status Manager (SSM) coordinates all the state changes related with the sever status,
The SSM will be implemented with 2 Design Patterns:

State Pattern (Coordinates State Changes)
Observer Pattern (Receives Events to trigger changes)

Tasks

[x] Implement the Observer Pattern
[ ] Create & Implement the State Pattern
[ ] Create Tests for each state change

Progress in this feature will halt a bit to reconsider the viability of implementing the State Pattern after gathering the all the requirements for status changes.


Schedule model testing every N training epochs

🚀 Feature
A check_test_every_n_epoch trainer option to schedule model testing every n epochs, just like check_val_every_n_epoch for validation.
Motivation
Sometimes validation and test tasks are very different. For instance, in unsupervised anomaly detection or segmentation, the training and validation set cannot contain anomalous samples, and therefore the set of metrics that can be computed on such set is limited.
Test metrics cannot also be computed on a second validation set that contains a portion of test data, because all parameters and hyperparameter optimization should be performed on clean (anomaly-free) samples.
The only way the user has to check test metrics is to run a test epoch, but currently pytorch-lightning allows to run a test epoch only after the trainer.fit() call has ended, by explicitly calling trainer.test().
Pitch
When initializing a Trainer with check_test_every_n_epoch: x every x epochs the fitting automatically stops, a test epoch is executed, logging and computing test metrics, and then the fitting resumes.
Alternatives
I tried to:

Calling self.trainer.test() from on_training_epoch_start module callback when the value of trainer.current_epochs becomes multiple of a desired value. Apparently, doing this works fine, but after calling the test method, the number of epochs continues to increase from the last value, but the trainer global_step is reset to the value it had when test was last called, creating the beautiful effect shown in figure and making logs unreadable.
Raising KeyboardInterrupt from on_training_epoch_start module callback to stop the fitting, testing, manually saving a checkpoint, and resuming training from the saved checkpoint in a while loop. Cumbersome, poor disk usage, cannot gracefully stop training with Ctrl+C anymore.
Computing test metrics in validation_epoch_end by manually looping on test set using pytorch only. Cumbersome, code duplication, makes test hooks useless, poor code readability.

Additional context
Tensorboard logging current epoch each step when trying solution 1:

How is this work going along? I would strong like this feature to be incorporated into pl

:+1:  I am also interested in this. In my use case, I use DDP, and performing a validation after every epoch on a per-rank basis is fine. However, after n epochs, I would like to test the current model only on one worker because my test involves the initialization of a kNN index which takes a lot of memory.

Me too, this would be really useful!

Thanks for raising this question!
Some questions that would need to be addressed are:

Testing could be run asynchronously as it doesn't block training. One could imagine launching a job taking the checkpoint path as an argument. However, this launching can be highly customized to the user or cluster needs, and specifying how to do this in the existing training and validation loop may not lead to a solution that works for everyone. What does scheduling for your use case entail?

Additionally, one might want to specify different resource requirements for testing: e.g. w/o gradients, you could increase your batch size. Having testing run periodically within training wouldn't make the best use of such resources. Do you see this as a need/requirement?

I also would want to know how to achieve this. Any help is appreciated!

I also want this feature as well
I want to know how the progress on the test set as well during training

Why this issue was tagged won't fixed. Can we raise some awareness on this ?

I'm also interested in this feature.

me too, had to do ugly hack to overcome this

me too, had to do ugly hack to overcome this
In my case my validation set is simulation images, and the test set is real images, so in this case its very important for me to run on test during training

Would you mind you sharing your solution? Thank you very much.

What I am really looking for is to schedule model prediction every n epochs, but either way works for me.
I'm currently working on a synthetic volume-to-volume model which is trained to predict 3D patches of volumes, hence my train- and validation-loaders return patches. At inference, multiple overlapping patches of a single volume are passed through the model and stitched together forming a full predicted volume. Due to the overlapping nature of patches, there is a massive memory advantage in having the inference data_loader return full volumes of images and not patches. I wish to optimize the similarity metrics of the stitched images, so I also need a way of scheduling prediction/testing during training.
I can share my hack, as I also have very different train and test datasetes. My lightning datamodule creates two dataloaders; one which contains my ordinary validation set and one which is actually my inference set of full volumes. The argument dataloader_idx is passed by lightning and tells you which dataloader generated the batch.
By checking the self.current_epoch, I can achieve the desired result.
Downside: the dataloader for inference loads the whole dataset at each epoch even though the data is only used every n epochs.
`
def validation_step(self, batch, batch_idx,dataloader_idx=None):
# Ordinary validation
if dataloader_idx == None or dataloader_idx == 0:
condition, real, mask  = batch
        for idx in range(len(self.generators)+len(self.discriminators)):

            loss = None

            ## Discriminator
            if idx % 2 == 1:
                dis_idx = self._opt_to_dis_idx(idx)
                loss = self._disc_step(real, condition,mask,dis_idx)

                self.log(f"Loss/val.Discriminator_{dis_idx}",loss,on_step=False,on_epoch=True,add_dataloader_idx=False)

            ## Generator
            elif idx %2 == 0:
                gen_idx = self._opt_to_gen_idx(idx)
                loss = self._gen_step(real, condition,mask,gen_idx)

                self.log(f"Loss/val.Generator_{gen_idx}",loss,on_step=False,on_epoch=True,add_dataloader_idx=False)

    elif dataloader_idx == 1 and self.current_epoch % self.hparams.train.test_every_n_epoch == 0:
        mrs, pets, masks, ids = batch
        for mr,pet,mask,id in zip(mrs,pets,masks,ids):
            out = self._predict_and_stitch(mr,idx=-1) # This is my testing function
            mse = torch.sum(torch.square(out[mask.squeeze()>0]-pet.squeeze()[mask.squeeze()>0]))
            self.log(f"Validation/MSE/{id}",mse,on_step=False,on_epoch=True)

`

What I am really looking for is to schedule model prediction every n epochs, but either way works for me.
I'm currently working on a synthetic volume-to-volume model which is trained to predict 3D patches of volumes, hence my train- and validation-loaders return patches. At inference, multiple overlapping patches of a single volume are passed through the model and stitched together forming a full predicted volume. Due to the overlapping nature of patches, there is a massive memory advantage in having the inference data_loader return full volumes of images and not patches. I wish to optimize the similarity metrics of the stitched images, so I also need a way of scheduling prediction/testing during training.
I can share my hack, as I also have very different train and test datasetes. My lightning datamodule creates two dataloaders; one which contains my ordinary validation set and one which is actually my inference set of full volumes. The argument dataloader_idx is passed by lightning and tells you which dataloader generated the batch.
By checking the self.current_epoch, I can achieve the desired result.
Downside: the dataloader for inference loads the whole dataset at each epoch even though the data is only used every n epochs.
def validation_step(self, batch, batch_idx,dataloader_idx=None):
        # Ordinary validation
        if dataloader_idx == None or dataloader_idx == 0:
            condition, real, mask  = batch
            for idx in range(len(self.generators)+len(self.discriminators)):

                loss = None

                ## Discriminator
                if idx % 2 == 1:
                    dis_idx = self._opt_to_dis_idx(idx)
                    loss = self._disc_step(real, condition,mask,dis_idx)
                    self.log(f"Loss/val.Discriminator_{dis_idx}",loss,on_step=False,on_epoch=True,add_dataloader_idx=False)

                ## Generator
                elif idx %2 == 0:
                    gen_idx = self._opt_to_gen_idx(idx)
                    loss = self._gen_step(real, condition,mask,gen_idx)

                    self.log(f"Loss/val.Generator_{gen_idx}",loss,on_step=False,on_epoch=True,add_dataloader_idx=False)

        elif dataloader_idx == 1 and self.current_epoch % self.hparams.train.test_every_n_epoch == 0:
            mrs, pets, masks, ids = batch
            for mr,pet,mask,id in zip(mrs,pets,masks,ids):
                out = self._predict_and_stitch(mr,idx=-1) # This is my testing function
                mse = torch.sum(torch.square(out[mask.squeeze()>0]-pet.squeeze()[mask.squeeze()>0]))
                self.log(f"Validation/MSE/{id}",mse,on_step=False,on_epoch=True)

I did not know of this feature of pytorch-lightning, it is still a hack but far better than any other "solutions" I have tried.
Maybe with a bit of patience one could actually extend base LightningModule, LightningDataModule and Trainer to automatically perform this in a cleaner way.

Would love to have this feature too!

I modified @ChristianHinge 's hack a bit to transparently train and test multiple times per epoch for any vanilla LightningModel, as I found that having that validation step function be used for each different model class results in a lot of boilerplate being copied and pasted which would obscure the actual model logic. This function requires a LightningDataModule that returns either just the validation dataloader or both the validation and the test dataloader, depending on whether the "test_every_n" flag is set or not.
In my LightningDataModule:
def val_dataloader():
    val_dataloaders = []
    if self.validation_set:
        val_dataloaders.append(DataLoader(...))
    if self.test_every_n > 0:
        val_dataloaders.append(self.test_dataloader())
    return val_dataloaders

And the wrapper function:
def patched_validation_step(self, batch, batch_idx, dataloader_idx):
    if dataloader_idx == 0:
        return self.do_validation(batch, batch_idx)
    elif dataloader_idx == 1 and self.current_epoch % self.test_every_n == 0:
        return self.test_step(batch, batch_idx)

def patched_validation_epoch_end(self, results):
    val_loop, test_loop = results
    on_vend = self.validation_end(val_loop)
    if len(test_loop):
        self.test_epoch_end(test_loop)
    return on_vend

def wrap_model(model, test_every_n=None):
    """Wraps a model so that it can be tested every N epochs"""

    if test_every:
        model.test_every_n = test_every_n
        model.do_validation = model.validation_step
        model.validation_end = model.validation_epoch_end

        model.validation_step = patched_validation_step.__get__(model, model.__class__)
        model.validation_epoch_end = patched_validation_epoch_end.__get__(model, model.__class__)
    return model

def unwrap_model(model):
    """Undoes the wrapping caused by `wrap_model`"""

    try:
        model.validation_step = model.do_validation
        model.validation_epoch_end = model.validation_end
        del model.test_every_n
    except AttributeError:
        # nothing to unwrap / already unwrapped
        pass
    return model

Finally, before calling trainer.fit(), set the flags and call the functions, and unwrap after training:
# tests every 5 epochs
datamodule.test_every_n = 5
wrap_model(model, datamodule.test_every_n)
trainer.fit(model, datamodule=datamodule)
# unwrap after training
unwrap_model(model)

The same downsides still apply, that is, the test dataloader iterates through the entire test dataset every validation epoch, even if it is not used at all. Still, I find it makes the code a bit cleaner.

One solution I am using is:

Monkey patch the method on_advance_end of the trainer before calling trainer.fit:

trainer.check_test_every_n_epoch = check_test_every_n_epoch
trainer.fit_loop.epoch_loop.on_advance_end = patched_on_advance_end.__get__(trainer.fit_loop.epoch_loop)

With a patched version of on_advance_end:

def patched_on_advance_end(self):
    """Runs validation and Checkpointing if necessary.

    Raises:
        StopIteration: if :attr:`done` evaluates to ``True`` to finish this epoch
    """
    # -----------------------------------------
    # VALIDATE IF NEEDED + CHECKPOINT CALLBACK
    # -----------------------------------------
    should_check_val = self._should_check_val_fx(self.iteration_count, self.is_last_batch)
    if should_check_val:
        self.trainer.validating = True
        self._run_validation()
        self.trainer.training = True

    should_check_test = _should_check_test_fx(self, self.iteration_count, self.is_last_batch)
    if should_check_test:
        self.trainer.testing = True
        self._run_validation()
        self.trainer.training = True

    # -----------------------------------------
    # SAVE LOGGERS (ie: Tensorboard, etc...)
    # -----------------------------------------
    self._save_loggers_on_train_batch_end()

    # update plateau LR scheduler after metrics are logged
    self.update_lr_schedulers("step", update_plateau_schedulers=True)

    self.total_batch_idx += 1

    # progress global step according to grads progress
    self._increment_accumulated_grad_global_step()

    if self.done:
        raise StopIteration

with:

def _should_check_test_fx(self, batch_idx: int, is_last_batch: bool) -> bool:
    """Decide if we should run validation."""
    if self.trainer.check_test_every_n_epoch <=0 :
        return False

    is_test_check_epoch = (self.trainer.current_epoch + 1) % self.trainer.check_test_every_n_epoch == 0
    if not is_test_check_epoch:
        return False

    ...

Of course this is a dirty solution as well since any chance of Pytorch Lighning can break this patch or produce issues.

Hi all! Just found this issue now.
#9254 contains a discussion on why this hasn't been implemented.
Please take a look. You can directly answer there to provide any arguments you think we missed.

Modifying the val_dataloaders can be very hard to do for external libraries.

You wouldn't need to modify the dataloaders as you can instead pass a list of them.

This way we get to know if our test metrics behave in the same way the validation metrics do.
If they don't correlate / val and test curves don't have similar shape, early stopping can be useless!

This is considered "cheating" as your training is being informed from its performance on the test set. The test set is supposed to be held out and only used after the best model (generally on validation) is selected.
There are many techniques to avoid inconsistencies between the different data partitions, from better shuffling to data drifting analysis.

Lightning can print a warning if the user is testing before training, but why remove this option completely?

If the user explicitly chose to do this, we wouldn't want to print a warning because it's his/her choice. However, we are not comfortable adding a flag to the Trainer that might signal our users that this practice is generally okay.

Yeah, I thought about it again and I agree with you. Probably I only needed this because pytorch-forecasting doesn't support multiple dataloaders (their pl_modules break), otherwise there is no need for testing every epoch. It really should be just another val dataloader.

any cleaner solutions for PL-1.6.1? looking forward to it.

Modifying the val_dataloaders can be very hard to do for external libraries.

You wouldn't need to modify the dataloaders as you can instead pass a list of them.

This way we get to know if our test metrics behave in the same way the validation metrics do.
If they don't correlate / val and test curves don't have similar shape, early stopping can be useless!

This is considered "cheating" as your training is being informed from its performance on the test set. The test set is supposed to be held out and only used after the best model (generally on validation) is selected.
There are many techniques to avoid inconsistencies between the different data partitions, from better shuffling to data drifting analysis.

Lightning can print a warning if the user is testing before training, but why remove this option completely?

If the user explicitly chose to do this, we wouldn't want to print a warning because it's his/her choice. However, we are not comfortable adding a flag to the Trainer that might signal our users that this practice is generally okay.

  People are still gonna do it when they need it, you're just making it messier.

seconding this thread to see if there's any updates

this is especially useful for robotics algorithms where we want to test rollout every n epochs. this will definitely be a useful feature


save to codepen (or other sharing platform)

Save HTML and CSS to a code sharing platform via API
find/create/integrate

[x] method to read out the needed HTML/CSS (possibly with cleaning out non-needed elements)

[Post to prefilled Codepen(http://blog.codepen.io/documentation/api/prefill/) Documentation

[x] method to build an API compatible JSON object
[x] method to send it to the API
[ ] method to display the URL the mockup was posted to
[ ] UI design to activate upload and show URL

relates to/solves #4

it works. I hope. If you look at the code, there is some escaping magic (mess?) going on, but I got it working. If you get a 500 Error from Codepen it is most likely not their fault, but mine/ours.

It seems that the export to CodePen is broken.

Indeed it looks broken on https://jdittrich.github.io/quickMockup
destination url http://codepen.io/pen/define doesnt exist


Login redirection doesn't return to IPT

When successfully authenticating on the IPT login via login.do, we are redirected back to the root URL of the hosted domain. Our deployment of IPT is served from an /ipt/ path. Navigating back to the IPT URL, we have a logged in session.
From https://github.com/gbif/ipt/blob/b6a4c2c26486763d03507da8589bba2c33f2a563/src/main/java/org/gbif/ipt/action/LoginAction.java#L114, it looks like the Referer is evaluated to determine a redirection URL, but excludes Referers ending with login.do and defaults to '/'.
I presume if we were serving from a subdomain out of the root context, this behavior wouldn't be noticed. Is this possibly a bug that presents to users that serve IPT from a path? I'm wondering if setting the redirect URL to the IPT base URL is a viable option?
To work around this I have manually set a Referer header to the IPT base URL on the reverse proxy.

Thank you for contacting us @joshbeard
We recently changed that, and now it redirects to the IPT base URL
https://github.com/gbif/ipt/blob/b6a4c2c26486763d03507da8589bba2c33f2a563/src/main/java/org/gbif/ipt/action/LoginAction.java#L109


Fix typo

This change was extracted from #1013, in order to isolate some of the changes.

@antkmsft please use more detailed PR titles in the future that describe the specifics/scope of the change, especially since they end up as commit messages.


This pop up

Describe your bug here.
it crashes when i load the storymode
Command Prompt/Terminal logs (if existing)
CHaxeToolkit\haxe\std/haxe/format/IsonParser.hx(line318)
C'\HaxeToolkit\haxe\std/haxe/format/IsonParser.hx(line99)
C\HaxeToolkit\haxe\std/haxe/format/JsonParser.hx(line57)
WeekData.hx (line 222)
WeekData.hx (line 190)
WeekData.hx (line 178)
StoryMenuState.hx (line 58)
flixel/FIxGame.hx (line 637)
flixel/FIxGame.hx (line 724)
flix el/FIxGame.hx (line 677)
flixel/FIxGame.hx (line 545)
openfl/events/EventDispatcher.hx (line 402)
openfl/display/DisplayObject.hx (line 1399)
Uncaught Error: Invalid char 105 at position 18
Please report this error to the GitHub page:

Are you modding a build from source or with Lua?
Lua
What is your build target?
Windows
Did you edit anything in this build?


[Fleet] Migrate ddtrace references to telem package

What does this PR do?
ddtrace accounts for 70% of downloader size, and a big chunk of installer too. Preparing the phasing out of the package by moving all references to telemetry package
Follow-up from Arthur's work
Motivation
Describe how you validated your changes

Possible Drawbacks / Trade-offs
Additional Notes

/merge

/merge

/merge


svgbob: 0.5.4 -> 0.6.2

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/svgbob/versions.
meta.description for svgbob is: "Convert your ascii diagram scribbles into happy little SVG"
meta.homepage for svgbob is: "https://github.com/ivanceras/svgbob"
meta.changelog for svgbob is: ""
Updates performed

Rust version update

To inspect upstream changes
Impact

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

/nix/store/av6ggd2vzz5pa9b3w665wjkdh04byyr5-svgbob-0.6.2/bin/svgbob passed the binary check.

1 of 1 passed binary check by having a zero exit code.

1 of 1 passed binary check by having the new version present in output.

found 0.6.2 with grep in /nix/store/av6ggd2vzz5pa9b3w665wjkdh04byyr5-svgbob-0.6.2

found 0.6.2 in filename of file in /nix/store/av6ggd2vzz5pa9b3w665wjkdh04byyr5-svgbob-0.6.2

directory tree listing: https://gist.github.com/b91bdcf9bfb7cbb41b2466caa77222a1

du listing: https://gist.github.com/6fe4aece824429c8762c1501542e27d2

Rebuild report (if merged into master) (click to expand)

4 total rebuild path(s)

1 package rebuild(s)

1 x86_64-linux rebuild(s)
1 i686-linux rebuild(s)
1 x86_64-darwin rebuild(s)
1 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
svgbob

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/av6ggd2vzz5pa9b3w665wjkdh04byyr5-svgbob-0.6.2 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A svgbob https://github.com/r-ryantm/nixpkgs/archive/88e3f20774ec11ba7e7b0dc72ab7cef77198b955.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/av6ggd2vzz5pa9b3w665wjkdh04byyr5-svgbob-0.6.2
ls -la /nix/store/av6ggd2vzz5pa9b3w665wjkdh04byyr5-svgbob-0.6.2/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

1 package built:

svgbob

Maintainer pings
cc @marsam for testing.

Result of nixpkgs-review pr 138914 run on x86_64-linux 1

1 package built:

svgbob


Multiple CMAKE_INSTALL_* entries ignored

I'm trying to compile arangodb 3.0 in Archlinux with custom cmake flags, unfortunately some flags are being ignored. To be precise, this is the process I'm following to compile arangodb:
mkdir build; cd build
cmake .. \
  -DCMAKE_INSTALL_PREFIX=/ \
  -DCMAKE_INSTALL_BINDIR=/usr/bin \
  -DCMAKE_INSTALL_SBINDIR=/usr/bin \
  -DCMAKE_INSTALL_DATAROOTDIR=/usr/share \
  -DCMAKE_INSTALL_SYSCONFDIR=/etc \
  -DCMAKE_INSTALL_LOCALSTATEDIR=/var \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_BUILD_TYPE=Release
make
make DESTIDIR="/a/test/directory" install

But once everything is compiled and installed, I find the following structure in the destination directory:
/
-   etc/
-   init.d/
-   sbin/
-   share/
-   usr/
    -   bin/
-   var/

Where I was expecting this structure:
/
-   etc/
-   init.d/
-   usr/
    -   bin/
    -   share/
-   var/

I thought the problem was easy to solve, a simple refactoring of arangodb/cmake/ArangoDBMacros.cmake should have done it. For example, the line
set(TRI_PKGDATADIR "${CMAKE_INSTALL_PREFIX}/share/arangodb3")

could be replaced with
set(TRI_PKGDATADIR "${CMAKE_INSTALL_DATAROOTDIR}/arangodb3")

Or the line
  set(TRI_SBINDIR "${CMAKE_INSTALL_PREFIX}/sbin")

with
  set(TRI_SBINDIR "${CMAKE_INSTALL_SBINDIR}")

But it seems there are many many string substitutions to do so that the installation respects all CMake entries.
I hope I've been able to articulate adequately, it's my first time trying to compile using CMake so I'm not sure if I've missed something.

Hi, this is what the rpm spec file does:
cmake
-DVERBOSE=On
-DUSE_OPTIMIZE_FOR_ARCHITECTURE=Off
-DCMAKE_BUILD_TYPE=RelWithDebInfo
-DCMAKE_C_FLAGS="${CFLAGS}"
-DCMAKE_CXX_FLAGS="${CXXFLAGS}"
-DCMAKE_C_COMPILER="%{C_COMPILER}"
-DCMAKE_CXX_COMPILER="%{CXX_COMPILER}"
-DFORCE_JEMALLOC_LIB=:libjemalloc.a
-DETCDIR=/etc
-DVARDIR=/var
-DCMAKE_INSTALL_PREFIX:PATH=/usr
-DCMAKE_SKIP_RPATH:BOOL=ON
-DCMAKE_USER_MAKE_RULES_OVERRIDE=
..
maybe that works out in the proper directory hierarchy?

Awesome, it worked! :smiley:
In the end I used:
cmake ..
But there are still some hardcoded paths into arangodb/cmake/ArangoDBMacros.cmake pointing to sbin, they should point to a SBIN variable/entry from CMake instead so if anybody wants to change the sbin installation directory, they could do it without any problem (I suppose the entry to use should be CMAKE_INSTALL_SBINDIR but I'm not sure).

We definitely still have some cmake cleanup on our TODO list, I hope we can streamline this as well, so a LHFS conforming install can be done with one argument.

We were indeed not using the intended cmake variables in all places. this has been fixed in devel (future 3.1 release) and your build instructions should yield the correct result now :)


Feat: Add Footer Section

Description

We need to add a footer section to the landing page.
For Assignee:

Ensure that you submit a PR for this issue within 3 days. Otherwise, you will be unassigned from this issue and it will be passed on to the next contributor who asked to be assigned to it.

Note: This issue is for JWoC 2024 participants. If you are not participating in this event, please refrain from asking to get assigned to this issue.

Hey @NaveenChandraAdhikari, since you suggested this idea, do you want to get assigned to this issue once the first coding period starts?

i am interested Please assign me

Definitely 😊
On Fri, Jan 12, 2024, 9:30 PM Sriparno Roy @.***> wrote:

Hey @NaveenChandraAdhikari https://github.com/NaveenChandraAdhikari,
since you suggested this idea, do you want to get assigned to this issue
once the first coding period starts?
—
Reply to this email directly, view it on GitHub
https://github.com/Sriparno08/Openpedia/issues/115#issuecomment-1889560555,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AKR3A45O4DGIUEBPDMBM4D3YOFM2HAVCNFSM6AAAAABBYDTKTGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBZGU3DANJVGU
.
You are receiving this because you were mentioned.Message ID:
@.***>

Great 👍

hello @Sriparno08 pls assign this to me

Hey @NaveenChandraAdhikari, requests made before 15th Jan, 2024 are not considered for assignment. You have to make one on or after 15th Jan, 2024 with the 'JWOC' keyword in your comment to be considered.
Hey @Ananyamishra08, you haven't included the 'JWOC' keyword in your comment.
The one who ticks all the boxes first will be assigned to this issue.

@Sriparno08 I would like to work on this issue under JWOC. Kindly assign it to me

hey @Sriparno08 i am sorry for the mistake .Can u reassign me this issue under JWOC

Hey @Ananyamishra08, you're assigned 👍
To Others:
The mentee who is assigned to this issue is chosen because he/she was the first one to ask for assignment on or after 15th Jan, 2024 with the 'JWOC' keyword in his/her comment.
Hey @zul132, you're already assigned to another issue. So, you will not be considered for this one.

Hey @NaveenChandraAdhikari, since you suggested this idea, do you want to get assigned to this issue once the first coding period starts?
@Sriparno08 that is something you have said

Correctly said, @NaveenChandraAdhikari.
But as the organizers notified, any comments made before 15th Jan, 2024 should not be considered for assignment. So unfortunately, all the mentors who have kept issues on hold for assignment to a particular mentee had to make them open for everyone.
Sorry for the inconvenience caused as the mentors were notified about it just before the first coding period starts.

@Sriparno08 please assign this issue to me under JWOC'24.


iam-config.yaml IDC configuration doesn't implement changes

Describe the bug
Config in iam-config.yaml doesn't run in the pipeline - it's ignored
To Reproduce
Put IDC configuration into iam-config.yaml:
providers: []
policySets: []
roleSets: []
userSets: []
identityCenter:
  name: ctIdentityCenter
identityCenterPermissionSets:
  - name: ISVPermissionSetViewOnly
    policies:
      awsManaged:
      - arn:aws:iam::aws:policy/ViewOnlyAccess
    sessionDuration: 60
identityCenterAssignments:
  - name: Assignment002
    permissionSetName: ISVPermissionSetViewOnly
    principalId: "82b53444-6051-7037-dda5-663a916ece5c"
    principalType: GROUP
    deploymentTargets:
      accounts:
      - someWorkloadAccount

Expected behavior
Creates permission sets for the group specified in the assignment
Please complete the following information about the solution:

[ 1.3.0] Version: [e.g. v1.1.0]

[ eu-west-1] Region: [e.g. us-east-1]

[ ] Was the solution modified from the version published on this repository?

[ ] If the answer to the previous question was yes, are the changes available on GitHub?

[ ] Have you checked your service quotas for the sevices this solution uses?

[ ] Were there any errors in the CloudWatch Logs?

Screenshots
If applicable, add screenshots to help explain your problem (please DO NOT include sensitive information).
Additional context
Add any other context about the problem here.

pebcac


Usage monthly report endpoint

Is your feature request related to a specific question or information need?
Currently, a monthly report for the number of total users and new users is generated each month.  Specifically answering the following questions:

How many users are currently registered?
How many users registered in the past month?

Describe the solution you'd like
An API endpoint to that provides total number of users and number of new users for prior month's date range.
E.g.
{
  "total_users": 100,
  "new_users": 23
  "start_date": "2022-09-01",
  "end_date": "2022-10-10"
}

Addressed by 9dd55281a92be7713ca5c062ab29045fa8b4fc13 and further by ac0af039bc4c73bdc55b74378ece4a8a21a415a1


tests: restore disabled tests and benches for BiLock

Hi! I was curious about BiLock internals (came there from its usage in split) so the easiest way to play with something is to poke into corresponding tests. Found out that tests and benchmarks for BiLock are disabled so I went ahead with reviving them to be up to date with current landscape.
Thank you your great work on the library!

Miri complains:
error: unsupported operation: integer-to-pointer casts and `ptr::from_exposed_addr` are not supported with `-Zmiri-strict-provenance`
   --> /home/runner/work/futures-rs/futures-rs/futures-util/src/lock/bilock.rs:178:31
    |
178 |                 Box::from_raw(n as *mut Waker).wake();
    |                               ^^^^^^^^^^^^^^^ integer-to-pointer casts and `ptr::from_exposed_addr` are not supported with `-Zmiri-strict-provenance`
    |
    = help: use Strict Provenance APIs (https://doc.rust-lang.org/nightly/std/ptr/index.html#strict-provenance, https://crates.io/crates/sptr) instead

Will it be desirable to go ahead with https://crates.io/crates/sptr as suggested in the error message?

Thanks @taiki-e! I've rebased it on top of current master.
CI / cargo +beta build (pull_request)  failed once with:
warning: spurious network error (10 tries remaining): [35] SSL connect error (error:1408F119:SSL routines:ssl3_get_record:decryption failed or bad record mac)
...
error: failed to verify the checksum of `byteorder v1.4.3`

Thanks for review! Fixed the comments, please take another look


Feature: add support for ingesting from rabbitmq super streams

This PR aims to add support for ingesting logs from a rabbitmq superstream, as a peer to Kafka or Kinesis. Since this supports scalable exactly once delivery, this provides a good gasket between RabbitMQ and Druid.
Currently, this is a WIP for a few reasons:

No support for username/password and some other features, I welcome suggestions here as I'm not exactly sure how those work yet and could use some guidance on how that plumbing works

The algorithm is pretty poor here, simply making a new rabbit consumer for each poll, it should be doing something similar to Kinesis where it fetches in the background, occasionally backing off if it gets too far ahead.

Testing is extremely bare bones, primarily because the design of the record supplier could change here, and design layout could change test layout here.

Documentation on how exactly to use it is missing. I've been testing it with the below supervisor spec, the primary configuration mechanism is the 'uri' which is used to configure how it connects to rabbitmq for reading metadata and for getting messages

The meat of the change is in RabbitStreamRecordSupplier, which has the components that interface with rabbitmq.
It uses two interfaces, one being the low level 'Client' interface to read the partitions from a super stream so that druid can distribute work, and the other being the 'Consumer' interface, the conventional interface which creates a thread that listens for messages for you, and calls back on the handler which puts the messages in queue.
The current interface simply manages a map of ConsumerBuilders as partitions are assigned, and then when it is polled for messages it will create consumers and wait for the timeout. Once the timeout has elapsed, it will close all the consumers, then collect any messages from the queue and return them. This should ensure no messages are lost, but probably isn't very efficient as it could be reconnecting more than necessary.
Release note
New: You can now ingest logs from RabbitMQ via super streams
This feature gives the ability to read directly from rabbitmq using the new super-streams feature. As super streams allows exactly-once delivery with full support for partitioning, it is now compatible with Druid's modern ingestion algorithm, without the downsides that the prior RabbitMQ firehose.
Note that this uses the RabbitMQ streams feature, and not a conventional exchange. You will need to make sure that your messages are in a super stream before consumption. For more information, see https://www.rabbitmq.com/streams.html.
In order to configure, create a new index task using the 'rabbit' type, and configure it's stream and URI to connect to the rabbitmq host.
{
  "type": "rabbit",
  "dataSchema": {
    "dataSource": <your super-stream>,
    }
  },
  "ioConfig": {
    "type": "rabbit",
    "stream": "api-audit",
    "uri": "rabbitmq-stream://localhost:5552",
    "taskCount": 1,
    "replicas": 1,
    "taskDuration": "PT1H"
  },
  "tuningConfig": {
    "type": "rabbit",
    "resetOffsetAutomatically": true
  }
}

Key changed/added classes in this PR
The following classes have been added to druid in this PR:

RabbitStreamSamplerSpec
RabbitStreamRecordSupplier
RabbitStreamIndexTaskTuningConfig
RabbitStreamIndexTaskModule
RabbitStreamIndexTaskIOConfig
RabbitStreamIndexTaskClientFactory
RabbitStreamIndexTask
RabbitStreamDataSourceMetadata
IncrementalPublishingRabbitStreamIndexTaskRunner
RabbitSupervisorTuningConfig
RabbitStreamSupervisorSpec
RabbitStreamSupervisorReportPayload
RabbitStreamSupervisorIOConfig
RabbitStreamSupervisorIngestionSpec
RabbitStreamSupervisor

This PR has:

[x] been self-reviewed.

[x] using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)

[x] added documentation for new or modified features or behaviors.
[x] a release note entry in the PR description.
[] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
[x] added or updated version, license, or notice information in licenses.yaml
[x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
[x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
[ ] added integration tests.
[x] been tested in a test Druid cluster.

@jamiechapmanbrn - let us know if you need any help from druid dev community in moving this PR forward.

I've marked this as ready for review. We're testing it currently in our cluster and it seems to be working fine. I've added documentation and extended the tests significantly, I'm not 100% sure whether it has enough coverage to pass the CI yet or not, but I'm running out of steam to work on those. They are relatively tricky for me since I don't work much with Java and I'm not as familiar with the supervisor structures/classes.

have you done any scale testing on this yet? I am wondering what kind of scale you have run it with so far.

Currently, our org is running with it with a relatively low scale, somewhere on the order of a few events per minute. What would you consider reasonable for scale testing?

@jamiechapmanbrn, I'll complete my review for this PR soon.
Could you please take a look at the Integration tests that are failing as well?

Okay, I rebased on the latest master and updated the pom.xml file to reference the latest. I'll have a look at any test failures that come when they're available.

Thank you @jamiechapmanbrn. Overall approach LGTM!
Have a few minor comments that I'll post soon.

That's great to hear!
I've rebased once again, and added additional test coverage. It should pass now. I'll take another look once the CI has done its magic.

I think I've fixed the pipeline issues.
I see a bunch of dependency pull problems for druid-iceberg-extensions that don't have any relation to this pr.
Error: Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: org.eclipse.aether.resolution.DependencyResolutionException: Could not find artifact org.apache.druid.extensions:druid-iceberg-extensions:jar:30.0.0-SNAPSHOT in  (https://repo1.maven.org/maven2/)
	at org.apache.druid.cli.PullDependencies.run(PullDependencies.java:333)
	at org.apache.druid.cli.Main.main(Main.java:112)
Caused by: java.lang.RuntimeException: org.eclipse.aether.resolution.DependencyResolutionException: Could not find artifact org.apache.druid.extensions:druid-iceberg-extensions:jar:30.0.0-SNAPSHOT in  (https://repo1.maven.org/maven2/)
	at org.apache.druid.cli.PullDependencies.downloadExtension(PullDependencies.java:418)
	at org.apache.druid.cli.PullDependencies.downloadExtension(PullDependencies.java:363)
	at org.apache.druid.cli.PullDependencies.run(PullDependencies.java:309)
	... 1 more

@jamiechapmanbrn Thank you!

That's great! Thanks for all the help!


modal behaviour changed

there is a change in the modal behaviour. It can be seen at the modal example on https://angular-ui.github.io/bootstrap/
basically when animation is turned off, it appears the screen behind the modal is very obviously deleted and redrawn as the modal is closed.

Hello,
I am not sure what you mean? Could you rephrase your issue or put some screenshot?

Hi,
I mean the Modal dialog window. if you look at the example on  https://angular-ui.github.io/bootstrap/ page in the section Modal (ui.bootstrap.modal)
click on large modal button and then do it again with the animation button false.
You will see that when the modal window is closed, when the animation is false, the view underneath is set to black  and then redrawn.

Ah, I see that. Will investigate.

What is happening is that the modal backdrop has a background color of #000 - when the opacity is 1, it will appear to be solid black...should give a good base to investigate though.

Ah, so the fade class is not applied when animations are disabled, so removing the in class will cause this effect to happen.


Create workflow to automate RBE image updates

This creates a new workflow that runs once per week or can be triggered manually to create a PR that automatically updates the list of sha256 hashes for the RBE docker containers to the latest versions available (for the SIG Build containers.)

Here is an example PR (note that I fixed the broken link after this): https://github.com/angerson/tensorflow/pull/9
I'm expecting that we'll have some problem with the CLA approval bot on this, but we'll probably be able to find a workaround.


Create a CollectionDescription object and use throughout

Useful for #274, #300
A collection description is an object with the following fields:

activeSetName
activeFilterNames
activeSortName
activeSortReversed

They can be canonically converted to a string, appropriate for being included in a URL (which sorts them), so they can be directly compared to one another.

This is non-trivial to disentangle. Logic to parse a URL is in src/actions/collection.js:updateCardSelector. And then query and final collections are tied fundamentally to the collection being used, in a multi-layer deep hierarchy of selectors, which would have to be duplicated for a secondary collection, or refactored a lot.
Although all of those nested selectors are probably not actually useful because they aren't exported anyway, so just add overhead to calculations and caching. (Although they might help when small things that only affect later stages of the pipeline change)

PartialMatch is a special case for query that can't be used in the same way that others can. ... Slash it's at least notionally similar to selectCollectionItemsThatWillBeRemovedOnPendingFilterCommit. But not close enough to be worth doing the same thing with it.

[ ] src/actions/collection.js:updateCollection should use CollectionDescription.equivalent()

For multi-part filterNames. Are they just normal filter names? Or are they special dynamic filters?

A dynamic filter has a pattern, and that pattern is consulted when parsing parts of a URL. But basically what it does is tell it that multiple parts of the URL are actually part of the same filter name.
So
all/stubs/updated/before/21-3-20/published/ would be parsed as `['stubs','updated/before/21-3-20', 'published']. The first time a new dynamic filter is found, it has a filter function created and stored in a dynamicFiltersMap, where the name is precisely, in this case, 'updated/before/21-3-20', and that points to a function that takes a fully-inflated card and returns true if it matches and false if it doesn't. The first time a dynamic filter is seen that doesn't exist, it goes to the dynamicFilterFactory, which takes the full descriptor and knows how to parse it into a functioning filter function, and then run it across all existing cards. And then from that point on, every new card that's updated will be run through the dynamicFilters.
Are dynamicFilters stored in state? Seems like they need to be because the list of supported filters at a given moment is tied to which filters are installed.
... Most filters, including TODO filters, and even section and tag filters, could use this dynamic machinery. (tags and sections are a bit faster right now by being specialized). Actually, that might not even be that hard to do--basically when a tag or section is found, we create a new dynamic filter based on it
Note that union operators are still handled differently, and can't be combined with dynamic filters

Conceptually the way filters work right now, is we have an (implied) set of filters, which is extended when we discover new sections or tags. These filters are effectively memoized versions of the filter funcs, but when that memoization happens is important not just for efficiency but also because it controls when the collection is allowed to change (so we don't thrash collections like read as you're reading).
State should contain a cardFilterNames, which is the set of filters that should be run and kept updated as new cards come in.

There are a number of types of filters:

concrete -  A literal object where the card_id is set to true if it's in the filter
inverse - Maps to the inverse of a given concrete filter.
union - A union of multiple concrete or inverse filters
configurable - A card filter, but that has so many possible version of itself that they're created only lazily, e.g. updated/before/2020-3-20, where there are nearly infinite possible dates, and date conditions.

There are also other ones that aren't only a function of a given card, like similar/CARD_ID/COUNT and descendants/CARD_ID/PLY, which need to look up information across multiple cards.

The machinery to select a collection in the first place is deeply tied up in a web of selectors. Will need to separate that so given a CollectionDescription, you can fetch it.

Problems of the design: a couple of the dynamic filters I want to do (similar-cards and descendants) require more than just the card to be passed in (state over all?). And collections have various states that should be memoized in their calculation: the set, the fully filtered set, the sorted set, and then the sorted labels. That implies memoization somewhere. There are likely only two sets at a time: the Active collection, and the Secondary collection. Maybe it's not that bad to have fewer of them.

Create a Collection type that takes a description, provided with the map of all sets, map of all cards, and the current filter sets. It then has a few getters (memoized, lazy): filteredCollection, sortedCollection, collectionLabels, and finally it can take a nextFilters to return which ones will be filtered out next time. StartCards are then applied after the fact by extending the card section and label section at the front and then removing duplicates.

Is similar/CARD-ID/COUNT and children/CARD-ID a set, or a filter? It seems useful to do both.
But that implies a sort ordering specific to it that is tied to an earlier dynamic filter. Maybe dynamic filters can produce output that can then be used in certain sorts after the fact.
Query machinery would use this, too.
So dynamic filters are not just given the card, they're given the card, all cards, and state. And they emit three things: a full-match, partial-match, and sortValue. The sortValue is stored and can be picked up by the sort function, passed in as a fourth argument.

URL parsing should take an operator and a list of how many additional pieces to consume. On each piece, if the piece has a count, push that count onto the stack until it's consumed, then pop off and read another. Each start-piece counts as consuming a piece. So updated/before/2020-3-20 would say updated: 1 and before:1. There isn't any validation at this stage, it's just how many pieces to consume off the path and assemble at the top level as one filter.
Then makeDynamicFilter(string) is memoized. it splits them up based on the path component, then looks up the factory func for the first bit, and passes the rest. It then parses, potentially using factory logic for the sub-bits, and returns a function (card, allCards, state) => [match, partialMatch, sortValue], which is run dynamically before the whole filter is run to create a faux filterMap to pass into the matching machinery as though that filter always existed. (TODO: verify that the cardsToBeRemoved logic works OK with this filter matching machinery)

Everything is now either done or captured in a new issue that references this one


chore(vscode): add recommended workspace extension

Checklist

[x] Fix linting errors
[x] Tests have been added / updated (or snapshots)

This change is

:tada: This PR is included in version 1.0.0-alpha.11 :tada:
The release is available on:

npm package (@alpha dist-tag)
GitHub release

Your semantic-release bot :package::rocket:

:tada: This PR is included in version 1.1.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


MM-19738 - Reply counts sometimes missing on Community Daily

Summary
Fixed fetching thread count (missing parameter in post sync)
Ticket Link
https://mattermost.atlassian.net/browse/MM-19738

Does this need a cherry pick? Is it for 5.18 only? Or does it also affect 5.17?

@grundleborg I am pretty sure improved thread fetching wasn't a part of 5.16


Question: Minimum Tx Transfer Size

Is there a minimum amount of data transferred via MM2S?  I have axidma_malloc'ed a 216 byte (54 32-bit words) tx buffer and called axidma_twoway_transfer with the size set to 216.  When I observe a transfer with ILA, I see 1024 bytes transferred.  The first 216 bytes are the data I intended to transfer followed by 808 bytes of 0.
Thanks,
Jim

I took a quick look over the backend Xilinx DMA driver and the AXI DMA user guide, and there doesn't seem to be any enforced minimum for the transfer. My driver also doesn't enforce any minimum size.
Can you send me a block diagram of where you have the ILA connection? Also, is this just a loopback design, or do you have custom IP downstream on the MM2S AXIS interface?

Thanks for writing me back.  I'm dumb.  I was not running updated software
that was sending the correct packet size.  The software was sending a 1 kB
packet.
The driver works great, but we do notice when sending a packet of 1024
32-bit words, the dma does not assert valid for 60 clocks after
transferring the first 512 words.  It does complete the other 512 words
correctly, but it just seems weird.
We don't notice the same behavior from a standalone no OS app.
Thanks again for responding and for all the great work you dinner on the
driver.
Jim
On Fri, Jul 13, 2018, 2:34 PM Brandon Perez<EMAIL_ADDRESS>wrote:

I took a quick look over the backend Xilinx DMA driver and the AXI DMA
user guide, and there doesn't seem to be any enforced minimum for the
transfer. My driver also doesn't enforce any minimum size.
Can you send me a block diagram of where you have the ILA connection?
Also, is this just a loopback design, or do you have custom IP downstream
on the MM2S AXIS interface?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/bperez77/xilinx_axidma/issues/64#issuecomment-404917271,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AmY3fbbPyB2qlOaADqsYlPrLJUpZ5s68ks5uGOg0gaJpZM4VKZhN
.

No worries, it's hard to keep everything straight when you have such a complex system like this.
That's interesting and quite weird. I've never noticed or heard of that behavior before (though I've never checked with ILA either). Since this behavior doesn't happen from the embedded app, I suspect it must be something specific with Linux. My initial though would be that the transaction is being split up into two separate transfers, but I think that the minimum width of buffer length register parameter value is larger than 10.
Are you familiar with how to run the AXI DMA test driver? I can give you some pointers if not,. This should help isolate if the issue is with the Xilinx backend DMA driver, or if it's with my driver.
One other thing to note is that the system call overhead is pretty high, so it will likely tank the performance of small transfer sizes like the 1 KB buffers that you're using (unless you're transfering a ton of them).

Actually it looks like the minimum width of the buffer length register can be 8, so it could be possible the transaction is automatically being split into an SG transaction (see page 24 of the Xilinx AXI DMA v7.1 User Guide. Just out of curiosity, what value did you set this parameter to?

Attached is a screenshot from ILA.  I have the DMA buffer length register width set to 14.  Unfotunately, since we got our project working, I have lost my ZedBoard so I can't help with troubleshooting.


When doing the go generate the follow error appears:

Loading input failed: unsupported version of go: exit status 2: flag provided but not defined: -compiled
using version 1.9

I am facing the same issue. Any answers?


Status of the examples

@tvyomkesh opened some issues (#9384, #9389) on errors in the examples (and some PRs to fix this). This raised the following question to me:
Apart from fixing the examples, I think we should think for a moment what we want to do with these examples. Why do we want them? What is their function?
And depending on that, do we want to update/extend them? Or maybe move them to the docs?
I personally never looked at these examples. And they are also quite outdated. Eg using the ols from pandas is somewhat (unofficially) deprecated for the use of statsmodels, and the finance module in matplotlib is deprecated starting from matplotlib 1.4. And to get such data, I think pandas-datareader should be used.
@jreback @TomAugspurger @shoyer

repeat of my comment:
I think that these should be checked to see if they are in the docs ; if not and appropriate add any
then blow away the entire directory
these are just duplicate, not maintained, not tested and pretty much superfluous to the docs

Agreed -- they really should go away. I wouldn't even bother fixing them up.

Yeah, I found it strange that only two examples were there that too outdated. I have closed the other fix related PRs and submitted a new one with deletes the examples folder

Will check any potential additions to documentation and update PR accordingly.


Feature/add tests

Since I'm using this Action and I'd like to make it less prompt to errors (like I did myself in the last PR, with a simple typo), I think it would be great to include some tests.
This PR includes some basic configurations and tests. I'm actually working on a refactor to make the script easier to read and change, but I'd like to have this base.
I was using a couple of workflows as an "end to end" way of testing. I can keep them for myself or you can adapt them to your needs. I found them useful. There are also some scripts to easily trigger those workflows.

Amazing work! Can we also add workflow-dispatch triggers (https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows#workflow_dispatch) to easily trigger test workflows through UI?

Amazing work! Can we also add workflow-dispatch triggers (https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows#workflow_dispatch) to easily trigger test workflows through UI?

Definitely. I tried to implement a workflow that would trigger the rest of them but then I realized that it's only possible using a personal access token. Then I implemented the small bash script.
The workflow_dispatch makes half of the job since you would need to lunch the other "simple tasks" first. I could add a delay step for those dummy checks so they give time to manually trigger those who wait. Does it make sense?

Definitely. I tried to implement a workflow that would trigger the rest of them but then I realized that it's only possible using a personal access token. Then I implemented the small bash script.
The workflow_dispatch makes half of the job since you would need to lunch the other "simple tasks" first. I could add a delay step for those dummy checks so they give time to manually trigger those who wait. Does it make sense?

kind of makes sense! let's try that :)

kind of makes sense! let's try that :)

Amazing! Really, really appreciate your initiative and love for clean testing. The final step would be an action that runs rspec on each push and pull_request, and a status badge for that in README. Also, maybe explain the testing flow in README too (not for Ruby tests, for action tests). Then I'll happily merge and bump the release to a minor version.

Amazing! Really, really appreciate your initiative and love for clean testing. The final step would be an action that runs rspec on each push and pull_request, and a status badge for that in README. Also, maybe explain the testing flow in README too (not for Ruby tests, for action tests). Then I'll happily merge and bump the release to a minor version.

Can you plz explain a bit how you use trigger-scripts. This is something novel to me and I'd like to learn

Can you plz explain a bit how you use trigger-scripts. This is something novel to me and I'd like to learn

Sure! Soon I'll update the readme with more instructions.
About the trigger scripts, is to be used in your local computer since if you execute it in a workflow, it would not have the expected result (triggering other workflows). It assumes you have this repo as your 'origin' remote. The script is simple: it creates a tag in the remote repo. The test-workflows are configured to be triggered when you push tags starting with a certain name (the name of the test). To keep the tags "clean", the workflows delete them after they're executed.

Sure! Soon I'll update the readme with more instructions.
About the trigger scripts, is to be used in your local computer since if you execute it in a workflow, it would not have the expected result (triggering other workflows). It assumes you have this repo as your 'origin' remote. The script is simple: it creates a tag in the remote repo. The test-workflows are configured to be triggered when you push tags starting with a certain name (the name of the test). To keep the tags "clean", the workflows delete them after they're executed.

Is it ready to be merged yet, or you'll give me a shout?

Is it ready to be merged yet, or you'll give me a shout?

Sorry for the delay. I've updated the README. Please have a look and let me know if you see anything to improve. English is not my first language and I may have mistakes 😅

Sorry for the delay. I've updated the README. Please have a look and let me know if you see anything to improve. English is not my first language and I may have mistakes 😅

Great! I’ll get back to it after the weekend

Great! I’ll get back to it after the weekend

Trying to test your work locally (again, thank you, this is amazing), but running into an issue:

What am I missing here?

Trying to test your work locally (again, thank you, this is amazing), but running into an issue:

What am I missing here?

I think is because you called the script from the project root directory.  Please try cd .github/trigger-scripts and then ./trigger-test-workflows.sh. We can also fix it to make it "callable" from anywhere in the project.

I think is because you called the script from the project root directory.  Please try cd .github/trigger-scripts and then ./trigger-test-workflows.sh. We can also fix it to make it "callable" from anywhere in the project.

 This is not the first time I’m facepalming with scripts. Gonna take another day to dig some more.

 This is not the first time I’m facepalming with scripts. Gonna take another day to dig some more.

I have some other features waiting for this to get merged

I have some other features waiting for this to get merged

All right, YOLO! :)

All right, YOLO! :)


ENH Allow finetuning

Add finetuning layers and basic learning algs. SGD with clipping should be enough since we are not currently planning to support learning from scratch.

:+1:


Missing trailing slash in Azure File Copy output parameter

Environment
Azure pipelines, Hosted
Issue Description
The containername isn't copy'ed over correctly to the output variable, missing a trailing slash in the end.
Current workaround to the Azure Resource Group Deployment as override parameter is:
-_artifactsLocation $(artifactsLocation)/ -_artifactsLocationSasToken $(artifactsLocationSasToken)
If I remove the / in the Azure Resource Group Deployment as override parameter the containername is missing in the _artifactsLocation parameter, somehow it must end with an trailing slash. Fix this either in the Azure File Copy or else allow the parameter in ARGD task without slash.
Task logs

@rfcdejong This has been fixed in Azure File Copy V2 a few days ago. The fix will be available after next deployment which can take a couple of weeks to reach all accounts.


Rework the names of metrics

Currently, the metrics appear to be fashioned after the PHP status page. While this is easy for those used to consuming the PHP status page to understand, it is not as easy for consumers of Prometheus to understand.
This work revises the metric names such that they are expressed in a more expected way to prometheus, such that users who are familiar with Prometheus but not with PHP are able to query PHP for state, and make some judgements as to its health.

Yo this isn't done yet. I'll delete this comment when it is. However, I wanted to write the merge commit message early -- each commit has it's own state, but there's some overall ideas I wanted to express with the PR.

This looks fine to me. Just let me know when ready to merge.
Sorry my notifications seem wonky, so I read the issues, PR, etc all out of order :)

Holaa
I've finished messing with this. Testing it got the following output:
# HELP phpfpm_accepted_connections_total Total number of accepted connections
# TYPE phpfpm_accepted_connections_total counter
phpfpm_accepted_connections_total 42
# HELP phpfpm_active_max_processes Maximum active process count
# TYPE phpfpm_active_max_processes counter
phpfpm_active_max_processes 3
# HELP phpfpm_active_processes Active process count
# TYPE phpfpm_active_processes gauge
phpfpm_active_processes 1
# HELP phpfpm_idle_processes Idle process count
# TYPE phpfpm_idle_processes gauge
phpfpm_idle_processes 2
# HELP phpfpm_listen_queue_connections Number of connections that have been initiated but not yet accepted
# TYPE phpfpm_listen_queue_connections gauge
phpfpm_listen_queue_connections 0
# HELP phpfpm_listen_queue_length_connections The length of the socket queue, dictating maximum number of pending connections
# TYPE phpfpm_listen_queue_length_connections gauge
phpfpm_listen_queue_length_connections 0
# HELP phpfpm_listen_queue_max_connections Max number of connections the listen queue has reached since FPM start
# TYPE phpfpm_listen_queue_max_connections counter
phpfpm_listen_queue_max_connections 0
# HELP phpfpm_max_children_reached_total Number of times the process limit has been reached
# TYPE phpfpm_max_children_reached_total counter
phpfpm_max_children_reached_total 0
# HELP phpfpm_scrape_failures_total Number of errors while scraping php_fpm
# TYPE phpfpm_scrape_failures_total counter
phpfpm_scrape_failures_total 0
# HELP phpfpm_slow_requests_total Number of requests that exceed request_slowlog_timeout
# TYPE phpfpm_slow_requests_total counter
phpfpm_slow_requests_total 0
# HELP phpfpm_total_processes Total process count
# TYPE phpfpm_total_processes gauge
phpfpm_total_processes 3
# HELP phpfpm_up able to contact php-fpm
# TYPE phpfpm_up gauge
phpfpm_up 1

I think *_processes could use some love, but this is all I have time for a the minute, and I wanted to get it mergeable.
<3 for patience.

@andrewhowdencom thanks! merging as is. we can address other metrics in a different PR.


problem with JS

this script good work in Joomla
    var controller6 = new ScrollMagic();
var d6 = new TimelineMax();
d6
.to(line2pr3, 0.1,  {opacity:1})
.fromTo(line2pr3, 1, {drawSVG:"0 0%"}, {drawSVG:"100% 0%", ease:Power1.easeInOut})
.to(circle2pr3, 0.6, {attr:{r:116, transformOrigin:"50% 50%"}, opacity:1,  ease:Power1.easeInOut});

var scene6 = new ScrollScene({triggerElement: "#pre001", triggerHook: 'onEnter', offset:'500',duration:'400'})
    .addTo(controller6)
        .setPin("#pre001", {pushFollowers: true})
    .setTween(d6.play());

when insert this to Grav - not working
.setPin work good, and work in there position, but tween stat immediately after page load...
what nuances in Grav?
Help with this please

How are you inserting into Grav?  What is it supposed to do? Do you have the actual ScrollMagic script installed and running?
Just pasting the JS by itself in isolation does not help. JS is run by the browser, not the webserver.  If the JS has been output to the browser in a script tag, Grav has done it's job.  99% chance it's user/setup/configuration error.

wrong order js files...  ScrollMagic need load after Tweenmax


Removed extra validation fields from user validator

issue  #246

Need to have clarification from the frontend engineer who worked on the signup form as what key are we using for the company name of the user. I highly doubt it's company_name. Some users in the DB have got company while some have got company_name
If we make any changes to the validators, the signup form will stop working which is not expected at this point of time. Let's keep it on hold till then.

Need to have clarification from the frontend engineer who worked on the signup form as what key are we using for the company name of the user. I highly doubt it's company_name. Some users in the DB have got company while some have got company_name
If we make any changes to the validators, the signup form will stop working which is not expected at this point of time. Let's keep it on hold till then.

yeah okay!

Can we also remove github_display_name and github_id since this is data coming from github, we shouldn't allow users to change it themselves.
This is last change, after which we can merge this one :P

yess okay :)

Looks good to me!
Thanks!
Let's wait for at least one more reviewer to have a look here as 1 mistake here can cause a big problem somewhere. Let's not merge till then.

yeah cool


Change macOS build target to 10.13

Changes macOS build target to 10.13, MoltenVK supports 10.15+ only, but changing target should ensure missing symbols are loaded dynamically.
Should fix https://github.com/godotengine/godot/issues/94200

Seems like some dependencies do need 10.15, not sure if it is worth patching, or we should bump min version to 10.15. We do patch it in the main repo - https://github.com/godotengine/godot/blob/master/thirdparty/glslang/patches/disable-absolute-paths-for-apple-compat.patch


Add attributes to pre tag

When using your plugin with Markdig and Prism I noticed that you only add the css class (.language-{languageCode}) to the code tag. But the pre tag, who is the parent tag from code, also needs this css class.
Is it possible to approve and merge this PR or leave me some comment if this PR couldn't be merged?
This is my first PR to someone others project, so maybe I'm doing some things wrong? If so, please let me know. :-)

Without the css class in the pre tag:

With the css class in the pre tag:

Without the css class in the pre tag:

With the css class in the pre tag:

Nevermind, I used MarkDig wrong. I called the Markdown.ToHtml() method twice in my code; once with the pipeline and once without the pipeline. That's why my HTML in my pre and codeblocks were messed up. This PR kan be deleted now. Sorry to bother you, but thanks for your plugin. I'm a happy user 👍

Nevermind, I used MarkDig wrong. I called the Markdown.ToHtml() method twice in my code; once with the pipeline and once without the pipeline. That's why my HTML in my pre and codeblocks were messed up. This PR kan be deleted now. Sorry to bother you, but thanks for your plugin. I'm a happy user 👍


Strongly recommend providing a link to documentation for MvcJsonOptions.AllowInputFormatterExceptionMessages

I think it would be extremely beneficial to provide a link to the following page in this article: https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.mvc.mvcjsonoptions.allowinputformatterexceptionmessages?view=aspnetcore-2.2
My team and I are currently working on a project that was flagged during a security scan for a potential XSS vulnerability in our Web API. This was due to the exception messages that are generated by Newtonsoft during the model binding pipeline. In our case, the field that was raising the flag was a DateTimeOffset. Our security testing team had posted a script tag into the field and the response from the server contained the following message:
{
    "errors": {
        "datetime": [
            "Could not convert string to DateTimeOffset: <script>hey!</script>. Path 'datetime', line 1, position 47."
        ]
    },
    "title": "One or more validation errors occurred.",
    "status": 400,
    "traceId": "0HLMF4A7SK4GI:00000001"
}

After several days of searching, we were able to suppress this message by adding the following to ConfigureServices:
services
    .AddMvc()
    .AddJsonOptions(options => options.AllowInputFormatterExceptionMessages = false)

I hope this helps someone in the future reading this page!

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 33c64844-bd39-46c9-8b52-192834fa625a
Version Independent ID: 94f92690-9791-a4eb-9256-30a497afccd1
Content: Prevent Cross-Site Scripting (XSS) in ASP.NET Core
Content Source: aspnetcore/security/cross-site-scripting.md
Product: aspnet-core
Technology: aspnetcore-security
GitHub Login: @Rick-Anderson
Microsoft Alias: riande

@Rick-Anderson Have you had a chance to look at this? Thanks!

@JamesNK is this an urgent security issue or can it wait for your teams next docathon?

I don't know. Returning a script tag in JSON doesn't seem insecure to me, since it should be encoded before it is set to any HTML. Check with @blowdart

It is insecure as a default behavior in my opinion. I realize that the client should encode it before rendering but this seems like it could be opt-out rather than opt-in to make it more secure for the majority of use cases.

I agree with both of you, the returning view/service should be encoding, but developers may not think to encode by default. If the mime type is correct though, i.e. the json mime type then there's nothing for us to do here, as the mime type would prevent XSS.

I'm torn @blowdart and @JamesNK
I understand both your perspectives since JSON is simply designed to serialize data. There is no responsibility for the server to automatically encode anything inside the JSON so I think we 100% agree on this point.
What I don't like is the fact that the default for AllowInputFormatterExceptionMessages is true in the current version (and presumably future versions) of .NET core. At the very least, I think it would be beneficial to the community to put something on the XSS article about this option. Even with a mime type of application/json, security scans pick up on this type of false positive all the time. In fact, we run two different pen testing tools and both caught this "potential vulnerability". Originally, I went the route of explaining to the security team that these were in-fact false positives. I've gone through the same headache twice now with separate issues since pen testing tools raise many false positives through a sort of narrow approach at detecting issues (don't get me wrong, they can also catch real issues but they often detect stuff that is perfectly safe).

Thanks for contacting us. We believe that the question you've raised have been answered. If you still feel a need to continue the discussion, feel free to reopen it and add your comments.