Significant behavior change not documented in release notes

v1 of the action ran editorconfig-checker on the files in the workspace:
https://github.com/editorconfig-checker/action-editorconfig-checker/blob/0527b11555ed46afd00df746f35b79cb7f7b68c2/src/entrypoint.sh#L3
v2 of the action makes editorconfig-checker available in the runner machine for use by subsequent steps in the job, but the action itself does not run editorconfig-checker.
The subsequent major version bump indicates to the existing users of the action that there was a breaking change, but there is no effective communication of what the change was. We are only given a list of seemingly non-breaking changes in the release notes. The users bumping the action in their workflows will need to do quite some investigation to discover that what is described incredibly vaguely as "refactor" in the release notes is actually FAR more than a refactoring (which by definition should not cause any change in behavior) and includes a complete change in the action's behavior.
Exacerbating the problem of poor communication is that there will not be any immediate symptom of the behavior change to the users. If their workflow runs were passing before bumping the action due to the files of their repository being .editorconfig compliant, they will only see that the workflow runs continue to pass after the bump. Maintainers may consider a browse of the release notes and a passing workflow run to be sufficient validation of the action bump and not realize that if they don't add a new step to their workflow to actually execute editorconfig-checker then they will no longer be getting any validation of the .editorconfig compliance of their files.
I think the behavior change is reasonable but it should be clearly communicated to the existing users in the release notes. I recommend providing some guidance in the migration of the workflows (the addition of the subsequent step to execute editorconfig-checker.

Could someone prepare something I can add to the release notes?
That would be great.

Could someone prepare something I can add to the release notes?

@mstruebing I will take responsibility for doing that. I'll add my proposed updated release notes in a comment here as soon as I am done. I'm working on it now.

@mstruebing here it is. I provide the complete contents of the release notes so that it can replace the current content wholesale rather than requiring a merge of the texts:
## What's Changed

### Breaking

#### Behavior change in #10

Prior to 2.0.0 the purpose of the action was to run **editorconfig-checker** on the files in the workspace. The purpose of the action is now solely to make **editorconfig-checker** available in the runner machine for use by subsequent steps in the job.

##### Migration Guide

A new step that invokes `editorconfig-checker` must be added to the workflow.

**Before:**

```yaml
- name: Check EditorConfig compliance
  uses: editorconfig-checker/action-editorconfig-checker@v1
```

**After:**

```yaml
- name: Install editorconfig-checker
  uses: editorconfig-checker/action-editorconfig-checker@v2

- name: Check EditorConfig compliance
  run: editorconfig-checker
```

#### Removal of `output` output in #10

Prior to 2.0.0 the action provided an [output](https://docs.github.com/actions/creating-actions/metadata-syntax-for-github-actions#outputs-for-docker-container-and-javascript-actions) named `output` that contained the output from running **editorconfig-checker** on the files in the workspace. This output could be [referenced](https://docs.github.com/actions/learn-github-actions/contexts#steps-context) by the action user in subsequent steps/jobs of their workflow.

Due to the change in the action's behavior, the output is no longer provided. If needed, you can capture and use the output in the shell code of the step in the workflow that invokes `editorconfig-checker`.

### Enhancement

- Add optional `github-token` input in #10
- Add optional `version` input in #10
- refactor by @septs in #10
- Bump @types/node from 17.0.21 to 20.6.4 by @dependabot in #14
- Bump @vercel/ncc from 0.33.3 to 0.38.0 by @dependabot in #15
- Bump @actions/github from 5.0.0 to 5.1.1 by @dependabot in #16
- Bump typescript from 4.5.5 to 5.2.2 by @dependabot in #17
- Bump @actions/tool-cache from 1.7.1 to 2.0.1 by @dependabot in #18
- chore: update actions core by @mstruebing in #19

## New Contributors

- @septs made their first contribution in #10
- @dependabot made their first contribution in #14

Full Changelog: v1...v2.0.0

I'm happy to make adjustments if you have any requests.

Thank you very much!
I've edited them right now.


Don't create other resources if db instance is not created

Description
Don't create other resources if the DB instance is not created
Motivation and Context
If I set create_db_instance = false I don't want the other resources to be created, eg. db_subnet_group, db_parameter_group, db_option_group
Breaking Changes
I don't believe this breaks anything
How Has This Been Tested?
Tested on Terraform Cloud with Terraform 0.12.29

@antonbabenko How does this look?

Hi,
Users may want to create DB option/parameter groups even without a DB instance, so I am not sure this is a good idea to have such a strong dependency.

Hi,
Users may want to create DB option/parameter groups even without a DB instance, so I am not sure this is a good idea to have such a strong dependency.

Hi Anton,
In that case, they can just use the aws_db_option_group/aws_db_parameter_group  resource directly. There's no reason to use the rds module just to create the option/parameter groups - it adds no value, unless you're also creating the DB instance.
R.

You are right, there is more than one way to get job done - use root module, use submodule, and use resources directly.
Using resources directly is not great, because it is not possible to version resources or do any kind of logic inside of it. It is better in 0.12+ for sure.
Users are using the root module to get desired functionality and there is no reason for them to change their code to use submodule (and do terraform state mv ...).
I am going to close this issue.

How about not ignoring the create_db_option_group setting? If the user sets create_db_option_group = false then don't create the DB option group. At the moment, that flag is ignored if the option_group_name is empty.


Error extracting screenshots (reduce empty array with no initial value)

The new code added in #338 causes a bug (should be easy to fix)
Results in:

Happens here:
https://github.com/whyboris/Video-Hub-App/blob/master/main-support.ts#L393
@kevinseelbach

@whyboris sorry about that , i think this will fix it
https://github.com/whyboris/Video-Hub-App/pull/354

No worries at all! I should have caught it -- I'm just unfamiliar with how reliable the ffmpeg output is when it comes to identifying streams.
As far as I can tell this happens in exceedingly rare cases -- I only had 2 bug reports occur and it hasn't happened to me 🤷♂
No rush -- and do let me know if you prefer I fix it -- seems like just setting an initial value in the .reduce will fix the problem.

As far as I can tell this happens in exceedingly rare cases -- I only had 2 bug reports occur and it hasn't happened to me 🤷♂

I think it would only happen on files that do not have any streams. I pushed some commits that should fix the issue? Just wrap it in a try/catch
@whyboris
https://github.com/kevinseelbach/Video-Hub-App/commit/11f24005c77cf23fb14746ddfc2bcfee9bdfc83d

Sorry I've not gotten back to VHA this week -- had been busy with my 3 other projects 😅

https://github.com/whyboris/Simplest-File-Renamer
https://github.com/whyboris/JSON-i18n-Editor
https://github.com/whyboris/Health-Impact-Fund
I'll try to merge in your change tomorrow (just want to test it for a second before merging).

Closed with #354 👍
Thank you @kevinseelbach 🤝

Closed with #354 👍
Thank you @kevinseelbach 🤝

Don't thank me too soon @whyboris
A file without metadata.streams will still throw an error at the next line after getBestStream is called:
      const stream = getBestStream(metadata);
      const fileDuration = metadata.streams[0].duration || metadata.format.duration;

Wouldn't it be ideal to wrap the entire extractMetadataForThisONEFile call in a try/catch somewhere so you don't have to worry about a single bad file crashing the entire app and extraction process? So far, I've needed to quit the app when errors are raised during scanning, because the progress bar just freezes wherever it is.

@kevinseelbach I may have a solution with #357 😅
I'm uneasy at this time about wrapping the whole extractMetadataForThisONEFile because I'm unsure how the app will behave with nothing returned ... I think with this we squashed the last possible bug in this method 🤞

That looks good!
Unrelated, but why the file scanning process (for example rescan) is sync and not async? Im guessing there is very good reason just trying to understand why

On Feb 11, 2020, at 6:27 PM, Boris<EMAIL_ADDRESS>wrote:

@kevinseelbach I may have a solution with #357 😅
I'm uneasy at this time about wrapping the whole extractMetadataForThisONEFile because I'm unsure how the app will behave with nothing returned ... I think with this we squashed the last possible bug in this method 🤞
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

There isn't a particularly good reason why the process is sync instead of async.
When the scan is happening I wanted to lock up the app so no state could change. It would require some care to not let users mess around with things while the extraction process is happening, so I took a shortcut: just disallow any interaction.
It would be good to let users at least minimize the app 😅 I think currently at least for a short period of time the app is just frozen.
I'm happy to accept a PR updating things for the better -- it's just a lower-priority than adding some cool features 🤷♂
All feedback & questions welcome! I may have not fully addressed your question as well.

Makes sense! On Mac at least you can definitely minimize or change apps, fyi, just can’t navigate or do anything within the app until scanning is complete
Sent from my iPhone

On Feb 12, 2020, at 4:25 PM, Boris<EMAIL_ADDRESS>wrote:

There isn't a particularly good reason why the process is sync instead of async.
When the scan is happening I wanted to lock up the app so no state could change. It would require some care to not let users mess around with things while the extraction process is happening, so I took a shortcut: just disallow any interaction.
It would be good to let users at least minimize the app 😅 I think currently at least for a short period of time the app is just frozen.
I'm happy to accept a PR updating things for the better -- it's just a lower-priority than adding some cool features 🤷♂
All feedback & questions welcome! I may have not fully addressed your question as well.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.


just terra celery

just terra celery to run a terra celery worker.  TERRA_CELERY_* environment variables are now intended to configure the terra celery worker (not celery workers for terra apps).  The celery queue and included modules are configured from the command line.
Terra apps are expected to provide their own environment variables (e.g., `TERRA_DSM_CELERY_*) and configure their own queues and included modules from the command line.
The new generic task terra.task.subprocess (in the celery queue terra) will allow celery to execute a shell command via subprocess.run.  Note this task will be run with the terra pipenv/virtualenv disabled.

I'm in the middle of rewriting how this works on dsm, so I hope you aren't planning on use the terra celery target there.

@andyneff - this is intended to run a terra app from celery.  For example, connect just terra celery to a remote redis instance and receive subprocess commands such as just run tool ....  I'd look for terra apps to configure their own celery workers, with their own queues and included modules.


Silent fail after timeout

First of all, great software!
I'm using 1FA on a non-root partition, with TRIALS and TIMEOUT set to 1.
If the YubiKey is disconnected, it falls back to asking a password. If I don't want to mount the partition, I need to press ENTER to skip.
Is there a way to auto-skip, set a timeout or disable the fallback?
Thanks!
PS: Is not the fallback to the encrypt hook as suggested in the README, I don't have it enable.

Hi, currently there is no option for auto-skip unlock. I think It was generally assumed that the root partition is locked so unlocking it isn't optional which isn't the case for you but for now there may be no workaround.


Remove allocation from select macro

If we replace Select with select! in the benchmarks, we get the following numbers:
bounded_select_both       Rust crossbeam-channel   0.639 sec
bounded_select_rx         Rust crossbeam-channel   0.685 sec
unbounded_select_both     Rust crossbeam-channel   0.595 sec
unbounded_select_rx       Rust crossbeam-channel   0.648 sec

After applying this PR the numbers become:
bounded_select_both       Rust crossbeam-channel   0.497 sec
bounded_select_rx         Rust crossbeam-channel   0.486 sec
unbounded_select_both     Rust crossbeam-channel   0.422 sec
unbounded_select_rx       Rust crossbeam-channel   0.478 sec

Not bad! Since select! is the most common use of selection in practice, this will alleviate any potential losses from #349.
A minor addition in this PR that is worth mentioning is that never() is now a const fn.
cc @tuxether @BurntSushi - this is a follow up to #349

bors r+


Client sends MALMOOK, but mission never starts

I get DEBUG: Looking for client, received reply from <IP_ADDRESS>: MALMOOK and keep calling peekWorldState(), but world_state.is_mission_running never evaluates to True

Suggestion: add world_state.did_mission_start flag, to help debug whether mission ever started.
Suggestion: has agent_host print out something useful when mission control messages are received.
Suggestion: have agent_host ping the Mod to find out whether still running.

I have a hunch this might be a thread deadlock issue caused by MissionRecordingSpec reuse, possibly related to #256.

Might be changing my mind... could this just be another case #118?

With the 0.17.0 release, we can investigate this issue more:

Use world_state.did_mission_start to diagnose if end of mission was getting triggered somehow.
See if the changes to MissionRecordSpec make a difference.

Reduced to P2 since there's a workaround: to put a timeout on waiting for the mission to start.

Closing this as we can no longer reproduce it.

I believe I've just reproduced this...
It's possible to accidentally "queue up" missions, due to a race condition in the client state machine. What happens is this:

Agent A tries to start a mission.
Dormant Minecraft client receives the MissionInit XML, returns  MALMOOK, and begins processing the XML. The client only leaves the "dormant" state when it has finished parsing the XML.
Agent A begins waiting for the mission to start.
Agent B tries to send a mission.
The Minecraft client is still processing the first MissionInit XML message, and is therefore still in the dormant state, so it believes it can accept this new mission too, and sends the MALMOOK response.
Agent B begins waiting for the mission to start.

Agent B's mission will eventually start, but only after A's has finished.
The more involved the XML, the longer it will take to process, so the more chance there is of catching the mod while it's in the dormant state. I've managed to create a queue of around ten missions. Obviously the tenth agent will be kept waiting while nine other missions get run.
In a single client case this is maybe not so bad, since the tenth agent will always have to wait its turn... but if there are, say, ten clients in the client pool, all agents could run simultaneously. Instead, nine clients will be sitting dormant while the first client does all the work in serial.


Are pre-trained SegNet model/weights made available?

Great work on the repo! Thank you!
I was planning to use SegNet as an intermediate step for another learning task. I have two options available:

Find a pre-trained model (weights, right out of the box), and test it on my input image.
Download this repo, and train it on the datasets following steps according to the Readme

Has anyone explored (1) above, before?
Thank you!

I wanna find a pretrained segmention network too. Did you find it?


Array Indices Corresponding to the Start and Ends of Encounter Windows

Is your feature request related to a problem? Please describe.
I would like to have a good way of determining the mission elapsed time at which encounters start and end.
I usually begin with an array of timesteps, and I would like to know which array indices correspond to the starts and ends of encounter windows.
Describe the solution you'd like
Define a new attribute/module, encounter_indices, which returns two lists defining the indices of the starts and ends in the time array.
Describe alternatives you've considered
Manually creating arrays to find np.where(alt > 30) and then breaking the arrays at discontinuities to find the starts and ends.
Additional context
My workaround code:
from celest import Satellite, GroundPosition

UTCTimeData = observation_times.utc.value.astype(str)
ECIvec = propagated_positions[:].xyz.T

toronto = GroundPosition(name="Toronto", coor=(43.662300, -79.394530))
torontoCoords = Satellite()

torontoECI = torontoCoords.getECI(posData=toronto.ECEFpos[np.newaxis, :], timeData = UTCTimeData)

finch = Satellite()
NdrAng = finch.getNdrAng(groundPos=toronto, posData=ECIvec, timeData=UTCTimeData)
# Get nadir angles of the spacecraft
Alt, _ = finch.getAltAz(groundPos=toronto, posData=ECIvec, timeData=UTCTimeData)
# Get Alt Az angles

altok = np.where(Alt > 0)
ndrok = np.where(NdrAng < 30)
IOI = np.intersect1d(altok, ndrok)

encounter_segments = np.split(IOI, np.where(np.diff(IOI) > 1)[0]+1)
encounter_segments

The Encounter class's _special_interp() method implements a very similar process as you outlined but is internal to the method. For the next release (i.e. Celest 0.2.0), I can incorporate a public method which can be used both in the _special_interp() method and for public access.

With the 0.2.0 release, the encounter indices functionality has been incorporated into the programs window generation workflow to allow for position and time interpolation before windows are generated. This approach uses a modified version of the _special_interp() method and only interpolates around regions where encounters are possible.
For making public the attitudes() function, the encounter indices problem will be addressed internally to prevent the need for user to juggle such information.
With this considered, raw encounter indices can be attained through the celest.encounter._analytical_math_utils._analytical_encounter_ind function, and processed encounter indices (considering lighting and solar constraint angle considerations) can be attained through the celest.encounter._window_utils._window_encounter_ind function.


Provide a mechanism to run only one tezos-node pod per k8s node.

We also turn this functionality on by default in pulumi if we are configured to have 1 node per vm.

I don't think we should be customizing such a specific requested feature like this directly in the nodes.yaml. We should templatize the affinity spec of the statefulsets however. A user should be able to specify any affinity spec they want for every node class. In oxheadinfra for example, we are going to want to provide different affinities for different sts's.
I think we should instead simply templatize the affinity spec so that a user can customize it however they want for each sts they spin up. In addition, we write a helper function in _helpers.yaml that contains the new added affinity yaml here. So that a user can still set the flag. But the main node.yaml file should not begin to accumulate many different specific user requested features.
And then obviously user provided affinities should be merged with globally set ones.

What would be the use case for that? Isn't that sort of undoing kubernetes?

Another thing that we could do is provide a way of setting the other resources beyond storage like CPU or RAM. That way if you want 1 tezos node per EC2 VM, say, you could request exclusive access to enough CPUs to make that a reality based on the instance type that you have selected.

Closing as this is probably not the right way to do it (per our weekly discussion).


Huge amount of files in Knowledge database

Hi Phen2gene authors
I'm using your tool in a cluster and I've an issue with the amount of files tha comprises the DB. This DB has the following folders:

Knowledgebase: 14259 files
skewness: 14259 files
weights: 14259 files

In my cluster I've a file number limit that it's reached with this DB. In the case of skewness, there is only one number per file and they could be merged in one file. The same for weights folder but it has more information. Could you give some solution to this problem?
Thank you in advance
Pedro Seoane

Use the REST API provided on https://phen2gene.wglab.org.  No reason to run
on your cluster if file limit is an issue. API is just as fast.
On Wed, Sep 21, 2022 at 6:26 AM seoanezonjic @.***>
wrote:

Hi Phen2gene authors
I'm using your tool in a cluster and I've an issue with the amount of
files tha comprises the DB. This DB has the following folders:

Knowledgebase: 14259 files
skewness: 14259 files
weights: 14259 files

In my cluster I've a file number limit that it's reached with this DB. In
the case of skewness, there is only one number per file and they could be
merged in one file. The same for weights folder but it has more
information. Could you give some solution to this problem?
Thank you in advance
Pedro Seoane
—
Reply to this email directly, view it on GitHub
https://github.com/WGLab/Phen2Gene/issues/31, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABSDYBFLNR5CL5ILLDRDAW3V7LPGNANCNFSM6AAAAAAQR5QKQQ
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Hi @jimhavrilla
I cannot use your API,  I'm working in a workflow that run entirely in my cluster and the computing nodes has not access to internet (split the workflow to perform the APi queries is not an option). Also the workflow runs intensively phen2gene over ~9000 hpo profiles in several executions so the API would be easily overloaded. I think that a simple reformat of some folders or a zipped db that could be used to extract only the files that will be used in the execution could work in this case.
Thank you in advance
Pedro Seoane

The API would be fine actually if you space it properly, but the lack of
internet does pose a challenge.
Sounds like a very specific issue, sorry but I cannot help you with your
specific issue by restructuring how phen2gene works.  It needs the
knowledgebase to run queries. I'm surprised that it will not work
considering the KB size is below 1 GB zipped, but your cluster can run so
many simultaneous queries.  Perhaps you can ask your administrator to
remove file limits? Since it's not a size issue that seems like a more
practical solution than us trying to create a version of Phen2Gene that
uses a DB or a smaller amount of files.
On Thu, Sep 22, 2022 at 4:44 AM seoanezonjic @.***>
wrote:

Hi @jimhavrilla https://github.com/jimhavrilla
I cannot use your API, I'm working in a workflow that run entirely in my
cluster and the computing nodes has not access to internet (split the
workflow to perform the APi queries is not an option). Also the workflow
runs intensively phen2gene over ~9000 hpo profiles in several executions so
the API would be easily overloaded. I think that a simple reformat of some
folders or a zipped db that could be used to extract only the files that
will be used in the execution could work in this case.
Thank you in advance
Pedro Seoane
—
Reply to this email directly, view it on GitHub
https://github.com/WGLab/Phen2Gene/issues/31#issuecomment-1254714165,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABSDYBBBPYLZF35L6YSRXO3V7QL5RANCNFSM6AAAAAAQR5QKQQ
.
You are receiving this because you were mentioned.Message ID:
@.***>

You can create a ram drive that for example stores all skewness files and weight files which occupies only a small portion of memory. This will avoid the limits of maximum number of files per partition.
It is best to actually remove the file number limit; for bioinformatics applications, it is common to have many hundreds of thousands of files.

Kai's idea is also a really good one.
On Thu, Sep 22, 2022 at 8:55 AM Kai Wang @.***> wrote:

You can create a ram drive that for example stores all skewness files and
weight files which occupies only a small portion of memory. This will avoid
the limits of maximum number of files per partition.
It is best to actually remove the file number limit; for bioinformatics
applications, it is common to have many hundreds of thousands of files.
—
Reply to this email directly, view it on GitHub
https://github.com/WGLab/Phen2Gene/issues/31#issuecomment-1254984428,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABSDYBGCKSDQJQGHKFSFZFDV7RJNVANCNFSM6AAAAAAQR5QKQQ
.
You are receiving this because you were mentioned.Message ID:
@.***>


Add kwargs handling for patchpixelsampler patch size

PatchPixelSamplerConfig has a patch_size item, but the current PatchPixelSampler init function doesn't override the value of patch_size with the corresponding kwargs value.
This is particularly problematic, because PatchPixelSamplerConfig is generated within the datamanager:
https://github.com/nerfstudio-project/nerfstudio/blob/9528a3f717fe28a6c6500199b5b37d4f2c6e10ca/nerfstudio/data/datamanagers/base_datamanager.py#L472
This PR simply adds the __init__ function for the sampler and adds a self.config.patch_size = self.kwargs.get(....
As an aside, it would be good if this override could happen automatically...

Like instead of:
            return PatchPixelSamplerConfig().setup(
                patch_size=self.config.patch_size, num_rays_per_batch=num_rays_per_batch
            )

Would this work?
            return PatchPixelSamplerConfig(
                patch_size=self.config.patch_size,
                num_rays_per_batch=num_rays_per_batch,
            ).setup()

I believe the latter should work.
On another note (maybe address in this PR or in some future), perhaps it would be better to have people manually change the datamanager's pixel_sampler to PatchPixelSampler in the method configs, instead of automatically generating it under the hood when patch_size > 1.
https://github.com/nerfstudio-project/nerfstudio/blob/9528a3f717fe28a6c6500199b5b37d4f2c6e10ca/nerfstudio/data/datamanagers/base_datamanager.py#L348


Revert "mps2_an385: use default stack sizes"

Many test cases failed while running code coverage report on mps2_an385 due to the this PR, so apply to revert it first.
This reverts commit 2162c39e957104997f0be0bec4698c6a3162a0fc.
While running test case by:  twister -T tests/kernel/ -p mps2_an385 --coverage
........
INFO    - Total complete:   98/  98  100%  skipped:   34, failed:   46
INFO    - 35 of 81 test configurations passed (43.21%), 46 failed, 34 skipped with 0 warnings in 838.47 seconds
INFO    - In total 818 test cases were executed, 279 skipped on 1 out of total 328 platforms (0.30%)
INFO    - 35 test configurations executed on platforms, 46 test configurations were only built.
INFO    - Generating coverage files...
Most of them failed due to stack overflow.
After this commit reverted:
INFO    - Total complete:   98/  98  100%  skipped:   34, failed:    6
INFO    - 75 of 81 test configurations passed (93.82%), 6 failed, 34 skipped with 0 warnings in 694.35 seconds
INFO    - In total 836 test cases were executed, 299 skipped on 1 out of total 329 platforms (0.30%)
INFO    - 75 test configurations executed on platforms, 7 test configurations were only built.
INFO    - Generating coverage files...
So I would like to revert this commit first, then I'll keep going on other 6 failed testing.

Wonder if you can do these in boards/arm/mps2_an385/Kconfig.defconfig instead. So the stack size only increases when CONFIG_COVERAGE=y.

Wonder if you can do these in boards/arm/mps2_an385/Kconfig.defconfig instead. So the stack size only increases when CONFIG_COVERAGE=y.

Yes, it definitely make sense stack should larger when coverage enabled . I'll refine this. Thank you for reminding this~

Please don't change default configuration for an entire board (the MPS2 is real hardware, even if we use it mostly in emulation) just to accommodate the needs of one test paradigm.
If you have to, you can define a separate board a-la qemu_x86_coverage with modified values.
Also: the actual numbers here seem outrageously large. I believe that the coverage tracking needs more stack space than the default. I am very suspicious that it needs an 8k idle stack...

Got it, it indeed need a proper stack size for code coverage option and should not waste our memory usage. I will refine this by applying proper stack size, thank you for the review!

starting to dislike adding board definitions for every other test use case, suggestion from @dcpleung  makes sense, probably we should also remove the qemu_x86_coverage one and do it the same way and enable this when running twister

starting to dislike adding board definitions for every other test use case, suggestion from @dcpleung makes sense, probably we should also remove the qemu_x86_coverage one and do it the same way and enable this when running twister

OK, got your point. I remember you mention that before. Will try to do this. Thank you for your review!


Git provider & add badges

Support for:

gitlab
github

usage of namespace in twig missing
migration would be nice
otherwise no objections from me


Work in progress - Async Route (now works)

New functions

Set-PodeRouteAsync
Add-PodeStopTaskRoute
Add-PodeGetTaskRoute

How to use the new function

Start-PodeServer -Threads 1 {

    Add-PodeEndpoint -Address localhost -Port $Port -Protocol Http
    New-PodeLoggingMethod -Terminal | Enable-PodeErrorLogging

    Enable-PodeOpenApi -Path '/docs/openapi' -OpenApiVersion '3.0.3'  -DisableMinimalDefinitions -NoDefaultResponses

    Add-PodeOAInfo -Title 'Async test - OpenAPI 3.0' -Version 0.0.1

    Enable-PodeOAViewer -Type Swagger -Path '/docs/swagger'

    Enable-PodeOAViewer -Editor -Path '/docs/swagger-editor'
    Enable-PodeOAViewer -Bookmarks -Path '/docs'
    $uSleepTime = 5
    $uMessage = 'coming from using'

    $global:gMessage = 'coming from global'
    $global:gSleepTime = 3
    Set-PodeState -Name 'data' -Value @{
        sleepTime = 5
        Message   = 'coming from a PodeState'
    }

    Add-PodeRoute -PassThru -Method Put -Path '/asyncUsing'    -ScriptBlock {

        Write-PodeHost "sleepTime=$($using:uSleepTime)"
        Write-PodeHost "Message=$($using:uMessage)"
        Start-Sleep $using:uSleepTime
        return @{ InnerValue = $using:uMessage }
    } | Set-PodeRouteAsync -ResponseType JSON, YAML

    Add-PodeRoute -PassThru -Method Put -Path '/asyncState'  -ScriptBlock {

        Write-PodeHost "state:sleepTime=$($state:data.sleepTime)"
        Write-PodeHost "state:MessageTest=$($state:data.Message)"
        for ($i = 0; $i -lt 20; $i++) {
            Start-Sleep $state:data.sleepTime
        }
        return @{ InnerValue = $state:data.Message }
    } | Set-PodeRouteAsync -ResponseType JSON, YAML

    Add-PodeRoute -PassThru -Method Put -Path '/asyncStateNoColumn'    -ScriptBlock {
        $data = Get-PodeState -Name 'data'
        Write-PodeHost 'data:'
        Write-PodeHost $data -Explode -ShowType
        for ($i = 0; $i -lt 20; $i++) {
            Start-Sleep $data.sleepTime
        }
        return @{ InnerValue = $data.Message }
    } | Set-PodeRouteAsync -ResponseType JSON, YAML

    Add-PodeRoute -PassThru -Method Put -Path '/asyncParam'   -ScriptBlock {
        param($sleepTime2, $Message)
        Write-PodeHost "sleepTime2=$sleepTime2"
        Write-PodeHost "Message=$Message"
        for ($i = 0; $i -lt 20; $i++) {
            Start-Sleep $sleepTime2
        }
        return @{ InnerValue = $Message }
    } -ArgumentList @{sleepTime2 = 2; Message = 'comming as argument' } | Set-PodeRouteAsync -ResponseType JSON, YAML

    Add-PodeGetTaskRoute -Path '/task' -ResponseType JSON, XML, YAML -In Path -TaskIdName 'pippopppoId'
    Add-PodeStopTaskRoute -Path '/task' -ResponseType JSON, XML, YAML -In Query -TaskIdName 'pippopppoId'

}

A clear and concise description of what has been changed.
Related Issue
A link/reference to the GitHub issue.
Examples
If relevant, include any examples of using what you have changed.
Additional Context
Add any other context or screenshots about the pull request request here.

Hey! I managed to get time to review 😄
I would recommend raising an Issue first for larger feature work, so it can be discussed before diving into the solution 😜
If I'm right, this is a wrapper for Routes, which sets the Route logic as an async task with optional routes for info retrieval?
If so, I'd recommend the following:

Re-use the existing Task functionality, adding any needed improvements there, rather than having 2 identical Task systems - plus they'll benefit from further improvements to Tasks in the future. If people need more threads, then Set-PodeTaskConcurrency is available to increase the amount used.

There's a ticket I have to improve Tasks, so if this needs to be done lemme know as I can look at this first instead of the locale work for Pode.Web.

Use a single Runspace Pool rather than multiple, even for the re-using of Tasks above, otherwise it will get resource heavy quickly.
In the Async.ps1 file keep the function naming consistent like with other files, for example:

Add-PodeQueryTaskRoute > Add-PodeAsyncQueryRoute
Add-PodeStopTaskRoute > Add-PodeAsyncStopRoute
Add-PodeGetTaskRoute > Add-PodeAsyncGetRoute
Set-PodeRouteAsync > Set-PodeAsyncRoute

Have a separate Remove-PodeAsyncRoute which calls Remove-PodeRoute, but also Remove-PodeTask - moving any runspace clean-up into here.
For Set-PodeAsyncRoute it might be an idea to allow a "WebhookUrl" to be supplied, so when the Task the Route invokes completes it could send a callout to a webhook.
For Get-PodeUserRequest, there's already Get-PodeHeader, so I'd just split these up as Get-PodeData, Get-PodeQuery, and Get-PodeParameter each with a -Name parameter. Placing them into the public Utilities.ps1 file.

Regarding the Get-PodeData, Get-PodeQuery, etc., I agree with your suggestion. I'll create a PR for this enhancement.
For the function names, I agree that your suggested names are better. I hadn't spent much time on naming, so this feedback is helpful.
However, I have a question about the need for a separate Remove-PodeAsyncRoute. Remove-PodeRoute seems to cover the functionality since you cannot remove an async route without removing the route itself.
On the topic of merging the functionality with PodeTask, I have some reservations. PodeTask serves a very specific purpose that doesn't align perfectly with async REST calls. The latest commit includes an option to specify the maximum number of threads that each route can execute, which is a crucial feature. Some routes cannot be run concurrently or must have a limited number of concurrent executions due to the heaviness of the process.

Regarding the webhook. In this context, a callback is the appropriate outbound method. I'm going to extend the callback already in place to allow a complete interpretation of the callback semantics https://swagger.io/docs/specification/callbacks/

Another part that still I need to implement is the security. So far anyone can see everything.
I need to us the roles and groups to limit access to the async results

specify the maximum number of threads that each route can execute, which is a crucial feature. Some routes cannot be run concurrently or must have a limited number of concurrent executions due to the heaviness of the process

I feel this is one that could be also achieved with Tasks as well, on top of #1037. There's need there to specify that certain tasks should only be run sequentially, and here we have a need to limit the number of a Task running concurrently, even sequentially at times - the two could probably be solved with the same solution, enabling the requirement here and enhancing Tasks at the same time:

Tasks when created and invoked, by default, use a global runspace pool; they can run concurrently, and have no concurrency limit. This'll be the default/common format for Tasks, and for Async Routes invoking them as well.

Then 2 potential options:

We introduce a new -Isolated switch on Add-PodeTask (and Set-PodeAsyncRoute), this enables a ParameterSet with advanced functionality to control threading - in this case likely just -MaxThreads for now, and setting this to 1 forces sequential processing only. If MaxThreads isn't supplied then the internal default of $PodeContext.Threads.Tasks is used.

Tasks with -Isolated create a separate Runspace Pool. Having this switch makes it safer so that people don't accidentally create a mass amount of Runspace Pools.

For example:
# global
Add-PodeRoute ... | Set-PodeAsyncRoute -ResponseContentType Json

# isolated and sequential
Add-PodeRoute ... | Set-PodeAsyncRoute -ResponseContentType Json -Isolated -MaxThreads 1

or,

This is one of thought about in the past. We have a Add-PodeRunspacePool public function which lets people specify a -Name and a -MaxThreads.

On Add-PodeTask and Set-PodeAsyncRoute there's a new -RunspacePoolName. If this is supplied then the Tasks run on the specified pool, if not passed then they run on the global Task pool. (It'll likely need protection to stop people running Tasks on other internal pools, hah)
This would also allow for an isolated pool for multiple select Tasks/Route Tasks to run on - rather than 1 to 1.

For example:
# global
Add-PodeRoute ... | Set-PodeAsyncRoute -ResponseContentType Json

# isolated and sequential
Add-PodeRunspacePool -Name 'CustomPool' -MaxThreads 1
Add-PodeRoute ... | Set-PodeAsyncRoute -ResponseContentType Json -RunspacePoolName 'CustomPool'

This way we don't have duplicated logic, and improve Tasks all around.

I like the idea of the isolated parameter.
But to be honest, I don't see a problem with having 1000 runspaces. A runspace uses no resources other than a small quantity of memory.
In a Pode project, I'm not expecting to see 1000 async routes; if that's the case, I doubt that all of them are used simultaneously. In the end, the number of running threads is the only thing that matters
At the moment, the way it works is like this :
Add-PodeRoute ... | Set-PodeAsyncRoute -ResponseContentType Json  -MaxThreads 2

As for the idea of using the same code for Task and Async, I'm only partially convinced it's feasible without compromising the compatibility with the current API.
ConvertTo-PodeEnhancedScriptBlock does all the magic by injecting the user code inside the "async" envelope, which is completely different from how the PodeTasks are managed.
The only similar thing is Start-PodeAsyncRoutesHousekeeper, but I want to find a way to remove it. I was thinking of using an individual scheduler to clean up each async process.

The callback implementation is completed.
Now, it is missing only the security part and the -Isolated switch

I’m looking at how to integrate SSE.
It’s a very useful feature when you using an async call from a browser

Documentation is done
the only part missing is SSE documentation and some minor fixes to the OpenAPI definition

I'm back from holiday, so I'll begin reviewing this one and the Runspace one as soon as I can :)

Runspace is simple
There are just 2 functions to make the debugging easier and a small document that explains that

Hey @mdaneri, I'm gradually getting through the review, just a slow one atm! Please try not to commit anything to the PR while I go through, as it'll confuse the ongoing review 😄 I'm hoping to finish the rest of the review this week.
While going through work the ContentType parameters reminded me of a feature I was toying with a few months back which might actually help out a lot here. I've been mapping the idea to the work here, and so far it seems like a good match; when I get chance I'll write it up, but in short it's an alternative to the Write-PodeXResponse functions and the way ContentTypes are figured out - and respecting the Accept header more, similar to how you have here.

I was thinking of making a small change, but I can postpone it to the next release.
In this implementation, when you query for an async task, there is no limit to the number of objects you can get back. I was thinking of adding a limit of 100 configurable.


Failed to check key count  on partition

v0.4.0, after 7 days running, showing error.

Failed to check key count on partition: 254: read tcp <IP_ADDRESS>:48176-><IP_ADDRESS>:3320: i/o timeout => distribute.go:65

Failed to check key count on partition: 160: context deadline exceeded => distribute.go:65

+1, the log met

The same issue from all members:
rpc-proxy-cacher-6c5d77d85-fp5fw rpc-proxy-cacher 2021/12/13 21:15:45 [ERROR] Failed to update routing table on <IP_ADDRESS>:3320: EOF => update.go:58
rpc-proxy-cacher-6c5d77d85-fp5fw rpc-proxy-cacher 2021/12/13 21:15:48 [ERROR] Failed to update routing table on <IP_ADDRESS>:3320: context deadline exceeded => update.go:58
rpc-proxy-cacher-6c5d77d85-fp5fw rpc-proxy-cacher 2021/12/13 21:15:48 [ERROR] Failed to update routing table on <IP_ADDRESS>:3320: context deadline exceeded => update.go:58
rpc-proxy-cacher-6c5d77d85-fp5fw rpc-proxy-cacher 2021/12/13 21:15:48 [ERROR] Failed to update routing table on <IP_ADDRESS>:3320: context deadline exceeded => update.go:58
rpc-proxy-cacher-6c5d77d85-fp5fw rpc-proxy-cacher 2021/12/13 21:15:48 [ERROR] Failed to update routing table on cluster: EOF => routingtable.go:238
rpc-proxy-cacher-6c5d77d85-fp5fw rpc-proxy-cacher 2021/12/13 21:15:49 [INFO] Routing table has been pushed by <IP_ADDRESS>:3320 => operations.go:87```

the problem is reproduced only under production traffic, therefore it is difficult to catch it on synthetic tests

Hi all,
I'm so sorry for my late response. The context deadline exceeded string is raised by the following function.
func (c *Client) conn(addr string) (net.Conn, error) {
	p, err := c.pool(addr)
	if err != nil {
		return nil, err
	}

	var (
		ctx    context.Context
		cancel context.CancelFunc
	)
	if c.config.PoolTimeout > 0 {
		ctx, cancel = context.WithTimeout(context.Background(), c.config.PoolTimeout)
		defer cancel()
	} else {
		ctx = context.Background()
	}

	conn, err := p.Get(ctx)
	if err != nil {
		return nil, err
	}

	if c.config.HasTimeout() {
		// Wrap the net.Conn to implement timeout logic
		conn = NewConnWithTimeout(conn, c.config.ReadTimeout, c.config.WriteTimeout)
	}
	return conn, err
}

This function tries to acquire a new connection from the pool for the given address, but it cannot do that. In order to inspect the problem, you may want to:

Disable PoolTimeout,
Increase the number of allowed connections in the pool.

I think the actual problem is much deeper. Because I see the following lines in the logs:
rpc-proxy-cacher-6c5d77d85-fp5fw rpc-proxy-cacher 2021/12/13 21:15:45 [ERROR] Failed to update routing table on <IP_ADDRESS>:3320: EOF => update.go:58
...
rpc-proxy-cacher-6c5d77d85-fp5fw rpc-proxy-cacher 2021/12/13 21:15:48 [ERROR] Failed to update routing table on cluster: EOF => routingtable.go:238

and (from ShawnHsiung)
Failed to check key count on partition: 254: read tcp <IP_ADDRESS>:48176-><IP_ADDRESS>:3320: i/o timeout => distribute.go:65

It seems that the network is unstable for these cases. Could you please give more information about your setup? Is that Kubernetes, right?
Source: https://github.com/buraksezer/olric/blob/b41f7632b01300ddab0549174a566141c34c2795/internal/transport/client.go#L124

Hey,
I double-checked the suspected parts of the code. Everything looks fine to me. I'm going to try to create load locally and inject errors into the cluster. It'll take some time. Do you guys give some info about your setup and configuration? Could you please maxConn and poolTimeout directives from your config? How many members do you have in the cluster?
Your help is appreciated

Another question, is the cluster works seamlessly despite these errors?

I just released v0.4.1. It improves error messages. I hope this version will increase our understanding of this problem.
See https://github.com/buraksezer/olric/releases/tag/v0.4.1
Probably there is nothing wrong with the code, but the network is unstable somehow. Because there is no proper and immediate way to close dead TCP sockets in the pool, sometimes Olric may try to use dead TCP sockets to send messages over the wire. This can explain unexpected EOF or Broken pipe kind error messages in the logs. Olric is designed to overcome this kind of problem. For example, the cluster coordinator constantly inspects key counts on the partitions. This check may fail occasionally. That error should be no problem at all.
I need to know the answers to these questions to debug the problem:

Does the network between pods/containers/virtual machines function properly? How do you know?
Does the cluster continue to function properly despite these errors?

Hey,
I double-checked the suspected parts of the code. Everything looks fine to me. I'm going to try to create load locally and inject errors into the cluster. It'll take some time. Do you guys give some info about your setup and configuration? Could you please maxConn and poolTimeout directives from your config? How many members do you have in the cluster?
Your help is appreciated

I think increase maxConn and poolTimeout will just delay the problem, not solve. I can reproduce the problem with single instance so I don't think it's the network unstable problem. And yes, I'm in k8s. And v0.4.0-beta.2 was ok.

I just released v0.4.1. It improves error messages. I hope this version will increase our understanding of this problem.
See https://github.com/buraksezer/olric/releases/tag/v0.4.1
Probably there is nothing wrong with the code, but the network is unstable somehow. Because there is no proper and immediate way to close dead TCP sockets in the pool, sometimes Olric may try to use dead TCP sockets to send messages over the wire. This can explain unexpected EOF or Broken pipe kind error messages in the logs. Olric is designed to overcome this kind of problem. For example, the cluster coordinator constantly inspects key counts on the partitions. This check may fail occasionally. That error should be no problem at all.
I need to know the answers to these questions to debug the problem:

Does the network between pods/containers/virtual machines function properly? How do you know?
Does the cluster continue to function properly despite these errors?

The network was interrupted maybe, but it's ok eventually and olric can not recovery(I know this because restart the cluster will fix the problem). And the cluster can not work properly while having these errors.

@hulucc
There is a configuration option for TCP KeepAlives.
If you use the YAML file to configure the nodes:
https://github.com/buraksezer/olric/blob/v0.4.2/cmd/olricd/olricd.yaml#L15
For embedded-member mode:
https://github.com/buraksezer/olric/blob/v0.4.2/config/client.go#L55
It is 300s in the sample configuration file but there is no default value currently.
I think  the  ma

@hulucc
There is a configuration option for TCP KeepAlives.
If you use the YAML file to configure the nodes:
https://github.com/buraksezer/olric/blob/v0.4.2/cmd/olricd/olricd.yaml#L15
For embedded-member mode:
https://github.com/buraksezer/olric/blob/v0.4.2/config/client.go#L55
It is 300s in the sample configuration file but there is no default value currently.
I think the root cause behind this story is buraksezer/connpool. It somehow fails to remove dead connections from the pool. I checked everything, but I cannot find any problem with it. We can also replace it with jackc/puddle.
You may try to reduce the TCP connection keep-alive. It seems that the Kubernetes networking layer drops connections without the termination phase.

And the cluster can not work properly while having these errors.

Failed to check key count on partition... is an unimportant error message. The coordinator node constantly tries to inspect the key count on the partitions. If one of the requests fails, it will try again shortly after.
Is there any error message in the logs? I need more details to debug that problem.

I saw this function in the Spiracle repository: https://github.com/LilithGames/spiracle/blob/ed2af92da1d13f1afa11b6a354db33048930655b/infra/db/db.go#L25
If you still use the same function or similar one, config.Config.KeepAlivePeriod is zero. That means TCP keepalive is disabled by default. I think we need to set a default value.
Could you please try to set 300s or lower? This is how it works in the server code: https://github.com/buraksezer/olric/blob/23cd56d53a7d0998740e5a19204fb482003a5fbf/internal/transport/server.go#L283
This is how Redis handles keepalives: https://redis.io/topics/clients#tcp-keepalive

I saw this function in the Spiracle repository: https://github.com/LilithGames/spiracle/blob/ed2af92da1d13f1afa11b6a354db33048930655b/infra/db/db.go#L25
If you still use the same function or similar one, config.Config.KeepAlivePeriod is zero. That means TCP keepalive is disabled by default. I think we need to set a default value.
Could you please try to set 300s or lower? This is how it works in the server code:
https://github.com/buraksezer/olric/blob/23cd56d53a7d0998740e5a19204fb482003a5fbf/internal/transport/server.go#L283
This is how Redis handles keepalives: https://redis.io/topics/clients#tcp-keepalive

Ok, I will try this. However, I can get this error with single embedded instance, olric will keep some tcp connect to itself?

Hello, I embedded Olric and I have the same issue. At the moment I'm using v0.4.5. This is my log:
2022/09/29 23:09:14 [ERROR] Failed to check key count on partition: 269: failed to acquire a TCP connection from the pool for: <IP_ADDRESS>:3320 timeout exceeded => distribute.go:65
2022/09/29 23:09:17 [ERROR] Failed to check key count on partition: 270: failed to acquire a TCP connection from the pool for: <IP_ADDRESS>:3320 timeout exceeded => distribute.go:65
2022/09/29 23:09:20 [ERROR] Failed to update routing table on <IP_ADDRESS>:3320: failed to acquire a TCP connection from the pool for: <IP_ADDRESS>:3320 timeout exceeded => update.go:58
2022/09/29 23:09:20 [ERROR] Failed to update routing table on cluster: failed to acquire a TCP connection from the pool for: <IP_ADDRESS>:3320 timeout exceeded => routingtable.go:238

I don't understand why there are 270 partitions, because I just use a couple of maps with just a few keys each. Anyway what happens here is that during this phase any operation seems to hang until the error appears for partition 270, then pending operations go on and it starts again from the first partition.
But why does olric need to open a tcp connection to itself?
This is my initialization code:
c := olric_conf.New("lan")
ctx, cancel := context.WithCancel(context.Background())
c.Started = func() {
    defer cancel()
}

olricSD := &OlricEmbeddedDiscovery{}
olricSD.olricConfig = c
sd := make(map[string]interface{})
sd["plugin"] = olricSD
c.ServiceDiscovery = sd

var err error
olricDb, err = olric.New(c)
if err != nil {
    log.Fatal(err)
}

go func() {
    err = olricDb.Start()
    if err != nil {
        fmt.Printf("********* Error starting Olric: %v\n", err)
        olricDb.Shutdown(ctx)
        cancel()
    }
}()

<-ctx.Done()

// Bootstrapping fs
stats, err := olricDb.Stats()
for id, m := range stats.ClusterMembers {
    fmt.Printf("%d -> %s\n", id, m.Name)
}

clusterUuid = stats.Member.ID

ownMap, err = olricDb.NewDMap(fmt.Sprintf("cluster-%d", stats.Member.ID))
if err != nil {
    return err
}
ownMap.Put("UUID", GetConfig().UUID)
ownMap.Put("node.uuid", GetConfig().UUID)
ownMap.Put("node.name", GetConfig().Name)

Hello,

At the moment I'm using v0.4.5.

Please consider upgrading to the latest stable version. Connection pooling dependency has received a few bug fixes.

I don't understand why there are 270 partitions because I just use a few maps with just a few keys each.

Olric is a partitioned data store. That mechanism distributes all your data among the cluster members. You can increase or decrease partition numbers, but prime numbers are good for equal distribution.

But why does olric need to open a tcp connection to itself?

It is not ideal; I know this, but it's completely unimportant. The coordinator node fetches all available nodes from the memberlist instance and sends a request to learn some statistical data from the members. The payload is quite low. I didn't want to add another code path to keep the complexity low. As I mentioned, the payload is only a few bytes long.

2022/09/29 23:09:14 [ERROR] Failed to check key count on partition: 269: failed to acquire a TCP connection from the pool for: <IP_ADDRESS>:3320 timeout exceeded => distribute.go:65

Does Olric run your local environment or Docket network? Something should be different in your environment. I need to know that difference to reproduce the problem on my side.
Did you check the previous messages?

For example, do you use any firewall software which blocks Olric? Something should be different in your network.

Sorry, maybe I had to mention the software embedding olric is running on a Raspberry Pi and I'm testing it with up to three nodes (most of the time just one). All nodes run on my home wifi network (I suppose it causes most of the issues when multiple nodes are involved).
Unfortunately the issue usually happens after running for several days (like the issue's creator), so it's very difficult to understand. To make it harder, it happens even if just one node runs all the time, so it doesn't seem to be directly related to network failures...
Anyway I will wait for the issue again to analyze it more accurately, because I'm dealing with other changes at the moment. I will let you know.

Hello, I see. Olric v0.4.7 has this fix: https://github.com/buraksezer/connpool/issues/4. This may fix the problem.

It looks promising! I will test it in the next days and let you know 😉


Allow device & shard count validation to be skipped

As a firebase test lab user, I may be offered access from time to time above the normal limits. As a flank & fladle user, I would like to be able to pass in customized config values to use my account to the fullest - but the configuration validation prevents it.
I would like the ability to skip config validation by passing in a flag at runtime. This flag could even be part of the config, it doesn't need to be a command line option. In the event of having an invalid config but passing the skip validation flag, Flank should crash (unparsable config) or pass the values provided (eg, 1000 shards, or FluffyBunny device type, etc).
As this is part of Flank that's (correctly!) built in, it's not currently possible to work around this when passing in otherwise invalid parameters.

For estimation

Add new option skip-config-validation
Update implementation for skipping o args validation if this option is used.


Add a null check for the class loader of OsgiUtils

org.apache.commons.compress.utils.OsgiUtils, introduced by commit 30a7750, calls OsgiUtils.class.getClassLoader().
This method can return null if the class is loaded by the bootstrap class loader.
When one of my CLI tools, which bundles commons-compress, used the -Xbootclasspath/a option for faster startup time, it started throwing a NullPointerException after upgrading commons-compress from 1.20 to 1.21.
Caused by: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because the return value of "java.lang.Class.getClassLoader()" is null
        at org.apache.commons.compress.utils.OsgiUtils.<clinit>(OsgiUtils.java:31)

This PR adds a null check for the class loader to avoid the NullPointerException.

@garydgregory Thank you for your reviewing. I updated the code in d2cdb3b and force-pushed it.


format_supported=npm i react-native-videoNO_EXCEEDS_CAPABILITIES"

Bug

Platform

Which player are you experiencing the problem on:

iOS
Android

Environment info

React native info output:
 // paste it here

Library version: x.x.x
Steps To Reproduce

...
Expected behaviour

Video should play with small resolution

Reproducible sample code

Video sample
If possible, include a link to the video that has the problem that can be streamed or downloaded from.

Issue without description...


Unable to resolve module merge

Is this normal?

I'm having the same issue. The module worked prior to upgrading to IOS 9.

Should be fixed in v0.3.4!


perf(material-experimental/mdc-chips): do not re-enter the Angular zone unnecessarily

@arturovt It's been a couple weeks since the last comment from @crisbeto - would you still like to continue trying to land this change?

Closing due to inactivity - feel free to re-open another PR if you'd still like to land this


Vue1 incompatibility

This component is currently not compatible with Vue 1.0.x. Since version 0.1.0 it uses transition element which is only available in Vue 2.x so when trying to show the toast we have this error in console:
[Vue warn]: Unknown custom element: <transition> - did you register the component correctly? For recursive components, make sure to provide the "name" option.

@cyberbobs thanks for reporting. I noticed that warning, but that should not affect the functionality.
I tried to suppress the warning but didn't work. Let me try again.

Hi @cyberbobs it should be no [Vue warn] in v0.2.0. Would you please help to verify it?


From regione and cf_amministrazione_appaltante to NUTS.

context
Indicators schema needs to generate also aggregation_id which can be instead of an int ranging from 1 to 20, their NUTS code. This should happen for each  geographical aggregation_type level i.e. provincia e comune, but also for cf_amministrazione_appaltante which has not really a geographical level (I found a file with C.A. registry information, me may use it to locate each cf_amministrazione_appaltante and then trace back its geo).
current behavior
not accounted.
expected behavior

option 1: manually define a function that maps each region (province and municipality) to its NUTS.

get_nuts_code <- function(region_name) {
  # get NUTS code for a given region name
  # regionName: string of the region name
  # returns: string of the NUTS code

  # dictionary of regions and their NUTS codes
  region_nuts_dict <- list(
    "Abruzzo" = "ITC1",
    "Basilicata" = "ITC2",
    "Calabria" = "ITC3",
    "Campania" = "ITC4",
    "Emilia-Romagna" = "ITC5",
    "Friuli-Venezia Giulia" = "ITC6",
    "Lazio" = "ITC7",
    "Liguria" = "ITC8",
    "Lombardia" = "ITC9",
    "Marche" = "ITD1",
    "Molise" = "ITD2",
    "Piemonte" = "ITD3",
    "Puglia" = "ITD4",
    "Sardegna" = "ITE1",
    "Sicilia" = "ITE2",
    "Toscana" = 	"ITE3",
    "Trentino-Alto Adige/Südtirol" =	"ITE4",
    "Umbria" = 	"ITE5",
    "Valle d'Aosta/Vallée d'Aoste" = 	"ITE6",
    "Veneto" = 	"ITE7")

# return the NUTS code for the given region name, if it exists in the dictionary. Otherwise, returns NULL.
  return(region_nuts_dict[region_name])
}

and then for provinces and municipalities (🤣)

option 2. have a data file to refer to (I found this .csv) and then we may vlookup. This is for region, we may abstractize it for each aggregation level.

vlookup_geo_entity <- function(region) {
  # read in the data from the csv file
  nuts_data <- read.csv("<path_to_csv>")

  # find the row with the matching region name
  row <- which(nuts_data$region == region)

  # return the NUTS code for that row
  return(nuts_data$NUTS[row])

Is there a reason to use base R instead of dplyer's joins?

This needs to be handled on the backedn of the application, in the database. Data should come with these information and the package should only be able to call the right column!


⚠ Bofrost.de has degraded performance

In 1257845, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1408 ms

Resolved: Bofrost.de performance has improved in a162f47 after 1 hour, 19 minutes.


SoC Data Discrepancy

I'm trying to figure out this data discrepancy between the web interface and the Grafana interface. The web interface correctly shows 74% SoC while the Grafana board shows 78%.
Could this be a matter of bad database data? I don't mind clearing the data out completely to test that. Is there a simple way to do that, by the way?

No need to clear all data. That's a design decision. The web interface always shows the most up-to-date data whereas Grafana's purpose is reporting and visualizing historical data. That's why there is, by default, no real-time dashboard (similar the Tesla-apiscraper) built into Grafana.

Oh, I see. I missed that distinction.
If only there were a way to combine the two methodologies... 🤔

Haven't tested it, but it should be possible to create a Text / HTML Panel in Grafana that includes an iFrame with the web interface.

I was considering something along those lines.


Example does not display bubble

I have run the example on 3 devices, with Android 7, 5.0.2, and 4.4.4 and the bubble does not display (though the handle does). In Android Studio, I set a break point at CountriesAdapter.getSectionTitle() and it never gets there.

You grab the handle and it doesn't show? Maybe I have merged something weird...

Oh, my mistake. I was simply scrolling without grabbing the handle. It works when grabbing then handle. My apologies.


PoC: decreasing performance load if widget is out of screen

Simple checking if current widget being rendered is in render area to skip those are not in. Am I right that the single point where we can get width and height the area being rendered to is imguiBeginScrollArea?

Here I don't render the widgets that are out of area they are rendered to. But another way is do not even make commands at all, that is do the checking while commands generating in imguiCollapse, imguiButton, etc functions. I am not sure if the commands are useful to have them even they will be discarded in rendering phase.


error running job: listen ip4:tcp <IP_ADDRESS>: bind: An invalid argument was supplied

PS> go version
go version go1.17.6 windows/amd64

PS> git status
On branch main
Your branch is up to date with 'origin/main'.

nothing to commit, working tree clean

PS> git log --pretty=format:'%h' -n 1
9b6ad5e

--
..
2022/03/09 17:29:32.543983 runner.go:112: error running job: listen ip4:tcp <IP_ADDRESS>: bind: An invalid argument was supplied.
2022/03/09 17:29:32.551535 runner.go:112: error running job: listen ip4:tcp <IP_ADDRESS>: bind: An invalid argument was supplied.
2022/03/09 17:29:32.550650 runner.go:112: error running job: listen ip4:tcp <IP_ADDRESS>: bind: An invalid argument was supplied.
2022/03/09 17:29:32.552248 runner.go:112: error running job: listen ip4:tcp <IP_ADDRESS>: bind: An invalid argument was supplied.
...

unless you run it with sudo this is kinda to be expected, see packetgen reference here
you should be able to see in the logs how many jobs were started overall to be able to assess how much impact these errors have on the overall effectiveness of the program. The only thing I could do here is improve logging to better indicate that this is kinda expected but there are couple of factors currently that make me shift focus towards other features (if I manage to do them it'll be huge)

terminal is running under admin role

apparently, this feature is not supported in windows itself, see relevant issue for the go runtime https://github.com/golang/go/issues/6786


Fzf should provide a way to dynamically rebind reload to change event after unbinding it

[x] I have read through the manual page (man fzf)
[x] I have the latest version of fzf
[x] I have searched through the existing issues

Info

OS

[x] Linux
[ ] Mac OS X
[ ] Windows
[ ] Etc.

Shell

[ ] bash
[x] zsh
[ ] fish

Problem / Steps to reproduce
RG_PREFIX="rg --column --line-number --no-heading --color=always --smart-case "
INITIAL_QUERY="${*:-}"
IFS=: read -ra selected < <(
  FZF_DEFAULT_COMMAND="$RG_PREFIX $(printf %q "$INITIAL_QUERY")" \
  fzf --ansi \
      --color "hl:-1:underline,hl+:-1:underline:reverse" \
      --disabled --query "$INITIAL_QUERY" \
      --bind "change:reload:sleep 0.1; $RG_PREFIX {q} || true " \
      --bind "ctrl-f:unbind(change)+change-prompt(2. fzf> )+enable-search+clear-query" \
      --bind "ctrl-r:unbind(ctrl-f)+change-prompt(1. ripgrep> )+disable-search+clear-query,change:reload(sleep 0.1; $RG_PREFIX {q} || true)" \
      --prompt '1. ripgrep> ' \
      --delimiter : \
      --preview 'bat --color=always {1} --highlight-line {2}' \
      --preview-window 'up,60%,border-bottom,+{2}+3/3,~3'
)
[ -n "${selected[0]}" ] && $EDITOR "${selected[0]}" "+${selected[1]}"

I have been trying to dynamically reload entries from $RG_PREFIX with the ctrl-r keybinding but  to no avail after unbinding the change event from reloading with the ctrl+f binding
I can manually reload $RG_PREFIX by chaining it to the ctrl-r keybinding .ie ctrl-r:unbind(ctrl-f)+change-prompt(1. ripgrep> )+disable-search+reload(sleep 0.1; $RG_PREFIX {q} || true) but that means i will have to click ctrl-r  every time the query string changes which isn't idle ,flexible or efficient like have the change event automatically do the reload for me when the query string changes
So I request that fzf provides a way to do this , I believe this is a limitation that can be overcomed.
I am by no means a guru with Go but I'm welling to learn and help anyone who will be willing to help fix this issue.
This discussion talks about this issue well maybe better than I did Dicussion on a similar issue
Wonderful tool you guys have built,Keep the great work up. :+1:

Related: #2564

Added rebind action to master branch.
#!/usr/bin/env bash

RG_PREFIX="rg --column --line-number --no-heading --color=always --smart-case "
INITIAL_QUERY="${*:-}"
IFS=: read -ra selected < <(
  FZF_DEFAULT_COMMAND="$RG_PREFIX $(printf %q "$INITIAL_QUERY")" \
  fzf --ansi \
      --color "hl:-1:underline,hl+:-1:underline:reverse" \
      --disabled --query "$INITIAL_QUERY" \
      --bind "change:reload:sleep 0.1; $RG_PREFIX {q} || true" \
      --bind "ctrl-f:unbind(change,ctrl-f)+change-prompt(2. fzf> )+enable-search+clear-query+rebind(ctrl-r)" \
      --bind "ctrl-r:unbind(ctrl-r)+change-prompt(1. ripgrep> )+disable-search+reload($RG_PREFIX {q} || true)+rebind(change,ctrl-f)" \
      --prompt '1. ripgrep> ' \
      --delimiter : \
      --preview 'bat --color=always {1} --highlight-line {2}' \
      --preview-window 'up,60%,border-bottom,+{2}+3/3,~3'
)
[ -n "${selected[0]}" ] && $EDITOR "${selected[0]}" "+${selected[1]}"

Please test it and let me know if it works as expected.

@junegunn Thanks for taking the time to work on this. The rebind action works just as it should.


Both yarn.lock and package-lock.json

Hi guys, I haven't checked this project for a long time, and I notice that there are both yarn and npm lock file now. Maybe this is caused by different contributors' MR. Anyway, I don't think this is good, at least it's not concise(?).
How about use only yarn instead?

Hey @RyanLiu0235, good to hear from you again! Yeah good catch, the package-lock.json must have made it into the last release. I'll review you PR right now and get it merged in!

This was resolved by another PR! Going to close this, but if you have any questions/comments feel free to reopen or make a new issue :octocat:


Stop JSContext

Is there a way to stop the execution of JSContext from another thread after calling evaluateScript?

Not in the default API, no.  There is no interrupt mechanism exposed.  It is, however, possible to do in JNI.  See issue #65.
I'm curious, though, as this is not the first time this question has been asked.  What is the scenario in which you need this?  JavaScript is designed to run in short bursts in a single thread.  Why do you have code that can get stuck?  This seems more like a bug than a feature.

Basically it is a script that analyzes data. The execution times can vary between a few seconds and several minutes - based on how much data you want it to process.
So it is not a big, it is doing what it is supposed to, but it would be nice to stop the script if the user decides to cancel that operation.

Ok, so it is not stuck in an endless loop.  It would eventually exit, but you don't want to wait for it.  Got it.  I ask because I am debating whether it is a common enough use case to warrant exposing a method in the API.

I really would appreciate it!

this is highly appreciated. it might be odd but one of my use case is to run short burst js code coming from the database. but sometimes the js code could be coded poorly by the user so i would like to be able to watch the runtime, if exceeds a certain timeout, then stop the code, and moves on to the next row in the database.

This feature now exists in 0.6.1.  See:
JSContextGroup.terminateExecution().


Fix docker-shim flag handling

This problem was causing CNI misconfiguration and consequently Virtlet failure on multi-node clusters.

This change is

Thanks for the fix! Please just make the CI pass. Also the doc string at https://github.com/Mirantis/virtlet/blob/a8ade7d17df143ad5d41f0bcc5b3213f8df06d0e/pkg/dockershim/dockershim.go#L135 says InitFlags rather than initFlags. LGTM otherwise

Fixed docstring. CI problem was a usual Travis flake.

Review status: 0 of 1 files reviewed at latest revision, all discussions resolved.

Comments from Reviewable

Reviewed 1 of 1 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


setFixedServiceURL() prevents getURL() from removing the QS parameters

My CAS client webapp (redirect.php) is behind a reverse proxy.
Its public URL is different from its internal one. Moreover, the public URL is under https but its internal is under http.
In brief, getURL() can't guess the right public URL to validate the ST, so I set the client webapp public URL with phpCAS::setFixedURL($public_URL_value).
Since the requested public URL has some parameters in its query string, it is slightly different from the fixed URL set.
Example :
requested : https://ego.iut-rodez.fr/caslogger/redirect.php?_cLt=eughvlgv
fixed : https://ego.iut-rodez.fr/caslogger/redirect.php
It seems that, if a fixed URL is set, getURL() doesn't remove the query string what then prevents the ST to be validated.
Log example :
1CD0 .=> phpCAS::setFixedServiceURL('https://ego.iut-rodez.fr/caslogger/redirect.php') [redirect.php:30]
1CD0 .<= ''
1CD0 .=> phpCAS::setNoCasServerValidation() [redirect.php:31]
1CD0 .|    You have configured no validation of the legitimacy of the cas server. This is not recommended for production use.   The original service was 'https://ego.iut-rodez.fr/caslogger/redirect.php?_cLt=7HHmTeinEw&_cLs=http://www.iut-rodez.fr/' and the supplied service was 'https://ego.iut-rodez.fr/caslogger/redirect.php'.
1CD0 .|    |    |    |    |    |    	</cas:authenticationFailure>
1CD0 .|    |    |    |    |    |    </cas:serviceResponse>
1CD0 .|    |    |    |    |    |     [CurlRequest.php:82]
1CD0 .|    |    |    |    |    <= true
1CD0 .|    |    |    |    <= true
1CD0 .|    |    |    |    => CAS_AuthenticationException::__construct(CAS_Client, 'Ticket not validated', 'https://cas2.iut-rodez.fr/cas/serviceValidate?service=https%3A%2F%2Fego.iut-rodez.fr%2Fcaslogger%2Fredirect.php&ticket=ST-834-9kAtXDWynCHPT4T5xItP-cas2.iut.rdz', false, false, '<cas:serviceResponse xmlns:cas=\'http://www.yale.edu/tp/cas\'>	<cas:authenticationFailure code=\'INVALID_SERVICE\'>		ticket 'ST-834-9kAtXDWynCHPT4T5xItP-cas2.iut.rdz' does not match supplied service.  The original service was 'https://ego.iut-rodez.fr/caslogger/redirect.php?_cLt=7HHmTeinEw&_cLs=http://www.iut-rodez.fr/' and the supplied service was 'https://ego.iut-rodez.fr/caslogger/redirect.php'.	</cas:authenticationFailure></cas:serviceResponse>', 'INVALID_SERVICE', 'ticket \'ST-834-9kAtXDWynCHPT4T5xItP-cas2.iut.rdz\' does not match supplied service.  The original service was \'https://ego.iut-rodez.fr/caslogger/redirect.php?_cLt=7HHmTeinEw&_cLs=http://www.iut-rodez.fr/\' and the supplied service was \'https://ego.iut-rodez.fr/caslogger/redirect.php\'.') [Client.php:2845]
1CD0 .|    |    |    |    |    => CAS_Client::getURL() [AuthenticationException.php:76]
1CD0 .|    |    |    |    |    <= 'https://ego.iut-rodez.fr/caslogger/redirect.php'
1CD0 .|    |    |    |    |    CAS URL: https://cas2.iut-rodez.fr/cas/serviceValidate?service=https%3A%2F%2Fego.iut-rodez.fr%2Fcaslogger%2Fredirect.php&ticket=ST-834-9kAtXDWynCHPT4T5xItP-cas2.iut.rdz [AuthenticationException.php:79]
1CD0 .|    |    |    |    |    Authentication failure: Ticket not validated [AuthenticationException.php:80]
1CD0 .|    |    |    |    |    Reason: [INVALID_SERVICE] CAS error: ticket 'ST-834-9kAtXDWynCHPT4T5xItP-cas2.iut.rdz' does not match supplied service.  The original service was 'https://ego.iut-rodez.fr/caslogger/redirect.php?_cLt=7HHmTeinEw&_cLs=http://www.iut-rodez.fr/' and the supplied service was 'https://ego.iut-rodez.fr/caslogger/redirect.php'. [AuthenticationException.php:95]
1CD0 .|    |    |    |    |    CAS response: <cas:serviceResponse xmlns:cas='http://www.yale.edu/tp/cas'>
1CD0 .|    |    |    |    |    	<cas:authenticationFailure code='INVALID_SERVICE'>
1CD0 .|    |    |    |    |    		ticket 'ST-834-9kAtXDWynCHPT4T5xItP-cas2.iut.rdz' does not match supplied service.  The original service was 'https://ego.iut-rodez.fr/caslogger/redirect.php?_cLt=7HHmTeinEw&_cLs=http://www.iut-rodez.fr/' and the supplied service was 'https://ego.iut-rodez.fr/caslogger/redirect.php'.
1CD0 .|    |    |    |    |    	</cas:authenticationFailure>
1CD0 .|    |    |    |    |    </cas:serviceResponse> [AuthenticationException.php:100]
1CD0 .|    |    |    |    |    exit()
1CD0 .|    |    |    |    |    -
1CD0 .|    |    |    |    -
1CD0 .|    |    |    -
1CD0 .|    |    -
1CD0 .|    -

Same problem with phpCAS1.3.2 and 1.3.4, didn't test with 1.3.3

phpcas should be able to deal with reverse proxied setups if the X_Forwarded Headers are properly set. The are evaluated by phpCAS. However some proxies do not properly set protocol and host infos.
I will check if I can spot a bug in regards the fixed URL.

Ok
Actually what I meant is that the behavior of getURL() differs in terms of query string management if the fixed URL is set or not.

fixed URL set => getURL() returns the fixed URL and don't care about removing the submitted URL query string before comparing them
no fixed URL => getURL() try to guess the host URL and removes all the QS to compare the guessed host URL and the submitted URL

Ad a side note, my reverse proxy sets correctly the x-forwarded-for header

Thanks for the details. With this info I can whip up a patch ;-)
A lot of people struggle because default proxy configs don't set the https protocol info or the loadbalancer use some vendor specific flags. (see othe bug for details: https://github.com/Jasig/phpCAS/issues/178) If all this is properly set up one should not need any fixed urls unless you have a very strange use case. At least this is the goal of the current code ;-)

When we set a fixed URL, we can't predict all the incoming parameters with (the query string). That's why the fixed URL is for instance necesseraly like :
https://myapp.domain/path/to/service

and not like
https://myapp.domain/path/to/service?this_fixed_query_string

I'm not expert with the CAS protocol, but imho the fixed URL should be compared the submitted URL without its query string.
Anyway, thank you for your time ;-)

I reviewed you debug log again. The problem is that the original ticket was requested with a service URL that was something like "https://ego.iut-rodez.fr/caslogger/redirect.php?_cLt=7HHmTeinEw&_cLs=http://www.iut-rodez.fr/". The comparison that goes wrong is CAS server sided. Your cas client requested and received the ticket with another URL that you are using during verification. The is checked server sided and results from a broken client setup. Please have a look at the debug log from first redirect to authentication and check the service URLs used.
I still have the opinion that a fixed URL is the wrong aproach and will create more issues. Please add some debug trace to the _getURL() and _isHttps(). Just add some phpCAS::trace('Parameter: '.$param); that dump all server variables to the debug log. This will enable you to find out where the url construction goes wrong. I'm pretty sure your proxy does not properly forward host, port or protocol. phpcas itself is very smart about URL generation.

Ok thank you
I will have a check about it (after next week, because I'll attend a conf)
I still think the problem is because of the fixed URL, because if I set no fixed URL and an easily guessable URL for getURL() with the same ticket and exchanges between the client and the CAS server, then the ticket is validated.
I will follow your advices for tracing more accurately and check if the reverse proxy sets really the x-forwarded-proto (I'm sure for ip and port). This will allow to avoid then the usage of the fixed URL.
Thanks again.

Hi,
I encountered the same kind of problem with a php web application (DokuWiki) and PhpCAS.
The problem was in fact à misconfiguration in my proxy virtual host and I could fix it.
I post my working configuration here :
https://github.com/Jasig/phpCAS/issues/178
If it could help.
Regards,

/
/-------------------------------------
|  Guy CARRÉ                          |
|  *************                      |
|  PostMaster - WikiMaster - SysAdmin |
|                                     |
| "Free Your Mind. Think Open Source" |
|  april.org                          |
|                                     |
|_____________________________________|
----- Mail original -----
De: "Jérôme Bousquié"<EMAIL_ADDRESS>À: "Jasig/phpCAS"<EMAIL_ADDRESS>Envoyé: Lundi 7 Décembre 2015 20:03:18
Objet: Re: [phpCAS] setFixedServiceURL() prevents getURL() from removing the QS parameters (#181)
Ok thank you
I will have a check about it (after next week, because I'll attend a conf)
I still think the problem is because of the fixed URL, because if I set no fixed URL and an easily guessable URL for getURL() with the same ticket and exchanges between the client and the CAS server, then the ticket is validated.
I will follow your advices for tracing more accurately and check if the reverse proxy sets really the x-forwarded-proto (I'm sure for ip and port). This will allow to avoid then the usage of the fixed URL.
Thanks again.
—
Reply to this email directly or view it on GitHub .

Ok thank you.
I'll check as aoon as I'm back from my conf.
I guess the same Apache directive than yours is missing on in my reverse proxy settings.
Thank both for your time.

Back.
I checked your suggestion. This works really well.
Thank you guys :-)
You were right : the fixed URL approach doesn't fit the URL with some variable parameters in the query string.


STORM-2251 Integration test refers specific version of Storm which should be project version

fix integration-test's pom.xml to have root pom as parent, and remove version and storm version

+1

+1


return extra lun info and allow create with direct_io_pattern

Great project! I've started to use the dsm/webapi module for a project of my own - a CLI for Synology iSCSI storage which you can see here.
Two features I would love to have for my CLI are:

Returning the DevAttribs along with the LUN info. My CLI allows creating a LUN with extra features like space reclamation and FUA/Sync cache. These are dev attributes which are already supported in the LunCreateSpec. However I can't display whether an existing LUN has these enabled or not since the dev attributes are not returned by the get/list methods.
Specify the I/O Policy during LUN creation. In the DSM webapp, you can see this option when editing a LUN, as shown in the first image below. I've tested it and this parameter can be passed along during LUN creation as well. By default, a LUN is created with a policy of 0 which maps to Buffered I/O. You can see this in the DSM SYNO.Core.ISCSI.LUN API response from the second image below. A value of 3 maps to the Direct I/O option.

This small PR adds both these features.

@chihyuwu Could you please look at this, I hope it will gets merged soon.
@pfrybar Could you rebase, so the repository maintainer can merge it, I hope it will get merged soon.


[Feature Request] Force spawn on join

I'd love to use this on my lobby where I'll be having different worlds and people spawn in random ones but I can't really do this with this plugin as I can't force spawn them when they join, so I thought I'd request that feature exactly.
So just a config setting that says forceSpawnOnJoin or something and when I set it to true, people will always be sent to one of the many random spawns when they join.
A permission to bypass it would perhaps also be cool, just in case, although I personally don't need it.

Added (:


rpc_support: Remove old host packages from containers

Closes-Bug: #845

How long will this task be around?  How long should cleanup tasks like this stick around?

I would guess it will be around for quite some time, probably for the lifetime of kilo and we can remove of for liberty

If that's the case, please close this bug and reopen one against kilo, master is liberty. :D


WIP: Adapting the cfssl download to work also with the other architectures and improving the travis.yml

This pull request implements:

A new way of downloading the correct binary for each architecture for the cfssl library
Improving the the travis.yml. In the previous approach the UT were run in a after_success step (https://docs.travis-ci.com/user/job-lifecycle/#breaking-the-build). That step will NOT mark the build as failed if the UT fail. Moreover the deploy procedure is trigger now only in the case that the commits are coming from the osixia/docker-light-baseimage repo.

OLD Description:
While trying to make the docker-openldap a multi arch image, I realised that we were downloading a x86_64 binary that was not running on the aarch machines (e.g. raspberry).
So this pull request implements a suggestion to solve this limitation.
With this pull request the source code of the cfssl library is being downloaded and compiled as a prebuild step. Then the produced binaries are included in the final image. (Currently I am using the official repo to download the source code, but that can be changed. I did not see any changes in the osixia repo, and I guessed that it is there only for creating and releasing the binaries).
Furthermore, I improved a little bit the travis.yml. In the previous approach the UT were run in a after_success step (https://docs.travis-ci.com/user/job-lifecycle/#breaking-the-build). That step will NOT mark the build as failed if the UT fail. Moreover the deploy procedure is trigger now only in the case that the commits are coming from the osixia/docker-light-baseimage repo.
I hope it is useful.
If something is not clear (or wrong), please let me know and I will try to explain it (fix it) :)

this image aims to be as light as possible,
i'll prefer dowload cfssl when needed and add all cfssl bin available in https://github.com/osixia/cfssl releases. Not sure what is the best way to achieve that. Any help welcomed :)

So the cfssl library is being download only and only if the user is using it. Did I get it right ?
If yes, then I can think a couple of choices:

Download the source code and compile it on demand (this of course will need some time when being executed).
use the uname -a (or the arch) command from bash to extract the correct ARCH and then download the correct binary from the repo (but of course we will have to precompile the binaries for all platforms -- Go supports that so it should not be problem).
Use the debian package (https://packages.debian.org/sid/devel/golang-cfssl).
Do not support cfssl on other architectures.

Which solution do you prefer/think it makes more sense?
KR,
Vasileios
P.S.: Disregard then also my PR in openldap image (https://github.com/osixia/docker-openldap/pull/333) :)

thank you very mutch,
i think we should first try cfssl precompiled for all plateforms, it was pretty fast, i hope the binaries are ok :
https://github.com/osixia/cfssl/releases
will try to adapt the download script accordingly

If you prefer I can also do it (but probably over the weekend :) ) . Now that you upload the pre compiled binaries it is quite easy !

This work in progress (sorry for the spam). Once it is ready I will leave a message :)

Hello @BertrandGouny ,
I think this one is ready. I also tested it with the openldap image and with the exception of UT 7, which only succeeds in amd64 (e.g. https://travis-ci.org/anagno/docker-openldap/jobs/560681136#L1031) everything is green. I believe that reason behind this incompatibility is that the test files were generated in an amd64 machine. I will try and investigate further and find out the reason of this strange behaviour. What do you think?
KR,
Vasileios

Introduced in 86392ca0c28e8776cd3c2dc28477b62b9fc8f484


[Identity] Doc formatting fixes

Fixed a few issues with docs not being rendered as expected on the api docs website.

ClientAssertionCredential docstring not showing.
TokenCachePersistenceOptions not being rendered properly.

Added a note in the DAC troubleshooting section for users who are directed there from an error regarding additionally_allowed_tenants configuration.

API change check
API changes are not detected in this pull request.


error: Unable to fetch any events from nvidia-smi: Error read |0: file already closed

Can someone help in solving the issue

Full stack trace and logs when you run nvidiagpubeat executable. ( Enclose in three back ticks otherwise logs will be unreadable )
What branch did you build.
Details on your environment.
Share the output of nvidia-smi command
What is output of ls /dev | grep nvidia | grep -v nvidia-uvm | grep -v nvidiactl | wc -l
What is output of nvidiagpubeat --query-gpu=utilization.gpu,utilization.memory,memory.total,memory.free,memory.used,temperature.gpu,pstate --format=csv

Full stack trace

2019-09-04T21:08:38.188+0530    INFO    instance/beat.go:607    Home path: [/home/ashwin/Downloads/beats_dev/src/github.com/ebay/nvidiagpubeat] Config path: [/home/ashwin/Downloads/beats_dev/src/github.com/ebay/nvidiagpubeat] Data path: [/home/ashwin/Downloads/beats_dev/src/github.com/ebay/nvidiagpubeat/data] Logs path: [/home/ashwin/Downloads/beats_dev/src/github.com/ebay/nvidiagpubeat/logs]
2019-09-04T21:08:38.188+0530    DEBUG   [beat]  instance/beat.go:659    Beat metadata path: /home/ashwin/Downloads/beats_dev/src/github.com/ebay/nvidiagpubeat/data/meta.json
2019-09-04T21:08:38.188+0530    INFO    instance/beat.go:615    Beat ID: 68386e1f-0080-4249-ae78-5278a46d79ac
2019-09-04T21:08:38.189+0530    INFO    [beat]  instance/beat.go:903    Beat info       {"system_info": {"beat": {"path": {"config": "/home/ashwin/Downloads/beats_dev/src/github.com/ebay/nvidiagpubeat", "data": "/home/ashwin/Downloads/beats_dev/src/github.com/ebay/nvidiagpubeat/data", "home": "/home/ashwin/Downloads/beats_dev/src/github.com/ebay/nvidiagpubeat", "logs": "/home/ashwin/Downloads/beats_dev/src/github.com/ebay/nvidiagpubeat/logs"}, "type": "nvidiagpubeat", "uuid": "68386e1f-0080-4249-ae78-5278a46d79ac"}}}
2019-09-04T21:08:38.189+0530    INFO    [beat]  instance/beat.go:912    Build info      {"system_info": {"build": {"commit": "unknown", "libbeat": "7.3.2", "time": "1754-08-30T22:43:41.128Z", "version": "7.3.2"}}}
2019-09-04T21:08:38.189+0530    INFO    [beat]  instance/beat.go:915    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.12.9"}}}
2019-09-04T21:08:38.192+0530    INFO    [beat]  instance/beat.go:919    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-09-03T20:40:11+05:30","containerized":false,"name":"linux-d4hc","ip":["<IP_ADDRESS>/8","::1/128","<IP_ADDRESS>/24","2405:201:e806:9f60:29d1:864b:af2b:f9f0/64","2405:201:e806:9f60:7a45:61ff:fec0:c319/64","fe80::7a45:61ff:fec0:c319/64"],"kernel_version":"4.12.14-lp151.27-default","mac":["c8:5b:76:68:99:f7","78:45:61:c0:c3:19"],"os":{"family":"","platform":"opensuse-leap","name":"openSUSE Leap","version":"15.1","major":15,"minor":1,"patch":0},"timezone":"IST","timezone_offset_sec":19800,"id":"1ae32b0454884a1cac7ab936ce597373"}}}
2019-09-04T21:08:38.193+0530    INFO    [beat]  instance/beat.go:948    Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/home/ashwin/Downloads/beats_dev/src/github.com/ebay/nvidiagpubeat", "exe": "/home/ashwin/Downloads/beats_dev/src/github.com/ebay/nvidiagpubeat/nvidiagpubeat", "name": "nvidiagpubeat", "pid": 2913, "ppid": 27508, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2019-09-04T21:08:37.370+0530"}}}
2019-09-04T21:08:38.193+0530    INFO    instance/beat.go:292    Setup Beat: nvidiagpubeat; Version: 7.3.2
2019-09-04T21:08:38.194+0530    DEBUG   [beat]  instance/beat.go:318    Initializing output plugins
2019-09-04T21:08:38.194+0530    INFO    [index-management]      idxmgmt/std.go:178      Set output.elasticsearch.index to 'nvidiagpubeat-7.3.2' as ILM is enabled.
2019-09-04T21:08:38.194+0530    INFO    elasticsearch/client.go:170     Elasticsearch url: http://localhost:9200
2019-09-04T21:08:38.195+0530    DEBUG   [publisher]     pipeline/consumer.go:137        start pipeline event consumer
2019-09-04T21:08:38.195+0530    INFO    [publisher]     pipeline/module.go:97   Beat name: linux-d4hc
2019-09-04T21:08:38.196+0530    INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2019-09-04T21:08:38.196+0530    INFO    instance/beat.go:422    nvidiagpubeat start running.
2019-09-04T21:08:38.196+0530    INFO    beater/nvidiagpubeat.go:57      nvidiagpubeat is running for ** test ** environment. ! Hit CTRL-C to stop it.
2019-09-04T21:08:39.205+0530    ERROR   beater/nvidiagpubeat.go:75      Event not generated, error: Unable to fetch any events from nvidia-smi: Error read |0: file already closed
2019-09-04T21:08:40.207+0530    ERROR   beater/nvidiagpubeat.go:75      Event not generated, error: Unable to fetch any events from nvidia-smi: Error read |0: file already closed
^C2019-09-04T21:08:41.178+0530  DEBUG   [service]       service/service.go:53   Received sigterm/sigint, stopping
2019-09-04T21:08:41.178+0530    DEBUG   [publisher]     pipeline/client.go:149  client: closing acker
2019-09-04T21:08:41.178+0530    DEBUG   [publisher]     pipeline/client.go:151  client: done closing acker
2019-09-04T21:08:41.178+0530    DEBUG   [publisher]     pipeline/client.go:155  client: cancelled 0 events
2019-09-04T21:08:41.184+0530    INFO    [monitoring]    log/log.go:153  Total non-zero metrics  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":20,"time":{"ms":24}},"total":{"ticks":60,"time":{"ms":64},"value":60},"user":{"ticks":40,"time":{"ms":40}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":5},"info":{"ephemeral_id":"f413aa9b-3ef6-4b77-998e-6e1f39166bc3","uptime":{"ms":3010}},"memstats":{"gc_next":4194304,"memory_alloc":1289056,"memory_total":3070424,"rss":23531520},"runtime":{"goroutines":8}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":4},"load":{"1":1.65,"15":2.23,"5":2.1,"norm":{"1":0.4125,"15":0.5575,"5":0.525}}}}}}
2019-09-04T21:08:41.185+0530    INFO    [monitoring]    log/log.go:154  Uptime: 3.016770154s
2019-09-04T21:08:41.185+0530    INFO    [monitoring]    log/log.go:131  Stopping metrics logging.
2019-09-04T21:08:41.185+0530    INFO    instance/beat.go:432    nvidiagpubeat stopped.

Beats branch  : 7.3
nvidiagpubeat branch  : withBeats7.3

OS: openSUSE Leap 15.1

ashwin@linux-d4hc:~> nvidia-smi
If 'nvidia-smi' is not a typo you can use command-not-found to lookup the package that contains it, like this:
    cnf nvidia-smi

ashwin@linux-d4hc:~> ls /dev | grep nvidia | grep -v nvidia-uvm | grep -v nvidiactl | wc -l
0

ashwin@linux-d4hc:~/Downloads/beats_dev/src/github.com/ebay/nvidiagpubeat> nvidiagpubeat --query-gpu=utilization.gpu,utilization.memory,memory.total,memory.free,memory.used,temperature.gpu,pstate --format=csv
Error: unknown flag: --query-gpu
Usage:
  nvidiagpubeat [flags]
  nvidiagpubeat [command]

Available Commands:
  export      Export current config or index template
  help        Help about any command
  keystore    Manage secrets keystore
  run         Run nvidiagpubeat
  setup       Setup index template, dashboards and ML jobs
  test        Test config
  version     Show current version info

Flags:
  -E, --E setting=value      Configuration overwrite
  -N, --N                    Disable actual publishing for testing
  -c, --c string             Configuration file, relative to path.config (default "nvidiagpubeat.yml")
      --cpuprofile string    Write cpu profile to file
  -d, --d string             Enable certain debug selectors
  -e, --e                    Log to stderr and disable syslog/file output
  -h, --help                 help for nvidiagpubeat
      --httpprof string      Start pprof http server
      --memprofile string    Write memory profile to this file
      --path.config string   Configuration path
      --path.data string     Data path
      --path.home string     Home path
      --path.logs string     Logs path
      --plugin pluginList    Load additional plugins
      --strict.perms         Strict permission checking on config files (default true)
  -v, --v                    Log at INFO level

Use "nvidiagpubeat [command] --help" for more information about a command.

#4 indicates nvidia-smi is not in PATH or not installed at all. nvidia-smi is NVIDIA GPU driver that can collect metrics from gpu cards.
I can add checks and throw appropriate error message, if this is the root cause of this issue.
#3 indicates 0 GPU Cards, on the current machine.
#5 was my typo.  Can you run
nvidia-smi --query-gpu=utilization.gpu,utilization.memory,memory.total,memory.free,memory.used,temperature.gpu,pstate --format=csv
Error: unknown flag: --query-gpu```

I noticed this from your logs
2019-09-04T21:08:38.196+0530    INFO    instance/beat.go:422    nvidiagpubeat start running.
2019-09-04T21:08:38.196+0530    INFO    beater/nvidiagpubeat.go:57      nvidiagpubeat is running for ** test ** environment. ! Hit CTRL-C to stop it.

And you are running on Suse Linux.
https://github.com/eBay/nvidiagpubeat#run-in-test-environment-macos indicates that "test" mode is supported on MacOS. Test mode uses localnvidiasmi and that executable is built on and for MacOS.

Do you want to work on https://github.com/eBay/nvidiagpubeat/issues/25 ? It will fix current issue.

Thanks deepujain. Installing nvidia-smi solved the issue


Support sessionStorage for persistence_type

Hello,
I see that Mixpanel supports localStorage and Cookie for persistence_type.
As known, localStorage and Cookies are shared between tabs, which can effect the expected behavior of the application, one tab can change the super properties of the other tab.
Supporting sessionStorage will enable the independency of the super properties keeping them encapsulated for that tab.
The shared resource between the tabs caused serious issues in our application that I had to solve by created my own super properties...

Interesting. For most applications, it's desirable to share super properties across tabs. But sessionStorage and localStorage are API-compatible, so it shouldn't be too much work to offer it as a persistence option for those who need it.

I found another issue in our system because of this...
"time_event" also stored in the cookies, different tabs with same event name will have wrong duration values...
How can we sort this issues quickly?


How to exclude an object property

Considering the following:
const OBJ = {
  a: [],
  b: [],
  c: [],
  d: []
}

// JSHint options
globals: {
  OBJ: false
}

How can a property (e.g. OBJ.c) be excluded so that is would generate an undefined error?

If you want to create an OBJ object with an unaccessible c property, you need to define a Proxy or something like that. It's not tied to JSDoc.
However, if you would like to exclude it from just JSDoc, you can just use
/**@type {{a:*[], b:*[], d:*[]}} */
const OBJ= { a:[], b:[], c:[], d:[]};

The globals option is used for outer globals, such as defining a script loader in a browser enviroment.
"globals":{"goog":false}
Check JSDoc Docs for more details.

JSHint isn't well-situated to track the runtime type of the bindings in your code. If you'd like to enforce restrictions like this, you might consider using Flow or TypeScript.

JSHint isn't well-situated to track the runtime type of the bindings in your code. If you'd like to enforce restrictions like this, you might consider using Flow or TypeScript.


Add some --help or help command to show usage options

I know there are some haskell libreries that do that automatically like optparse-applicative. But since you have quite a custom handling of args maybe that's not the best option.
BTW nice project :smiley:

I totally forgot about this, I will check how to do. Thank you :)

Added with 40f928e, should be in the next release!


Task #4111576: Add TFS upgrade script

What
Add an upgrade script to the TFS plugin.
Why ?
_https://redmine.mellanox.com/issues/4111576_
How ?
Add merge_configuration_files.py to merge user and new configurations (The file was added to ufm_sdk_tools).
Add upgrade.sh script that calls the Python script.
Add upgrade.sh to the TFS plugin container.
Testing ?
merge_configuration_files.py was tested manually.
Special triggers
Use the following phrases as comments to trigger different runs

bot:retest rerun Jenkins CI (to rerun GitHub CI, use "Checks" tab on PR page and rerun all jobs)
bot:upgrade run additional update tests

bot:retest

bot:retest

bot:retest


perf(es/lexer): Use jump table for skip_space

Description:
I'll compare performance with simd
Related issue (if exists):

This makes parser about 3% faster