Compiling issue cChoco module

Hello,
We are trying to compile the cChoco module (https://github.com/PowerShellOrg/cChoco). But we're having issue's doing that (yes I know puppet also supports Chocolatey, but we want to do it the DSC way).
CentOS Linux release 7.3.1611 (Core)
ruby 2.3.3p222 (2016-11-21 revision 56859) [x86_64-linux]
The steps that we took:
In the following directory:
[root@puppetmaster02 custom]# pwd
/etc/puppetlabs/code/environments/production/modules/dsc/build/vendor/custom

git clone https://github.com/PowerShellOrg/cChoco.git
verified that the directory name and the PSD1 filename are the same.
Directory structure looks like this:

[root@puppetmaster02 cChoco]# ls -l
total 32
drwxr-xr-x. 2 root root 4096 Feb 20 14:53 AppVeyor
-rw-r--r--. 1 root root  914 Feb 20 14:53 appveyor.yml
-rw-r--r--. 1 root root  495 Feb 20 14:53 cChoco.psd1
drwxr-xr-x. 6 root root 4096 Feb 20 14:53 DSCResources
-rw-r--r--. 1 root root 1580 Feb 20 14:53 ExampleConfig.ps1
drwxr-xr-x. 2 root root 4096 Feb 20 14:53 Examples
-rw-r--r--. 1 root root 2202 Feb 20 14:53 readme.md
drwxr-xr-x. 2 root root 4096 Feb 20 14:53 Tests
[root@puppetmaster02 cChoco]# pwd
/etc/puppetlabs/code/environments/production/modules/dsc/build/vendor/custom/cChoco

after making sure all the files are in place we did the following:

4a) in folder:
[root@puppetmaster02 build]# pwd
/etc/puppetlabs/code/environments/production/modules/dsc/build
4b) runned the following command
bundle exec rake dsc:resources:import["build/vendor/custom"]
4c) good results!
[root@puppetmaster02 build]# bundle exec rake dsc:resources:import["build/vendor/custom"]
(in /etc/puppetlabs/code/environments/production/modules/dsc)
Importing custom types from '/etc/puppetlabs/code/environments/production/modules/dsc/build/vendor/custom'
Copying vendored resources from /etc/puppetlabs/code/environments/production/modules/dsc/build/vendor/custom to /etc/puppetlabs/code/environments/production/modules/dsc/lib/puppet_x/dsc_resources
Adding custom types to '/etc/puppetlabs/code/environments/production/modules/dsc/import/dsc_resources'

now we need to rebuild the module with the following command:
bundle exec rake dsc:types:build

5a)
[root@puppetmaster02 build]# pwd
/etc/puppetlabs/code/environments/production/modules/dsc/build
5b)
run the command: bundle exec rake dsc:types:build
5c)
We're getting a lot of things now. But at the end we have the following:
open /etc/puppetlabs/code/environments/production/modules/dsc/import/dsc_resources/xWindowsUpdate/DscResources/MSFT_xMicrosoftUpdate/MSFT_xMicrosoftUpdate.schema.mof [pragma]
open /etc/puppetlabs/code/environments/production/modules/dsc/import/dsc_resources/xWindowsUpdate/DscResources/MSFT_xWindowsUpdate/MSFT_xWindowsUpdate.schema.mof [pragma]
open /etc/puppetlabs/code/environments/production/modules/dsc/import/dsc_resources/xWindowsUpdate/DscResources/MSFT_xWindowsUpdateAgent/MSFT_xWindowsUpdateAgent.schema.mof [pragma]
open /etc/puppetlabs/code/environments/production/modules/dsc/import/dsc_resources/xWordPress/DscResources/MSFT_xWordPressSite/MSFT_xWordPressSite.schema.mof [pragma]
rake aborted!
NoMethodError: undefined method empty?' for nil:NilClass /etc/puppetlabs/code/environments/production/modules/dsc/build/dsc/psmodule.rb:13:in version'
(erb):81:in block (2 levels) in build_dsc_types' /etc/puppetlabs/code/environments/production/modules/dsc/build/dsc/manager.rb:187:in block (2 levels) in build_dsc_types'
/etc/puppetlabs/code/environments/production/modules/dsc/build/dsc/manager.rb:186:in open' /etc/puppetlabs/code/environments/production/modules/dsc/build/dsc/manager.rb:186:in block in build_dsc_types'
/etc/puppetlabs/code/environments/production/modules/dsc/build/dsc/manager.rb:178:in each' /etc/puppetlabs/code/environments/production/modules/dsc/build/dsc/manager.rb:178:in build_dsc_types'
/etc/puppetlabs/code/environments/production/modules/dsc/build/dsc.rake:228:in block (3 levels) in <top (required)>' /usr/local/rvm/gems/ruby-2.3.3/bin/ruby_executable_hooks:15:in eval'
/usr/local/rvm/gems/ruby-2.3.3/bin/ruby_executable_hooks:15:in `'
Tasks: TOP => dsc:types:build
(See full trace by running task with --trace)
I've runned the task with the --trace and then we have the following output
open /etc/puppetlabs/code/environments/production/modules/dsc/import/dsc_resources/xWindowsUpdate/DscResources/MSFT_xWindowsUpdate/MSFT_xWindowsUpdate.schema.mof [pragma]
open /etc/puppetlabs/code/environments/production/modules/dsc/import/dsc_resources/xWindowsUpdate/DscResources/MSFT_xWindowsUpdateAgent/MSFT_xWindowsUpdateAgent.schema.mof [pragma]
open /etc/puppetlabs/code/environments/production/modules/dsc/import/dsc_resources/xWordPress/DscResources/MSFT_xWordPressSite/MSFT_xWordPressSite.schema.mof [pragma]
rake aborted!
NoMethodError: undefined method empty?' for nil:NilClass /etc/puppetlabs/code/environments/production/modules/dsc/build/dsc/psmodule.rb:13:in version'
(erb):81:in block (2 levels) in build_dsc_types' /usr/local/rvm/rubies/ruby-2.3.3/lib/ruby/2.3.0/erb.rb:864:in eval'
/usr/local/rvm/rubies/ruby-2.3.3/lib/ruby/2.3.0/erb.rb:864:in result' /etc/puppetlabs/code/environments/production/modules/dsc/build/dsc/manager.rb:187:in block (2 levels) in build_dsc_types'
/etc/puppetlabs/code/environments/production/modules/dsc/build/dsc/manager.rb:186:in open' /etc/puppetlabs/code/environments/production/modules/dsc/build/dsc/manager.rb:186:in block in build_dsc_types'
/etc/puppetlabs/code/environments/production/modules/dsc/build/dsc/manager.rb:178:in each' /etc/puppetlabs/code/environments/production/modules/dsc/build/dsc/manager.rb:178:in build_dsc_types'
/etc/puppetlabs/code/environments/production/modules/dsc/build/dsc.rake:228:in block (3 levels) in <top (required)>' /usr/local/rvm/gems/ruby-2.3.3/gems/rake-10.5.0/lib/rake/task.rb:240:in block in execute'
/usr/local/rvm/gems/ruby-2.3.3/gems/rake-10.5.0/lib/rake/task.rb:235:in each' /usr/local/rvm/gems/ruby-2.3.3/gems/rake-10.5.0/lib/rake/task.rb:235:in execute'
/usr/local/rvm/gems/ruby-2.3.3/gems/rake-10.5.0/lib/rake/task.rb:179:in block in invoke_with_call_chain' /usr/local/rvm/rubies/ruby-2.3.3/lib/ruby/2.3.0/monitor.rb:214:in mon_synchronize'
/usr/local/rvm/gems/ruby-2.3.3/gems/rake-10.5.0/lib/rake/task.rb:172:in invoke_with_call_chain' /usr/local/rvm/gems/ruby-2.3.3/gems/rake-10.5.0/lib/rake/task.rb:165:in invoke'
/usr/local/rvm/gems/ruby-2.3.3/gems/rake-10.5.0/lib/rake/application.rb:150:in invoke_task' /usr/local/rvm/gems/ruby-2.3.3/gems/rake-10.5.0/lib/rake/application.rb:106:in block (2 levels) in top_level'
/usr/local/rvm/gems/ruby-2.3.3/gems/rake-10.5.0/lib/rake/application.rb:106:in each' /usr/local/rvm/gems/ruby-2.3.3/gems/rake-10.5.0/lib/rake/application.rb:106:in block in top_level'
/usr/local/rvm/gems/ruby-2.3.3/gems/rake-10.5.0/lib/rake/application.rb:115:in run_with_threads' /usr/local/rvm/gems/ruby-2.3.3/gems/rake-10.5.0/lib/rake/application.rb:100:in top_level'
/usr/local/rvm/gems/ruby-2.3.3/gems/rake-10.5.0/lib/rake/application.rb:78:in block in run' /usr/local/rvm/gems/ruby-2.3.3/gems/rake-10.5.0/lib/rake/application.rb:176:in standard_exception_handling'
/usr/local/rvm/gems/ruby-2.3.3/gems/rake-10.5.0/lib/rake/application.rb:75:in run' /usr/local/rvm/gems/ruby-2.3.3/gems/rake-10.5.0/bin/rake:33:in <top (required)>'
/usr/local/rvm/gems/ruby-2.3.3/bin/rake:22:in load' /usr/local/rvm/gems/ruby-2.3.3/bin/rake:22:in '
/usr/local/rvm/gems/ruby-2.3.3/bin/ruby_executable_hooks:15:in eval' /usr/local/rvm/gems/ruby-2.3.3/bin/ruby_executable_hooks:15:in '
Are we doing something wrong? :-)

But we're having issue's doing that (yes I know puppet also supports Chocolatey, but we want to do it the DSC way).

That's fine, just keep in mind that the DSC way is much more limited currently than the Puppet supported puppetlabs/chocolatey module (which supports completely offline scenarios better than cChoco. Also not chocolatey/chocolatey - the supported module has chocolateysource, chocolateyconfig, and chocolateyfeature as well as package).  However it is hoped that the DSC will achieve a bit more parity at some point this year as compared to the Puppet provider.
As for the issues you are running into, I imagine @jpogran or someone else will answer that.

@nealpeters86 Believe it or not, the manifest file parser had a bug (line 21) in it which meant it expected single quotes in the psd1 file, and the cChoco DSC Resource is the first to use double quotes.
After editing the regex, we correctly parse the file. I'll put up a PR in abit with the fix
~/src/puppetlabs/modules/dsc   footest ●  git s
 M build/dsc/psmodule.rb
?? lib/puppet/type/dsc_cchocoinstaller.rb
?? lib/puppet/type/dsc_cchocopackageinstaller.rb
?? lib/puppet/type/dsc_cchocosource.rb
?? lib/puppet_x/dsc_resources/cChoco/

Note that the cChocoPackageInstallerSet Composite DSC Resource won't be parsed as Composite DSC Resources don't have a MOF schema file and can't be parsed at this time.

Addressed in #282

Hi,
Puppet will be shutting down the github issues tab in this repository as there already is a publicly accessible issue tracker on March 24th, 20017.
If you want to continue the discussion from this issue, please follow the ticket already created from this issue (typically MODULES-XXXX) in our issue tracker.
If you would like to file a new issue, please follow the template here to create a new ticket.
Thanks,
James Pogran

Hello! We are doing some house keeping and noticed that this issue has been open for a long time.\n\nWe're going to close it but please do raise another issue if the issue still persists. 😄


'bigquery' key error when running compare-reports fails to produce diff summary

First of all, I am really enjoying this tool! Unfortunately I have come across this bug which is blocking a rollout to the wider team so I am hoping that there is a quick fix!
Describe the bug
When comparing two piperider reports, a warning "bigquery" is returned and no comparison summary is generated.
$ piperider compare-reports --base /target_prod_profile/run.json --target /outputs/latest/run.json --output ./comparison_report
────────────────────────────── Comparison report ───────────────────────────────
Selected reports:
  Base:   /target_prod_profile/run.json
  Target: /outputs/pre-release-20231206213745/run.json
Warning:
'bigquery'
Got problem to generate changeset.
Comparison report:
/data_warehouse/comparison_report/index.html

Reproduce
Unfortunately, I cannot provide the manifest jsons but I will try my best to describe the issue and steps taken.

I generated a run report on production and I run this across all models in prod

dbt compile -t prod
piperider run --dbt-target prod --debug --report-dir $CI_PROJECT_DIR

When I open an MR, I run a dbt run on the modified and new models only in staging

dbt run --fail-fast -t pre-release --select "state:modified.body+ state:modified.configs+ state:new+" --defer --state /target_prod

I create the staging piperider run report only on the models above

piperider run --select "state:modified.body+ state:modified.configs+ state:new+" --state /target_prod --dbt-target pre-release --debug --report-dir $CI_PROJECT_DIR

I then compare the two reports

piperider compare-reports --base /target_prod_profile/run.json --target /outputs/latest/run.json --output ./comparison_report

What's strange is that the diff summary report works for some MRs but not others. I have tried to find the common trait but I am unable to.
The MR and subsequent report comparison that works is a very simple test case where I add a text column to an existing table with a constant value e.g.
...
"apples" as fruit,
...

Looking at the comparison report, row and col information for both base (production) and target (staging) are recorded.
What I have tried

Tried locally and in my CI environment for the failed cases
Tested the run.json of the MR that did work above both locally and in the CI pipeline and it still works
Tried different combinations of adding cols, removing cols, enforcing data contracts etc
piperider diagnose passes

I attached a debugger and I tried to figure out what was going on.

The GraphDataChangeSet object fails to be created
This is due to the function call for list_changes_in_unique_id failing
This fails because when it invokes the dbt task, the key error for bigquery (hence the warning printed on the console), the AdapterContainer's  lookup_adapter function is called which attempts to extract the bigquery adapter using the bigquery key

    def lookup_adapter(self, adapter_name: str) -> Adapter:
        return self.adapters[adapter_name]

Relevant code linked here
Expected behavior
Diff summary reports for dbt models that have been changed.
Example output below from the successful MR comparison
Selected reports:
  Base:   /target_prod_profile/run.json
  Target: /outputs/pre-release-20231206165611/run.json
Impact Summary:
  Code Changes: added=0, removed=0, modified=2
  Resource Impact: potentially_impacted=7, assessed=7, skipped=0, impacted=5
Comparison report:
/data_warehouse/comparison_report/index.html
Comparison summary:
/data_warehouse/comparison_report/summary.md

Desktop (please complete the following information):

OS: macOS local env and ubuntu in CI env
Python Version 3.10
Version v0.41 piperider
dbt-core 1.7.2
dbt-bigquery 1.7.2

Hi @Stochastic-Squirrel
We're having some difficulty in reproducing the issue. We'll continue to look into this.
It looks like you already ran PipeRider with --debug ? If not, I would suggest that as a further debugging step. There may be a lot of output -  If there's nothing sensitive in the output you could share that with us. Either attach it here, or email to<EMAIL_ADDRESS>In the meantime, you could try out an Impact Report manually by using the DBT Manifest Analyzer in PipeRider Cloud:

Log into PipeRider Cloud
Click the Analyze tab
Upload two manifest files into the Manifest Analyzer

I'll follow up on this when we have had more success reproducing the issue.
Thanks,
Dave

Hi Dave, thanks for reaching out!
I have been using the --debug flag throughout and nothing is printed to the console. Unfortunately I haven't been able to glean any more info!
I tried the website now and I experienced a server error when attempting uploads
Sentry event id: 229c6e399c2b48d08f61682dff0ac69a
I have tried uploading a single manifest as well as two at the same time.
Unfortunately I don't feel comfortable sharing the manifests in their entirety. I'll try to cut them down to a minimal set.
What are the essential keys needed? I am thinking of maybe isolating a single table that does not contain any sensitive information.
However, I do have an update on what causes the error!
I experimented some more and I noticed that it is only when changing the model YAML files that the error occurs. Changes to the SQL models seem to work fine.
Here are some scenarios that causes the error

Changing the data type of an existing column in prod array<string> to array
Renaming an existing column in prod fruit to fruits_and_vegetables in staging
Switching enforcement of contracts on and off has no effect
Adding a new column to the model YAML in staging that is not in prod does raise an error

In my case, I have to tweak model yamls for SQL models that are affected by a change e.g. data type change.
I hope this makes it a bit easier to recreate the issue on your end.

@Stochastic-Squirrel
Hi, thanks for your information. I know there is some privacy concern for providing the real run.json. So Is it possible to provide the two run.json from a dummy project? It would help us to reproduce the issue.
Expected reproduce steps

Download your two run.json
runpiperider compare-reports --base run_base.json --target run.json --output ./comparison_report

Get error result:Selected reports:
  Base:   run_base.json
  Target: run.json
Warning:
'bigquery'
Got problem to generate changeset.
Comparison report:
/data_warehouse/comparison_report/index.html

thanks @popcornylu. I'll try to reproduce this error in a dummy project as soon as I can!


Only single color of lamp reported with A0201 and A0202

The alarms A0201 and A0202 can only report a single color at a time.
If more than one lamp with different colors are gone on the same signalgroup it cannot be reported properly. The simple solution is to clarify that the TLC should send a comma separated list of all colors which are broken, for instance, "red,yellow"

related: #69


Implement requestFullscreen method & callback

Hi,
The current implementation works fine but fullscreen does not rotate video to landscape, if we implement requestFullscreen method callback then the developer can force the app to rotate to landscape mode.

Hey, @ssanju01 thank you for reporting this, we will be publishing the new version of the library these days with the fullScreen landscape mode.
Regards, Philip


📝 Add deprecation notice to README, move to SQLModel

📝 Add deprecation notice to README, move to SQLModel

@tiangolo - Any chance of merging this deprecation warning?
(The CI failures are obviously unrelated.)

This was covered by another PR at some point :sweat_smile:


I am unable to install it

`Microsoft Windows [Version 10.0.10240]
(c) 2015 Microsoft Corporation. All rights reserved.
C:\Users\DELL>cd desktop\suliteos
C:\Users\DELL\Desktop\suliteos>npm i -D postcss-loader
[            ......] \ finalize:code-point-at: sill finalize C:\Users\DELL\Desktop\suliteos\node_modules\fsevents\node_
`
stuck at this for a long time and I tried numerous times

@Tanuj69 problem in npm, try to reinstall npm


chore(); subgraphs; add manifest to gitignore

Added subgraph.yaml to main .gitignore

looksrare, opensea-v2, x2y2, and yeti-finance need to be converted to our current ci/cd

Great initiative!


Lens protocol research

[ ] Create Profile
[ ] Edit Profile
[ ] Follow Module(s)
[ ] Submit Publication
[ ] Edit Publication
[ ] Like Publication
[ ] Comment on Publication

Here's my working doc
https://docs.google.com/document/d/1umxdOLACBpVANaJ2UKCxrvRTiAD1aJ65tnXnp1HY6TU/edit#


ENH: first draft of MPL artist

Minimal datashader aware matplotlib artist.
from datashader.mpl_ext import DSArtist
import matplotlib.pyplot as plt
import matplotlib.colors as mocolor

fig, ax = plt.subplots()
da = DSArtist(ax, df, 'dropoff_x', 'dropoff_y',  ds.count('passenger_count'), norm=mcolors.LogNorm());
ax.add_artist(da); ax.set_aspect('equal');

fig.colorbar(da)

This is using DS to just do the binning and then re-using mpl's existing normalization and color-mapping tools.

attn @mdboom @astrofrog

Looks great, thanks!  I'll try it out and merge if it's all ok.

This requires the 2.0 beta to work (lost track of when that private class
created).  The beta is on conda forge.
I also have an idea on how to make the connection to the ds pipeline more
general.
On Mon, Jul 18, 2016, 15:42 James A. Bednar<EMAIL_ADDRESS>wrote:

Looks great, thanks! I'll try it out and merge if it's all ok.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/bokeh/datashader/pull/200#issuecomment-233436143, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAMMhRZXXwWdY-3jikR95WuOm-4-8Jxqks5qW9cVgaJpZM4JN8jc
.

@tacaswell - would it make sense to expose (as public) the currently private class and private _make_image methods so that we can reliably rely on them for examples like this?

Another reasonable option would be to move this artist into MPL and tweak the API so that all of the datashader dependency is injected as a pipeline argument.

@tacaswell, I'm not sure how to get the mpl 2.0 beta from conda-forge.  It's only offering me 1.5.2:
0172-jbednar:~> conda install -c conda-forge  matplotlib
Using Anaconda Cloud api site https://api.anaconda.org

# All requested packages already installed.
# packages in environment at /Users/jbednar/anaconda:
#
matplotlib                1.5.2               np111py27_4    conda-forge

You need to ask for the rc channel as well
conda install -c conda-forge/label/rc -c conda-forge matplotlib
Sorry for not being clearer about that.
On Thu, Jul 21, 2016 at 12:52 PM James A. Bednar<EMAIL_ADDRESS>wrote:

@tacaswell https://github.com/tacaswell, I'm not sure how to get the
mpl 2.0 beta from conda-forge. It's only offering me 1.5.2:
0172-jbednar:~> conda install -c conda-forge  matplotlib
Using Anaconda Cloud api site https://api.anaconda.org
All requested packages already installed.
packages in environment at /Users/jbednar/anaconda:

matplotlib                1.5.2               np111py27_4    conda-forge
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bokeh/datashader/pull/200#issuecomment-234314709, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAMMhSUHgM9ephj1kchTcYO49ksip2blks5qX6PCgaJpZM4JN8jc
.

Very nice!  I couldn't get scroll zooming to work, but box zooming was very snappy.  I had to make some edits for Python2 compatibility:
0172-jbednar:~/datashader/datashader> diff mpl_ext.py~ mpl_ext.py
14c14
<         super().__init__(ax, **kwargs)
---
>         super(DSArtist,self).__init__(ax, **kwargs)
48c48
<         return (*self.axes.get_xlim(), *self.axes.get_ylim())
---
>         return self.axes.get_xlim() + self.axes.get_ylim()

We'd want to include a runnable example with the distribution, so I adapted your snippet above into a new file examples/nyc_taxi_mpl.py:
import pandas as pd
df = pd.read_csv('data/nyc_taxi.csv',usecols=['dropoff_x','dropoff_y', 'passenger_count'])

import datashader as ds
from datashader.mpl_ext import DSArtist
import matplotlib.pyplot as plt
import matplotlib.colors as mcolors

fig, ax = plt.subplots()
da = DSArtist(ax, df, 'dropoff_x', 'dropoff_y',  ds.count('passenger_count'), norm=mcolors.LogNorm(), cmap='viridis_r');
ax.add_artist(da); ax.set_aspect('equal');

fig.colorbar(da)
plt.show()

This pair of files worked well for me, anyway!  Note that I reversed the colormap, so that it works better on a white background:

scroll zooming is not one of the default interactions which is why it didn't work :smiling_imp:
The reversed color map does look much better.

Overall, it seems like this approach will work well for simple datashader pipelines, as in the case illustrated above (basically anything supported by datashader.pipeline.Pipeline).  But it won't support more complex pipelines, where it's not just the agg (i.e., reduction) that needs to be overridden, but the pipeline itself.  E.g. in the census example, there are user-defined operations on the aggregate array before it is displayed:
tf.colorize(agg.where(agg.sel(race='w') < agg.sel(race='b')).fillna(0), color_key, how='eq_hist')

I'm not sure how a user could inject the agg.sel operation into the DSArtist, which would mean they'd have to copy that class and edit it to do what should be a simple operation.
Those examples also use colorize, which takes categorical information that I'm not sure how to integrate in this approach, if matplotlib is doing the colorizing.
It seems like it would be more general if mpl_ext coud support a create_image() callback instead of the current approach, as datashader.InteractiveImage does now. so that users could supply any arbitrary pipeline in just a few lines of code.  Supporting a create_aggregate() callback would also be useful, so that users could employ mpl's own colormapping, though I'm not sure how that would work for categorical information.

It seems like it would be more general if mpl_ext coud support a create_image() callback

Another reasonable option would be to move this artist into MPL and tweak the API so that all of the datashader dependency is injected as a pipeline argument.

These two suggestions may amount to the same thing; if so then it's clear how to move forward!

My current best thought is to have the users provide a callback which has a signature like
def ds_cb(canvas, data):
    return float_or_int_img

mpl has support for discrete color maps (if your norm returns integers the values are used as direct lookups in the color table).
Given that mpl users already know how to use the mpl colorization code (one hopes), I would greatly prefer that level get delegated to us, but making this class smart enough to check if it got back a NxM or NxMx4 is not too much work (or it might just fall through correctly now).
I agree that we seem to be in agreement.
attn @story645 (who is a GSOC student working on integrating categorical plotting into mpl)

I agree that we'd want as much of the processing to use mpl's code as is practical, to help integrate it more easily into mpl users' workflows and make it more familiar to them. Let datashader do what datashader is best at, and let mpl handle the rest!
mpl's discrete colormap support may work for categorical information, but I don't know enough about it to be sure.  datashader.tf.colorize() does use discrete colors, but it then (a) mixes those discrete colors according to the counts in each category for that pixel, and (b) adjust the alpha value of the color from a continuous range, depending on the total (relative) count.  So the result is an arbitrarily large set of colors, starting from the nominally discrete colormap like the 5 base colors used here:

Not sure if that's similar to what mpl supports or will support.

There is not support of the catagorical blending (yet but we have been talking about generalizing the norm/colormap chain for a while now).

Ok, then it sounds like supporting both NxMx1 and NxMx4 would be good in the meantime.

I'd love to get matplotlib support into datashader.  Any progress on addressing some of the issues above?

Sorry, I have been swamped with other work.

@tacaswell @jbednar Have there been any updates to this?
Looking at the travis output, it seems to work fine with python 3.  I will see how far I can get with the example above.

I realize my question is more related to usage (and probably just demonstrates my unfamiliarity with datashader and matplotlib) and is not specifically related to accepting/updating this PR.  This was just the best resource I found when searching how to use datashader with matplotlib.  Let me know if you prefer I move this to stack overflow and I will delete this.
Following the above example, with DSArtist definition from this PR, I almost have what I need.  I am not sure how to get matplotlib coloring to normalize the same way datashader does by default.  It is so much more faint than 'eq_hist' (the default), 'log', and even 'linear'.
Here is what I get using datashader with 'eq_hist':

This is the much fainter version I get using matplotlib:

My end goal is to use matplotlib to add the axes, labels, and red success points (blue are the ~30,000 trials, red are the ~10 successes) then plot this using a script in a headless environment (on a remote server without X).
This screenshot shows what I have been trying.  Any thoughts @tacaswell?

For now, this PR's discussion is fine as a place to collect anything about MPL support for datashader.
I'm surprised that you aren't seeing comparable results between MPL's shading and the linear shading in datashader.  It would be good to post a side by side comparison using the same colormap and ranges with linear mapping; those should be at least very nearly the same regardless of who is doing the colormapping.

As a side note, I am including a reference to this in mpl's GSoC ideas list.

Great!  I'd be happy to work with a GSoC-er to help make this move forward.  We are hoping to have funding for datashader start up again soon, and we'll be putting various functions in place that will help make it simpler to build legends, colorbars, etc.  Those functions should help any downstream plotting library to summarize what's in the plot accurately and easily.

I have had some difficulty using/defining matplotlib palettes as they are not as simple as the lists used by datashader and bokeh but I these should both be the reversed viridis palette.  Also, I am not sure how to change the MPL shading scheme.  The example above using norm=mcolors.LogNorm() which I took to be the same as datashader's how='log' option.  So I used these for the plots below.
# some setup definitions
width = 6
height = width/3.0
x_range = [0, 40]
y_range = [trial['energy'].min(), trial['energy'].max()]
w = int(width * 100)
h = int(height * 100)

# code for the datashader version
from bokeh.palettes import Viridis256 as palette
palette.reverse()
canvas = datashader.Canvas(plot_width=w, plot_height=h, x_range=x_range, y_range=y_range)
agg = canvas.points(trial, 'chi2', 'energy', agg=datashader.count())
img = datashader.transfer_functions.shade(agg, cmap=palette, how='log')
img

# code for matplotlib version
fig, ax = plt.subplots(figsize=(width, height), dpi=400)
da = DSArtist(ax, trial, 'chi2', 'energy', agg=datashader.count(), norm=mcolors.LogNorm(),
              cmap='viridis_r')
ax.add_artist(da)
ax.set_xlabel(r'$\chi^2$')
ax.set_ylabel(r'LR Docking Score')
ax.set_xlim(x_range)
ax.set_ylim(y_range)

plt.show()

I expected these to look essentially the same but the difference is obvious. I am not sure how to account for this but it is possible I simply do not understand how to use matplotlib well enough. In case I am not actually using logarithmic binning in MPL, here is the datashader plot using how='linear':

and hera is the datashader plot using how='eq_hist':

Overall, the matplotlib plot looks too sparse, like I should not need to use datashader.  That said, here is what the plot looks like without datashader:
plt.plot(trial.chi2, trial.energy, '.', ms='.1')

In addition to the general question on how to make the matplotlib version match the datashader version, I have two specific questions.

Is it possible to use equal histogram shading in matplotlib, the datashader option how='eq_hist'?  If so, how would I do this?
How do I give matplotlib a colormap with a single color, or technically white and some other color?  This seems to be the default for datashader but if I want to manually select the color I could define it simply using a list, e.g., cmap=['white', 'firebrick']:

transfer_functions.shade(canvas.points(df, 'x', 'y', agg=ds.count()),
                                       cmap=['white', 'firebrick'], how='log')

In addition to the general question on how to make the matplotlib version match the datashader version

To answer that, can you please post the same size image from both mpl and datashader colormapping, using a grayscale colormap, with linear mapping?  You might have already provided enough info above, but I can't find any pair of images that should truly be mathematically identical, which is always the safest place to start.  Grayscale should be comparable across all libraries.

Is it possible to use equal histogram shading in matplotlib, the datashader option how='eq_hist'?  If so, how would I do this?

I'm not aware of any histogram equalization option in matplotlib or bokeh, or else we probably would have just used those instead of adding our own to datashader.  It would be very convenient if plotting libraries would support eq_hist directly, which would make it simpler to have meaningful colorbars, legends, and hover information.  MPL is welcome to steal our eq-hist code; it's only 15 lines of Numpy-based Python, adapted from scikit-image.

How do I give matplotlib a colormap with a single color, or technically white and some other color?  This seems to be the default for datashader but if I want to manually select the color I could define it simply using a list, e.g., cmap=['white', 'firebrick']:

transfer_functions.shade(canvas.points(df, 'x', 'y', agg=ds.count()), cmap=['white', 'firebrick'], how='log')

See: http://matplotlib.org/examples/pylab_examples/custom_cmap.html

See http://matplotlib.org/users/colormapnorms.html for details of how the color mapping process in mpl works.
Also, turn the DPI down on the mpl plots, the spatial bins passed to datashader are set by the physical pixels in the axes.

Thanks for the links.  Based on those, I created a black and white palette and colormap that should match
from matplotlib.colors import ListedColormap
palette = ['white', 'black']
cmap = ListedColormap(palette)

then defined the image size and plot ranges
width = 600  # in units of pixels
height = 300  # in units of pixels
x_range = [0, 40]
y_range = [-100, 20]

then generated the datashader plot using how='linear'
canvas = datashader.Canvas(plot_width=w, plot_height=h, x_range=x_range, y_range=y_range)
agg = canvas.points(trial, 'chi2', 'energy', agg=datashader.count())
img = datashader.transfer_functions.shade(agg, cmap=palette, how='linear')

and matplotlib plot using linear normalization: norm=mcolors.Normalize()
dpi = 100
x_inches = width/dpi
y_inches = height/dpi
fig = plt.figure(figsize=(x_inches, y_inches), dpi=dpi)
ax = plt.axes([0., 0., 1., 1.], frameon=False, xticks=[], yticks=[])
da = DSArtist(ax, trial, 'chi2', 'energy', agg=datashader.count(),
              norm=mcolors.Normalize(), cmap=cmap)
ax.add_artist(da)
plt.savefig('mpl.png', dpi=dpi, transparent=True)

Note that datashader uses pixel unit and matplotlib uses inches and dpi.  You can see the code I used to convert between these and eliminate the axes on the matplotlib plot so the image uses the entire space.
I verified these are each 600x300 pixels
(datashader) ➜  odin: test/> file mpl.png datashader.png
mpl.png:        PNG image data, 600 x 300, 8-bit/color RGBA, non-interlaced
datashader.png: PNG image data, 600 x 300, 8-bit/color RGBA, non-interlaced

Here is an example using the sample data from the pipeline notebook:
import datashader
import matplotlib.pyplot as plt
import matplotlib.colors as mcolors
import numpy as np
import pandas as pd
from matplotlib.colors import ListedColormap

np.random.seed(1)
num=100000

dists = {cat: pd.DataFrame(dict(x=np.random.normal(x,s,num),
                                y=np.random.normal(y,s,num),
                                val=val,cat=cat))
         for x,y,s,val,cat in
         [(2,2,0.01,10,"d1"), (2,-2,0.1,20,"d2"), (-2,-2,0.5,30,"d3"), (-2,2,1.0,40,"d4"), (0,0,3,50,"d5")]}

df = pd.concat(dists,ignore_index=True)
df["cat"]=df["cat"].astype("category")
palette = ['gray', 'black']  # white created issues with the low blending into the background
cmap = ListedColormap(palette)

width = 300  # in units of pixels
height = 300  # in units of pixels
x_range = [-15, 15]
y_range = [-15, 15]

canvas = datashader.Canvas(plot_width=width, plot_height=height, x_range=x_range, y_range=y_range)
agg = canvas.points(df, 'x', 'y', agg=datashader.count())
img = datashader.transfer_functions.shade(agg, cmap=palette, how='log')
img = datashader.transfer_functions.shade(agg, cmap=palette, how='linear')  # uncomment for linear
img

dpi = 100
x_inches = width/dpi
y_inches = height/dpi
fig = plt.figure(figsize=(x_inches, y_inches), dpi=dpi)
ax = plt.axes([0., 0., 1., 1.], frameon=False, xticks=[], yticks=[])

da = DSArtist(ax, df, 'x', 'y', agg=datashader.count(),
              norm=mcolors.LogNorm(), cmap=cmap)
# da = DSArtist(ax, df, 'x', 'y', agg=datashader.count(),  # uncomment for linear
#              norm=mcolors.Normalize(), cmap=cmap)
ax.add_artist(da)
ax.set_xlim(x_range)
ax.set_ylim(y_range)
plt.savefig('mpl_sample_data.png', dpi=dpi, transparent=True)

Here are the logarithmic results.
datashader

matplotlib

Here are the linear results.
datashader

matplotlib

For matplotlib, I think you want a LinearSegmentedColormap (which interpolates between colors), not a ListedColormap (which uses only the provided colors).

Or just use the existing gray color map.

It is not clear how to use the same color map for both datashader, which takes a list of strings, and matplotlib, which takes a more custom format.  This may be the root problem.
I changed to using LinearSegmentedColormap.from_list('name', palette, N=100) and they still differ.  I am not sure what N to use to make the two colormaps comparable.
I will update my comment above using the sample data to reflect this new colormap definition and the generated plots.

Datashader uses a 256-color palette, so in this case N=256 (or use "gray" as suggested).

Using LinearSegmentedColormap.from_list('name', palette, N=256) I was able to get the logarithmic plots to match.  While the linear plots do not match, neither looks helpful anyway.  Since matplotlib does not offer equalized histogram (yet?), I think the logarithmic option with this colormap definition provides the best solution.  I will try this with the real data.

Looks like the mpl linear version isn't respecting the NaN mask in the same way, but it's good to see the logarithmic versions matching.

Another quirk of either matplotlib or jupyter is the plot looks very different in jupyter from the saved version.  I just repeated this with my real data and in jupyter the colors went back to incredibly faint.  (note that I want to use white and blue but this shows red and blue because I worried the white was not visible)
Here is the jupyter version (right click and save)

and here is the version saved right before running plt.show()

then here is the datashader version

Looks to me like this is happening because different renderers are drawing at different resolutions. My hypothesis is that the inline backend is using the hi-dpi option (presumably because you have a macbook) and therefore sampling at a higher resolution you get when saving the datashader plot directly or using matplotlib to save it.

Right -- the results will vary a lot at different resolutions, by design, though you can use tf.spread() or tf.dyn_spread() to ensure that individual dots are visible at high resolutions.

BTW, note that recent dev releases of HoloViews now support datashader, with matplotlib or any other backend. Here's an example: https://anaconda.org/jbednar/census-hv-mpl/notebook

I noticed this thread on twitter, it reminded me to put in a matplotlib backend for vaex based in ipympl. The code lives here and might be useful for this discussion, since it tries to attach a similar problem. What might be useful is the debounced decorate that I use for instance here that will only execute after 0.5 seconds have passed, to avoid many update when moving and zooming. It only works when there is an ipykernel, for Qt you need a different debounce method (should have that code somewhere).

datashader is not made available on pypi, a solution that could also support "pypi-compatible" alternatives, like "mpl-scatter-density",  would be great.

I couldn't do:
import from datashader.mpl_ext import DSArtist
here's the error:
DLL load failed: The specified module could not be found.

@ruiyangl, to get this experimental code, you would have to check out the branch of datashader associated with this PR, and use that instead of any released datashader version.  We'd be happy to merge support for Matplotlib into Datashader whenever this PR can be completed.  In the meantime, you can use HoloViews+Matplotlib to see static Datashader output inside a Matplotlib plot as mentioned above.

Is this available in the current version?

The HoloViews support for Matplotlib+Datashader is in any recent version, but is only Agg (image) based (not interactive).  No one has ever finished up this PR and made it mergeable, but if anyone wants to take it on, I'd be very happy to help get it merged! Meanwhile, https://github.com/astrofrog/mpl-scatter-density does similar things (though only for points).

datashader is not made available on pypi

BTW, Datashader is available on PyPi, nowadays (since 2018 at least).

I'm pretty new to datashader but plan on integrating it heavily into my ecosystem.  Forgive my ignorance but does DSArtist allow one to convert datashader objects to matplotlib ax? Or does it actually create the plots natively with matplotlib?
Just to be clear, there is no way to use datashader layouts with matplotlib because that would defeat the purpose of plotting large complex graphs?

This PR allows creating a Matplotlib artist that uses Datashader internally; it doesn't accept any Datashader pipelines you've already created.  So there's no conversion involved.  I'm not sure what you mean by a Datashader layout or why that would defeat the purpose, but the purpose of this PR is to create an object that will interactively re-draw on zooming; you can already create a matrix that you can plot manually with Matplotlib.  If that's not what you're asking, please elaborate!

I'm not sure what you mean by a Datashader layout or why that would defeat the purpose, but the purpose of this PR is to create an object that will interactively re-draw on zooming; you can already create a matrix that you can plot manually with Matplotlib.

Sorry, by datashader layout I meant the hammer_bundle algorithm http://datashader.org/api.html#datashader.bundling.hammer_bundle
I noticed it returns many more rows than there are nodes.  What are these rows?  Is it possible to use this output to plot using matplotlib natively?
Regarding "defeating the purpose" I was referring to fact that the output of hammer_bundle is a very large dataframe.  If those are arc coords then it may take a while plotting in matplotlib vs. datashader.
Hope that makes sense!

Ah, I see what you mean.  I don't think this PR will help you.  This PR hard-codes a call to cvs.points() to plot a 2D histogram of points, whereas a bundled-edge network graph requires line plotting, not point plotting.
Each row of the output of hammer_bundle is a line segment that approximates one bit of a curve connecting two nodes. Given that there are typically more edges than nodes and given that each curve will be approximated by many line segments, yes, there will be many more rows in the output than nodes.
These rows can be visualized using any plotting program capable of plotting line segments, including presumably Matplotlib, if you want to write code for that library.  But you are correct that for large networks the number of line segments involved will typically be too large for most plotting programs.
With that in mind, you can use Datashader on the resulting segment dataframe to create a rectangular array with the fully rendered output, and then display that in any plotting program capable of plotting rectangular arrays, including Matplotlib.
This PR won't be useful for any of those options unless it is heavily generalized, so I'd recommend just sticking with the functions described at Datashader.org, using the output from them with whatever your favorite plotting library is.

Reopening this PR so that we don't forget about it; the work that there is to do is relatively minor and at the Datashader side, so if we ever get a chance to look at it, we should be able to merge this.

Closing in favor of #939.


[CLI]: Python agent missing traceback

Describe the bug
I'm having the same issue as in #2387, specifically for the python (not CLI) version of Wandb agent when --count=1.
I'm using the VSCode debugger, and when an exception in my function is raised I only see something like the following (and nothing in the two log files relating to my own source code).
wandb: Find logs at: /tmp/wandb/wandb/run-20220425_212502-6q4aki1t/logs
wandb: ERROR Run 6q4aki1t errored: ValueError('too many values to unpack (expected 2)')

Stepping through the python code, what gets printed out is repr(exc) not the traceback, and the exception is never re-raised. None of the other if/else blocks below this that use self._exceptions get entered and the code just hits a return statement and exits.
Is this a bug or are you intentionally trying not to log a traceback? It makes development almost impossible when you can't see the traceback without setting a breakpoint in the Wandb codebase.
Thanks,
Collin
Additional Files
No response
Environment
WandB version: 0.12.15
OS: Linux
Python version: 3.8.13
Versions of relevant libraries:
Additional Context
No response

Specifically, for my use case it would be sufficient to modify the error logging here to be something like:
logger.error(f"Run {run_id} errored: {repr(exc)}\n{traceback.print_exc()}")
wandb.termerror(f"Run {run_id} errored: {repr(exc)}\n{traceback.print_exc()}")

instead of
logger.error(f"Run {run_id} errored: {repr(exc)}")
wandb.termerror(f"Run {run_id} errored: {repr(exc)}")

One more related question, why are we forcing the Python agent to be threaded? Specifically why use pyagent here rather than let it use the normal multiprocessing agent by calling run_agent() two lines below?

Hi Leslie, I was just trying to get my workflow functioning, I don't have time to wait for a bug fix with some deadlines approaching. I've since stopped using the agent as I can't get consistent behavior out of it (repeated runs, multiprocessing issues, etc.). I opened another issue but decided to move on from the agent for now. Thanks.
Closing since you've opened an internal ticket for the traceback.


RBAC: tell people to use upstream system: bindings+roles when possible, limit endpoints permissions

I added endpoints permissions to clusterroles. These are excessive. Since the endpoints object is hardcoded to live in kube-system, better would be to copy upstream's "leader-lock-" role+rolebinding pattern to limit the permissions to kube-system https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/testdata/namespace-roles.yaml#L57
Moreover, once the upstream bootstrap system: roles are updated for this leaderelection stuff we should tell people to use those instead of distributing our own RBAC objects, where possible (e.g. nfs-provisioner needs extra permissions...most others don't)

maybe make the namespace for the endpoints object configurable eventually. That would be even more secure, in some sense, since we could isolate the provisioner's permissions over endpoints to its own namespace.


Pre-trained GoogLeNet does not work with DIGITS 6.0.0 and NVIDIA Caffe 0.16.4

Hello,
upon trying to fine tune the pre-trained GoogLeNet downloaded from the Model Store 6.0, while running DIGITS 6.0 and using Caffe 0.16.4 the following error appears:
ERROR: Check failed: axis_index < num_axes() (2 vs. 2) axis 2 out of range for 2-D Blob with shape 64 1024 (65536)
Is this a known issue?

DIGITS 6.0 targets Caffe 0.15.. this is because 0.16 have problems or incompatibilities.


Trying to clone tfs to git

Hello,
I'm actualy trying to clone an old TFS repository into a new Git repository with history.
The project name is containing spaces and i think is causing the error.
error: the path $/ABCD - FID/Payment/PaymentV2/PaymentV2-DEV-LOT2 you want to clone doesn't exist!
Even using quotes/double quotes, nothing worked.
Something i made or a bug ?
Fun fact, i can clone the whole repository using
git tfs clone -d http://xxxx:8080/tfs/DSICW/ $/. C:\Temp\GIT-Import
I have the branches tree and i want to clone the last branch:
 $/ABCD - FID/Payment/PaymentV2/PaymentV2-MAIN [*]
 |
 +- $/ABCD - FID/Payment/PaymentV2/PaymentV2-INT
 |  |
 |  +- $/ABCD - FID/Payment/PaymentV2/PaymentV2-DEV
 |  |  |
 |  |  +- $/ABCD - FID/Payment/PaymentV2/PaymentV2-DEV-LOT2

The execution command line :
git tfs clone -d http://xxxx:8080/tfs/DSICW/ "$/ABCD - FID/Payment/PaymentV2/PaymentV2-DEV-LOT2" C:\Temp\GIT-Import
and the log
Command run:git tfs clone -d http://tfs2012pro-v1:8080/tfs/DSICW/ $/ABCD - FID/Payment/PaymentV2/PaymentV2-DEV-LOT2 C:\Temp\GIT-Import
No authors file used.
git-tfs version <IP_ADDRESS> (TFS client library <IP_ADDRESS> (MS)) (64-bit)
Looking for assembly StructureMap.resources, Version=<IP_ADDRESS>, Culture=fr-FR, PublicKeyToken=e60ad81abae3c223 ...
Trying to get HKEY_LOCAL_MACHINE\Software\WOW6432Node\Microsoft\VisualStudio\11.0|InstallDir
Looking for assembly StructureMap.resources, Version=<IP_ADDRESS>, Culture=fr-FR, PublicKeyToken=e60ad81abae3c223 ...
Looking for assembly StructureMap.resources, Version=<IP_ADDRESS>, Culture=fr, PublicKeyToken=e60ad81abae3c223 ...
Looking for assembly StructureMap.resources, Version=<IP_ADDRESS>, Culture=fr, PublicKeyToken=e60ad81abae3c223 ...
git command: Starting process: git init
Initialized empty Git repository in C:/Temp/GIT-Import/.git/

git command time: [00:00:00.2339217] init
No .gitignore file specified...
Looking for assembly Microsoft.TeamFoundation.VersionControl.Common.resources, Version=<IP_ADDRESS>, Culture=fr-FR, PublicKeyToken=b03f5f7f11d50a3a ...
Looking for assembly Microsoft.TeamFoundation.VersionControl.Common.resources, Version=<IP_ADDRESS>, Culture=fr-FR, PublicKeyToken=b03f5f7f11d50a3a ...
Looking for assembly Microsoft.TeamFoundation.VersionControl.Common.resources, Version=<IP_ADDRESS>, Culture=fr, PublicKeyToken=b03f5f7f11d50a3a ...
Looking for assembly Microsoft.TeamFoundation.VersionControl.Common.resources, Version=<IP_ADDRESS>, Culture=fr, PublicKeyToken=b03f5f7f11d50a3a ...
GitTfs.Core.GitTfsException: error: the path $/ABCD - FID/Payment/PaymentV2/PaymentV2-DEV-LOT2 you want to clone doesn't exist!
   à GitTfs.Util.GitTfsCommandRunner.Run(GitTfsCommand command, IList`1 args)
   à GitTfs.GitTfs.Main(GitTfsCommand command, IList`1 unparsedArgs)
   à GitTfs.GitTfs.Run(IList`1 args)
   à GitTfs.Program.Main(String[] args)
error: the path $/ABCD - FID/Payment/PaymentV2/PaymentV2-DEV-LOT2 you want to clone doesn't exist!
You may be able to resolve this problem.
- To discover which branch to clone, you could use the command :
git tfs list-remote-branches http://xxxx:8080/tfs/DSICW/

I can't myself reproduce this, as I don't have any TFS path with a space in it.
But there is https://stackoverflow.com/questions/9173802/git-tfs-how-do-i-clone-a-tfs-project-that-contains-spaces
suggesting a diffferent quoting and the issues #933 and #940 which suggest that the space might need to be URL encoded, e.g $/ABCD%20-%20FID/Payment/PaymentV2/PaymentV2-DEV-LOT2
Please report back what actually solved it so we can either enhance the documentation or add some integrated URL encoding

Thanks for quick reply !
By the way, i'm using the windows batch, cmd with the <IP_ADDRESS>
With the url encoding is still the same :
error: the path $/ABCD%20-%20FID/Payment/PaymentV2/PaymentV2-DEV-LOT2 you want to clone doesn't exist!
I've already founded the both issues but not worked for me...
Same with double slash //
error: the path $//ABCD%20-%20FID//Payment//PaymentV2//PaymentV2-DEV-LOT2 you want to clone doesn't exist!
use quotes $/"Awesome Project" instead of "$/Awesome Project/" not worked also
error: the path $/ABCD%20-%20FID/Payment/PaymentV2/PaymentV2-DEV-LOT2 you want to clone doesn't exist!
The thing i do not understand is that command seems working

Hey,
I launch the source of git-tfs with VS2019 and list all branches with QueryRootBranchObjects. I found in VS debug that "$/ABCD - FID/" use a long - and when i copy paste inside windows batch or any cmd, it replace it with short -
Maybe it is the origin of my error but I don't know how to put long - .... Any ideas ?

You mean the hyphen character '-' has a different UTF8 character value ?
AFAIK cmd doesn't support utf8. Perhaps try to clone to clone in git bash?
But in there the quoting is a little different and you need to prefix with MSYS_NO_PATHCONV=1, e.g.
MSYS_NO_PATHCONV=1 git tfs clone -d http://xxxx:8080/tfs/DSICW/ $/ABCD%20-%20FID/Payment/PaymentV2/PaymentV2-DEV-LOT2" /c/Temp/Git-Import

otherwise the MSYS logic kicks in and tries to converts /ABCD... to a windows path

@Sowz Did you have made some progress?

We are trying to clean up old/stale issues. As this issue hasn't seen any updates (or progress with pull requests) in quite a while, the issue will be closed. In case further information is provided, we can revisit that and reopen the issue again.


cleanup(cmd,pkg): properly manage ubuntu flavors at builders level.

What type of PR is this?
/kind cleanup
Any specific area of the project related to this PR?
/area cmd
/area pkg
What this PR does / why we need it:
This PR aims at resolving an issue we have while using driverkit as a go pkg: ubuntu flavors were translated in driverkit to just "ubuntu" in the persistentValidateFunc of root.go; this was the wrong place since while using driverkit as a pkg, you won't call those methods.
To be much cleaner, we now have:

a write only, package local builderByTarget map, that has been renamed builder.byTarget (most of the lines change come from that)
when we need read access to the map, we already had the builder.Factory method; it is now universally used, since the map is no more readable from outside. Here there is the logic to convert ubuntu-$flavor to ubuntu before getting the builder
Side effect of the above: the log when building will now show real target that was requested (eg: ubuntu-aws instead of ubuntu)

Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

NONE

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Note, since this needs to be ported into https://github.com/falcosecurity/dbg-go to fix our drivers build pipeline, i will need to make a patch release for driverkit once this is merged :)


Update the update section of the docs

Bring the Code Updates section of the documentation up to date.

Finally resolved by 19c5fd6872dd43331abac0d8ab8758c95eee9dc0.


[KED-1491] Drop Python 3.5 support in CircleCI config

Description
Kedro has stopped supporting Python 3.5, and this has broken the CI builds in Kedro plugins
Development notes
Remove Python 3.5 job in CI
QA notes
N/A
Checklist

[x] Read the contributing guidelines
[x] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
[ ] Updated the documentation to reflect the code changes
[ ] Added new entries to the RELEASE.md file
[ ] Added tests to cover my changes

Legal notice

[x] I acknowledge and agree that, by checking this box and clicking "Submit Pull Request":

I submit this contribution under the Apache 2.0 license and represent that I am entitled to do so on behalf of myself, my employer, or relevant third parties, as applicable.

I certify that (a) this contribution is my original creation and / or (b) to the extent it is not my original creation, I am authorised to submit this contribution on behalf of the original creator(s) or their licensees.

I certify that the use of this contribution as authorised by the Apache 2.0 license does not violate the intellectual property rights of anyone else.

And package/setup.py :)
python_requires=">=3.5, <3.8" will be python_requires=">=3.6, <3.8"

@yetudada @921kiyo done, thanks for the reviews!


Remove invasive styles

Questions
Answers

Description?
No module should override global theme styles unless absolutely necessary. This is just wrong!

Type?
improvement

BC breaks?
no

Deprecations?
no

Fixed ticket?

How to test?
Use any theme with header background other than white, or change the header background to black in the console on a new selector, or the theme selector, should be enough. Or just test with classic/hummingbird if the module works as expected

https://watch.screencastify.com/v/d3NiiSdLuk83Q8TYqSC2

There is a problem somewhere in your testing method, what you modified in the console shouldn't exist as it's removed by @SharakPL
Try to clear your cache by opening the console, going to network tab and hit the "Disable cache" checkbox

Questions	Answers
Description?	No module should override global theme styles unless absolutely necessary. This is just wrong!
Type?	improvement
BC breaks?	no
Deprecations?	no
Fixed ticket?
How to test?	Use any theme with header background other than white, or change the header background to black in the console on a new selector, should be enough. Or just test with classic/hummingbird if the module works as expected

Hello @SharakPL,
Can you create an issue to have a few steps to follow to test your pr ? Moreover it will allow you to have something to put in fixed tickets.
Thank you =)

Hello @AureRita
Originally this PR doesn't impact the current header display or colors, there is nothing for test!
It just allows the developers control the header of their themes.
And why a module must control the theme header's color?!

Thanks @AureRita

Thanks @SharakPL & @AureRita


python-scipy doesn't register properly in python's searchpatch

installing python-scipy with nix-env, it properly shows up in ~/.nix-profile/lib/python2.7/site-packages/scipy. but in ipython i see the following:
In [1]: import scipy
...
ImportError: No module named scipy
if i set PYTHONPATH=~/.nix-profile/lib/python2.7/site-packages, it works as expected. is this how it's supposed to work?

I encounter this problem today as a new user and I am very confused how nix handles python package.
I did nix-env -i python2.7-numpy.  After that, import numpy in python 2.7 returns No module named numpy. Is this a bug? Or I am expected to do something more complex in order to make my numpy work such as creating an environment for all the python packages?  But that is not what I want.  I hope numpy is accessible in my user environment, not a special development environment.
Of course I can set PYTHONPATH=~/.nix-profile/lib/python2.7/site-packages as a workaround, but I think that will confuse python3.


Implement addition of annotation values on single cells in grid editor based on clean history

This PR replaces #172, which had a botched history due to intermittent rebases on develop.

Pull request checklist
Please check if your PR fulfills the following requirements:

[x] Tests for the changes have been added (for bug fixes / features)
[x] Docs have been reviewed and added / updated if needed (for bug fixes / features)
[ ] The Unreleased section in CHANGELOG.md has been amended to reflect the changes in this PR
[x] Build (mvn verify) was run locally and any changes were pushed
[x] The pull request is against the correct branch (master for bug fixes, develop for new functionality)

Pull request type

Please check the type of change your PR introduces:

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no API changes)
[ ] Build related changes
[ ] Documentation content changes
[ ] Other (please describe):

What is the current behavior?

Issue Number: #94
What is the new behavior?

When merged, this PR fixes #94:

Annotations can now be added to single empty cells, both for token annotations and for span annotations
When an annotation value is set to "" or null, the annotation is removed
Note: explicit removal of annotations via a respective command will be fixed with #95.
When a span has no more annotations after removal of annotations, the span itself is also removed
Note: explicit removal of spans via a respective command will be fixed with #99.

Does this introduce a breaking change?

[ ] Yes
[x] No

Does this introduce new dependencies?

[ ] Yes
[x] No

If this introduces new dependencies:

[ ] Dependencies and citation templates have been updated where necessary

Other information

Checklist for maintainers
Releases

[ ] Do NOT push the green merge button on GitHub.
Follow the latest version of the developer/maintainer docs for making releases.
[ ] License and citation information for dependencies are complete (each dependency has a folder in /THIRD-PARTY/).

Fixed the exception reported in https://github.com/hexatomic/hexatomic/pull/176#pullrequestreview-413322412 which was due to a false assumption in processing annotation namespaces and names based on column value.

Thanks for the review and comments, I'll merge if/when the build passes :).

As it builds now (there was a new Salt release which updated the P2 repo) and I'm merging it :smiley:


Change the "join the chat" link to point to the Discourse

On the mybinder front page we have this section:

The "join the chat" section points to gitter. How do people feel about point this to Discourse?

It's been changed at some point :smile:


Externalize currency name in payout string

Hey Fugi, was watching the VOD from last night and thought I'd check out the code to see how you implemented the new stuff. Noticed this bug and thought I'd help.

Thanks!!


Update stable version in repo from 1.1.4 to 1.1.5

Think about update stable version to 1.1.5
Version: stable=1.1.4 (243 days old) => latest=1.1.5 (15 days old)
Installs: stable=1441 (75.6%), latest=166 (8.71%), total=1906
Click to use developer portal
Click to edit
Do not close this issue manually as a new issue will be created if condition for update still exists.
Please drop a comment if any reason exists which blocks updating to version 1.1.5 at this time.
Note: This is an automatically generated message and not personally authored by bluefox!
@mcm1957 for evidence

Is there any reason to delay updating this adapter at stable any longer?
Please drop a commit if you know any reason to refrain from updating.
If there is no response until 29.10.2024 we could decide to update anyway

done


Instaloader problem while getting images from hashtag with Sidecar

I am experienced PHP developer but having very basic skills in python. But I am in desparate need of using Instaloader utility. I am facing some issues in fetching all the images of perticular hashtag. The only problem is sidecar images. I am describing my problem in 3 approaches I have used.
If you can guide me in Any of them, it will solve my problem.

I want to extract images of particular hashtag. Images should also include all the images of sidecar structure. But instaloader cli utility only fetches first photo of the sidecar. Is there any way to get all the images from sidecar using cli utility
When I use python code to get all the posts of hashtag, it shows wrong post type.
for post in L.get_hashtag_posts('cloudyweather2017'):
print(post.typename)
This code gives me
GraphImage
GraphImage
Whereas one of them is a sidecar. I believe this is an issue.
In instagram PHP scraper, I found that when we get all posts using hashtag - We are not getting the sidecar images whereas if we try to fetch details of particular post - We are getting all the sidecar images. I tried to replicate the same behaviour here. But its not working. Following is the code.(P.S. I am getting too many timeouts in PHP scraper. that is why I want to use instaloader.)
from instaloader import Instaloader, Post
L = Instaloader()
post = Post.from_shortcode(L.context, 'BYkZsY2Avgz')
print(post.typename)
When I execute this code, I am getting following error
AttributeError: 'Instaloader' object has no attribute 'context'
I have taken reference of this example from Instaloader

Thanks for reporting. This was indeed a bug. We fixed it in version 4.0.4 and your approaches should work fine now.

Hello @Thammus Can you please let me know which of the issue has been solved?

Wiith version 4.0.4 a simple invocation of Instaloader like instaloader "#cloudyweather2017" should download all targeted posts including all pictures and videos inside sidecars.
Also when using Instaloader as a module the typename now gets correctly determined for sidecar posts:
import instaloader
L = instaloader.Instaloader()
for post in L.get_hashtag_posts('cloudyweather2017'):
    print("Type of {}: {}".format(post, post.typename))

This code is expected to output:
Type of <Post Bbuhp3Th7X6>: GraphImage
Type of <Post BYkZsY2Avgz>: GraphSidecar

If you want to download the posts simply call L.download_post() from inside the loop while specifying the post object and a target string.
In the same manner the code of your third approach should work just fine.


Filling with lava does not work

From what I can tell, using lava (Material.Lava) as the block to fill does not work.
I first tried this
BlockChanger.setSectionCuboid(upperCorner, lowerCorner, new ItemStack(Material.GLASS), false);
This worked fine, the section was filled with glass.
I then just changed the material type (nothing else), and it didn't work. Instead of filing with lava, it just filled with air.
BlockChanger.setSectionCuboid(upperCorner, lowerCorner, new ItemStack(Material.LAVA), false);
I also tried the asynchronous counterparts, and the SetCuboid() method to no avail.
Server Info

Also tested on 1.19.3
Latest version of BlockChanger

Thanks for the report. I will look into it when I am free.

Fixed https://github.com/TheGaming999/BlockChanger/commit/d7c8134f89f1703eb0532a171a20540fa870f739


RFS: Fixed or fluid responsive font sizes?

Having tested the new RFS feature in 4.3, I am not sure how this is supposed to fit in the bootstrap way of handling responsive layout, which is designed around fixed breakpoints (xs,sm,md,lg,xl). These breakpoints are not required if I intend to design my page to be fluently scalable. Does this mean the breakpoint-approach might become deprecated in the near future?
If not, wouldn't it make sense to make RFS behave closer to the bootstrap paradigm? By this I mean that font-sizes (at least by option) don't scale fluently, but jump to the respective container-size (xs,sm,md,lg,xl). Because it looks weird if you slowly decrease viewport size and while the container keeps the same width for a while, font-size continually changes.
On the other hand, if I prefer the fluid change of font-size, I'd also want my container to resize fluently, to a maximum of 1200px.
It would be nice if there was a feature to control if I want to make use of breakpoints (for containers and fonts) or have fluid scaling (also both for containers and fonts).
Sorry if I completely misunderstood RFS.

Does this mean the breakpoint-approach might become deprecated in the near future?

No, there are no plans to move away from the breakpoints.

On the other hand, if I prefer the fluid change of font-size, I'd also want my container to resize fluently, to a maximum of 1200px.

We're probably going to do more with containers in v5. Not sure yet what, but a some approaches are mentioned here: https://github.com/twbs/bootstrap/pull/25631
The idea behind RFS was to provide a solution to rescale font sizes without having to worry about text becoming unreadable or losing it's relation to other text sizes (h1 will always be bigger as h2 ect..) The fluid rescaling turned out to be the easiest for font sizes because we don't have to generate media queries for all breakpoints.


Modifying the blazor component settings  (#12)

Appsetting supdate

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 65.90%. Comparing base (4d0327b) to head (446757c).

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##              dev      #18   +/-   ##
=======================================
  Coverage   65.90%   65.90%
=======================================
  Files          32       32
  Lines         352      352
  Branches       28       28
=======================================
  Hits          232      232
  Misses        106      106
  Partials       14       14

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


CLI interface

As the project grows I occasionally find use-cases where a dedicated cli might be useful e.g.:

skbuild-core version to get the dynamic version
skbuild-core configure/build/install perform the cmake/wheel commands manually
This one I've noticed would be useful in packaging since the os packager might add system flags/options to be used, and it would be good to reuse those.

For the backend I am biased towards click so far especially for designing with sub-commands

Yes, I've been planning to work on scikit-build-cli eventually. That would ideally happen around the time scikit-build's backend gets replaced by scikit-build-core.

Work ongoing here: https://github.com/scikit-build/scikit-build-cli


Replace 2 separate cloud FAQs with a single unified version

There are currently 2 versions of the Rancher PAYG FAQ.
One for Azure, One for AWS.
https://github.com/rancher/rancher-docs/blob/main/versioned_docs/version-2.7/integrations-in-rancher/cloud-marketplace/aws-marketplace-payg-integration/aws-marketplace-payg-integration.md
https://github.com/rancher/rancher-docs/blob/main/versioned_docs/version-2.7/integrations-in-rancher/cloud-marketplace/azure-marketplace-payg-integration/azure-marketplace-payg-integration.md
Can we replace the 2 FAQ sections with a single unified FAQ?
Reason: Simpler updates in the future.
I already have that unified version in .md format here.
Happy to upload via a PR, but it will need linking from the the main docs.
Let me know how you want to proceed.
Many Thanks

There are currently 2 versions of the Rancher PAYG FAQ. One for Azure, One for AWS.
https://github.com/rancher/rancher-docs/blob/main/versioned_docs/version-2.7/integrations-in-rancher/cloud-marketplace/aws-marketplace-payg-integration/aws-marketplace-payg-integration.md
https://github.com/rancher/rancher-docs/blob/main/versioned_docs/version-2.7/integrations-in-rancher/cloud-marketplace/azure-marketplace-payg-integration/azure-marketplace-payg-integration.md
Can we replace the 2 FAQ sections with a single unified FAQ? Reason: Simpler updates in the future.
I already have that unified version in .md format here. Happy to upload via a PR, but it will need linking from the the main docs.
Let me know how you want to proceed.
Many Thanks

Hi @stephenmogg! Although I understand the ask to simplify updates for the future, it may be better to keep the FAQ sections separate. For example, If a user is getting started with the AWS PAYG option, they will land on the top-level page, AWS Marketplace Pay-as-you-go (PAYG) Integration, which contains information/FAQs specific to AWS Marketplace users. If we create a unified FAQ, the AWS user would have to navigate outside the "AWS Marketplace Pay-as-you-go (PAYG) Integration" section to see a list of FAQs. There are some small differences between the AWS and Azure FAQs, and it seems like more work for the user to figure those out with a unified FAQ.


Chore/98 invalidate jwt on reset password

closes #98
sends the user to the login page with an error when they click an expired jwt link where the jwt was expired because the user has been updated since it was generated.
@myieye I can't get the test to work, it's kinda hard to follow and figure out what I need to do to click on the link again.

@hahn-kev

I fixed backend/Testing/Browser/EmailWorkflowTests.cs
Testing.Browser.UserPageTest.CanResetPassword is failing, because if a user goes to their settings and tries to change their password they get a 403. It seems that it's become exclusive to the reset-password audience?
Most links seems to currently fail, because there's some imprecision somewhere. When trying to verify my email it says the link is expired, because 2023-12-05T15:30:51.8288550+00:00 == 2023-12-05T15:30:51.8288552+00:00: False

Thanks, I'm seeing this date issue as well. I also noticed that the test has a weird failure mode.
When the test does a submit of the reset form, the reset fails (due to a bug), however it then just sends the user to the home page, without any indication that the password change failed. This is also a bug in the front end, if a password reset fails (for any reason) we need to tell the user, not just send them home.

@myieye test's are working again, please take another look.


lfs_mount: LFS_ERR_INVAL raised because of unfound superblock

I would like to use LittleFS to log some information into an internal flash memory (stm32l4). While running the boot count example, lfs_format and lfs_mount are done successfully:
lfs_trace:3583: lfs_format -> 0
lfs_trace:3588: lfs_mount(20000304, 0800e9b0 {.context=00000000, .read=08009e9d, .prog=08009cb5, .erase=08009a9d, .sync=08009ec1, .read_size=4, .prog_size=8, .block_size=2048, .block_count=64, .block_cycles=500, .cache_size=2048, .lookahead_size=32, .read_buffer=00000000, .prog_buffer=00000000, .lookahead_buffer=00000000, .name_max=0, .file_max=0, .attr_max=0})
lfs_trace:3717: lfs_mount -> 0
lfs_trace:2480: lfs_file_open(20000304, 2000037c, "boot_count", 103)
lfs_trace:2327: lfs_file_opencfg(20000304, 2000037c, "boot_count", 103, 0800e830 {.buffer=00000000, .attrs=00000000, .attr_count=0})
lfs_trace:3737: lfs_fs_traverse(20000304, 08009ffd, 20000304)
lfs_trace:3821: lfs_fs_traverse -> 0
lfs_trace:2467: lfs_file_opencfg -> 0
lfs_trace:2484: lfs_file_open -> 0
lfs_trace:2823: lfs_file_write(20000304, 2000037c, 200017e0, 4)
lfs_trace:2948: lfs_file_write -> 4
lfs_trace:3060: lfs_file_rewind(20000304, 2000037c)
lfs_trace:2954: lfs_file_seek(20000304, 2000037c, 0, 0)
lfs_trace:2983: lfs_file_seek -> 0
lfs_trace:3067: lfs_file_rewind -> 0
lfs_trace:2489: lfs_file_close(20000304, 2000037c)
lfs_trace:2676: lfs_file_sync(20000304, 2000037c)
lfs_trace:2727: lfs_file_sync -> 0
lfs_trace:2508: lfs_file_close -> 0
lfs_trace:3727: lfs_unmount(20000304)
lfs_trace:3729: lfs_unmount -> 0

But then when the MCU resets, I am unable to mount the file system (and so the boot_count file is re-created):
lfs_trace:3588: lfs_mount(20000304, 0800e9b0 {.context=00000000, .read=08009e9d, .prog=08009cb5, .erase=08009a9d, .sync=08009ec1, .read_size=4, .prog_size=8, .block_size=2048, .block_count=64, .block_cycles=500, .cache_size=2048, .lookahead_size=32, .read_buffer=00000000, .prog_buffer=00000000, .lookahead_buffer=00000000, .name_max=0, .file_max=0, .attr_max=0})
lfs_trace:3727: lfs_unmount(20000304)
lfs_trace:3729: lfs_unmount -> 0
lfs_trace:3722: lfs_mount -> -22

-22 is LFS_ERR_INVAL (invalid parameter). Using breakpoints, I have found out that the superblock was not found and that it failed here (lfs_mount(), l.3695):
// found superblock?
    if (lfs_pair_isnull(lfs->root)) {
        err = LFS_ERR_INVAL;
        goto cleanup;
    }

The flash memory does not get erased in between. The configs seem good to me, but maybe I have missed something here. Moreover, I have tried to write and update boot count in a for loop after the first mount, and it worked fine (boot count gets updated), so I assume that writing and reading operations are not the issue here.
Did I miss something? Is there a bug?
Thanks,
Tristan

Ditto, same issue here. Littlefs has no issue with lfs_format and returns with a code of 0. lfs_mount immediately after fails.

Ah ok, my write of the superblock is indeed failing in lfs_format. However it seems that lfs_format erroneously just ignores the fact the read-back and checksum don't match the write.

Alright, found the flaw on my end. I was writing blocks successfully BUT, block 1 was overwriting block 0 due to an address masking fault on my end. This would imply that lfs_format doesn't do any kind of full sanity check post formatting to ensure all blocks are in place before continuing.

Hi, sorry about not being able to look into the original issue. It does sound like it might be something similar to what @marekr ran into.
@marekr this is quite interesting. I thought we were doing a sanity check after formatting by re-fetching the superblock (after a format the only data on disk is the superblock):
https://github.com/ARMmbed/littlefs/blob/master/lfs.c#L3648
But after changing lfs_format to write both superblocks (https://github.com/ARMmbed/littlefs/pull/193), we now write the second block after the sanity check.
It's surprising that the first fetch succeeds but that mount fails, my guess is that the uninitialized second block happens to have a lingering revision count that makes it look more recent than the superblock we just wrote.
Moving the sanity-check fetch to the last operation in lfs_format should catch this. Thanks for pointing it out!

@geky others are probably tripping on this too.
This issue after a first look is probably the same thing with format being misleading https://github.com/ARMmbed/littlefs/issues/441

Just as a note I had the same issue but moving the sanity check to after the force compaction function it cleared up the problem.

Ah! I marked this as "needs fix" so I wouldn't forget about it, and then forgot about it. My bad.
Pushed up the fix here: https://github.com/littlefs-project/littlefs/pull/508, thanks for the reminder.

@geky - not a problem - I am the same way - too many distractions going on.  Thanks for the update.


feature add: Add 2 args to narrow down cores range

There are many cpu cores in MP server, the more available cores
benchmarked, the more time is needed. This patch tries to add 2
args begin_core and end_core to narrow down cores range, so the
measurement time would be reduced.
Signed-off-by: Pascal Xie<EMAIL_ADDRESS>Signed-off-by: Jianhua Xie<EMAIL_ADDRESS>

You don't need this. c2clat will detect available cores from the cgroup it's running in. If you run it with taskset -c 5-10 c2lat it will only run for cores in range 5 to 10.


Wire removal not deployed on partial deployment

Current Behavior
When removing a wire and then doing a partial deployment (Modified Flows), the wire remains active.
Expected Behavior
The wire should not remain active
Steps To Reproduce
Here's a video demonstrating the issue. I cannot reproduce with a simple inject node so perhaps related to the node-red-contrib-homekit-bridged node
https://www.youtube.com/watch?v=1UCOPFSmDBY
And forum discussion: https://www.youtube.com/watch?v=1UCOPFSmDBY
Example flow
[
    {
        "id": "f2a73127dd903706",
        "type": "homekit-service",
        "z": "fa0dc791ef6c0b53",
        "isParent": true,
        "hostType": "0",
        "bridge": "409001a1.3e7a78",
        "accessoryId": "",
        "parentService": "",
        "name": "Flippy",
        "serviceName": "Switch",
        "topic": "",
        "filter": false,
        "manufacturer": "NRCHKB",
        "model": "1.6.1",
        "serialNo": "Default Serial Number",
        "firmwareRev": "1.6.1",
        "hardwareRev": "1.6.1",
        "softwareRev": "1.6.1",
        "cameraConfigVideoProcessor": "ffmpeg",
        "cameraConfigSource": "",
        "cameraConfigStillImageSource": "",
        "cameraConfigMaxStreams": 2,
        "cameraConfigMaxWidth": 1280,
        "cameraConfigMaxHeight": 720,
        "cameraConfigMaxFPS": 10,
        "cameraConfigMaxBitrate": 300,
        "cameraConfigVideoCodec": "libx264",
        "cameraConfigAudioCodec": "libfdk_aac",
        "cameraConfigAudio": false,
        "cameraConfigPacketSize": 1316,
        "cameraConfigVerticalFlip": false,
        "cameraConfigHorizontalFlip": false,
        "cameraConfigMapVideo": "0:0",
        "cameraConfigMapAudio": "0:1",
        "cameraConfigVideoFilter": "scale=1280:720",
        "cameraConfigAdditionalCommandLine": "-tune zerolatency",
        "cameraConfigDebug": false,
        "cameraConfigSnapshotOutput": "disabled",
        "cameraConfigInterfaceName": "",
        "characteristicProperties": "{}",
        "waitForSetupMsg": false,
        "outputs": 2,
        "x": 710,
        "y": 80,
        "wires": [
            [
                "91d1ce8202137f5d"
            ],
            []
        ]
    },
    {
        "id": "91d1ce8202137f5d",
        "type": "debug",
        "z": "fa0dc791ef6c0b53",
        "name": "debug 17",
        "active": true,
        "tosidebar": true,
        "console": false,
        "tostatus": false,
        "complete": "true",
        "targetType": "full",
        "statusVal": "",
        "statusType": "auto",
        "x": 860,
        "y": 80,
        "wires": []
    },
    {
        "id": "409001a1.3e7a78",
        "type": "homekit-bridge",
        "bridgeName": "Demo 1",
        "pinCode": "153-10-538",
        "port": "",
        "allowInsecureRequest": false,
        "manufacturer": "NRCHKB",
        "model": "Demo",
        "serialNo": "1.1.2",
        "customMdnsConfig": false,
        "mdnsMulticast": true,
        "mdnsInterface": "",
        "mdnsPort": "",
        "mdnsIp": "",
        "mdnsTtl": "",
        "mdnsLoopback": true,
        "mdnsReuseAddr": true,
        "allowMessagePassthrough": true
    }
]

Environment
Installed via Docker nodered/node-red:latest

Node-RED version: 4.0.2
Node.js version: v20.15.1
npm version: 10.7.0
Platform/OS: Linux 6.5.0-41-generic x64 LE
Browser: Safari

Can you update the link to the discussion please, you appear to have included the youtube link twice and I can't find it on discourse.nodered.org

(Edited link to forum discussion)

Hi @cwagdev,
Can you check if v1.7.1 of NRCHKB has solved your problem?

Hi @cwagdev, Can you check if v1.7.1 of NRCHKB has solved your problem?

Confirmed working! Thanks!!


Update README.md regarding flow label creations

flow label comes from entropy

Pull Request Functional Test Report for #3606 / a9c5aaf19d74fc077608735ace84c451d8abbf30
No tests identified for validation.
Help

Pull Request Test Coverage Report for Build<PHONE_NUMBER>4
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 55.268%

Totals

Change from base Build<PHONE_NUMBER>0:
0.0%

Covered Lines:
1983

Relevant Lines:
3588

💛  - Coveralls


Add kernelCTF CVE-2023-4623_lts_cos

Hi,
netlink_write_noerr(nl_sock_fd, &new_qfq_qdisc) is called twice intentionally. The first call creates the QFQ qdisc and the second call triggers ROP. I added a comment about this and the 4-byte offset.
I removed the unused gadget and some other unnecessary code. I also added a bit more detail on why the side-channel doesn't work on the github action instance.

I removed the unused gadget and some other unnecessary code. I also added a bit more detail on why the side-channel doesn't work on the github action instance.

Thanks!

netlink_write_noerr(nl_sock_fd, &new_qfq_qdisc) is called twice intentionally. The first call creates the QFQ qdisc and the second call triggers ROP. I added a comment about this and the 4-byte offset.

So I guess the first call detects in tc_modify_qdisc that this is a new Qdisc, so a new Qdisc is created, but does not trigger the modified qfq_qdisc_ops.change function pointer because it jumps to the create_n_graft label and skipping the call to qdisc_change which would call sch->ops->change (which is the overwritten function pointer). Do I understand correctly?

I also have a new question, which I don't understand fully, you mention that

The new_qfq_qdisc message is constructed with two consecutive TCA_OPTIONS attributes

Why do you need two messages and separate the jop_buf and rop_buf messages (instead of just using one rop_buf)? Only the mentioned stack pivot (push rsi ; jmp qword ptr [rsi - 0x70]) could be found in the kernel binary, but there were no e.g. push rsi ; jmp qword ptr [rsi] or similar gadget in the kernel?
Btw you can send two TCA_OPTIONS because the attribute parsing code, simply does not check that you are using the same attribute multiple times, but only uses the latest one (which is just a pointer into the middle of user-supplied message structure)? That's kind of a bug in itself which at the end makes exploitation a bit easier, right?

So I guess the first call detects in tc_modify_qdisc that this is a new Qdisc, so a new Qdisc is created, but does not trigger the modified qfq_qdisc_ops.change function pointer because it jumps to the create_n_graft label and skipping the call to qdisc_change which would call sch->ops->change (which is the overwritten function pointer). Do I understand correctly?

That's right.

I also have a new question, which I don't understand fully, you mention that

The new_qfq_qdisc message is constructed with two consecutive TCA_OPTIONS attributes

Why do you need two messages and separate the jop_buf and rop_buf messages (instead of just using one rop_buf)? Only the mentioned stack pivot (push rsi ; jmp qword ptr [rsi - 0x70]) could be found in the kernel binary, but there were no e.g. push rsi ; jmp qword ptr [rsi] or similar gadget in the kernel?

It was just chosen to make the ROP chain slightly simpler, since alternative JOP gadgets like push rsi ; jmp qword ptr [rsi + 0x2e] would jump to an address in the middle of the ROP chain.

Btw you can send two TCA_OPTIONS because the attribute parsing code, simply does not check that you are using the same attribute multiple times, but only uses the latest one (which is just a pointer into the middle of user-supplied message structure)? That's kind of a bug in itself which at the end makes exploitation a bit easier, right?

Yes. For the same effect the first rtattr could instead have an rta_type that is never checked for by the syscall.


Watchpoint sizes

Watchpoints seem to only work for the sizes 4 and 8, not 1 and 2. This means that watchpoints for chars and shorts will not be triggered. The cause of this seems to be in the hardware debug API, as setting a size 1 or 2 watchpoint does not generate a debug exception at all, but it is unclear why. The way we are setting the BAS field of the DBG_WCR register looks valid, as for a size 1 watch point we set the BAS field to '0b00000001'  and for a size 2 to '0b00000011', which should select the first and first two bytes of the word pointed to by the watchpoint respectively as per the ARMv8 TRM.

Fixed by https://github.com/alwin-joshy/seL4/commit/72c794536aa678288e7e9334ead49a0aaf5d1449. Will close when PR is made and merged.


User SNBAT paste the same verification code twice

Expected Behavior
An error message that prevents the user from pasting the same verification code twice.
Current Behavior
The user can paste the same code twice, which leads to failing verification.

This can happen in the time between the user clicks paste and the time we process and ingest the code. With the new identity changes, we now validate the code via a contract call.
The solution is to disable the button temporarily while we process the code and accept/reject it.


usershell -> sdk

let's use another name
let's make sure filters are working correctly ignoring anything but lowercase letters for completion, no _, __, or Class names.

Superseeded by https://github.com/threefoldtech/js-ng/issues/135


Debugging C# - VSCode hangs when breakpoint hit

Issue Type: Bug

I am using VSCode to debug my C# ASP.NET Core project running on Kestrel on my Windows laptop.
I set a breakpoint in my C# code and trigger the feature in my app (in the browser) that causes that breakpoint to be hit.
I see VSCode's task bar button begin to flash, indicating that the breakpoint has been hit.
I click the task bar button to bring VSCode to the foreground.
VSCode is hung - the entire UI is non-responsive. The line with the breakpoint is not highlighted.
I click back on the browser window and VSCode immediately becomes responsive again and highlights the line with the breakpoint. I can then work with the debugger, which works fine until the next time a breakpoint is hit; then the whole process repeats.

This sequence of steps happens the same way every time. I have tried a sample ASP.NET Core project and got the same behavior, so it is nothing specific to my code. I have also tried disabling all extensions except the C# extension (required for debugging) and it still happens.
VS Code version: Code 1.53.2 (622cb03f7e070a9670c94bae1a45d78d7181fbd4, 2021-02-11T11:48:04.245Z)
OS version: Windows_NT x64 10.0.19042

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz (12 x 2592)

GPU Status
2d_canvas: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: enabledopengl: enabled_onprotected_video_decode: enabledrasterization: enabledskia_renderer: enabled_onvideo_decode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
31.71GB (18.94GB free)

Process Argv
--crash-reporter-id 28325b46-87fa-4884-ae6b-3bd5373edc01

Screen Reader
no

VM
67%

Extensions (11)

Extension
Author (truncated)
Version

vscode-sqlite
ale
0.11.1

awarest-align
awa
1.1.0

markdown-preview-github-styles
bie
0.1.6

vscode-eslint
dba
2.1.14

EditorConfig
Edi
0.16.4

rainbow-csv
mec
1.8.1

git-graph
mhu
1.28.0

csharp
ms-
1.23.9

powershell
ms-
2020.6.0

vscode-gitextensions
pmi
1.1.0

local-history
xyz
1.8.1

A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vstry244cf:30256637
pythonvsdeb440:30248342
pythonvsded773:30248341
vstes627cf:30244335
pythonvspyt875:30251590

I'm also seeing this when a "debugger" statement is hit in my Vue script section. The behavior reminds me of my Win32 days when we would call LockWindowUpdate() while doing some messy painting to reduce flicker, and it didn't get unlocked. I'm now in the habit of doing a quick double Alt-Tab or double tapping the Windows key (to bring up/hide the Start menu) which has the effect of unlocking it.

Sounds related to this https://github.com/microsoft/vscode/issues/114459
Since we use the electron focus window API. Also assigning to @deepak1556 since he would know if we should close this as duplicate

It certainly is the same issue as #116615 which was closed as a duplicate of  #114459, and starting with code --disable-features=CalculateNativeWinOcclusion solves the problem for me too, so I would say yes, this is a duplicate of that issue.

Thanks for letting us know. Then let's close as duplicate, and if @deepak1556 thinks I am mistaken we can reopen

Nope all good, thanks for triaging the issue!

Confirmed that code --disable-features=CalculateNativeWinOcclusion fixes the problem on my system. Thanks for your help!


Fix get-password for gce

#4094 Tested by deleting local ~/.kubernetes_auth and spinning up an e2e cluster. I had two failures running e2e, one of which may be a flake (pd.sh), but this should at least fix the cluster validation problem for jenkins.
cc @zmerlynn, @brendandburns

Merging after hours to hope for some e2e runs.

This appears not to have fixed things.  Do we need to worry about file locations in the jenkins install?

Or perhaps there is more than one problem...  I'm looking at this.


Please add sodium to php-7.2

ISSUE TYPE

Feature request

SUMMARY

Magento2 is not installing.
Warning: Use of undefined constant SODIUM_CRYPTO_PWHASH_ALG_ARGON2ID13 - assumed 'SODIUM_CRYPTO_PWHASH_ALG_ARGON2ID13' (this will throw an Error in a future version of PHP) in /shared/httpd/gql-ce/graphql-c
  e/lib/internal/Magento/Framework/Encryption/Encryptor.php on line 153

Goal

Something similar to
RUN echo "deb http://deb.debian.org/debian stretch-backports main" >> /etc/apt/sources.list
RUN apt-get update && apt-get -t stretch-backports install -y
libsodium-dev
RUN pecl install -f libsodium-1.0.17 / RUN docker-php-ext-install sodium

For now, I think you can use PHP7.1. I'm doing it on local dev and on my production server.

For Magento 2.3.2
Make sure that you’re running the the latest PHP version 7.2.19 which contains a version of libsodium of at least 1.0.13 (we run 1.0.17 right now).
https://support.hypernode.com/changelog/release-6364-magento-2-preinstall-updated-to-2-3-2/
http://blog.grafzahl.io/magento-warning-after-updating-to-2-3-2/

How can I manually upgrade the libsodium to 1.0.13?

I was able to get this working. (Using the php 7.2 box)

ssh into the box

go root (sudo su)

run command:

wget https://download.libsodium.org/libsodium/releases/libsodium-1.0.18.tar.gz \     && tar xfvz libsodium-1.0.18.tar.gz \     && cd libsodium-1.0.18 \     && ./configure \     && make && make install \     && pecl install -f libsodium

docker-compose kill / docker-compose up -d (or however you manage your instances)

1.0.13 or later of libsodium is needed for Magento 2.3.2 onwards so this breaks if using PHP 7.2 and the latest Magento version.

@stale unstale

@TomashKhamlai DId you try using PHP 7.3?

PHP 7.3 and 7.2 have the same sodium version installed (1.0.11). As @robsoned mentioned using 7.1 can serve as a temporary solution. It's suboptimal, but it works as this version has no sodium installed at all so there are no problems with it (if there's no sodium, SHA256 is used instead of Argon2id)

Thank you


Readme Enchantment

I don't know how to make an introduction so, I'm saying the things directly.
I saw so many people who wants 32 bit FFmpeg for their system. That's the exact reason why I made this repo - github.com/sudo-nautilus/FFmpeg-Builds-Win32
Could you please add the repo link in readme.md. It will be helpful for those who want 32 bit builds,
That's it

I can't really guarantee that you keep the same level of maintenance to those builds. As in, porting over new library versions, security upgrades, etc.
So I'm not too keen on advertising something I can't give any guarantees about.
The scripts in this repo are perfectly capable of building 32 bit executables, there just is no asset space left to do so in the auto-builds without sacrificing more useful other build variants.
So if someone for some reason still needs a 32 bit variant, building that manually is possible relatively easily.
Overall the need for 32 bit builds seems very low though. 64 bit Windows systems are the norm for over a decade at this point.

I can't really guarantee that you keep the same level of maintenance to those builds. As in, porting over new library versions, security upgrades, etc.
So I'm not too keen on advertising something I can't give any guarantees about.
The scripts in this repo are perfectly capable of building 32 bit executables, there just is no asset space left to do so in the auto-builds without sacrificing more useful other build variants.
So if someone for some reason still needs a 32 bit variant, building that manually is possible relatively easily.
Overall the need for 32 bit builds seems very low though. 64 bit Windows systems are the norm for over a decade at this point.

But anyway people including me use that,
I'm currently writing a script to sync the fork

Ok synced fork

I can't really guarantee that you keep the same level of maintenance to those builds. As in, porting over new library versions, security upgrades, etc.
So I'm not too keen on advertising something I can't give any guarantees about.
The scripts in this repo are perfectly capable of building 32 bit executables, there just is no asset space left to do so in the auto-builds without sacrificing more useful other build variants.
So if someone for some reason still needs a 32 bit variant, building that manually is possible relatively easily.
Overall the need for 32 bit builds seems very low though. 64 bit Windows systems are the norm for over a decade at this point.

you can do that, sure. It will be maintained at the same level of your repo. There' s also 32 bit guys nowadays. The scripts require Docker in order to Build the desired variant.  Docker is not supported on 32 bit windows. And that's why We* cant build FFmpeg. Even though the need of 32 bit builds seems low, there are some guys who actually need em.

One day I tried to build FFmpeg. There were a long list of errors. And I searched each of them in the internet and saw that there are so many people just like me. In your repo I went through the issues and understood there are people who needs 32 bit builds. These people and me are referred here as 'we'.

https://github.com/BtbN/FFmpeg-Builds/issues/61#issuecomment-813067664


Making issue prior to PyBaMM developer training for test

Description
I suggest adding a graphical representation for documents and for developers.
Motivation
To start contribution.
Possible Implementation
No response
Additional context
No response

Can you give more details about what you mean by a graphical representation?

Can you give more details about what you mean by a graphical representation?

 Following long instructions either as a PyBaMM user or a first-time developer, going back and forth to different pages, and reading long instructions can be replaced by a simplified flow diagram (_**in some cases**_). If not possible, put some photos illustrating operations and steps of the procedures to follow in platforms (such as Github,...) to make it more visible to new users or developers.

Thanks


Terminal cannot display correctly

VSCode Version: 1.25
OS Version: macOS 10.13.5

Here's the screenshot, move side bar right will have this issue:

Any update?

This is happening to me too

For now you can add this to the end of C:\Program Files (x86)\Microsoft VS Code\resources\app\out\vs\workbench\workbench.main.css
.xterm-screen{overflow-x:hidden}

I believe this is already fixed, can you see if you can reproduce in the Insiders build?

hey @Tyriar, yes this looks like it is working in the latest insiders build, thanks
could you link to the issue you've tagged this as a duplicate of? so that I can follow it and get updated for when it makes it into usual release

@saborrie there were 2 issues that caused this in various cases https://github.com/Microsoft/vscode/issues/51479 & https://github.com/Microsoft/vscode/issues/54230


chore: rename workflow

Coverage remained the same at 100.0% when pulling 0c8edfc0aa42147d732644bd143f0e350806de85 on rename-workflow into c33ec8db912dcc6237cc06498e8ba38d4e1b9746 on main.


Integrate Rockset as Vectorstore

Description
This PR adds Rockset as a vectorstore for langchain. Rockset is a real time OLAP database which provides a fast and efficient vector search functionality. Further since it is entirely schemaless, it can store metadata in separate columns thereby allowing fast metadata filters during vector similarity search (as opposed to storing the entire metadata in a single JSON column). It currently supports three distance functions: COSINE_SIMILARITY, EUCLIDEAN_DISTANCE, and DOT_PRODUCT.
This PR adds rockset client as an optional dependency.
We would love a twitter shoutout, our handle is https://twitter.com/RocksetCloud
Before submitting

Integration test: https://github.com/anubhav94N/langchain/blob/master/tests/integration_tests/vectorstores/test_rocksetdb.py
Example notebook: https://github.com/anubhav94N/langchain/blob/master/docs/modules/indexes/vectorstores/examples/rockset_vector_database.ipynb
Ran make format and make lint locally

Who can review?
@hwchase17, @dev2049 can you help review please?

thanks @anubhav94N!


Game completely crashes upon clicking any of the compacted buttons

When clicking singleplayer, resourcepacks, or multiplayer the game immediately crashes with compact ui installed.
I have no idea how this is happening, I only have compact ui and fabric api installed.
Log: https://paste.ee/p/FsaMk

try using java 17?

I can't reproduce this.. if you can feel free to reopen the issue..

I'm also getting this same issue. https://pastebin.com/bnA3LqVj

Compact UI 1.19 3.0.1 build 10 (latest)
Minecraft 1.19.2
Quilt Loader 0.17.3
Java 17

Duplicate of #1


add CDNJS version badge in README.md

This badge will show the library version on CDNJS!

Hi @jaubourg Sorry to bother you.
May I know can this PR can be merged?
I think it would better to close it if it can't be merged.
Thank you. :)