Makefile: Fix libgrpc++.so link

The library is built as libgrpc++.so.6, but installed as libgrpc++.so.1.
Signed-off-by: Bernhard Walle<EMAIL_ADDRESS>

Thank you for your pull request. Before we can look at your contribution, we need to ensure all contributors are covered by a Contributor License Agreement.
After the following items are addressed, please respond with a new comment here, and the automated system will re-verify.

User @bwalle isn't covered by a CLA. They will need to complete the form at https://identity.linuxfoundation.org/projects/cncf

Regards,
CLA GitHub bot

Something missing here?

Is there a corresponding issue for this?

Makefile is generated, you need to update the corresponding template too.
I have a feeling that this was fixed already by another PR - can you check if the problem still exists on the latest HEAD?

Seems fixed here? https://github.com/grpc/grpc/pull/16832

Yes, it was fixed. The pull request is more than 6 month old!


concurrent.futures error

What happened?
I ran mriqc using the docker version on one participant and received the error code: exception calling callback for <Future at 0x2aaac0288e50 state=finished raised BrokenProcessPool>
I am running the latest docker version on a Mac Studio with an M2 chip and macOS Sonoma
What command did you use?
docker run --platform linux/amd64 -it --rm -v /Volumes/clippdata/IDP/bids2:/data:ro -v /Volumes/clippdata/IDP/mriqc:/out nipreps/mriqc:latest /data /out participant --participant_label 001

What version of the software are you running?
24.1.0
How are you running this software?
Docker
Is your data BIDS valid?
Yes
Are you reusing any previously computed results?
No
Please copy and paste any relevant log output.
No response
Additional information / screenshots

Hi @madeleinebarter, this looks like a memory error. Although I'm actively working on making this more reliable, in the case of having many subjects you may want to play a little with the options --omp-nthreads and --nprocs.
Can you elaborate more on the characteristics of your images (number of images, modalities, etc.) and your machine (number of cpus, memory available, etc.)?
It would be also helpful to run mriqc a bit more verbose (e.g., set -vv or -vvv).
Finally, copying and pasting the last config file generated at <work-directory>/config-<timestamp>-<unique-id>.toml would also be of help.

Hey @oesteban , I also have had a similar error for some time now, " ERROR    | concurrent.futures | exception calling callback for <Future at 0x7f622a65b8d0 state=finished raised BrokenProcessPool>"
I also tried implementing some solutions to get around this problem.

Tried limiting memory usage using --mem. Same error.
Tried using --use-plugin options. "Got a strange "File Not Found Error" for the attached plugin.yml file.
Currently using the following command docker run -it --rm -v /home/test_data/Nifti:/data:ro -v /home/test_data/mriqc2:/out nipreps/mriqc:latest /data /out participant --participant_label 01 --no-sub --omp-nthreads 1 --nprocs 1 --mem 4

System Information:
Running MRIQC version 24.1.0, on Linux (8GB) and MAC OS(16 GB), latest Docker Version, platforms, but got the same error in both cases.
I have attached the log files (linux and MAC OS) for your reference.
crash-20240507-073505-root-UploadMetrics.a0-4ce7c5f2-2f4c-4dc8-a8d3-b684affa8fd9.txt
mriqc-20240507-083540_d7ea3439-f37f-449c-9158-9ae617df4d5f.log
MACOS_mriqc-20240507-081646_441bbab0-acb9-43c4-9304-c6a9e99cff5a.log
Thanks in advance

@Armos05 - by looking at your crashfile, it seems our server was down when you ran MRIQC.
Can you try again with the same arguments? If it still crashes, can you run MRIQC while adding the argument --no-sub?

@oesteban, Thanks for looking into it.
MRIQC works now!!
Although, I was unable to perform group analysis, MRIQC can calculate all the parameters into the json file but there are no visual renders of those values into the graphs to look at in the final HTML file.
Here the config file:
[environment]
cpu_count = 8
exec_env = "singularity"
free_mem = 13.2
freesurfer_home = "/opt/freesurfer"
overcommit_policy = "heuristic"
overcommit_limit = "50%"
nipype_version = "1.8.6"
synthstrip_path = "PosixPath('/opt/freesurfer/models/synthstrip.1.pt')"
templateflow_version = "24.2.0"
total_memory = 15.365371704101562
version = "24.1.0.dev0+g3fe90466.d20240417"
[execution]
ants_float = false
bids_dir = "/data"
bids_dir_datalad = false
bids_database_dir = "/out/.bids_db"
bids_database_wipe = false
cwd = "/tmp"
datalad_get = true
debug = false
dry_run = false
dsname = ""
float32 = true
layout = "BIDS Layout: /data"
log_dir = "/out/logs"
log_level = 25
modalities = [ "T1w", "T2w", "bold", "dwi",]
no_sub = true
notrack = false
output_dir = "/out"
participant_label = [ "01", "32", "33",]
pdb = false
reports_only = false
resource_monitor = false
run_uuid = "20240521-083912_05591ae9-5501-48d5-9296-462c064a37a7"
templateflow_home = "/templateflow"
upload_strict = false
verbose_reports = false
webapi_url = "https://mriqc.nimh.nih.gov:443/api/v1"
work_dir = "/tmp/work"
write_graph = false
[workflow]
analysis_level = [ "group",]
biggest_file_gb = 1.1229652797928313e-10
deoblique = false
despike = false
fd_thres = 0.2
fd_radius = 50
fft_spikes_detector = false
min_len_dwi = 7
min_len_bold = 5
species = "human"
template_id = "MNI152NLin2009cAsym"
[nipype]
crashfile_format = "txt"
get_linked_libs = false
local_hash_check = true
nprocs = 2
omp_nthreads = 2
plugin = "MultiProc"
remove_node_directories = false
resource_monitor = false
stop_on_first_crash = true
[settings]
file_path = "/out/logs/config-20240521-083912_05591ae9-5501-48d5-9296-462c064a37a7.toml"
start_time =<PHONE_NUMBER>.2356057
[execution.bids_filters]
[workflow.inputs]
t1w = [ "/data/sub-01/anat/sub-01_T1w.nii.gz", "/data/sub-32/anat/sub-32_T1w.nii.gz", "/data/sub-33/anat/sub-33_T1w.nii.gz",]
bold = [ "/data/sub-01/func/sub-01_task-rest_bold.nii.gz", "/data/sub-32/func/sub-32_task-rest_bold.nii.gz", "/data/sub-33/func/sub-33_task-rest_bold.nii.gz",]
[nipype.plugin_args]
maxtasksperchild = 1
raise_insufficient = false

Can you try again with the same arguments? If it still crashes, can you run MRIQC while adding the argument --no-sub?

I got the exact same error even if I run with --no-sub. Is mriqc trying to fetch something from the server which can not be turned off? e.g., templateflow?
BTW, I run mriqc from docker without --mem at first, mriqc ate all free memory and swap, made my computer frozen.
My environment is openSUSE Tumbleweed, with nipreps/mriqc:24.0.0, which report 24.1.0.dev0 in its output.

For my latest try, I limited the memory with --mem 80 (the workstation has 128G), mriqc still used all memory resources during some processing (without any output after the 'Generating visual report' info and warning about 'background was too small', before the crash, just like in OP's screenshot). This time the workstation survived after mriqc crashed, fortunately.
It seems that version 24.1.0.dev0 do not obey the memory limits? I misunderstood this option?

I get the same problem running MRIQC 24.0 via singularity on a HPC cluster. v23.1 works fine with the exact same command/options.

@suxpert @hugofluhr this bug report is too general and has branched into several different issues so it is really difficult to identify what's the target.
May I ask you to file fresh new issues for the problems you are experiencing? It'd be great to get access to your logs and crashfiles.
Closing this one for the time being.


can't see console messages...

can't see console messages...
I am running the following setup and don't see any of the report errors..
any help is appreciated:
gulp.task('x_ts-compile', function () {
    var sourceTsFiles = [config.allTypeScript, //path to typescript files
        config.libraryTypeScriptDefinitions]; //reference to library .d.ts files

    var tsResult = gulp.src(sourceTsFiles)
        .pipe(sourcemaps.init())
        .pipe(tsc(tsProject, undefined, tsc.reporter.longReporter()));

    tsResult.dts.pipe(gulp.dest(config.tsOutputPath));
    return tsResult.js
        .pipe(sourcemaps.write('.'))
        .pipe(gulp.dest(config.tsOutputPath));
});

gulp.task('x_server_systemjs', ['x_ts-compile', 'x_watch2'], function () {
    process.stdout.write('Starting browserSync and superstatic...\n');
    browserSync({
        port: 8003,
        open: false,
        files: ['index.html', '**/*.js'],
        injectChanges: true,
        logFileChanges: true,
        logLevel: 'debug',
        logPrefix: 'angular2_ts:',
        notify: true,
        reloadDelay: 0,
        server: {
            baseDir: './',
            middleware: superstatic({debug: false})
        }
    });
    opn('http://localhost:8003/src/public/index_sys.html')
});

I can't replicate your behavior. Do you have any reason why you're using process.stdout.write instead of console.log in the second task? Do I understand correctly that the compilation works (eg JavaScript files are generated) but you don't see any compile errors?

I ended up using a diff solution so closing this, tx for the reply...


enh: remove stmt2_get_fields

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 77.50%. Comparing base (4779825) to head (7f78719).

Additional details and impacted files
@@            Coverage Diff             @@
##              3.0     #364      +/-   ##
==========================================
+ Coverage   77.46%   77.50%   +0.03%
==========================================
  Files         142      142
  Lines       18077    18001      -76
==========================================
- Hits        14004    13952      -52
+ Misses       3283     3267      -16
+ Partials      790      782       -8

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Dylib not loaded

I can't figure out what dependancy this is:
/opt/local/lib/libreadline.8.dylib
Any help would be appreciated
Thanks :)

Full log please.

If error occurs in my fork you must open issue in my fork.

Ok Thanks. I'll open an issue over there. :)

I can't open an issue over there, there is nowhere to open the issue. I think that this is missing some dependancies, I ran the command to install the dependencies, but some are still missing(I think, I don't know for sure.) Full log here:
./bundlegen -d iPhone9,4 -i 11.4.1
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/xdelta3/version.py:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
from distutils.version import StrictVersion
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/util/ssl_.py:260: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = SSLContext(ssl_version or PROTOCOL_TLS)
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connection.py:407: DeprecationWarning: ssl.match_hostname() is deprecated
match_hostname(cert, asserted_hostname)
Creating Firmware Bundle for iPhone9,4, iOS 11.4.1
[1] Grabbing decryption keys...
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/util/ssl_.py:260: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
context = SSLContext(ssl_version or PROTOCOL_TLS)
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connection.py:407: DeprecationWarning: ssl.match_hostname() is deprecated
match_hostname(cert, asserted_hostname)
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connection.py:407: DeprecationWarning: ssl.match_hostname() is deprecated
match_hostname(cert, asserted_hostname)
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/urllib3/connection.py:407: DeprecationWarning: ssl.match_hostname() is deprecated
match_hostname(cert, asserted_hostname)
[2] Patching bootchain...
Traceback (most recent call last):
File "/Users/tobinredekop/Downloads/Inferius-master 2/./bundlegen", line 412, in
main()
File "/Users/tobinredekop/Downloads/Inferius-master 2/./bundlegen", line 296, in main
delta.write(xdelta3.encode(original.read(), new_file.read()))
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/xdelta3/main.py", line 89, in encode
return _xdelta3.execute(new_value, original, flags, 0)
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

I don't see dylib problems, it was my bad that I didn't provide dylib for something. Looks like you already installed all dependencies, nice!
Now the problem is in xdelta3 library. It's outdated. To fix this problem please use only Python 3.9.0, not 3.10.
 I uninstalled Python 3.10 and installed Python 3.9.0, and I'm still getting the same error. Log:
Restoring custom IPSW
dyld: Library not loaded: /opt/local/lib/libreadline.8.dylib
Referenced from: /Users/tobinredekop/Downloads/Inferius-master 2/./bin/irecovery
Reason: image not found
[ERROR] Device in DFU mode not found. Exiting.

There! Uninstalled Python 3.10, and tried again. This time it gave me another module not found error. Log:
./bundlegen -d iPhone9,4 -i 11.4.1
Traceback (most recent call last):
File "/Users/tobinredekop/Downloads/Inferius-master 2/./bundlegen", line 4, in
from mwclient import Site
ModuleNotFoundError: No module named 'mwclient'

pip install -r requirements.txt

It looks like you never heard about Python before

Sorry, I thought that I had installed the dependencies. Thanks for the fix :).

It still says that I am missing libreadline.8.dylib.

It still says that I am missing libreadline.8.dylib.
EDIT: I did a bunch of research online about libreadline, and it looks like something that should only be required in Linux distros, such a Debian or Arch Linux. It sounds like macos should not need this...

I'll do a small script to properly install Inferius. Go to https://github.com/vadimszzz/Inferius and click „Watch“. It will be soon.

Many thanks! I will go over there and watch it, best of luck with your coding!


The registry should link projects to the account that created them

When request is submitted via the API the user will be authenticated against their SSO account. It should be recorded who requested the project set be created.

PR created for this change.


Add support for Read operation for custom resources

Currently CITF does not support Read operation for custom resources. This support needs to be added.
The client should be able to pass customresourcedefinition name and customresource and should get yaml/json of the resource.
Please refer to this file to see existing supported code.
Please comment if you want to be assigned for this issue.
Help links:
Contributors guide
hacktoberfest2018 channel on slack. Ping us here if you want to discuss anything around hacktoberfest

@princerachit I would like to work on it. To be clear the input parameters will be crdName, customresource, namespace and output should be to in yaml format.

I have a doubt. Should the logic be based on executing kubectl command or through client-go. If it is through client-go I think we need Custom resource struct to unmarshal/deserialize.

@krishnakarthik1309
We define the structs in a separate package. And yes you will get to know the structs for the CR that is already there in other projects. You will need to utilise that.

@krishnakarthik1309
You will require to use openebs client here which will do the job.
cc @Akash4927

HI @krishnakarthik1309 Instead of a struct, I think we can use the clientset of the CRDs that are already present in openebs/maya and openebs/node-disk-manager repo.

This has been addressed.
See following:
https://github.com/openebs/CITF/issues/45
https://github.com/openebs/CITF/issues/56
https://github.com/openebs/CITF/issues/48
https://github.com/openebs/CITF/issues/47
https://github.com/openebs/CITF/issues/46


Document kubectl subresource support

Ref: https://github.com/kubernetes/enhancements/issues/2590

/assign @chrisnegus
/milestone 1.24
/sig cli

/retitle [WIP] Document kubectl subresource support

/assign @soltysh

Hi @nikhita , since subresource support is introduced as alpha in v1.24, this PR should target the dev-1.24 branch, can you change the target branch?
Thank you

@reylejano @sftim base branch fixed.

@divya-mohan0209 addressed all comments. PTAL.

Lgtm, thanks!

@sftim can you help lgtm + approve?

Preview of updated pages:
https://deploy-preview-32604--kubernetes-io-main-staging.netlify.app/docs/reference/kubectl/cheatsheet/
https://deploy-preview-32604--kubernetes-io-main-staging.netlify.app/docs/reference/kubectl/conventions/
Technical lgtm was provided in comment
/lgtm
/approve


Implement Metadata as needed by CARTO VL

CartoVL is using the SQL API to obtain metadata about the dataset/query used, including a sample of the data.
We must avoid using the SQL API from CartoVL and obtain this data from the Maps API to avoid requiring both Maps API and SQL API authorization keys.
At this point we want to implement quickly what cartovl needs, and eventually refactor it into something more reasonable and efficient.
We could just implement now an ad-hoc endpoint performing the exact same queries/data processing we do now in CartoVL.
Or, if we consider the effort will be similar (I'm inclined to think so), to implement it as optional metadata returned by the map instantiation. This will offer opportunities for optimization (now or later) since some of the metadata may be already computed/used, we save requests and could also save queries by combining requested metadata with e.g. the needs of getAggregationMetadata).
We could add a parameter to request metadata, e.g. "metadata": { sample: true, rowCount: true, columnStats: true } and the data could be added to the existing metadata.stats in the response. This could be nicely encapsulated in the
setLayerStats function of the Windshaft-cartodb maps controller.
Details
What CartoVL does now
All the metadata CartoVL requests now is actually needed.
Tne windshaft module encapsulates all SQL API requests through getSQL, which is used by the next functions which are called to prepare the metadata (in _getMetadata):

getSample(conf, sampling)

query: SELECT * TABLESAMPLE BERNOULLI / random() < x
=> metadata.sample

getFeatureCount(query, conf)

query: SELECT COUNT(*) FROM ${query}
=> metadata.featureCount

getColumnTypes(query, conf)

query: select * from ${query} limit 0 => .fields (name: {type})
=> metadata.columns
=> metadata.categoryIDs, categoryIDsToName
Note that, depending to the field type, columns are categorized as numeric, date or category (all strings).

getNumericTypes(names, query, conf) (executed for numeric columns)

query: SELECT min($name), max($name), ...FROM ${query}
=> metadata.columns (name, type, min, max, avg, sum)
COUNT(*) computed but not used (?)

getDatesTypes(names, query, conf) (executed for date columns)

query:  SELECT min($name), max($name) FROM ${query}
=> metadata.columns (name, type, min, max)

getCategoryTypes(names, query, conf) (executed for category columns)

query: SELECT COUNT(*), ${name} FROM ${query} GROUP BY ${name}
=> metadata.categoryIDs, categoryIDsToName
=> metadata.columns (name, type, categoryNames, categoryCounts)

getGeometryType(query, conf)

query: SELECT ST_GeometryType(the_geom) FROM ${query}
=> windshaft.geomType

In addition to the metadata (categoryIDs, columns, featureCount, sample).
geomType is kept in windshaft object, used to deterimine if aggregation is possible and to decode MVT.
What the tiler already does at instantiation
The module query-utils of Windshaft-cartodb contains some functions to fetch metadata about the query.
In particular a function getAggregationMetadata used to determined if aggregation should be applied which returns:

An estimate of the row count

using CDB_EstimateRowCount (as meta.stats.estimatedFeatureCount)

the geometry type

using SELECT ST_GeometryType(${geom}) FROM (${query}) WHERE ${geom} IS NOT NULL LIMIT 1

When default aggregation is used (sampling) the columns of the original query are obtained with a LIMIT 0 query (in getLayerAggregationColumns) to set the columns layer parameter.
The map instantiation response contains layergroup.metadata.layers[0].meta.stats.estimatedFeatureCount (which could be extended for additional metadata). It also contains layergroup.metadata.layers[0].meta.stats.aggregation which could be used for aggregated stats at some point.

Whether it's a good idea to implement the metadata request in the map instantiation endpoint(s), or add a new specific endpoint.
Then we can start with the implementatin; other details can be decided later/after some experimentation:

Categories: now CartoVL requests all individual values with counts for any string column that appears in the query [FIXME: which appears in the query or which is used in viz?] This would be expensive in terms of data transfer for columns with many distinct values, such as ids in a large table.

We could alleviate this but returning only the top N categories, or by not returning individual values for columns with many individual values according to PG stats. But this has a problem: if a filter column='value' is used in such a columns it may not work (in the client) if the particular value is not in the metadata stats (because string values are mapped to floats for execution in WebGL). For simple cases the filter will be executed in the server, but not in general (e.g. if combined with AND with an expression not executable in the db)

We can use estimated or actual counts for the total row count. Or we can leave it to the client to request one or another.
Per type statistics (min, max for dates, min, max, avg, sum for numbers, etc.) can be automatically decided by the server or requested by the client.

Experimental map instantiation metadata is now available in #952
But there's a problem with returning metadata at instantiation and how we use it now at the client;
Carto VL is using metadata for these two details of the instantiation:

The number of features in the (unaggregated) data source is used to decide whether to use backend filters
The column type is used to cast date columns to text (so they survive through MVT)

Possible solutions

A. Add a separate Maps API endpoint to only get the data without instantiation
B. Re-instantiate the map if date columns are present or size is not large enough for filters
C. Add conditional filtering (based on num. rows) and casting to the Maps API (or conditional variant/conditional query: one for small datasets, other for larger ones)
D. Make use of metadata for instantiation unncessary:

date columns could be identified by user by having to use them always inside special functions to them (day, hour, timebuckets...) that also will allow to use future dimensions API revision and solve current problem with time aggregation.
backend filter could always be applied

Since MVT does not support date/time types, it would be nice to be able to cast those types into something (text strings or epoch numbers) that can be transferred in the MVT.

Automatically for all time/date columns in the tiler: when generating MVTs we detect column types with a LIMIT 0 query and wrap the query with explicit column selection.
On demand: the Maps API include some option to request column casts and proceed altering the query as in the previous case.
Automatically in the MVT Mapnik plugin (and in ST_ASMVT)

@rmdrodiguez has mentioned that Mapnik doesn't currently support time/date for styling, and implementing the automatic casting at the plugin level would not only make those types available in MVTs, but would allow to use them to style raster tiles.

I would like a flavor of D.
Regarding the timestamp management, I think it would be best if Maps API automatically cast it to a usable form. Ideally, it would be compressed in some way (no strings).
Regarding filters, I would move the conditional logic to Maps API. I wouldn't apply filters every time since we saw this is overkill for most maps (small and medium datasets) since they won't be able to instantly refilter with just client-side logic, and the MVT sizes would be small even without filtering taken into account.
Basically, I think Maps API should return an instantiated map and a flag saying if filters were applied or not (similar to aggregation). When the filters change in the client, CARTO VL should re-instantiate if the flag indicates that Maps API filtered in the last instantiation.

I reopen this to be closed after deployment.

Closing this.


fix: removing links to stale on-chain verification repos and replacing them with latest

current docs link to repos that are no longer maintained

Thanks!


Feature/django migration

creates a how-to installation file for django_migration branch

"Help us" (user number entry) page and links removed from frontend.


[master] added if enabled for certManager and webhooks resource

Good day
This feature is required for fine-tuning the environment in our clusters, specifically, we want to be able to disable all resources within the scope of a release.
custom-values.yaml
# Disabled all by default
global:
  podSecurityPolicy:
    enabled: false
    useAppArmor: false
  rbac:
    create: false

certManager:
  enabled: false

serviceAccount:
  create: false
  automountServiceAccountToken: false

securityContext:
  runAsNonRoot: false

prometheus:
  enabled: false
  servicemonitor:
    enabled: false
    honorLabels: false

webhook:
  enabled: false
  securityContext:
    runAsNonRoot: false
  serviceAccount:
    create: false
    automountServiceAccountToken: false
  hostNetwork: false

cainjector:
  enabled: false

  securityContext:
    runAsNonRoot: false

  serviceAccount:
    create: false
    automountServiceAccountToken: false

startupapicheck:
  enabled: false

  securityContext:
    runAsNonRoot: false

  serviceAccount:
    create: false
    automountServiceAccountToken: false

installCRDs: false

helm template . -f custom-values.yaml

Result, get release without resources

/assign @munnerz

disable all resources within the scope of a release

Can you please describe a use case where you would need to use these parameters (i.e why would you ever deploy the release without cert-manager controller and webhook)?
I am also concerned that having these parameters may suggest to users that webhook is optional which it is not.

The company has requirements for activating webhooks and controllers across releases. There are also patches that add our custom resources and want to deploy them as part of a cert-manager release without cert-manager resources.

If I understand correctly you're just trying to split out CRDs? We do publish a separate CRDs file, can you just utilize that to ensure that CRDs are applied before helm releases are installed if you have some workflow where you aren't sure of the ordering of the releases? https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml.
Otherwise I think you might need to describe the flow you are trying to achieve in a bit more detail, I'd be interested to hear what requirements there are for webhooks and controllers.

Sorry, it's hard to convey the whole idea in English.
CRD is one of the options.
The most important message is to separate into two releases in one controller in another webhook, in the third CRD, in the fourth pieces that are needed for network policies and monitoring.

Good day. I will try to explain again why we need this patch.
In our company, there are requirements for charts to meet certain criteria, such as the ability to completely disable all components in the chart, so that detailed manipulation can be performed later.
For example, to split into two separate releases, one with a separate controller, one with a separate webhook, and one with a separate CRD.
To perform such manipulation, we need to be able to disable each resource.
Example:
[root@control1 bash]# helm list
NAME                      	NAMESPACE      	REVISION    STATUS  	CHART                        	APP VERSION
cert-manager              	pfm-certmanager	1           deployed	cert-manager-v1.7.0-v2       	v1.7.0
cert-manager.clusterissuer	pfm-certmanager	2           deployed	cert-manager-v1.7.0-v2       	v1.7.0
cert-manager.crd          	pfm-certmanager	3           deployed	cert-manager-v1.7.0-v2       	v1.7.0
cert-manager.monitoring         pfm-certmanager 1           deployed	cert-manager-v1.7.0-v2       	v1.7.0

We also have a similar need to split out deployment of different parts of the project. Namely as we have cluster administrators who may manage the lifecycle of the CRDs, webhook and cainjector (which are all cluster scoped components) and the controller (which can be installed by ‘namespace admins’ in namespaced mode).
I think it should be very clearly documented the use case for these options if we add them, so people don’t assume it’s possible to use just a subset of components in isolation.

How can I help to speed up?

?

?

ping

ping

Colleagues, here is a mistech chart update, please confirm the changes.

@munnerz

@fraima could you signoff your commits? https://cert-manager.io/docs/contributing/sign-off/

@fraima As @munnerz said, we agree that we could create value by adding this as a usecase to our Helm chart. However, it should be clearly explained that you are supposed to run ALL components (possibly in multiple releases). Could you explain that eg. in the values.yaml file?

@fraima could you signoff your commits? https://cert-manager.io/docs/contributing/sign-off/

DONE

As @munnerz said, we agree that we could create value by adding this as a usecase to our Helm chart. However, it should be clearly explained that you are supposed to run ALL components (possibly in multiple releases). Could you explain that eg. in the values.yaml file?

DONE

hi, can you help me? I not understand how to fix DCO job.
The problem is caused by the merge commits that don't have a signoff message.
I'll update the PR for you so this issue is fixed.

/ok-to-test

Please update the issuer. Need a label of approval.

need merge

We've had a ready PR for a year now - please apply it.

We've had a ready PR for a year now - please apply it.

@munnerz @inteon @irbekrm ^


How to dynamically produce and consume topic data

After creating a Producer, I want to send data to multiple different topics, or create a Consumer, I want to consume data from multiple topics

For example, first subscribe to topic:persistent://public/default/my-topic:
Consumer consumer = client.newConsumer()
.topic("persistent://public/default/my-topic")
.subscriptionName("my-subscription")
.subscribe();
Then I want to dynamically subscribe to other topics: persistent://public/default/aaa or persistent://public/mynamesapce/bbb without restarting the service

When producing data, I want to dynamically send it to different topics. The topics may be in different namespaces and different tenants.

Like mqtt, a producer can send data on any topic, and a consumer can subscribe to topics dynamically

You can use client.newConsumer().topicsPattern() for multiple topics.

When producing data, I want to dynamically send it to different topics. The topics may be in different namespaces and different tenants.

In Pulsar client, you must create a separate producer for sending to a different topic. There are many ways to achieve what you are describing. The simplest approach is to create a producer, send the message and then close the producer. However, for most cases, this wouldn't be efficient, and producer caching is needed.
This challenge is handled in the Reactive Streams adapter for Pulsar Java Client, for example. However, if you aren't using Reactive Streams it would be an overkill to use the library only for producer caching.
Here is a code example in the Pulsar code base of producer caching:
https://github.com/apache/pulsar/blob/18f2f4a9c1dab7eec5c7c9590b76aca17ee1ead8/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/sink/PulsarSink.java#L148-L170
The problem with that solution is that the producers accumulate and don't get closed automatically. Closing idle producers can be achieved by using Caffeine cache as the "container". Caffeine's removalListener feature can be used to close the producer before disposing it.
I hope this helps implementing your use case.


.aar files from dependent android binding libs are not included in the nuget pack

My project is a .net 7 android binding lib, which depends on two other binding projects (ZoomCommonLib and Lottie.Android)
I'd like to create a single nuget package without having to bother with nugets for the two dependent projects. So your project is ideal.
However the .aars are not brought over. Since XF => MAUI, .jars must now appear in the same directory as dlls, as they are no longer embedded in the .dll (source - https://learn.microsoft.com/en-us/dotnet/maui/migration/android-binding-projects)
Nuget reference setup:
  <PackageReference Include="Teronis.MSBuild.Packaging.ProjectBuildInPackage" Version="1.0.0">
    <PrivateAssets>all</PrivateAssets>
    <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
  </PackageReference>

Project reference section:
  <ItemGroup>
    <ProjectReference Include="..\Lottie.Android\Lottie.Android.csproj" PrivateAssets="all" />
    <ProjectReference Include="..\ZoomCommonLib.MAUI\ZoomCommonLib.MAUI.csproj" PrivateAssets="all" />
  </ItemGroup>

Resulting Nuget package explorer - dlls from the dependent projects are brought in correctly, but the .aars are missing

Can .aars be brought in as standard, or is there an option to include them in IncludeAssets?
Changing to
 <IncludeAssets>all</IncludeAssets>
Did not change anything.

I got you.
Your problem has nothing to with  Teronis.MSBuild.Packaging.ProjectBuildInPackage. You need to add custom files always manually. This is so by default.
To add custom files you can use either thr TFM-agnostic approach:
https://learn.microsoft.com/en-us/nuget/reference/msbuild-targets#packing-a-file-without-an-extension
or the TFM-specific approach:
https://learn.microsoft.com/en-us/nuget/reference/msbuild-targets#targetsfortfmspecificcontentinpackage
You want the last.

Nice one @teneko


MVC app without preserveCompilationContext has no controllers

Without preserveCompilationContext: true - MVC's IAssemblyProvider won't include any assemblies, and thus the app won't find any controllers.

@pakrym,  what's the work item tracking this: https://github.com/aspnet/Mvc/pull/4261#issuecomment-194603013?

This looks like it's fixed with the newest DependencyContext (which we have a private copy of now). I'll leave this open until we pick up a new build.

This is fixed in the most recent builds of Mvc. Should remain fixed when we move to the newest build of DependencyModel.


zephyr kernel Kconfig USE_STDC_LSM6DS3TR and hal_st CMakeLists.txt lsm6ds3tr-c variable name mismatched (hyphen sign special case)

Describe the bug
zephyr kernel Kconfig USE_STDC_LSM6DS3TR and hal_st CMakeLists.txt lsm6ds3tr-c variable name are mismatched, this is becuase there is hyphen sign in lsm6ds3tr-c, and Kconfig need take care of it. Current implementation in zephyr code is to remove '-c'. And we need to the same in hal_st to support it.
Zephyr Knernel implementation:
https://github.com/zephyrproject-rtos/zephyr/blob/main/modules/Kconfig.st#L145
config USE_STDC_LSM6DS3TR
	bool

hal_st implementation:
https://github.com/zephyrproject-rtos/hal_st/blob/master/sensor/stmemsc/CMakeLists.txt#L61
set(stmems_pids
...
  lsm6ds3tr-c
...
  )
foreach(stmems_pid ${stmems_pids})
  string(TOUPPER ${stmems_pid} pid_to_upper)
  if(CONFIG_USE_STDC_${pid_to_upper})
    zephyr_include_directories(
        ${stmems_pid}_STdC/driver/
        )
    zephyr_library_sources(
        ${stmems_pid}_STdC/driver/${stmems_pid}_reg.c
        )
  endif()
endforeach()

To Reproduce
This leads to lsm6ds3tr-c register lib unusable, select USE_STDC_LSM6DS3TR is not able to load lsm6ds3tr-c
	select USE_STDC_LSM6DS3TR

To Fix
A propsal fix of hal_st: (singal patch)
diff --git a/sensor/stmemsc/CMakeLists.txt b/sensor/stmemsc/CMakeLists.txt
index 215cab6..e55560c 100644
--- a/sensor/stmemsc/CMakeLists.txt
+++ b/sensor/stmemsc/CMakeLists.txt
@@ -74,7 +74,8 @@ set(stmems_pids

 foreach(stmems_pid ${stmems_pids})
   string(TOUPPER ${stmems_pid} pid_to_upper)
-  if(CONFIG_USE_STDC_${pid_to_upper})
+  string(REPLACE "-C" "" pid_to_upper_ ${pid_to_upper} )
+  if(CONFIG_USE_STDC_${pid_to_upper_})
     zephyr_include_directories(
         ${stmems_pid}_STdC/driver/
         )

A more neatly fix is to patch both zephyr and hal_st, convert hyphen (-) to undersocre (_)
zephyr:
USE_STDC_LSM6DS3TR -> USE_STDC_LSM6DS3TR_C
hal_st
diff --git a/sensor/stmemsc/CMakeLists.txt b/sensor/stmemsc/CMakeLists.txt
index 215cab6..e8f7389 100644
--- a/sensor/stmemsc/CMakeLists.txt
+++ b/sensor/stmemsc/CMakeLists.txt
@@ -74,7 +74,8 @@ set(stmems_pids

 foreach(stmems_pid ${stmems_pids})
   string(TOUPPER ${stmems_pid} pid_to_upper)
-  if(CONFIG_USE_STDC_${pid_to_upper})
+  string(REPLACE "-" "_" pid_to_upper_ ${pid_to_upper} )
+  if(CONFIG_USE_STDC_${pid_to_upper_})
     zephyr_include_directories(
         ${stmems_pid}_STdC/driver/
         )

sensor dirver:
	select USE_STDC_LSM6DS3TR_C

Expected behavior
N/A
Impact
N/A
Logs and console output
N/A
Environment (please complete the following information):
N/A
Additional context
N/A

Closing with the correct reason as this was unproperly marked as "completed".
@JiapengLi would you be maybe willing to work on a PR to fix this?


Hope I can access to rawscore...

I changed
final double detectionThreshold = 0.7;
to 0.7 but I think sometimes even if rawscore value is about 0.5 , then it triggered....

with
result
        ?.listen((event) => {
              print("Recognition Result: " +
                  event["recognitionResult"].toString()),

I can't get rawScore value,
if I can get it, I can filter out on my self.
Thank you for great plugin.

Hi @kotran88,
Set the following value below to true.
 TfliteAudio.loadModel(
      model: 'assets/google_teach_machine_model.tflite',
      label: 'assets/google_teach_machine_label.txt',
      outputRawScores: true, //Set this true
      inputType: 'decodedWav');

Let me know if this solves your problem


Fine tuning ...

Hello,
You have provided the pre-trained model of VOC. I have a small dataset with 2 classes, which I annotated based on VOC and I want to fine-tune it. Would you please guide me through the process?

Maybe this helps
https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html

Thank you but that one is a classification task but yours is a semantic segmentation.
Besides, may I ask you how much is the best accuracy of this model on VOC evaluation?

I am not the author and I dont know the accuracy.
For fine-tuning, I think the methods are similar.
Just use your training data to fine-tune the layers you want and freeze the other layers.

I think this layer should be truncated and fine-tuned with a different number of classes:
model.add(Convolution2D(21, 1, 1, activation='linear', name='fc-final'))
otherwise the rest of the code uses the pre-trained network to train on anew dataset, but with 21 classes.

Also this layer inside context:
model.add(Convolution2D(21, 1, 1, name='ct_final'))
This makes it tricky for me, but the author certainty can do or give some tips.

Correct, first the number of features in those two layers to 2. Then, follow the keras blog to load the pretrained weights for all the other layers. If you want, you can freeze those layers, and then you can start training

Actually you load the pre-trained model already in the code, as I see. Am I right?
Therefore I should pop or delete these two layers and add new ones OR freeze the weights just before these layers to be trained with new class parameters.
In case of freezing, may I ask you I should freeze till what layer number?
Because these are two different layers in two different locations I'm little bit confused and I need your help. what I know is classification models which we only deal with one layer and examples are all about this, not the situations like we have here

Start by changing the model definitions to reduce the output dimension to 2 (the number of classes you want). This function will then error out: change it so that you don't load the weights for the last layer.
 You may want to look into freezing the weights for the existing layers, but it may not be required.


Docker image - vulnerabilities

Environment
Self-Hosted (Docker)
System
Docker (Various)
Version
3.1.1
Describe the problem
I have been running dashy (Docker image) on both Windows and Linux, I have noticed that, there are a number of Critical and Serious vulnerabilities with the image. Scout on Windows lists these very well, on both the latest and Auto tags. Are there any plans to address these?
I love this method of displaying links/apps etc, I'm very concerned of continuing to use it with these vulnerabilities.
Additional info
No response
Please tick the boxes

[X] You have explained the issue clearly, and included all relevant info
[X] You are using a supported version of Dashy
[X] You've checked that this issue hasn't already been raised
[X] You've checked the docs and troubleshooting guide
[X] You agree to the code of conduct

Hi
We take security quite seriously, could you share which vulnerabilites exactly you mean?
The ones displayed by node/npm/yarn ?
I think they have been discussed before but were marked as non-critical for dashy, but we can defenitely take a look.

Hello,
I use the following system to review the vulnerabilities of docker images, Docker Scout, which is embedded in the Windows Docker Environment, you will see from below that there are a number of vulnerabilities in the latest tag, I have also checked the auto tag which has also many vulnerabilities. Would suggest you run the Windows docker environment yourself have a interactive view of the issues.


Missing doc_values property in DateMapping

https://www.elastic.co/guide/en/elasticsearch/guide/master/doc-values.html says that doc_values parameter is enabled for date fields, but there is not such property in DateMapping. Could you please help me figure it out? Thanks a lot!

Thanks for reporting @Crassirostris.  Looks like this is also missing for the boolean and geo point field mapping types.


httpPayload does not work on union shapes

$version: "1.0"

namespace com.amazonaws.simple

use aws.protocols#restJson1

@restJson1
service SimpleService {
    operations: [
        HttpPayloadUnion,
// No AWS protocol supports this: https://github.com/awslabs/smithy/blob/3a0d0233b2cbffebae3e14459f8aea2e83978c72/smithy-aws-traits/src/main/java/software/amazon/smithy/aws/traits/protocols/ProtocolHttpPayloadValidator.java#L99-L100
//      HttpPayloadList,
//      HttpPayloadMap,
    ],
}

@http(uri: "/http-payload-union", method: "POST")
operation HttpPayloadUnion {
    input: HttpPayloadUnionInputOutput,
    output: HttpPayloadUnionInputOutput,
}

@http(uri: "/http-payload-list", method: "POST")
operation HttpPayloadList {
    input: HttpPayloadListInputOutput,
    output: HttpPayloadListInputOutput,
}

@http(uri: "/http-payload-map", method: "POST")
operation HttpPayloadMap {
    input: HttpPayloadMapInputOutput,
    output: HttpPayloadMapInputOutput,
}

structure HttpPayloadUnionInputOutput {
    @httpPayload
    unionShape: UnionShape
}

structure HttpPayloadListInputOutput {
    @httpPayload
    listShape: ListShape
}

structure HttpPayloadMapInputOutput {
    @httpPayload
    mapShape: MapShape
}

union UnionShape {
    listVariant: ListShape,
    mapVariant: MapShape
}

list ListShape {
    member: String
}

map MapShape {
    key: String,
    value: String
}

Generates in operation_ser.rs:
pub fn serialize_payload_http_payload_union_output(
    payload: &std::option::Option<crate::model::UnionShape>,
) -> Result<std::vec::Vec<u8>, aws_smithy_http::operation::BuildError> {
    let payload = match payload.as_ref() {
        Some(t) => t,
        None => return Ok(),     ■■ supplied 0 arguments
    };
    Ok(
        crate::operation_ser::serialize_member_com_amazonaws_simple_synthetic_http_payload_union_output_union_shape(payload)?
    )
}

Relevant generator code:
https://github.com/awslabs/smithy-rs/blob/ca2c7883e7c252b7686314b404ecc11d408460f5/codegen-core/src/main/kotlin/software/amazon/smithy/rust/codegen/core/smithy/protocols/HttpBoundProtocolPayloadGenerator.kt#L257-L281
The httpPayload trait:

can be applied to structure members that target a string, blob, structure, union, document, map, or list.

However, we're only handling string, blob, document, and structure shapes.
Of note is that no AWS protocol supports list or map, but HttpBoundProtocolPayloadGenerator is protocol-agnostic (in that it handles all protocols that support HTTP binding traits). That's something to take into account if/when we ever support a non-AWS protocol where binding to list or map shapes is allowed.
When we fix this bug, we should ensure to include an else branch to that when that aborts code generation with an informative message, and contribute HTTP protocol tests to awslabs/smithy.

I think we're blocked on https://github.com/awslabs/smithy/issues/1462.


Question: What's the quickest path to clone an ES instance?

What command should I be running to clone one ES instance (all indexes, mappings, etc) to another instance. I tried copying index-by-index across two AWS ES instances and the indexes weren't created on the destination instance. 😕
Thanks!

Mapping (which will create the index)
Analyzer
data.

check out https://github.com/taskrabbit/elasticsearch-dump#multielasticdump, which can do it all.

@evantahler Thanks! I imported the analyzer before the mapping so I guess that's why I don't have indexes? I'll give it another go with the order you provided.
I'll also take a look at multielasticdump, thanks!

This is not only worth documenting in the readme; I would go as far as to add --type=all to do it right from the get-go.

@MarkHerhold that's a great feature request, let me know if you need help getting started on that pull request!

Subtle :-)


Is there any way to clear the queue?

I want my crawler to iterate a number of times and clear the queue for each iteration. Can you show me an example on how to do that?

What do you want to iterate over? A set of pages or an entire site? And is it necessary to clear the queue, or do you just need to re-queue a set of URL's?

Clearing the queue completely is the best option for my use case

There's no inbuilt method to clear the queue, but you can easily assign a fresh queue (and allow the old one to be garbage collected) like this:
// When resetting
mycrawler.queue = new Crawler.queue()

Let me know if that satisfies your use case!
As an aside — and in the interests of understanding what features we should prioritise, if you wouldn't mind sharing why you want to clear the queue, it'd be much appreciated! I'm very curious. :)

Hey, just closing this issue. If my comment above didn't help, and you're still in need of something different, feel free to reopen the issue with a bit more detail. :)

About to open feature request issue and found this.. IT works. Many Thanks

@cgiffard
My previous comment might be wrong or I miss something when I refactor the code.
I try to reuse the module, give a new path and craw.
var crawler = new Crawler("http://www.example.com/");
// repeat this
function craw(path) {
crawler.initialPath = path;
crawler.queue = new Crawler.queue();
crawler.start();
}
//
craw('/Category.aspx?category=abc');
craw('/Category.aspx?category=xyz');
It does not grab different url. It only pull the first URL.

@mike-aungsan You might be better off making a new crawler for each crawl, if you're trashing the queue each time.

@cgiffard  Many Thanks.


TestStep variables proof of concept

PoC for https://github.com/linuxboot/contest/issues/25
Introduced a new interface for step plugins:
type StepsVariables interface {
	Add(tgtID string, name string, v interface{}) error
	Get(tgtID string, name string, v interface{}) error
}

that can add/read variables from current/previous steps.
A mapping between external->internal variables is represented as StepVariablesMapping type which was added to step bundles.
In order to keep PR short, will add parameters template + better testing including e2e in the next PR

Get(tgtID string, name string, v interface{}) error

It puts the value into v?

Get(tgtID string, name string, v interface{}) error

It puts the value into v?

yes, probably I should rename method/variable for more clearity. v -> out

Codecov Report

Merging #83 (92fe7df) into main (775e930) will decrease coverage by 0.31%.
The diff coverage is 52.87%.

:exclamation: Current head 92fe7df differs from pull request most recent head d34f891. Consider uploading reports for the commit d34f891 to get more accurate results

@@            Coverage Diff             @@
##             main      #83      +/-   ##
==========================================
- Coverage   66.57%   66.26%   -0.32%
==========================================
  Files         157      157
  Lines        9578     9717     +139
==========================================
+ Hits         6377     6439      +62
- Misses       2522     2581      +59
- Partials      679      697      +18

Flag
Coverage Δ

e2e
50.75% <32.50%> (-0.62%)
:arrow_down:

integration
57.54% <34.68%> (-0.59%)
:arrow_down:

unittests
48.65% <46.54%> (-0.02%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/jobmanager/bundles.go
70.83% <ø> (ø)

pkg/pluginregistry/errors.go
33.33% <0.00%> (-66.67%)
:arrow_down:

pkg/test/step.go
73.33% <ø> (ø)

plugins/teststeps/echo/echo.go
8.00% <0.00%> (ø)

plugins/teststeps/randecho/randecho.go
6.66% <0.00%> (ø)

plugins/teststeps/sshcmd/sshcmd.go
1.20% <0.00%> (ø)

tests/plugins/teststeps/channels/channels.go
0.00% <0.00%> (ø)

tests/plugins/teststeps/hanging/hanging.go
0.00% <0.00%> (ø)

tests/plugins/teststeps/noreturn/noreturn.go
33.33% <0.00%> (ø)

tests/plugins/teststeps/panicstep/panicstep.go
0.00% <0.00%> (ø)

... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 775e930...d34f891. Read the comment docs.

Get(tgtID string, name string, v interface{}) error

It puts the value into v?

yes, probably I should rename method/variable for more clearity. v -> out

Let's at least rename the variable v in the interface to in and out accordingly?

My main concern here is that the PR contributes into tech debt we have in TestRunner. Let me quote myself from another chat:

Basically it is a mess of different ideas intertwined in muddy implicit fuss.

My main concern here is that the PR contributes into tech debt we have in TestRunner. Let me quote myself from another chat:

Basically it is a mess of different ideas intertwined in muddy implicit fuss.

But I guess solving this issue is a whole another epic story.

I would say the main tech debt is in convoluted state machine implementation. If it will be moved into a separate logic, everything will be straightforward. Saving additional info at the end of the TestRunner.Run is straightforward

would also be useful to have a "permitted outputs" type of array in the plugin itself, and the framework checks if that's valid when emitting; like the Events one. this would also be able to be extracted as docs, otherwise there's no way of knowing what output a plugin produces without reading the code

would also be useful to have a "permitted outputs" type of array in the plugin itself, and the framework checks if that's valid when emitting; like the Events one. this would also be able to be extracted as docs, otherwise there's no way of knowing what output a plugin produces without reading the code

I don't know, on the other hand it makes you each time to specify all the variables when using the plugin. We can add this later as improvement

makes you each time to specify all the variables when using the plugin
yes, but it can also act as a filter; if youre only interested in some outputs, you only write those for that specific plugin. The main point is to make things obvious for the reader, and adds some safety on top.
eg.

- name: cmd
  label: cmd0
  parameters:
    - ...
  outputs:
    - stdout

- name: echo
  label: echo0
  parameters:
    - message: "cmd said {{step_output `cmd0.stdout`}}"

# this would be an error
- name: echo
  label: echo1
  parameters:
    - message: "cmd said {{step_output `cmd0.stderr`}}"

And to get a list of that permitted outputs one still needs to read either a doc or code
only valid for the first writer, these descriptors are usually a write once, read many kind of thing so it would have value for the readers

only valid for the first writer, these descriptors are usually a write once, read many kind of thing so it would have value for the readers
If you mean copy-paste of test descriptors :)

Anyway we can add it later if we start to feel the need of that kind of documentation

Rebased the PR. @mimir-d could you please restamp it?

Found a bug introduced by my previous PR, the fix is in https://github.com/linuxboot/contest/pull/121


Beeg Grabber not working

[ x] I'm reporting a broken site support
[ x] I've verified that I'm running youtube-dl version 2021.12.17
[ x] I've checked that all provided URLs are alive and playable in a browser
[x ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x ] I've searched the bugtracker for similar issues including closed ones

Verbose log
/usr/bin/python3 /usr/local/bin/youtube-dl --verbose https://beeg.com/-0607688354383627
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', 'https://beeg.com/-0607688354383627']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.6.8 (CPython) - Linux-4.18.0-348.7.1.el8_5.x86_64-x86_64-with-centos-8.5.2111
[debug] exe versions: ffmpeg 4.2.5, ffprobe 4.2.5
[debug] Proxy map: {}
[generic] -0607688354383627: Requesting header
WARNING: Falling back on generic information extractor.
[generic] -0607688354383627: Downloading webpage
[generic] -0607688354383627: Extracting information
ERROR: Unsupported URL: https://beeg.com/-0607688354383627
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
ie_result = ie.extract(url)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 534, in extract
ie_result = self._real_extract(url)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/generic.py", line 3489, in _real_extract
raise UnsupportedError(url)
youtube_dl.utils.UnsupportedError: Unsupported URL: https://beeg.com/-0607688354383627
Description
Beeg Grabber isn't working anymore. It says Unsupported URL while url is working correctly.
Thank you.

A load of changes at that site. Should be fixed by PR #29861 but apparently that also needs work now. After a small change:
# python -m youtube_dl -F -v 'https://beeg.com/-0607688354383627'
[debug] System config: []
[debug] User config: [u'-f', u'(best/bestvideo+bestaudio)[height<=?1080][fps<=?60][tbr<=?1900]']
[debug] Custom config: []
[debug] Command-line args: [u'-F', u'-v', u'https://beeg.com/-0607688354383627']
[debug] Encodings: locale ASCII, fs ASCII, out ASCII, pref ASCII
[debug] youtube-dl version 2021.06.06
[debug] Git HEAD: 041cd20a7
[debug] Python version 2.7.1 (CPython) - Linux-2.6.18-7.1-7405b0-smp-with-libc0
[debug] exe versions: ffmpeg 4.1, ffprobe 4.1
[debug] Proxy map: {}
[Beeg]<PHONE_NUMBER>383627: Downloading webpage
[Beeg]<PHONE_NUMBER>383627: Downloading JSON metadata
[info] Available formats for<PHONE_NUMBER>83627:
format code   extension  resolution note
fl_cdn_multi  mp4        unknown
fl_cdn_240    mp4        240p
fl_cdn_480    mp4        480p
fl_cdn_720    mp4        720p
fl_cdn_1080   mp4        1080p      (best)

What you one say

After applying #29861 fix and typing you command I get:
[root@ tmp]# /usr/bin/python3 -m youtube_dl -F -v 'https://beeg.com/-0607688354383627'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-F', '-v', 'https://beeg.com/-0607688354383627']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.6.8 (CPython) - Linux-4.18.0-348.7.1.el8_5.x86_64-x86_64-with-centos-8.5.2111
[debug] exe versions: ffmpeg 4.2.5, ffprobe 4.2.5
[debug] Proxy map: {}
[Beeg]<PHONE_NUMBER>383627: Downloading webpage
[Beeg]<PHONE_NUMBER>383627: Downloading JSON metadata
Traceback (most recent call last):
File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/main.py", line 19, in
youtube_dl.main()
File "/usr/local/lib/python3.6/site-packages/youtube_dl/init.py", line 475, in main
_real_main(argv)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/init.py", line 465, in _real_main
retcode = ydl.download(all_urls)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 2078, in download
url, force_generic_extractor=self.params.get('force_generic_extractor', False))
File "/usr/local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 808, in extract_info
return self.__extract_info(url, ie, download, extra_info, process)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
return func(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
ie_result = ie.extract(url)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/common.py", line 534, in extract
ie_result = self._real_extract(url)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/beeg.py", line 181, in _real_extract
return self._old_real_extract(url, video_id, webpage)
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/beeg.py", line 88, in _old_real_extract
for rcnt, api_path in reverse_enumerate(('', 'api.')):
File "/usr/local/lib/python3.6/site-packages/youtube_dl/extractor/beeg.py", line 86, in reverse_enumerate
return zip(reversed(zip(*enumerate(l))[0]), l)
TypeError: 'zip' object is not subscriptable
Thank you.

There's a Python2-3 issue with zip(). You can't get index 0 of the Py3 version: instead you have to get the next(...) of it; conversely you can't get the next(...) of the Py2 version. Options are to import future_builtins in Py2, making the definitions of zip() compatible, or to write next(iter(zip(...) or, less good, tuple(zip(...))[0], either of which works for both Py2 and 3.
But, as I noted, there are other (site-related) issues that prevent the PR from working. I'll update it soon.


areas (km² or m²) does not seem to work

I liked to calculate the areas of the voronoi regions,
but in my example they was much smaller than they should be (0.11715301m² for whole of Berlin).
When I tried do fix the bug, I took a look at random_points_and_area.py
and printet the sum of the areat there
area_sum=0
for a in poly_areas:
   print("area is: %s"%a)
    area_sum=area_sum+a
print("area_sum is: %s"%area_sum)
print("area_sum from shape is: %s"%calculate_polygon_areas([area_shape], m2_to_km2=True))

where I got:
area is: 38047.414248663496

area is: 81558.23012468345

area is: 45652.50539830615

area is: 37314.4644019487

area is: 85299.03205647763

area is: 97225.72636558591

area is: 60036.51052754671

area is:<PHONE_NUMBER>7811942

area is:<PHONE_NUMBER>8640596

area is: 28103.058823670108

area is: 68784.31109230645

area is: 36575.84996001104

area is: 16695.936399656446

area is: 15257.965532032309

area_sum is:<PHONE_NUMBER>954138

area_sum from shape is: [863999.57229541]

But according to Wikipedia it should be: 505.970 km²

Thanks for spotting this. I used the wrong projection before calculating the areas. One should always use an Equal Area projection before calculating areas. I switched to LAEA Europe projection and now the sum is almost as reported on Wikipedia:
areas in km²:
[19766.86024891 31168.503297   34305.73485662 31272.51239666
 26466.88487366 33067.69869796 31443.55697406 30221.83664307
 58889.06936024 20570.89231156 26280.49994667 22134.7166086
 39027.36182935 68215.05103782 18857.893836   10619.19585911]
sum:
502308.2687772701

The small difference comes from the fact that the example only uses mainland Spain without the islands.
You should make sure to use an Equal Area projection in your project, too!


python310Packages.siobrultech-protocols: 0.5.0 -> 0.6.0

Description of changes
https://github.com/sdwilsh/siobrultech-protocols/releases/tag/v0.6.0

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
22.11 Release Notes (or backporting 22.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 176499 run on x86_64-linux 1

5 packages built:

home-assistant-component-tests.greeneye_monitor
python310Packages.greeneye-monitor
python310Packages.siobrultech-protocols
python39Packages.greeneye-monitor
python39Packages.siobrultech-protocols


Performance Query

Hi,
As per the docs below are suggested when designing events and rules:

Shorten number of fields inside rules, the less key in the rules, the short path to find them out.
Shorten number of fields inside event, the less key inside event, the less attempts will be required to find out rules.

As per my knowledge, the rules are matched with the input event using a state machine. So the number of keys inside the rule matters for performance. But how does the number of fields inside a event matter for performance.
Suppose we have a rule with 2 keys. If the input event has 10 keys or 100 keys, the matching is done only for the keys inside rule correct? In this case it does 2 matches irrespective of number of keys in event correct?

It's complicated. The process of matching has two phases - the incoming event has to be "flattened" - turned into a list of fields, i.e. key/value pairs, and sorted.  Then the flattened form is run over the machine.  Typically, the flattening takes >50% of the execution time.
Only fields that appear in one or more Rules in the Machine are included in the flattened output, the rest are bypassed quite efficiently. However, running over all the data that will not be used in matching is not free.
So the matching time is related to the number of fields that are mentioned in one or more rules, and the total size of the event.

got it. Thanks


sh: can't create /system/vendor/build.prop: Read-only file system

Laptop configs from 'neofetch' command
OS: Ubuntu 22.04.3 LTS x86_64
Kernel: 6.2.0-39-generic
CPU: AMD Ryzen 7 4800H with Radeon Graphics (16) @ 2.900GHz
GPU: AMD ATI 05:00.0 Renoir
GPU: NVIDIA 01:00.0 NVIDIA Corporation TU117M
Genymotion Version 3.6.0 | Revision 20231219-22ea4e2c8f.
For the emulated phone, I use the 'Custom Phone', the first option on the list. But I adjusted the RAM to be 4096 MB.
Get stuck at the 'tee' command in the second step, which reports the only one line as the title shows.(If I need to be '--verbose', simply tell me.)
My attempt:
:/ #  ls -l /system/build.prop:
-rw------- 1 root root 3987 2023-09-08 14:21 /system/build.prop
:/ #  chmod 622 /system/vendor/build.prop
chmod: chmod '/system/vendor/build.prop' to 100622: Read-only file system
Now I totally have no idea what is going on.

Have you tried to run this command before running tee command?
adb shell

su

mount -o rw,remount /

Thank you for your screenshot. It seems that I directly clicked the "copy" button for the commands,
'su
mount -o rw,remount / '
so then second line is actually not executed. Sorry for my bad.


Add PeerJ community

While we await content, we should add the community and its about page using placeholder text.

Close with the placeholder description, or leave open until it is provided?

Will close because https://github.com/libero/prc/issues/134 will cover the description.


Update BASE_VERSION to master-2022-02-09T22-48-14

@istio-testing: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-pilot-istiodremote_istio_release-1.13
98cf7176e8faa7a8c496dfde47eba8eb786d27c1
link
true
/test integ-pilot-istiodremote_istio_release-1.13

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Don't need this one either.


Bugfix infinite looping with references 5.2.1 and below

Swapping out array contents by multiple references while looping over an array recursively in a version of PHP where both of these things alter and depend on the array pointer is not a bright idea.
In the face of overwhelming apathy from the many many users that complained since this bug was introduced to such ancient PHP versions months ago I've decided to just disable reference detection entirely below 5.2.2.


Bug: Files generated via swift_proto_library are not accessible on Xcode file tree

Describe the bug

Maybe, this should be classified as a feature request rather than a bug.
Files generated via swift_proto_library are not appeared in the Bazel Generated Files section of the Xcode file tree.
Therefore, jump-to-definition and autocomplete work well, but file search and syntax highlighting do not.
Probably, this is related to https://github.com/buildbuddy-io/rules_xcodeproj/issues/1744
Reproduction steps

Define swift_proto_library target like below, and add it to the xcodeproj.
proto_library(
    name = "example_proto",
    srcs = glob([
        "*.proto",
    ]),
)

swift_proto_library(
    name = "example_proto_swift",
    deps = [":example_proto"],
)

swift_library(
    name = "example_lib",
    ...
    deps = [":example_proto_swift"]  ,
)

ios_unit_test(
    name = "example_test",
    deps = [":example_lib"],
)

xcodeproj(
   name = "xcodeproj",
   ...
   top_level_targets = [
       top_level_target(":example_test", target_environments = ["simulator"]),
   ],
)

Expected behavior

When an Xcode project is generated, rules_xcodeproj places the files to be generated in xxx/bin/external/yyyy under Bazel Generated Files as missing files, and then to be available after you build the target.
All the following Xcode features are enabled for the generated files and symbols.

Jump to definition
Auto completion
Syntax highlighting
Xcode file search
Xcode text search

Versions

Bazel: 6.0.0-pre.20220804.3
rules_xcodeproj: 1.2.0
rules_apple: d04edf4136b4818662d7126582ec257aee372339 (not a release tag for some reason)
rules_swift: 6153a848f747e90248a8673869c49631f1323ff3 (not a release tag for some reason)

Additional context

We also have a target defined using rules_proto_grpc, and the files generated by it appear on the Xcode file tree. It has a rule just to compile the protos and then the files are consumed by the official swift_library rule.

Addressed by https://github.com/MobileNativeFoundation/rules_xcodeproj/pull/2484.


Use file-based logs for self-hosted

A TODO in the future is to allow for pluggable logger backends.

#387  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @acco and the rest of your teammates on  Graphite


Enable loop

Small changes that allow programmer to set the loop bytes

How well-supported is the loop setting these days? Back 10 years or so ago, I found that lots of browsers just totally ignored it and looped forever anyway. So I worry it might cause confusion to expose the loop bytes - "I tried to turn off looping, why is my gif still looping when I view it in Chrome?"
I'd also prefer to preserve API compatibility by adding new parameters at the end of the parameter list - with this change as is, if some client is setting bitdepth they will see a behavior change if they update the .h.

How well-supported is the loop setting these days? Back 10 years or so ago, I found that lots of browsers just totally ignored it and looped forever anyway.

I made a little test here, drag and drop the gif to play on browser and also putting an img in a html file to see those that respect the loop, these are the results with loop = 1:

Browser
Drag and drop
HTML image

Chrome
✔
✔

FireFox
✔
✔

IE
✔
❔

Android HTML Reader
-
✔

Firefox Android
-
✔

Chrome Android
-
✔

IE don't play the gif image, so no way to know if it works.
Also, I tested in some others applications:

App
Works

Gmail Web
✔

Gmail Android
✔

DIscord Web
✔

Android Galery
❌

So I worry it might cause confusion to expose the loop bytes - "I tried to turn off looping, why is my gif still looping when I view it in Chrome?"

How about a comment just like the comment in the delay parameter?
The delay value is the time between frames in hundredths of a second - note that not all viewers pay much attention to this value.

I'd also prefer to preserve API compatibility by adding new parameters at the end of the parameter list - with this change as is, if some client is setting bitdepth they will see a behavior change if they update the .h.

I can change the parameter order if you want.

Sorry for the delay.
Now I reduced the comment size, changed loop to repeat (which makes more sense), and the repeat variable is on the end of the argument list


Conference room: Firefox ESR version can't load conference room

Description: Currently it is not possible to join the conference with a Firefox ESR browser (version 102.10.0esr (64-bit)). The normal Firefox release instead loads the jitsi app without problems.
Error message:

Reproduction:
pre-conditions: Open an instance with Firefox ESR. Setup the conference room feature for a meeting.

open the meeting
join the conference room. Spinning Wheel appears.

What should happen:
The Firefox ESR version should also open the jitsi conference room.

Tested it on Firefox 115.3.1esr: It still doesn't work. The loading spinning wheel in the livestream window spins without loading the Jitsi conference. But, the error message didn't appear anymore. I guess, that the reason of issue switched.


How can I get the api key from the developer console?

Also, it would be nice to specify that in the documentation

Hi,
I'm afraid I can't help you here as I haven't worked with mobile applications for more than 6 years now.
I'd be more than happy to accept a PR, though.


fix: headless pull with imported resources

Description of changes
Changed the headless parameter validation when pulling a project with imported resources:

If no headless parameters are present for a given resource, use parameters loaded from ParameterStore.
If any headless parameters are given for a particular resource, then all headless parameters must be present or the CLI will throw an error like it does today.

Description of how you validated changes
Ran e2e locally and manual testing. Full e2e.
Checklist

[x] PR description included
[x] yarn test passes
[x] Tests are changed or added
[x] Pull request labels are added

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Codecov Report

Merging #12063 (ad9c1db) into cdkv2 (379e360) will decrease coverage by 3.56%.
The diff coverage is 52.48%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##            cdkv2   #12063      +/-   ##
==========================================
- Coverage   50.18%   46.62%   -3.56%
==========================================
  Files         702      826     +124
  Lines       33976    37978    +4002
  Branches     6976     7762     +786
==========================================
+ Hits        17050    17707     +657
- Misses      15447    18604    +3157
- Partials     1479     1667     +188

Impacted Files
Coverage Δ

...sync-simulator/src/data-loader/opensearch/index.ts
45.45% <0.00%> (ø)

...s/amplify-appsync-simulator/src/type-definition.ts
100.00% <ø> (ø)

...sync-simulator/src/velocity/util/dynamodb-utils.ts
6.55% <0.00%> (ø)

...y-appsync-simulator/src/velocity/util/map-utils.ts
10.00% <ø> (ø)

...tor/src/velocity/util/transform/dynamodb-filter.ts
4.93% <0.00%> (ø)

...ync-simulator/src/velocity/util/transform/index.ts
21.05% <0.00%> (ø)

...appsync-simulator/src/velocity/value-mapper/map.ts
86.88% <ø> (ø)

...ackages/amplify-category-auth/src/commands/auth.ts
0.00% <0.00%> (ø)

...es/amplify-category-auth/src/commands/auth/help.ts
0.00% <0.00%> (ø)

.../amplify-category-auth/src/commands/auth/import.ts
0.00% <0.00%> (ø)

... and 523 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


A timeout function. Example: `setTimeout(_.timeout(function () {console.l

A timeout function.
Example:
setTimeout(
  _.timeout(
    function () {
      console.log('done');
    },
    function () {
      console.log('ouch');
    },
   100
  )
, 200);

Right -- looking at this again, I don't think that this API helps you in the Ajax case. It's too awkward to have to pass the Ajax call into _.timeout, and then also use the function result of _.timeout so that the expires callback could be canceled. Nice idea though.


Parsing logic for SHOW SCHEMAS

Codecov Report

:exclamation: No coverage uploaded for pull request base (datafusion-sql-planner@ec7f2a6). Click here to learn what that means.
The diff coverage is n/a.

@@                    Coverage Diff                    @@
##             datafusion-sql-planner     #697   +/-   ##
=========================================================
  Coverage                          ?   67.37%
=========================================================
  Files                             ?       73
  Lines                             ?     3638
  Branches                          ?      753
=========================================================
  Hits                              ?     2451
  Misses                            ?     1047
  Partials                          ?      140

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Specify model size in the InferenceService CRD

Would be nice having a new parameter in the InferenceService CRD that allows user to specify the model size (the size in bytes), avoiding the MODEL_MULTIPLIER factor to estimate the size.
Is your feature request related to a problem? If so, please describe.
The heuristic used to calculate the model size (model size on disk * MODEL_MULTIPLIER) is not always accurate because the amount of  memory used by a model on a GPU could be greater and sometimes it could be possible to face OOM errors. Due to this problem the number of total models that can stay loaded on the GPU is not estimated correctly.
We already faced this issues using Triton as serving runtime.
Describe your proposed solution
New parameter in the InferenceService CRD that allows user to specify the model size, avoiding the MODEL_MULTIPLIER factor to estimate the size.

I'm using Triton's Python backend to load models from Hugging Face Hub. The disk size of config.pbtxt and model.py is around 12K, so MODEL_MULTIPLIER ends up being the average model size, which can vary  from 5G to 25G!
This difference impacts model placement decisions. We really need a better way to estimate/set model size.


fix license

What's changed?
[improve]fix github unknown licenses found

Checklist

[ ]  I have read the Contributing Guide
[ ]  I have written the necessary doc or comment.
[ ]  I have added the necessary unit tests and all cases have passed.

Add or update API

[ ] I have added the necessary e2e tests and all cases have passed.

ok


Pro: pharos license assign for another cluster

Fixes #1194
Makes it possible to assign a license to another cluster by giving --cluster-id and --cluster-name. The --config is not needed in that case and there will be no attempt to connect to the cluster.
$ pharos license assign --cluster-id abcd --cluster-name foo abcd-defg-123
FDSKJVNDSKJNVkjdsnvcksdjnckdsjnckjsd.CJDNkcjdsnkcjdnskcjn.dskvdsvdsv
$ pharos license assign --cluster-id abcd --cluster-name foo abcd-defg-123
ERROR: License is already in use

The pharos license inspect can output the cluster id and name with --cluster-info:
$ pharos license inspect -c cluster.yml --cluster-info
=> Sharpening tools ...
...
...
Cluster ID: 37cxxveb-4961-11e9-9xxx51-0yyy1f34da68
Cluster name: lively-sound-7972

@nevalla PTAL

--print-subscription-token is a bit long flag to type. How about --output|-o=token?

So, an option with only one possible value or some shorter flag?

atm just a shorter flag. Name of the option came from kubectl
 -o, --output='': Output format. One of:
json|yaml|wide|name|custom-columns=...|custom-columns-file=...|go-template=...|go-template-file=...|jsonpath=...|jsonpath-file=...

Should there then also be --output default?
Feels a bit weird to write something like:
    option %w(-o --output), "FIELD", 'possible values: token', attribute_name: :output_field do |field|
      signal_usage_error "valid values for --output are 'token'" unless field == 'token'
      field
    end

Maybe drop the option and always just print the jwt when --cluster-id given

Maybe drop the option and always just print the jwt when --cluster-id given

I'm with that. @jakolehm any opinions?

Maybe drop the option and always just print the jwt when --cluster-id given

I'm with that. @jakolehm any opinions?

SGTM


⚠ Bofrost.be has degraded performance

In de77e71, Bofrost.be (https://www.bofrost.be) experienced degraded performance:

HTTP code: 200
Response time: 880 ms

Resolved: Bofrost.be performance has improved in 44f23e4.


Introduce OneOf and AnyOf types

This merges into master - previously merged into v3 branch.

bors r+

bors r+

bors r+


CVE description is not mapped correctly

CVEs mirrored from the NVD are missing a description.

Cause appears to be that mirror-service puts the CVE description in the vulnerability.description field, whereas the API server is expecting it to be in the vulnerability.detail field:

https://github.com/DependencyTrack/hyades/blob/6a80fa570b7d03be12bf11295304491062738e6d/mirror-service/src/main/java/org/hyades/vulnmirror/datasource/nvd/NvdToCyclonedxParser.java#L65
https://github.com/DependencyTrack/hyades-apiserver/blob/1125891290ccbd57fe90db3b0975b9123d3d642b/src/main/java/org/dependencytrack/parser/hyades/ModelConverterCdxToVuln.java#L40

Using description for titles / summaries and details for the actual, full-length description, makes the entire thing a bit ambiguous - as demonstrated by this bug. I wonder if instead we should use a dedicated vulnerability property for title / summary.
The property should follow a proper taxonomy, e.g. something like dtrack:vuln:summary.

There already is a registered namespace for dependency-track: https://github.com/CycloneDX/cyclonedx-property-taxonomy#registered-top-level-namespaces


Integrated terminal results in vscode freezing when increasing scrollback to 10000 lines

VS Code Version: 1.91.1
OS Version: Darwin arm64 23.5.0

When I set the scrollback to 10000 lines the whole IDE becomes laggy or freezes whenever I resize the terminal or whenever I run clear command in the terminal. It seems to me that modern hw should handle 10000 lines just fine and for example the default Mac OS terminal has no issue with that. I need the larger scrollback since I get a lot of logs when debugging.

/duplicate https://github.com/microsoft/vscode/issues/219559


Symbolic visualization

Hello, so i found this repo really fascinating but can i extract params from each layer and somehow visualize it like in the kan paper or do you plan working on this later?

Hi there, @NQTri00 ! Thanks for your feedback. It seems a nice and helpful feature, I'll add this in the next few days, thanks for your suggestion

Looking forward to your open-source visualization code！


Feat/add helm build options

Changes

Adds KeyValueOption for generic key-value flags in cli
Adds CLIPathOption to store generic CLI tools inside the base image
Adds hash.HashFolder method to generate a digest hash based on a folder's content
Adds maps package for map manipulation
Move artifacts package into apis/artifacts/v1alpha1
Deprecates artifacts in favor of apis/artifacts/v1alpha1
Adds command/exec for more complex command execution (copied from kubernetes-sigs/kind)
Fixes logger context to make the same for command/logger and knative.dev/pkg/logging
Context option adds validation to check if provided context exists

Submitter Checklist
As the author of this PR, please check off the items in this checklist:

[ x ] spec PR link included
[ x ] Follows the commit message standard
[ x ] Meets the contributing guidelines (including
functionality, content, code)
[ x ] Test cases with documentation and functionality works as expected using current and related github repos (MUST deploy and check)
[ x ] Release notes block below has been filled in or deleted (only if no user facing changes)

Release Notes
- Adds KeyValueOption for generic key-value flags in cli
- Adds CLIPathOption to store generic CLI tools inside the base image
- Adds `hash.HashFolder` method to generate a digest hash based on a folder's content
- Adds `maps` package for map manipulation
-  Move `artifacts` package into `apis/artifacts/v1alpha1`
- Deprecates `artifacts` in favor of `apis/artifacts/v1alpha1`
- Adds `command/exec` for more complex command execution (copied from kubernetes-sigs/kind)
- Fixes logger context to make the same for `command/logger` and `knative.dev/pkg/logging`
- Context option adds validation to check if provided context exists

为了deprecate pkg/artifacts 那个包，复制了代码到 pkg/apis/artifacts/v1alpha1 导致sonarqube duplications 指标超标了，等其它地方都改成新的包之后那部分代码会移除


Add Swift Package Manager compatibility

I had to move a lot of files in the cmark lib to make it compatible with the swift package manager. I tested the project with XCode with the sample and everything works. I tested also with a simple project using SPM and it works as well. After the Merge, the project will need a version Tag x.x.x in order to be clonable by SPM as a dependency.

Sorry I didn't see this PR this week-end I'll review it ASAP. I've got a lot of things to do today.

Ok I forgot a bit. Must be a side effect of working 12h a day I think...
I merge it right away !


Fat JAR support with Maven

I was wondering if it's possible to adjust the plugin so it just updates the main fat jar (made with Shade plugin) instead of copying every possible dependency.

For Maven we don't even have a plugin, so I see no reason why you couldn't adapt the example pom to use Shade. I wouldn't recommend it though, as even the smallest code change in your actual app jar would cause every user to have to download the entire app again. Having many small files is a major advantage, and I don't see a single disadvantage to that approach. Do you?

Normally I would agree, but in this specific case I am in need of a single jar file for this particular problem. As it turns out, it is easier for my project to be deployed through a single file, since it is not very big to begin with. Since only a handful of collaborators will have access to it as well, I don't believe it will be too tasking when I update it.
One quick question Edvin, if I may. I have managed to adjust the code as I wanted it, but I still require some external files to be uploaded and included in the app.xml. Is there an elegant way of achieving this? I realize this doesn't exactly pertain to your work here, but I would really appreciate your help on the matter (I'm coming from a heavy C++ background so I'm still getting the hang of this). Thank you for your quick reply.

I see. The easiest solution would be to put those external files in the folder you point to when you generate the manifest. If that's not an option, you can for sure read in the Manifest and add several files to it manually before writing it out again. That could be a separate built step you execute with the maven exec:jar goal perhaps.
Some pseudo code:
FXManifest manifest = JAXB.unmarshal(file, FXManifest.class);
LibraryFile externalFile = new LibraryFile(basePath, filePath);
manifest.files.add(externalFile);
JAXB.marshal(manifest, file);

I hope this is what you're looking for :)

Could you provide an example where to put those files when generating the manifest? I am having trouble deducing how exactly is the manifest being made.

@Valrog there is now an example in the README.md where, in this case, one file is copied from the src/main/resources directory into the target/app directory. Basically it looks like this:
<execution>
                    <id>copy-whatsnewfile</id>
                    <phase>package</phase>
                    <goals>
                        <goal>exec</goal>
                    </goals>
                    <configuration>
                        <executable>jar</executable>
                        <workingDirectory>${app.dir}</workingDirectory>
                        <arguments>
                            <argument>uf</argument>
                            <argument>fxlauncher.jar</argument>
                            <argument>-C</argument>
                            <argument>${project.basedir}/src/main/resources</argument>
                            <argument>whatsnew.html</argument>
                        </arguments>
                    </configuration>
                </execution>

In your case where you want to only use the fatjar its possible by adjusting the way the target/app directory is filled. Normally its done by copying all the dependencies into the directories. Just copy the fatjar into it and dont use the copy-dependencies plugin

@ronsmits I see, this is a much more elegant solution than what I've come up with. Thanks!

@Valrog thanks I will close the ticket


Investigation into DB queries happening during Syncing

Description
This is an investigation into the database queries being ran during syncing which is the slowest process in db-sync.
Currently to sync from nothing to the tip of the chain with no data omitted from the configurations. So procedure can take days! (an exact duration hasn't been ascertained as I haven't been able to keep it running in a single go!)
First place of investigation was to use a tool called DbBadger which looks at the output of the postgresql logs and forms a report.
DbBadger Report Overview
Time frame for this report is about 40 minutes of syncing time which was through epoch 352 on mainnet.

The first surprising results was that the high volume of queries were actually SELECT at a whopping 84%
Looking closer at those SELECT queries here is a list of the most prominent offenders:

Number of times executed
id
query

1,427,906
1
SELECT "stake_address"."id" FROM "stake_address" WHERE "stake_address"."hash_raw" = ?;

774,307
2
SELECT "multi_asset"."id" FROM "multi_asset" WHERE ("multi_asset"."policy" = ?) AND ("multi_asset"."name" = ?);

696,813
3
SELECT "tx"."id", "tx"."hash", "tx"."block_id", "tx"."block_index", "tx"."out_sum", "tx"."fee", "tx"."deposit", "tx"."size", "tx"."invalid_before", "tx"."invalid_hereafter", "tx"."valid_contract", "tx"."script_size" FROM "tx" WHERE "tx"."hash" = ?;

118,598
4
SELECT "script"."id" FROM "script" WHERE "script"."hash" = ?;

Queries in codebase
I'd imagine each query will require it's own PR, and I will add more as we proceed. For now here is a look at 1:
queryStakeAddress
             |--> queryStakeAddrWithCacheRetBs
             |                              |--> queryStakeAddrWithCache
             |                              |                      |--> adjustEpochRewards
             |                              |                      |                   |--> insertBlockLedgerEvents
             |                              |                      |--> deleteReward
             |                              |
             |                              |--> queryRewardAccountWithCacheRetBs
             |                                                      |--> queryOrInsertRewardAccount
             |                                                                           |--> insertWithdrawals
             |                                                                           |--> queryOrInsertStakeAddress
             |
             |--> queryStakeAddrAux
                              |--> queryStakeAddrWithCacheRetBs
                                                          |--> queryStakeAddrWithCache
                                                                                |--> adjustEpochRewards
                                                                                |--> deleteReward

That's a great analysis.
Some explanations for the high percentage of queries:

Some insertions insert hundrend of thousands of entries, that's why they appear to be small in numbers
In most case we decide to use autoincremental ids of referenced hashes. For example in reward.addr_id, epoch_stake.addr_id we use the id instead of the hash directly. Resolving this id requires a query. However replicating the hashes would significantly increase disk space.

That being said, it doesn't mean that there is not room for improvement.

I'd imagine each query will require it's own PR, and I will add more as we proceed.

Do you have some specific solution in mind?

In most case we decide to use autoincremental ids of referenced hashes. For example in reward.addr_id, epoch_stake.addr_id we use the id instead of the hash directly. Resolving this id requires a query. However replicating the hashes would significantly increase disk space.

My thought was to create a cache which is populated after a new stake_address is inserted.
We'd populate the cache by having the insert of the stake address return the id it was assign and it's hash. Then this could be added to a local Map eg:
type StakeCache = Map StakeAddressHashRaw StakeAddressId

data SyncEnv = SyncEnv {
  ...
  envStakeCache :: (StrictTVar IO StakeCache)
  }

then prior to creating Reward or EpochStake we could do a lookup against envStakeCache to get the StakeAddressHashId from the cache.
The part I'm unsure about is the deletion of values in the cache. Is there a way for us to know how many times the same  StakeAddressHashRaw is likely to be called or is it a case that we'd only do one per raw hash?

Take that back I can see we have type StakeAddrCache = Map StakeCred DB.StakeAddressId which should be doing what I'm thinking but I can't spot it being populated in the first place.
I'm just trying to log around here exactly what is happening:
queryStakeAddrAux ::
  MonadIO m =>
  CacheNew ->
  StakeAddrCache ->
  StrictTVar IO CacheStatistics ->
  Network ->
  StakeCred ->
  ReaderT SqlBackend m (Either (DB.LookupFail, ByteString) DB.StakeAddressId, StakeAddrCache)
queryStakeAddrAux cacheNew mp sts nw cred =
  case Map.lookup cred mp of
    Just addrId -> do
      liftIO $ hitCreds sts
      case cacheNew of
        EvictAndReturn -> pure (Right addrId, Map.delete cred mp)
        _ -> pure (Right addrId, mp)
    Nothing -> do
      liftIO $ missCreds sts
      let !bs = Ledger.serialiseRewardAccount (Ledger.RewardAccount nw cred)
      mAddrId <- mapLeft (,bs) <$> queryStakeAddress bs
      case (mAddrId, cacheNew) of
        (Right addrId, CacheNew) -> pure (Right addrId, Map.insert cred addrId mp)
        (Right addrId, _) -> pure (Right addrId, mp)
        (err, _) -> pure (err, mp)

You can look at how the queryOrInsertRewardAccount and queryOrInsertStakeAddress are used, which are passed one of the three values
data CacheNew
  = CacheNew
  | DontCacheNew
  | EvictAndReturn

EvictAndReturn is used when the stake is deregistered. DontCacheNew is used on simple outputs that use the stake credential and on reward adjustments (that's relatively new). CacheNew is used all other times, it on registration, receiving of rewards, withdrawals etc, since db-sync assumes they will be reused.
DontCacheNew was used on simple outputs, since an address that was simply used in an output without being registered is not expected to be reused many times. So it allows to have a cache which is reduced in size. It seems though there are addresses that are repeatedly used, which are never registered. These trick the cache, which never caches them.
Ideally we would have some metrics about how many addresses are the not registered used stake addresses and what is their usage distribution. Caching everything would solve the queries issue, but it may have a negative impact in memory.

Some stats about different implementations

StrictSeq + StrictMap
memory 16.3GB
Epoch 432 [15:27 -> 15:58] ~29mins
Stake Addresses: cache sizes: 1287547 and 31415, hit rate: 57%, hits: 1821628, misses: 1321280
TxId: cache size: 299608, cache capacity: 300000, hit rate: 81%, hits: 717021, misses: 158255
Epoch 433 [15:58:01 -> 16:30] ~32mins
Stake Addresses: cache sizes: 1286934 and 48715, hit rate: 79%, hits: 5124040, misses: 1338764
TxId: cache size: 287078, cache capacity: 300000, hit rate: 83%, hits: 1672377, misses: 334003

Master - 3 x Maps
memory 22.4GB
Epoch 432 [09:19 -> 09:52] ~33mins
Stake Addresses: cache sizes: 1287547 and 31415, hit rate: 57%, hits: 1821628, misses: 1321280
TxId: cache size: 291335, cache capacity: 300000, hit rate: 81%, hits: 715294, misses: 159982
Epoch 433 [09:52 -> 10:23] ~35mins
Stake Addresses: cache sizes: 1286934 and 48715, hit rate: 79%, hits: 5124040, misses: 1338764
TxId: cache size: 267288, cache capacity: 300000, hit rate: 82%, hits: 1662167, misses: 344213


login

How to login as admin? What is the pass?

Admin isnt redirected to admin dashbboard, rather it is redirected to store dashboard can you please provide me the information....

i suggest

Admin isnt redirected to admin dashbboard, rather it is redirected to store dashboard can you please provide me the information....

i suggest you dont use this application. because here you found lot of bug. i developed this appication when i learn codeigniter. thank you.


Handle llama_cpp_python installation based on gpu & cpu availability

We have been using this with cuda enabled machine but when installing this node, it always installs llama_cpp_python, which is cpu version.
For us when we changed to llama_cpp_python_cuda, it works faster & LLM models are able to load quickly.
Closes #6 #3
Test: After this changes, it automatically picks up the llama cpp installation based on CPU or GPU availability.
Let me know if any further changes are needed.

Could you explain how the change in this PR is doing anything different from what the requirements.txt did before? You seem to do the same checks for system and python version and are selecting the same wheels in case they match. The main change is that the requirements.txt file installed both versions of llama-cpp before (cpu and cuda) and with this PR only one of them would be installed.
It's perfectly fine for both packages to be installed, I explicitly have code in the node: See code here
that tries to load the cuda version first and only falls back to using the cpu version if that failed.

Could you explain how the change in this PR is doing anything different from what the requirements.txt did before? You seem to do the same checks for system and python version and are selecting the same wheels in case they match. The main change is that the requirements.txt file installed both versions of llama-cpp before (cpu and cuda) and with this PR only one of them would be installed.
It's perfectly fine for both packages to be installed, I explicitly have code in the node: See code here
try:
    Llama = importlib.import_module("llama_cpp_cuda").Llama
except ImportError:
    Llama = importlib.import_module("llama_cpp").Llama

that tries to load the cuda version first and only falls back to using the cpu version if that failed.

Yes You are right I think, This might be a problem on our end as requirements are not persistent. I installed the node & saw both cpu & gpu llama python. I think we don't need to do this change.
Key problem is that our session doesn't persist the requirements. So Let me once try on our end, this PR might not be needed. Thanks for the clarification.

Key problem is that our session doesn't persist the requirements. So Let me once try on our end, this PR might not be needed. Thanks for the clarification.

OK, that makes sense. And if you don't persist installed packages, you will also see similar problems with all kinds of other comfyui nodes. Let me know if you encounter any more problems.