Metrics don't follow Prometheus best practices

Describe the bug
Hi,
I was able to collect the Falco metrics with Grafana Alloy and forward them to my Grafana Cloud account:
I'm noticing anyway some issues with the metrics.
First, they are not following the best practices of Prometheus. For example, we have a metric by rule loaded in Falco, and not a metric with label to create series:
falcosecurity_falco_rules_Polkit_Local_Privilege_Escalation_Vulnerability_CVE_2021_4034_total{raw_name="rules.Polkit_Local_Privilege_Escalation_Vulnerability_CVE_2021_4034"}
falcosecurity_falco_rules_Java_Process_Class_File_Download_total{raw_name="rules.Java_Process_Class_File_Download"}
falcosecurity_falco_rules_Modify_Container_Entrypoint_total{raw_name="rules.Modify_Container_Entrypoint"}
falcosecurity_falco_rules_Decoding_Payload_in_Container_total{raw_name="rules.Decoding_Payload_in_Container"}
falcosecurity_falco_rules_Basic_Interactive_Reconnaissance_total{raw_name="rules.Basic_Interactive_Reconnaissance"}
...

The issues with that pattern are:

it declare a serie even when the rule has not been triggered
more memory consumption for Prometheus
impossible to create dynamic graphs in grafana

The good practice would be to create a metric and use labels:
falcosecurity_falco_rules_total{rule_name="Basic_Interactive_Reconnaissance"}
falcosecurity_falco_rules_total{rule_name="Privilege_Escalation_Vulnerability_CVE_2021_4034"}
falcosecurity_falco_rules_total{rule_name="Java_Process_Class_File_Download"}
falcosecurity_falco_rules_total{rule_name="Modify_Container_Entrypoint"}
...

This pattern, happens with a lot of different metrics: These:
falcosecurity_falco_memory_rss_bytes{raw_name="memory_rss_bytes"}
falcosecurity_falco_memory_vsz_bytes{raw_name="memory_vsz_bytes"}
falcosecurity_falco_memory_pss_bytes{raw_name="memory_pss_bytes"}

Should be:
falcosecurity_falco_memory_bytes{type="rss"}
falcosecurity_falco_memory_bytes{type_name="vsz"}
falcosecurity_falco_memory_bytes{type_name="pss"}

etc, etc
Another issue is the labels are not relevant at all, we have raw_name for all series, when they are supposed to give context.
See my example:
falcosecurity_falco_falco_sha256_rules_file_override_rules_info{raw_name="falco_sha256_rules_file_override_rules"
falcosecurity_falco_falco_sha256_rules_file_falco_sandbox_rules_info{raw_name="falco_sha256_rules_file_falco_sandbox_rules"
falcosecurity_falco_falco_sha256_rules_file_falco_incubating_rules_info{raw_name="falco_sha256_rules_file_falco_incubating_rules"
falcosecurity_falco_falco_sha256_rules_file_falco_rules_info{raw_name="falco_sha256_rules_file_falco_rules"

Should be:
falcosecurity_falco_sha256_rules_info{rule_file="override_rules"}
falcosecurity_falco_sha256_rules_info{rule_file="falco_sandbox_rules"}
falcosecurity_falco_sha256_rules_info{rule_file="falco_incubating_rules"}
falcosecurity_falco_sha256_rules_info{rule_file="file_falco_rules"}

It's important to have clear labels, and possibly the same from one metric to another to allow to create filters in the dashboards.
They are also used to create dynamic graphs (thanks to group_by).
I also think, more labels should be added, to allow to filter all graphs easily, see what I did in Falcosidekick:
falco_events{hostname="aks-agentpool-10286953-vmss000000",k8s_ns_name="kube-system",k8s_pod_name="azure-ip-masq-agent-bhsqt",priority="Informational",rule="Launch Sensitive Mount Container"} 1
falco_events{hostname="aks-agentpool-10286953-vmss000000",k8s_ns_name="kube-system",k8s_pod_name="cloud-node-manager-jxg2r",priority="Informational",rule="Launch Sensitive Mount Container"} 2
falco_events{hostname="aks-agentpool-10286953-vmss000000",k8s_ns_name="kube-system",k8s_pod_name="csi-azuredisk-node-qctfm",priority="Informational",rule="Launch Privileged Container"} 2
falco_events{hostname="aks-agentpool-10286953-vmss000000",k8s_ns_name="kube-system",k8s_pod_name="csi-azuredisk-node-qctfm",priority="Informational",rule="Launch Sensitive Mount Container"} 3
falco_events{hostname="aks-agentpool-10286953-vmss000000",k8s_ns_name="kube-system",k8s_pod_name="csi-azurefile-node-d28zb",priority="Informational",rule="Launch Privileged Container"} 2

It allows to scope to a namespace, a priority, etc.
How to reproduce it
Enable the metrics and scrape the /metrics endpoint.
Expected behaviour

Screenshots

Environment

Falco version: 0.38.1

System info:

Cloud provider or hardware configuration:
OS: Amazon Linux 2 (EKS)

Kernel:

Installation method: Helm

Additional context

Hey @Issif no opinion re what's best to create better dashboards and other benefits.
Note that we also have the output rules in JSON where each metric is it's own key, which explains a bit why the rules counter etc are their own metric.
Makes sense:

[ ] (1) Rules counters, have rule as label only (falco repo changes)
[ ] (2) sha256 of rules files (falco repo changes)
[ ] (3) raw_name can be removed no issue with it (libs repo changes, lower priority)

I would not change:

falcosecurity_falco_memory_rss_bytes to be more consistent with the output rules and avoid a mess  when fetching the libs metrics I would not do it differently for just those 3 metrics. CAdvisor has its own series for each resource utilization metric as well.
Adding the hostname as const label is however against best practices, instead leave it as it's own metric, see the libs comment. https://github.com/falcosecurity/libs/blob/0ec2ad8422b2ac06a034f335c7768c5ab0d13736/userspace/libsinsp/metrics_collector.h#L160-L164

@Issif do you want a patch release to address (1) and (2)? (3) would only work for the next release.
CC @leogr

@Issif
One additional question: the rules counter always had bunch of labels including the original rule name, you did see them? As you didn't include them above in your example outputs.
slight deviation in naming suggestion, but exact idea you proposed @Issif for items under 1.) and 2.)
# TYPE falcosecurity_falco_falco_sha256_rules_files_info gauge
falcosecurity_falco_falco_sha256_rules_files_info{raw_name="falco_sha256_rules_files",file_name="falco_rules test"} 1
# HELP falcosecurity_falco_falco_sha256_rules_files_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_rules_files_info gauge
falcosecurity_falco_falco_sha256_rules_files_info{raw_name="falco_sha256_rules_files",file_name="falco-sandbox_rules"} 1
# HELP falcosecurity_falco_falco_sha256_rules_files_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_rules_files_info gauge
falcosecurity_falco_falco_sha256_rules_files_info{raw_name="falco_sha256_rules_files",file_name="falco-incubating_rules"} 1
# HELP falcosecurity_falco_falco_sha256_rules_files_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_rules_files_info gauge
falcosecurity_falco_falco_sha256_rules_files_info{raw_name="falco_sha256_rules_files",file_name="falco-deprecated_rules"} 1
# HELP falcosecurity_falco_falco_sha256_rules_files_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_rules_files_info gauge
falcosecurity_falco_falco_sha256_rules_files_info{raw_name="falco_sha256_rules_files",file_name="falco_rules"} 1
# HELP falcosecurity_falco_falco_sha256_config_files_info https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_falco_sha256_config_files_info gauge
falcosecurity_falco_falco_sha256_config_files_info{raw_name="falco_sha256_config_files",file_name="falco"} 1

# HELP falcosecurity_falco_rules_matches_total https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_rules_matches_total counter
falcosecurity_falco_rules_matches_total{raw_name="rules_matches_total"} 10
# HELP falcosecurity_falco_rules_counters_total https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_rules_counters_total counter
falcosecurity_falco_rules_counters_total{raw_name="rules_counters",priority="4",rule_name="Read sensitive file untrusted",source="syscall",tags="T1555, container, filesystem, host, maturity_stable, mitre_credential_access"} 10

rules_counters now only exported for rules with a counter greater than O.
As pointed out the hostname should not be a label and its own metric aka it remains the same, for example:
# HELP falcosecurity_evt_hostname_info https://falco.org/docs/metrics/
# TYPE falcosecurity_evt_hostname_info gauge
falcosecurity_evt_hostname_info{raw_name="hostname",hostname="test"} 1

One additional question: the rules counter always had bunch of labels including the original rule name, you did see them? As you didn't include them above in your example outputs.

Just missed them when I cropped the lines.

As pointed out the hostname should not be a label and its own metric aka it remains the same, for example:

For the counters for memory, it's ok, they are not dynamic like the one for the rules. Good point for the hostnames too. Thanks @incertum

@Issif mind checking above example outputs once more for correctness. Re-pushed to not accidentally remove the sha256 ...

Your proposal seems good to me.
I'm just wondering if this metric is useful:
# HELP falcosecurity_falco_rules_matches_total https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_rules_matches_total counter
falcosecurity_falco_rules_matches_total{raw_name="rules_matches_total"} 10

With
# HELP falcosecurity_falco_rules_counters_total https://falco.org/docs/metrics/
# TYPE falcosecurity_falco_rules_counters_total counter
falcosecurity_falco_rules_counters_total{raw_name="rules_counters",priority="4",rule_name="Read sensitive file untrusted",source="syscall",tags="T1555, container, filesystem, host, maturity_stable, mitre_credential_access"} 10

Without specifying a label, the count will be the same.

We have it for the JSON output rules. We can remove it for Prometheus if you want given the refactor.

@Issif do you want a patch release to address (1) and (2)? (3) would only work for the next release. CC @leogr

Since metrics feature is "incubating", I don't see any urgency to deliver the fixes. So, I'd avoid an hotfix release, unless we have compelling reasons to do so.

I also think, more labels should be added, to allow to filter all graphs easily, see what I did in Falcosidekick:
falco_events{hostname="aks-agentpool-10286953-vmss000000",k8s_ns_name="kube-system",k8s_pod_name="azure-ip-masq-agent-bhsqt",priority="Informational",rule="Launch Sensitive Mount Container"} 1
falco_events{hostname="aks-agentpool-10286953-vmss000000",k8s_ns_name="kube-system",k8s_pod_name="cloud-node-manager-jxg2r",priority="Informational",rule="Launch Sensitive Mount Container"} 2
falco_events{hostname="aks-agentpool-10286953-vmss000000",k8s_ns_name="kube-system",k8s_pod_name="csi-azuredisk-node-qctfm",priority="Informational",rule="Launch Privileged Container"} 2
falco_events{hostname="aks-agentpool-10286953-vmss000000",k8s_ns_name="kube-system",k8s_pod_name="csi-azuredisk-node-qctfm",priority="Informational",rule="Launch Sensitive Mount Container"} 3
falco_events{hostname="aks-agentpool-10286953-vmss000000",k8s_ns_name="kube-system",k8s_pod_name="csi-azurefile-node-d28zb",priority="Informational",rule="Launch Privileged Container"} 2

It allows to scope to a namespace, a priority, etc.

I agree on this. falco-exporter provides a similar behavior:
https://github.com/falcosecurity/falco-exporter/blob/master/pkg/exporter/exporter.go#L29-L43

I also think, more labels should be added, to allow to filter all graphs easily, see what I did in Falcosidekick:
falco_events{hostname="aks-agentpool-10286953-vmss000000",k8s_ns_name="kube-system",k8s_pod_name="azure-ip-masq-agent-bhsqt",priority="Informational",rule="Launch Sensitive Mount Container"} 1
falco_events{hostname="aks-agentpool-10286953-vmss000000",k8s_ns_name="kube-system",k8s_pod_name="cloud-node-manager-jxg2r",priority="Informational",rule="Launch Sensitive Mount Container"} 2
falco_events{hostname="aks-agentpool-10286953-vmss000000",k8s_ns_name="kube-system",k8s_pod_name="csi-azuredisk-node-qctfm",priority="Informational",rule="Launch Privileged Container"} 2
falco_events{hostname="aks-agentpool-10286953-vmss000000",k8s_ns_name="kube-system",k8s_pod_name="csi-azuredisk-node-qctfm",priority="Informational",rule="Launch Sensitive Mount Container"} 3
falco_events{hostname="aks-agentpool-10286953-vmss000000",k8s_ns_name="kube-system",k8s_pod_name="csi-azurefile-node-d28zb",priority="Informational",rule="Launch Privileged Container"} 2

It allows to scope to a namespace, a priority, etc.

I agree on this. falco-exporter provides similar behavior: https://github.com/falcosecurity/falco-exporter/blob/master/pkg/exporter/exporter.go#L29-L43

Hey @leogr which additional labels are you referring to?
Re the hostname, please see  https://github.com/falcosecurity/falco/issues/3271#issuecomment-2206918760 or https://github.com/falcosecurity/libs/blob/0ec2ad8422b2ac06a034f335c7768c5ab0d13736/userspace/libsinsp/metrics_collector.h#L160-L164 we have it as separate / own metric following the recommendations in the documentation.
Re pod name and namespace name:  Would you mind getting me up to speed how you would retrieve this in Falco? IMO given that they can change per event it may be best suited to do that analyses based on the logs in your data lake and instead focus on the simple rules counters in Prometheus. The Falco rules counters are currently not partitioned by namespace or pod name. I can imagine that such granular groupBy queries can become very large in large deployments?


Use LatLonCover for landcover data

Replaces hard coded CSV specific to QUEST 2023 with logic to lookup each lat/lon coordinate and provide specific details for each location via LatLonCover. LatLonCover uses the CropScape API to lookup land cover data.
Fetching land cover data takes about 5 seconds per observation. Since this process takes so long the process for retrieving an iNaturalist observation CSV. Only the observations necessary for display are initially retrieved. Once the display observations are retrieved we start building the observation CSV in the background.
Occasionally the CropScape API failed with a read timeout error, requiring some retry logic to provide a reliable API endpoint.

I created this as a draft PR since we may be demoing the app Monday.

Actual problem: when I go to upload a spreadsheet that has the satellite RGB values, those columns aren't showing up. sat_url shows up as an image_url option, but the RGB columns don't show up (so if it's just those, there isn't even the option to visualize).
I believe this was set in parseCSV.ts and related files that weren't changed...

@egrace479 Could this be because those rgb columns all have the same value? There is logic that filtered out columns that don't have more than one value. At some point we discussed changing the UI to let the user know about ignored columns.

Actual problem: when I go to upload a spreadsheet that has the satellite RGB values, those columns aren't showing up. sat_url shows up as an image_url option, but the RGB columns don't show up (so if it's just those, there isn't even the option to visualize).
I believe this was set in parseCSV.ts and related files that weren't changed...

@egrace479 Could this be because those rgb columns all have the same value? There is logic that filtered out columns that don't have more than one value. At some point we discussed changing the UI to let the user know about ignored columns.

It seems there are only two distinct values per column. But I'm still getting the same problem with this other CSV that has more variation in those columns (I did a fetch for dan_equids).

It looks like dan_equids now has a observation that doesn't have lat/lon. The numeric columns are all empty for row p36.

It looks like dan_equids now has a observation that doesn't have lat/lon. The numeric columns are all empty for row p36. So javascript throws out the columns because they are not all numeric values.

So that's why lat/lon columns also aren't showing up for auxiliary columns...We're still getting the land cover because it fills in 0 for all entries that don't have lat/lon values too.

Also, did a sanity check and everything does seem to  function just fine when I remove the row with the missing values.

I created issue #68 to better explain why certain columns have been excluded from the visualization.

For item 1 above:

An odd thing (which really isn't terribly important) is that the preview rows show the land cover values in early columns, but when you download the spreadsheet they're way at the end (I tested with also adding RGB).

This looks to also be a pre-existing problem I can reproduce this on the production Andromeda instance. The screen displays extra columns based on the order of keys in the first object (which seem to be alphabetic).

I'm just noticing that "small" and "big"/"large" aren't defined in the app or in LatLonCover repo, which is linked to in the Fetch image data tab notes.
Maybe simply adding a link to the outputDataDictionary.txt file in the notes is sufficient?

I'm just noticing that "small" and "big"/"large" aren't defined in the app or in LatLonCover repo README, which is linked to in the Fetch image data tab notes.
Maybe simply adding a link to the outputDataDictionary.txt file in the notes is sufficient?

There is a hover-text explaining each on the Fetch Data Tab. It's part of the latlonCover package (and there is a link to the data dictionary explaining it in that README), so I don't know that it's really necessary. We don't have much documentation for the fetch data portion, which should maybe be fleshed out a bit more on the main README along with more explanation of Andromeda, the UI and all that is in this repo. Happy to create a new branch to update our documentation after this PR is addressed.

I only see this for hover-text:

But yes, this could go into a separate documentation effort 👍

I only see this for hover-text: ...
But yes, this could go into a separate documentation effort 👍

I could've sworn the "big" and "small" had hovertext indicating the size. Should come from this no? @johnbradley wasn't that in the original PR?

Here is the html for big and small:
https://github.com/Imageomics/Andromeda/blob/c8b8d9b4b0f9da6ae33169947937e0f908a09e5a/andromeda-ui/components/LatLonCoverNote.tsx#L11-L13
Perhaps the tooltips where too subtle.

We could consider changing the color of the font or otherwise indicating they're special beyond the italics, but I think everything is functional and works well as currently set up in this PR to proceed. The few areas we think could be improved are documented and are currently behaving in that manner, so ✅🚀

@thompsonmj @egrace479 How does this look?

I'd vote to have the dimensions in parentheses right behind those bold terms rather than as a tooltip.


[Submodule] Add Flash attention v2

Flash attention v2 was released last week and it is claimed to give good speed up compared to existing implementations. I integrated flash v2 into our Relax CUTLASS BYOC and got 2-3 img / sec speed up on SD v1.5 compared to our existing flow that uses the xformer kernel.
The original code depends on libtorch so it is hard to integrate. A version without torch dep was created in https://github.com/tlc-pack/libflash_attn and I'm adding it as a submodule.
@tqchen @vinx13 @yzh119 @sunggg

let us also update https://github.com/apache/tvm/blob/main/LICENSE#L248 and https://github.com/apache/tvm/tree/main/licenses

Very cool work! This will substantially help with TVM performance on stable diffusion and LLM training workloads


Can't start VSCode 1.56.1 on macOS 10.10

VS Code Version: 1.56.1
OS Version: macOS 10.10

VSCode can't open in OS X 10.10, it says that it requires macOS 10.11, but VSCode official site says that macOS 10.10 is still supported.
Steps to Reproduce:

Open VSCode in macOS 10.10
See error message

Does this issue occur when all extensions are disabled?: Yes

Sorry about that, the supported version in the website should be updated. The runtime version used by vscode 1.56 only supports >= 10.11

the latest version of vscode crashes immediatly on start on OSX 10.11 ... when starting from cmd line it says Illegal Instruction: 4 ... I grabbed an older Version 1.51 from another Mac and that one works ... is it possible that  your minimum supported runtime version is already higher than 10.11 ?

@mithrendal This happens since VSC v1.56.0 under MacOS 10.11.6, the previous VSC v1.55.2 works fine instead.
Take a look at the VSC generated crash reports ...
https://github.com/microsoft/vscode/wiki/Native-Crash-Issues#creating-and-symbolicating-local-crash-reports
... it is most likely due to spdlog.node in Electron.

Hi veriotis ... cool thing ... thanks for the info ...
here comes the VSC generated crash dump
da31a834-4fb9-4569-8d67-82ae8070c7d5.dmp.zip
and here the crash log reported to the OS
crashlog.txt
can we still download the older working version 1.55.2 from the official site ?

@mithrendal You can find and download the working 1.55.2 version from the official site's Updates lists, see ...
https://code.visualstudio.com/updates/v1_55

@mithrendal for crash on 10.11 please follow https://github.com/microsoft/vscode/issues/123374 for updates, thanks!

Webpage docs has been updated and will be reflected with next release.


FAILURE 70

Hello, thanks for this project. Sadly, I only got it to work once, but the latency was amazing!!
So, when I run sudo node server-rpi.js the webpage is served fine, but when I click start video, I see the following error in the console and nothing happens:
Incomming action 'REQUESTSTREAM' raspivid -t 0 -o - -w 960 -h 540 -fps 12 Failure 70
now, I got it working once by running this command I found in issue #40 and also running server-rpi on another ssh terminal. The raspivid command says it was serving on port 5001, and the video played fine on 8080. Sadly, I wasn't able to reproduce this brief success. Any tips?
raspivid -t 0 -o - -w 960 -h 540 -fps 25 -pf baseline | nc -k -l -v 5001
I am obviously missing some basics here... any help is appreciated

Ok, got working inconsistently, then started everything again on a fresh install and it started working fine on just server-rpi.js - thanks for the great project!


[Bug]: Aether Equipment not visible on posed armorstand

General Info

[X] I am running a modpack
[ ] I can reproduce this issue consistently in single-player
[X] I can reproduce this issue consistently in multi-player
[X] I have searched for this issue previously and it was either (1) not previously reported, or (2) previously fixed and I am having the same problem.
[ ] I am crashing and can provide my crash report(s)
[X] I am using the latest version of the mod

Loader version
Forge 47.2.17 / NeoForge 47.1.83
Minecraft version
1.20.1
Mod version
2.1.2
Java version
Jdk <IP_ADDRESS>
Issue Description
On a server, Aether mod equipment does not display on a posed armor stand. This issue is not present in single-player. The equipment is visible in a single-player world. Can consistently recreate within multi-player.
Additional Information
No response

This is not related to Armor Poser, all the mod does is apply NBT to the vanilla armor stand. The armor stand rendering stays vanilla or another mod is changing it


PSScriptAnalyzer fails on DSC configuration file on macOS - Unable to load DLL 'libmi.so'

Steps to reproduce
git clone https://github.com/OctopusDeploy/OctopusDSC.git
cd OctopusDSC
git checkout ce3f797347bec9c0366c33c9b6c3f3e6ddeef6f3
cd OctopusDSC/Examples
Invoke-ScriptAnalyzer -Path ./cOctopusSeqLogger.ps1

Expected behavior
Normal script analysis
Actual behavior
PS /Users/me/OctopusDSC> Invoke-ScriptAnalyzer ./OctopusDSC/Examples/cOctopusSeqLogger.ps1
Invoke-ScriptAnalyzer : Parse error in file /Users/me/OctopusDSC/OctopusDSC/Examples/cOctopusSeqLogger.ps1:  System.DllNotFoundException: Unable to load DLL 'libmi.so': The specified module or one of its dependencies could not be found.
 (Exception from HRESULT: 0x8007007E)
   at Microsoft.Management.Infrastructure.Native.NativeMethods.MI_Application_InitializeV1(UInt32 flags, String applicationID, IndirectPtr extendedError, DirectPtr application)
   at Microsoft.Management.Infrastructure.Native.MI_Application.Initialize(String applicationId, MI_Instance& extendedError, MI_Application& application)
   at Microsoft.Management.Infrastructure.Internal.CimApplication.GetApplicationHandle()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Lazy`1.CreateValue()
   at Microsoft.Management.Infrastructure.Internal.CimApplication.get_Handle()
   at Microsoft.Management.Infrastructure.Serialization.CimMofDeserializer..ctor()
   at Microsoft.Management.Infrastructure.Serialization.CimMofDeserializer.Create()
   at Microsoft.PowerShell.DesiredStateConfiguration.CimDSCParser..ctor(OnClassNeeded onClassNeeded)
   at Microsoft.PowerShell.DesiredStateConfiguration.Internal.DscClassCache.ImportClasses(String path, Tuple`2 moduleInfo, Collection`1 errors)
   at Microsoft.PowerShell.DesiredStateConfiguration.Internal.DscClassCache.Initialize(Collection`1 errors, List`1 modulePathList)
   at Microsoft.PowerShell.DesiredStateConfiguration.Internal.DscClassCache.LoadDefaultCimKeywords(Dictionary`2 functionsToDefine, Collection`1 errors, List`1 modulePathList, Boolean cacheResourcesFromMultipleModuleVersions)
   at Microsoft.PowerShell.DesiredStateConfiguration.Internal.DscClassCache.LoadDefaultCimKeywords(Collection`1 errors)
   at System.Management.Automation.Language.Parser.ConfigurationStatementRule(IEnumerable`1 customAttributes, Token configurationToken) at line 7 column 15.
At line:1 char:1
+ Invoke-ScriptAnalyzer ./OctopusDSC/Examples/cOctopusSeqLogger.ps1
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : ParserError: (ParserError:String) [Invoke-ScriptAnalyzer], ParseException
+ FullyQualifiedErrorId : Parse error in file /Users/me/OctopusDSC/OctopusDSC/Examples/cOctopusSeqLogger.ps1:  System.DllNotFoundException: Unable to load DLL 'libmi.so': The specified module or one of its dependencies could not be found.
 (Exception from HRESULT: 0x8007007E)
   at Microsoft.Management.Infrastructure.Native.NativeMethods.MI_Application_InitializeV1(UInt32 flags, String applicationID, IndirectPtr extendedError, DirectPtr application)
   at Microsoft.Management.Infrastructure.Native.MI_Application.Initialize(String applicationId, MI_Instance& extendedError, MI_Application& application)
   at Microsoft.Management.Infrastructure.Internal.CimApplication.GetApplicationHandle()
   at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Lazy`1.CreateValue()
   at Microsoft.Management.Infrastructure.Internal.CimApplication.get_Handle()
   at Microsoft.Management.Infrastructure.Serialization.CimMofDeserializer..ctor()
   at Microsoft.Management.Infrastructure.Serialization.CimMofDeserializer.Create()
   at Microsoft.PowerShell.DesiredStateConfiguration.CimDSCParser..ctor(OnClassNeeded onClassNeeded)
   at Microsoft.PowerShell.DesiredStateConfiguration.Internal.DscClassCache.ImportClasses(String path, Tuple`2 moduleInfo, Collection`1 errors)
   at Microsoft.PowerShell.DesiredStateConfiguration.Internal.DscClassCache.Initialize(Collection`1 errors, List`1 modulePathList)
   at Microsoft.PowerShell.DesiredStateConfiguration.Internal.DscClassCache.LoadDefaultCimKeywords(Dictionary`2 functionsToDefine, Collection`1 errors, List`1 modulePathList, Boolean cacheResourcesFromMultipleModuleVersions)
   at Microsoft.PowerShell.DesiredStateConfiguration.Internal.DscClassCache.LoadDefaultCimKeywords(Collection`1 errors)
   at System.Management.Automation.Language.Parser.ConfigurationStatementRule(IEnumerable`1 customAttributes, Token configurationToken) at line 7 column 15.,Microsoft.Windows.PowerShell.ScriptAnalyzer.Commands.InvokeScriptAnalyzerCommand

Invoke-ScriptAnalyzer : Parse error in file /Users/me/OctopusDSC/OctopusDSC/Examples/cOctopusSeqLogger.ps1:  Unexpected token '{' in expression or statement at line 8 column 1.
At line:1 char:1
+ Invoke-ScriptAnalyzer ./OctopusDSC/Examples/cOctopusSeqLogger.ps1
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : ParserError: (UnexpectedToken:String) [Invoke-ScriptAnalyzer], ParseException
+ FullyQualifiedErrorId : Parse error in file /Users/me/OctopusDSC/OctopusDSC/Examples/cOctopusSeqLogger.ps1:  Unexpected token '{' in expression or statement at line 8 column 1.,Microsoft.Windows.PowerShell.ScriptAnalyzer.Commands.InvokeScriptAnalyzerCommand

Invoke-ScriptAnalyzer : Parse error in file /Users/me/OctopusDSC/OctopusDSC/Examples/cOctopusSeqLogger.ps1:  Unexpected token '}' in expression or statement at line 22 column 1.
At line:1 char:1
+ Invoke-ScriptAnalyzer ./OctopusDSC/Examples/cOctopusSeqLogger.ps1
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : ParserError: (UnexpectedToken:String) [Invoke-ScriptAnalyzer], ParseException
+ FullyQualifiedErrorId : Parse error in file /Users/me/OctopusDSC/OctopusDSC/Examples/cOctopusSeqLogger.ps1:  Unexpected token '}' in expression or statement at line 22 column 1.,Microsoft.Windows.PowerShell.ScriptAnalyzer.Commands.InvokeScriptAnalyzerCommand

Environment data

> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      6.0.0-rc.2
PSEdition                      Core
GitCommitId                    v6.0.0-rc.2
OS                             Darwin 17.3.0 Darwin Kernel Version 17.3.0: Thu Nov  9 18:09:22 PST 2017; root:xnu-4570.31.3~1/RELEASE_X86_64
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

@matt-richardson This issue should be opened in the PSScriptAnalzer repo.p

It appears to be an underlying issue, rather than related to PSScriptAnalyzer. Most likely an issue loading DSC. (I was short cutting the process of logging it over there and then getting told it should be logged over here.)

It is not supported.

It is not supported.

You ask for detailed information when someone files an issue, but then you close without any kind of response.
I dont want to use libmi. I want to use PSScriptAnalyzer on a DSC configuration file on macOS.

You ask for detailed information when someone files an issue, but then you close without any kind of response.
I dont want to use libmi. I want to use PSScriptAnalyzer on a DSC configuration file on macOS.

@matt-richardson It is old history and a plan is to exclude all dependencies on libmi.  MSFT team has started new cross platform DSC support project. See summary of the PR #13399.

@matt-richardson It is old history and a plan is to exclude all dependencies on libmi.  MSFT team has started new cross platform DSC support project. See summary of the PR #13399.

@matt-richardson Last year, I improved PSScriptAnalyzer to be able to analyze DSC dependencies on non-Windows Unix systems as well when using the -SaveDscDependency switch in PR https://github.com/PowerShell/PSScriptAnalyzer/pull/1246, which was first released in PSScriptAnalyzer 1.18.1. Can you please try again with the latest version of PScriptAnalyzer by running Install-Module PSScriptAnalyzer -Force and restarting the shell in case PSScriptAnalyzer was already loaded before installing the newer version.

@matt-richardson Last year, I improved PSScriptAnalyzer to be able to analyze DSC dependencies on non-Windows Unix systems as well when using the -SaveDscDependency switch in PR https://github.com/PowerShell/PSScriptAnalyzer/pull/1246, which was first released in PSScriptAnalyzer 1.18.1. Can you please try again with the latest version of PScriptAnalyzer by running Install-Module PSScriptAnalyzer -Force and restarting the shell in case PSScriptAnalyzer was already loaded before installing the newer version.


[Feature][Executor] Add K8S Executor for task dispatching

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description

Still WIP.
Discussions are more than welcome.

Why we need it?

Higher resource utilization - The core idea is one task per pod. With the help of K8S metrics server, users could clearly get the resource consumption data for each task (both CPU and Memory) so that they could adjust and allocate suitable amount of resources based on the data to each specific task pod and achieve global minimum resource consumption. In the future, we might even use K8S VPA to automatically adjust the resource allocation.
Better monitoring (pod level, cloud native, K8S metrics server)
Easier fault tolerance
Faster iteration, for updating task plugin, there is no need to upgrade all workers.

How to design it?
Compatibility

Do some decoupling and abstraction work before adding a K8S executor. Make executor configurable. If users choose not to use K8S Executor, he will be able to use the current way - one worker per pod.

Fault Tolerance

Make DS worker stateless in K8S environment (one task per pod)
DS worker does not need to rely on ZK. We could start a thread in K8S executor to interact with K8S watcher to subscribe to the state change event of worker pod. Since one task per pod, there is no need to decide which pod to resume the task execution if worker crashes, just start a new one.

Communication With Master

Is it possible to use a simpler way instead of Netty for communication between worker and master in K8S environment? Such as a message queue? Since there is one task per pod and K8S Executor could kill the pod through K8S API server, looks like not necessary to use Netty server and client in worker in K8S environment. For logging, we could use PV or remote logging. Maybe we could find a simpler way for K8S Executor to pass TaskExecutionContext into worker.

Combined With DS Hadoop Executor

For difference, we temporarily call current executor as DS HadoopExecutor.
There is no perfect solution, only trade-off. Pros and cons?
detach(async) vs attach(sync) - sync task, resource usage hard to estimate -> use Hadoop Executor (one worker per pod); async task, resource usage easy to estimate -> use K8S Executor (one task per pod)
Two queues for two executors: HadoopTaskPriorityQueue and K8STaskPriorityQueue
For Hadoop Executor (one worker one pod), we could add worker pod elasticity feature, also mentioned in #9337 One possible solution is to use K8S HPA and KEDA with the number of queued and running tasks as the metric.

General Design (Not Detailed)
K8S Executor

With VPA: WIP

DS K8S Worker: WIP

Pros & Cons

Compound Solution (K8S Executor + Hadoop Executor)

Asynchronous Tasks (Detach)
Synchronous Tasks (Attach)
Maximum Resource Utilization Rate
Elastic Worker

RoadMap & Milestones

Sub-Tasks
Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Is it possible to use a simpler way instead of Netty for communication between worker and master in K8S environment? Such as a message queue?

+1, I think if k8s watcher is in master, we can use event queue locally.

Just some drafts, will redraw later:

For #9337 - KEDA + HPA for worker pod scaling (one worker one pod). PS: this is a solution for Hadoop Executor Elasticity, not for K8S Executor:

KEDA: https://github.com/kedacore/keda

FYI:
K8S HPA: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
K8S VPA: https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler

@MonsterChenzhuo

Hi, @EricGao888 , I'm interested in this issue. If help is needed, I am willing to participate in it.

Hi, @EricGao888 , I'm interested in this issue. If help is needed, I am willing to participate in it.

@rickchengx Thanks for reaching out! Feel free to participate in the discussions and design : )

A brief roadmap with several milestones:

Later this week, I will send a proposal email to dev mailing list : )

Hi, @EricGao888 , it's a promising work! I'm willing to make more contribution to it.

There is no perfect solution, only trade-off.

+1, different users may have different targets of latency and resource utilization, some wish to maximize throughput under limited resources, and others may desire high resource utilization. When using k8s executor, because of stateless and one task per pod design, it can easily meet resource utilization target, however, some problems like cold start of container will affect latency. The problem will be exaggerated when task execution time is short, at this time, hadoop executor is better.

I have a question. In the scenario where each task has one pod, how to determine the resources of each pod?

I think we can reopen this issue, anybody who'd like to implement it can leave a message

目前正在开发弹性资源组件，

原理，资源分两条线a线 现有资源，b线 待定资源
a)	4资源请求->5a 分配可用资源-> 6a 请求使用资源-> 7a 提供资源->8a 提交任务
b)	4资源请求->5b 分配待定资源-> 6b 请求新worker-> 7b 部署worker->8b 注册/报告资源
a线是分配现有资源；b线请求新资源，新资源注册后成为现有资源，变成a线资源，在a线分配
该组件时通用的弹性资源，准备接入组件，增加弹性能力，如，datax，xxl-job，eventbridge，dolphin-scheduler
下图是开发架构：

实现业务资源消费者和launcher，下图xxl-job集成设计

xxl-job的master负责实现业务资源消费者，分片后，分发器按分片(组)申请资源；申请资源后，发布指令，新建执行器


How to properly run the 3Dgan Keras model?

Hello,
I'd like to know how to properly run the Keras based 3Dgan model? Which of the branches is recommended? It seems that master branch was not developed for a long time.
I've tried to run the model on Nvidia V100 with TF2 and Keras2 but I'v got some syntax errors.
Please, let me know if there are any guidelines within repository about how to run the model in proper way.
Thanks and regards,
Adam

Hello Adam
This repo is developing in different branches
Master branch does not work on TF2. You can try https://github.com/svalleco/3Dgan/tree/frehm/Conv2D https://github.com/svalleco/3Dgan/tree/frehm/Conv2D
But the network architecture is different. We still solve a 3D problem, but use 2D convolutions to do so
Cheers
Sofia

On 7 Dec 2021, at 19:46, akrzgc @.***> wrote:
Hello,
I'd like to know how to properly run the Keras based 3Dgan model? Which of the branches is recommended? It seems that master branch was not developed for a long time.
I've tried to run the model on Nvidia V100 with TF2 and Keras2 but I'v got some syntax errors.
Please, let me know if there are any guidelines within repository about how to run the model in proper way.
Thanks and regards,
Adam
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/svalleco/3Dgan/issues/10, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACABGKXH6RHAYIX5VEJH4DLUPZI7DANCNFSM5JR6MSRA.
Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

Hello Sofia,
Thank you for such a quick answer!
It is not exactly my requirement to run the model with TF2. I'd like to know how to run the model in proper way - if it needs TF1 it is not a problem :)
Ideally I'd like to know:

which branch is the recommended one?
what are the packages versions needed for running the training on the selected branch?
which of the scripts runs the 3Dgan model (there are few scripts with "Train" in the name on different branches)?
how can I get the original dataset?
if the dataset is not public - what should be the shape and size of synthetic random generated dataset?

Thanks and regards,
Adam


Non existing metric mapped to a random one.

Tried running this policy:
policies:
   - name: bad-metric
     description: |
       Non existing metric mapped to a random one.
     resource: azure.vm

     filters:
     - type: metric
       metric: NonExistingMetric
       aggregation: average
       op: ne
       threshold: 17
       timeframe: 12

     actions:
      - type: tag
        tag: bug
        value: NonExistingMetric

Expected an error message, got the tag assigned.
Also tried le 0.75 condition, got one instance assigned the tag and one not. Looks like if a metric is not found a random metric is picked up.
VM OS - Linux, Cent OS.

Created a ticket for the Azure REST API:
I am running the script provided below according to
https://docs.microsoft.com/en-us/rest/api/monitor/metrics/list
Expecting to get Network metrics, but getting CPU metrics.
script:
AAD_TOKEN=az account get-access-token --resource "https://management.azure.com/" --query accessToken -o tsv
curl -H "Authorization: Bearer $AAD_TOKEN" https://management.azure.com/subscriptions/d36d0808-a967-4f73-9fdc-32ea232fc81d/resourceGroups/IVANCUSTODIAN/providers/Microsoft.Compute/virtualMachines/ivancustodian1/providers/microsoft.insights/metrics?api-version=2018-01-01&metricnames=Network+In
response:
{"cost":0,"timespan":"2018-08-23T18:44:48Z/2018-08-23T19:44:48Z","interval":"PT1M","value":[{"id":"/subscriptions/d36d0808-a967-4f73-9fdc-32ea232fc81d/resourceGroups/IVANCUSTODIAN/providers/Microsoft.Compute/virtualMachines/ivancustodian1/providers/Microsoft.Insights/metrics/Percentage CPU","type":"Microsoft.Insights/metrics","name":{"value":"Percentage CPU","localizedValue":"Percentage CPU"}, ... truncated

This might be python SDK issue, repro:
from azure.common.client_factory import get_client_from_cli_profile
from azure.mgmt.monitor import MonitorManagementClient
monitorClient = get_client_from_cli_profile(MonitorManagementClient)
subscription = "{subscription}"
resourceGroup = "{resourceGroup]"
vmname = "{vm name}"
resource = "subscriptions/{}/resourceGroups/{}/providers/Microsoft.Compute/virtualMachines/{}" . format(subscription, resourceGroup, vmname)
metrics_data = monitorClient.metrics.list(resource, metric="NonExistingMetric", raw=True)
metricId = metrics_data.output.value[0].id
print(metricId)
expected error message, actual:
/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Compute/virtualMachines/{vm}/providers/Microsoft.Insights/metrics/Percentage CPU

PR #2808

Fixed with PR #2808


[fix][broker] Fix consumer does not abide by the max unacks limitation

Fixes #16667
Motivation
Fix the consumer does not abide by the max unacks limitation.
The root cause is the dispatcher only cares about the flow permits while
dispatching messages but doesn't care if the dispatched messages
will exceed the max unacks limitation.
The issue is easy to reproduce if max_unacks < receiver_queue_size
Modifications
Calculate the dispatch messages count for a consumer with
max_unacks - unacks
Verifying this change
A new test added to verify the consumer will not receive
messages more than max unacks limitation
Documentation
Check the box below or label this PR directly.
Need to update docs?

[ ] doc-required
(Your PR needs to update docs and you will update later)

[x] doc-not-needed
(Please explain why)

[ ] doc
(Your PR contains doc changes)

[ ] doc-complete
(Docs have been already added)

I think this PR can also fix https://github.com/apache/pulsar/issues/16663

Is a similar fix needed for other dispatcher implementation classes such as PersistentDispacherSingleActiveConsumer?

Is a similar fix needed for other dispatcher implementation classes such as PersistentDispacherSingleActiveConsumer?

@lhotari Oh, good point. I will take a look at the Key_Shared subscription, It's might different from Shared subscription. For the single active consumer subscription, we don't have unacks limitations for now, so we don't need a fix for it.


overwrite Regmod model to take advantage of the sparse data type

Add a intermediate regmod class to host the features we wish to have in regmod for sparse model...
Change the name from regmodsm to spxmod stands for "Space x Model"

This PR is intended to be the last large PR for this branch

Thanks very much @AHsu98! I really appreciate the review! Later on we can chat more on the design!


Is do_parse! + opt! broken?

Should this work?
named!(a_bug <(&[u8], Option<&[u8]>)>, do_parse!(
    a:tag!("a") >>
    b: opt!(tag!("b")) >>
    ((a, b))
  ));

#[cfg(test)]
mod tests {
    use nom::IResult::Done;

    #[test]
    fn a_bug() {
        assert_eq!(super::a_bug(&b"ab"[..]), Done(&b""[..], (&b"a"[..], Some(&b"b"[..]))));
        assert_eq!(super::a_bug(&b"a"[..]), Done(&b""[..], (&b"a"[..], None)));
    }
}

If you're using nom 2.0 it's this issue: https://github.com/Geal/nom/blob/master/doc/upgrading_to_nom_2.md#parsers-returning-incomplete-instead-of-an-error-on-empty-input
It can be fixed by adding a complete inside the opt!:
named!(a_bug <(&[u8], Option<&[u8]>)>, do_parse!(
    a:tag!("a") >>
    b: opt!(complete!(tag!("b"))) >>
    ((a, b))
));

Okay that works but still seems like a bug to me (principle of least suprise etc).
Surely opt! should always convert incomplete into None?

I'd think that makes sense, it can be tricky to recognize what's going wrong there. I only know due to recently having to deal with the issue in upgrading to nom 2.0.
I can't comment exactly on any technical reasons why it was changed, but it looks like it might be a consequence of these issues #217 and #226, found through the nom 2.0 tracking issue.

Just noticed that the opt! macro is specified to return "Incomplete if it did not have enough data to decide".

@danielbodart I agree with the principle of least surprise, but here, the real surprise does not come from nom's operators, it comes from the underlying use case: working on data streams like network sockets or buffered files.
If your parser must recognize an optional "abcd", but the TCP packets chunk it in "ab" then "cd", the parser should tell you that it does not have enough data to decide with "ab", because if it returns None already, then it will not interpret "cd" correctly, although the data is correct.
I tried with nom to make robust parsers, and incomplete data is one of the biggest sources of errors.

Hmmm that sounds sensible but I think you actually have a broken grammar IMHO, you should never end with a trailing opt in a grammar for exactly this reason.
I'm actually using nom for parsing HTTP exactly according to the RFC and there is not a single case in the whole grammar that I don't have to force opt!(complete!(...)) and I test with fragmented data.
I've used many parser combinator libraries and am the author of a couple (in Java) and have never come across an optional combinator that isn't optional!

I was not talking about a trailing opt. This could happen right in the middle of your format. If you don't believe me, let's take a look at a case in your HTTP parser (I'm not familiar with your code, so there may be issues I have not seen, but I trust you'll get the point):
this line has a opt!(complete!: https://github.com/danielbodart/http-handler.rust/blob/master/src/grammar.rs#L100
and the result is used in: https://github.com/danielbodart/http-handler.rust/blob/master/src/grammar.rs#L111
let's say you get the header Accept-Encoding: gzip, deflate\r\n, but with the value chunked like this:

gzi
p, d
eflate

Since this parser ends up merging all the subslices (and the content of this header is not particularly meaningful), it won't have an effect in the end, but let's see how the individual parts will be recognized by field_vchar: gzi, p,, d, eflate.
While in that case, you would like to see: gzip,, deflate.
That's why, if you cannot work on the complete data at once, the optional part should contain a terminating token of some kind. If the token isn't there, ask for more data then reparse.
Can you help me find a better way to handle that kind of behaviour?

To me seems least surprising that if an optional is not decidable with the current data it would signal that by default and the whole thing gets forwarded.
You do not want drop an optional because the chunk you have is incomplete generally, IMHO.

@Geal I don't think we are going to agree, a field_vchar might be usable as standalone parser but it's not meant to be. It doesn't matter where you chunk it because you would never use field_vchar on it's own. As soon as you put it into the bigger combinators of header_field or headers or http_message chunking just works.
Following the HTTP RFC by the letter was a pretty great experience showing how well designed the grammar is and I was able to do it by following the grammar exactly. Every test just worked first time except for anything optional, hence why I originally raised this issue.
I fundamentally think that any grammar that ends with an optional element is broken as it's just ambiguous. Excluding those then I think returning incomplete is a trade-off between performance and usability. I prefer usability and having to re-parse a larger chunk as any optional element will always be contained in a larger parser.
Or maybe I'm miss understanding your point. I'm pretty chilled about the whole thing, Feel free to close this. I just have to always remember to do opt!(complete!()). I've actually been meaning to just create a optional! macro anyway.
I'd be interested to see how many projects using nom are doing opt!(complete!()) vs opt!() on it's own.

@danielbodart as I mentioned the way your parser is designed, the chunking inside optional components won't affect you, but it might affect others.
I feel there are two points here where we misunderstand each other:

you talk about trailing optional elements, while I talk about optional elements that can appear anywhere. I agree that trailing optional elements are painful to do if we take optional parsers and incomplete behaviour into account, and there are often better ways to handle them
"I prefer usability and having to re-parse a larger chunk as any optional element will always be contained in a larger parser": this is where we agree, since the pattern of a parser returning Incomplete to let the calling code reparse with a larger chunk is one of the design decisions in nom. But optional elements can be much larger. Like, in mp4, it's a convention to have the metadata block at the beginning (as it's better for streaming), but the original spec tells you to put it at the end. So you make it optional, and adapt the state machine's behaviour to its presence.

I feel I should t least add more documentation to opt! to explain the issue and let people decide what they want. I'd be interested in a survey of how people use that combinator as well.

@Geal

That's why, if you cannot work on the complete data at once, the optional part should contain a terminating token of some kind. If the token isn't there, ask for more data then reparse.

Just today I wrote a parser where every element is opt!. And will be writing another one in a couple of days. Because that's what the grammars are. Now in this case, the input is a short string always in memory, so I don't care about incremental processing, but other grammars may exist where the end-of-input is significant. HTTP/0.9 comes to mind as example (the body is terminated by end-of-connection).

Can you help me find a better way to handle that kind of behaviour?

I think it should be possible to signify the end-of-input condition to the parser. It is not needed often, because most formats do have way to recognise complete input for error checking, but exceptions exist.

I was not talking about a trailing opt. This could happen right in the middle of your format.

By the way—and I may be misunderstanding things as I didn't find good documentation for this—doesn't the same problem occur for take_while!, digits and similar?
Say I have:
#[macro_use]
extern crate nom;

use std::str;
use std::str::FromStr;

named!(
    integer<i32>,
    map_res!(
        map_res!(
            nom::digit,
            str::from_utf8
        ),
        FromStr::from_str
    )
);

fn main() {
    println!("{:?}", integer("123"));
}

This prints Done([], 123). However, say my input is actually 12345, split to 123 and 45. How will the parser know to return and reparse the whole thing when the first part seemed successful?
It it won't, that would be a bug. If it will, wouldn't the same approach work on the empty inputs?

Hit this about a week ago. I'm writing a parser for an xml-like syntax and have opted to tokenize first and then parse (after an initial all-in-one approach got a bit hairy). It was a bit surprising to have a do_parse on a token of "10%" (where the percentage sign is optional) return Incomplete just because the chain of parsers had an opt at the end.
I've read the rest of the thread and I can see why this happens, just wanted to give another example of confusion. I probably wouldn't have tripped up if the docs for opt! mentioned the issue (perhaps as a note below the example or something).

Hit this issue today. I'm writing a parser for a format where semicolon is optional at the end. So this was parsing correctly:
COMMAND(ARG);

but this was giving me Incomplete:
COMMAND(ARG)

If docs for opt! had a suggestion to use opt!(complete!(tag_s!(";"))), I would have wasted less time.

There is now some more documentation on using opt with trailing data in a992cb2. This might get easier once #144 advances too.


Queue Priority in Routing Profile

The routing profile exposed by Agent.getConfiguration().routingProfile and Agent.getRoutingProfile(), does not provide the priority assigned to each queue contained in the routing profile.  Is there a way to obtain this priority information from the routing profile assigned to the agent?

Hi aossey@, unfortunately currently there is no way of doing this. We will consider this feature request and prioritize it accordingly. Thanks for the feedback.


RFC: Ingress/Egress into the Postgres

Rendered

After looking over this, particularly with @lassizci's and @hlinnaka's comments:

I think security wise it's better to do [eBPF setup] on orchestrator layer

and

With VMs, we should load the eBPF module in the host kernel rather than the guest

I wonder if this may be better left to the NeonVM controller (at least for VMs). There are some benefits from compute_ctl though:

It gives a unified interface for pods vs VMs
compute_ctl already knows which endpoint it is; there's no extra attribution

However, I'd be concerned about:

What permissions are required to set up an eBPF program? Does that create a wider attack surface for pod-based ? (maybe this doesn't matter, given that we're moving to VMs)
If an attacker gains access to a compute node and takes over the eBPF reporting, do we have any way to tell that something suspicious is afoot?

As I can see it, the benefits of doing this in the NeonVM controller are:

We already have consumption tracking implemented in the autoscaler-agents for CPU time and active time; networking could be easy to add there
If we want the eBPF programs to be added to the host kernel, we already have some NeonVM parts per-host, so we could do that setup there (or with a similar daemonset). Or, we can just use the VM runner (pod that runs QEMU), because we already have communication with it
If the VM is compromised, we still get accurate data - so long as the pod running the VM is not also compromised.

As I see it, the difficulties with implementing it in NeonVM are:

There's limits on accuracy of what we report — given the switch to compute pooling, we're going to have some inaccuracy on what we measure (happy to elaborate on this more)
Network traffic will include compute_ctl (and the vm-informant[^informant]), not just postgres
Setting/getting the current traffic quantity must go through k8s; we have to decide how often we update the amount, which will always be a little bit out of date. This may dramatically increase the rate of k8s operations.
Handling VM migrations is tricky — we kind of have two pods for the same VM, with only one pod actually running (the other just sending/receiving data to/from the other). This problem should be fine, but need to be careful about.

Given all that, I think if the tricky bits with doing it via NeonVM are manageable (they might not be!), we should implement it there. Other than that, I don't know.
[^informant]: reminder: the vm-informant is the autoscaling component inside the VM

Could you describe how we would ignore network usage for those data streams?
We can just look at the IP address of the incoming/outgoing packet to decide

which means that extensions could also access this mmapped area and reset the counters. How would we protect against that?
As I understand it, you need the file descriptor to the BPF program to perform the mmap, so I'm not sure if this is an issue.

If you reset the value first, and then send it to a billing metrics collector, how would this handle failed requests?
I'd handle failed requests at the messaging layer - just retry sending failed messages until they work.


Create default page

Create a default page that contains information about getting started with Beacon.
It should be created on the site boot, after this line: https://github.com/BeaconCMS/beacon/blob/46147338b65b02b7e406dd1cbbe875f5561cf2e5/lib/beacon/boot.ex#L34
The page will be created on path / unless a page is already created.

Closed by https://github.com/BeaconCMS/beacon/pull/475


Rewrite data management user guide a bit

putting this up for people to look at, but I want to work on it more

Is this still active?  And does it make sense to look at now?

@danielskatz I just re-read this text, and I'm happy for it to be merged into master.

@benclifford  - I've made a bunch of small comments in-line.  In addition, I think it would make sense to add a little general text under the staging providers header that explains at that point the difference between the FTP, HTTP, HTTPS: separate task staging and FTP, HTTP, HTTPS: in-task staging sections that follow, something like - if you have this situation, use this part below...

The rest of this looks good to me

I've gone ahead and made the changes that are small.
I think the one more thing that is needed is to add a little general text under the staging providers header that explains at that point the difference between the FTP, HTTP, HTTPS: separate task staging and FTP, HTTP, HTTPS: in-task staging sections that follow, something like - if you have this situation, use this part below...

@danielskatz why don't we merge this now. Since you've already done a fair bit of testing with staging, can you try re-reading some of this and recommend changes in a separate PR?

I suggest we just merge this PR for now - It already has a bunch of changes I added a few weeks ago, and I don't have a good suggestion for the part that I thought still needed something more.


Add Dockerfiles and instructions for using docker containers

Dockerfile creates an image from the master branch of this repository.
Dockerfile.dev creates an image from the local repository tree.
Added instructions to README.md with how to use these containers in a simple use case.

Continuing @Manouchehri 's PR with an additional development dockerfile. #3

Thank you for continuing the work on Docker support! I will review the changes and will let you know.

Sorry for submitting that PR with obvious errors. I'm still getting used to the github workflow.

No problem ;-).

Responded to indentation comments. :)

Thank you, merged!


Application Control event tags not showing in the Wizard - for example 15 "Failed WHQL check."

Hello,
One question concerning kernel mode drivers (*.sys), that have failed the WHQL check with Application control event tag 15 "Failed WHQL check." - as it is listed here from Microsoft:
https://learn.microsoft.com/en-us/windows/security/application-security/application-control/windows-defender-application-control/operations/event-tag-explanations
Is it possible to parse this for creating a policy with the "Convert Event Log to WDAC policy" function in the Wizard? Actually it doesn't show this, but it is in the event logs defined and will be created if an SYS Driver fails WHQL check.
As an example: the new "logi_lamparray.sys" driver from the new dynamic lighting feature in Windows 11 with Logitech devices fails with event tag 15, but you can't see this in the wizard - so i still use the very nice WDAC Toolkit PS Module from Matt Graeber (see https://github.com/mattifestation/WDACTools/blob/master/WDACAuditing.psm1) which can show this tag as you can see in this screenshot:

Would be amazing if this could be added in a future version of the wizard, it would help a lot if you have a WDAC policy with strict WHQL enforcement - thanks in advance!

Great suggestion, @SebCT!

@SebCT Hi
Correct me if I'm wrong but WHQL doesn't mean more security, it only means more compatibility with the OS as in more tests have been done with the driver. So if your intention for strictly enforcing WHQL drivers is to boost security then it just won't do it.
https://learn.microsoft.com/en-us/windows-hardware/drivers/install/whql-release-signature
That's why for more security, we need to actually have a zero-trust strategy.

You can use WDAC Wizard and WDACConfig module which also reveals that driver you're referring to when you deploy a strict kernel-mode policy in audit mode, offers the native PowerShell GUI for selecting individual logs, I happen to have the same hardware as you. The combination of both the Wizard and module offer so many features and capabilities and they just keep getting better 🙂

@SebCT Hi
Correct me if I'm wrong but WHQL doesn't mean more security, it only means more compatibility with the OS as in more tests have been done with the driver. So if your intention for strictly enforcing WHQL drivers is to boost security then it just won't do it.
https://learn.microsoft.com/en-us/windows-hardware/drivers/install/whql-release-signature
That's why for more security, we need to actually have a zero-trust strategy.
You can use WDAC Wizard and WDACConfig module which also reveals that driver you're referring to when you deploy a strict kernel-mode policy in audit mode, offers the native PowerShell GUI for selecting individual logs, I happen to have the same hardware as you. The combination of both the Wizard and module offer so many features and capabilities and they just keep getting better 🙂

Hello!
Thanks for your reply, i know your WDAC-module a little bit and it's really an amazing work, will get to know it better the next time and will try it out :-)
Concerning the strict WHQL enforcement: is important for denying cross-signed WHQL drivers and it should be the norm since Win10 1607 to sign kernel drivers striclty WHQL - here are some links, one is from Matt Graeber, i learned a lot from his blogs and online material since 2016 (i started with the former device guard in 2016, it was very difficult, WAC is a lot better now), and another one from Geoff Chappell concerning the danger behind cross-signed drivers:
Adventures in Extremely Strict Device Guard Policy Configuration Part 1 — Device Drivers:
https://posts.specterops.io/adventures-in-extremely-strict-device-guard-policy-configuration-part-1-device-drivers-fd1a281b35a8
-> "I also applied the “Required: WHQL” option as well because I wanted to apply as many rules as possible where WHQL co-signing validation was also performed."
Back doors for cross signed drivers:
https://www.geoffchappell.com/notes/security/whqlsettings/index.htm
-> "For Windows 10, Microsoft took the opportunity to draw a line and make it significantly more difficult for anyone outside Microsoft to sign drivers that Windows will load in anything like ordinary use. Microsoft’s first-known public statement of motivation for Driver Signing changes in Windows 10 was:
We’re making these changes to help make Windows more secure. These changes limit the risk of a driver publisher’s signing keys being lost or stolen and also ensures that driver publishers are strongly authenticated.
For most practical effect, all new drivers must be signed directly by Microsoft.The days of Windows users being exposed to cross-signed drivers that can have been signed by just about anyone, possibly for mischief, possibly without being traceable, are numbered. Though this goal surely could be met securely without going so far as requiring that drivers can load only if signed by Microsoft, there doesn’t look to be any reasonable dispute that the goal is worthy or even that cross-signed drivers would better be gone."
Some Links from MS:
https://techcommunity.microsoft.com/t5/windows-hardware-certification/driver-signing-changes-in-windows-10-version-1607/ba-p/364894
https://learn.microsoft.com/en-us/windows/security/application-security/application-control/windows-defender-application-control/design/select-types-of-rules-to-create

https://learn.microsoft.com/en-us/windows-hardware/drivers/install/kernel-mode-code-signing-policy--windows-vista-and-later-
https://learn.microsoft.com/en-us/windows-hardware/drivers/install/deprecation-of-software-publisher-certificates-and-commercial-release-certificates

@SebCT
Well, the thing is, those articles aren't up to date, the threats they were trying to mitigate have changed significantly since then. We're in the era of BYOVDs, where totally-legit Microsoft-signed drivers are being used maliciously because they have vulnerabilities.
WHQL enforcement won't help it at all, one step further than that is enforcing EV Certificates for drivers but even that won't help with today's threat model as I explained in my article.
If you really want to vet your drivers and lock down your kernel, you need to follow that scenario and remove the blind trust to any entity whatsoever (except for the 3 root certificates used by the Windows itself) and that is the only way to stay protected against BYOVD scenarios and much more.
Nowadays, threat actors don't care if a driver is WHQL signed or not and that's the point I was trying to explain in my comment, WHQL certification does not include vulnerability scanning of drivers, it only guarantees that the hardware driver is compatible with the OS.
I just like when I decide to do something, do it once and right, now on my systems, when I install some random app such as OBS, its kernel-mode virtual webcam driver doesn't automatically load because I didn't allow it.
When I install system informer or WireShark, its kernel-mode driver doesn't run even though it's WHQL signed and so on, it just doesn't matter anymore.
Even if the whole world says something is trustworthy, as long as your personalized policy on your computer doesn't allow a kernel-mode file to run, it shouldn't run, that's really powerful stuff that Windows has got since many years ago!

@SebCT Well, the thing is, those articles aren't up to date, the threats they were trying to mitigate have changed significantly since then. We're in the era of BYOVDs, where totally-legit Microsoft-signed drivers are being used maliciously because they have vulnerabilities.
WHQL enforcement won't help it at all, one step further than that is enforcing EV Certificates for drivers but even that won't help with today's threat model as I explained in my article.
If you really want to vet your drivers and lock down your kernel, you need to follow that scenario and remove the blind trust to any entity whatsoever (except for the 3 root certificates used by the Windows itself) and that is the only way to stay protected against BYOVD scenarios and much more.
Nowadays, threat actors don't care if a driver is WHQL signed or not and that's the point I was trying to explain in my comment, WHQL certification does not include vulnerability scanning of drivers, it only guarantees that the hardware driver is compatible with the OS.
I just like when I decide to do something, do it once and right, now on my systems, when I install some random app such as OBS, its kernel-mode virtual webcam driver doesn't automatically load because I didn't allow it.
When I install system informer or WireShark, its kernel-mode driver doesn't run even though it's WHQL signed and so on, it just doesn't matter anymore.
Even if the whole world says something is trustworthy, as long as your personalized policy on your computer doesn't allow a kernel-mode file to run, it shouldn't run, that's really powerful stuff that Windows has got since many years ago!
P.S you can use WDAC Wizard and its GUI to achieve the scenario i explained in my article as well. Just follow it and use the Wizard to remove the unnecessary signers from the default Windows template.

I know the BYOVD scenario, so i always used an strict allowlist on my systems and i only use the few EKU's you mentioned on my systems (i also do not want flight signing EKUs). Of course it's something not everyone wants to do because it can be painful - but strict driver allowlisting (in combination with the recommended driver block list from MS) is the configuration i really like.
And as i meant with thses articles, even if they are older now, i want to block those legacy cross-signed or badly signed drivers, so i use the WHQL setting only for this particular scenario. Wireshark still works for me by the way with WHQL enforced, and i have to say i don't had much problems with this settings - it was the first time since several years that i saw such a block now with this logitech driver (which i don't need anyway, GHUB doesn't use it).


[FEATURE REQ] GA OpenTelemetry support

Library name
Azure.Core
Please describe the feature.
We have a partner ask to GA OTel tracing plugins across all languages in [Zn], this is an uber-issue to track .NET work in this space.
Here're the sub-tasks:

Support telemetry schema versioning (attribute names). User should be able to set OTel schema version from an enum we provide and DiagnsoticScope should map attribute names according to the given version. This should not affect the DiagnosticSource path.
Align with OTel messaging semantic conventions
Functional and perf testing and other formal GA prereqs

Label prediction was below confidence level 0.6 for Model:ServiceLabels: 'Event Hubs:0.16724567,Service Bus:0.13776705,Azure.Core:0.09390852'

The HTTP part is stable - done in https://github.com/Azure/azure-sdk-for-net/pull/39682, https://github.com/Azure/azure-sdk-for-net/pull/39637
Closing this issue since messaging is staying experimental for the time being.


More examples

[ ] sender
[ ] receiver
[ ] listener
[ ] transaction posting
[ ] transaction retirement


Add support for Scala 2.13

Remove deprecated scala.collection.JavaConversions
Add scala.collection.JavaConverters
Clean up build and update dependencies

Thanks for your contribution, and I merged it @lolgab 👍


"Proposal: json-file logging driver should rotate logs by default

Description
Default logging config for docker is "until my computer runs out of disk space"
When doing some performance testing I noticed that after a couple weeks my services were crashing. After digging into the issue I noticed that some of my servers were out of disk space.
I know you can set log-opt manually to these parameters, in daemon config. I think it might be safer out of the box to just do that instead of growing unbounded. Thoughts?
Steps to reproduce the issue:

Create a very small server (disk space)
Install docker with no config changes
Run a container that generates a lot of logs
wait until there's no disk space left

Describe the results you received:
Issues running commands on server (cause there is no disk space left)
Describe the results you expected:
Sane defaults for logging configs to prevent docker from consuming the entirety of available disk space
Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
    Client:
     Version:      1.12.3-cs3
     API version:  1.24
     Go version:   go1.6.3
     Git commit:   781bb16
     Built:        Thu Oct 27 17:45:19 2016
     OS/Arch:      linux/amd64

    Server:
     Version:      1.12.3-cs3
     API version:  1.24
     Go version:   go1.6.3
     Git commit:   781bb16
     Built:        Thu Oct 27 17:45:19 2016
     OS/Arch:      linux/amd64```

**Output of `docker info`:**

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 11
Server Version: 1.12.3-cs3
Storage Driver: devicemapper
 Pool Name: docker-202:1-149203-pool
 Pool Blocksize: 65.54 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: ext4
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 697.2 MB
 Data Space Total: 107.4 GB
 Data Space Available: 6.352 GB
 Metadata Space Used: 1.638 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.146 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.77 (2012-10-15)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge overlay null host
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 3.13.0-96-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.676 GiB
Name: ip-172-31-37-98
ID: 5WMS:A4PC:JLN4:6VV7:3UHO:5WPF:WLG4:HW2O:ID3L:GUD4:QWWZ:2ODA
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: fxdgear
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
 <IP_ADDRESS>/8```

Do we have log rotation for other logging drivers like syslog.

@milindchawre It's not up to docker to specify log rotation for services outside of docker's control.

Generally, I'm +1 to this, as I have seen people not realizing it's not rotating, and running out of disk space. Will this break people that collect these logs? (I know people shouldn't, but I know it's done, e.g. kubernetes seems to symlink all these log-files)

We discussed this in the maintainers meeting, and think that it would be difficult to set a real default size for json-logs; we are thinking of taking a similar approach as graph-drivers; and pick the best available logging driver (e.g. use journald by default, but fall back to json-logs) wdyt? @fxdgear

I think "best" in this case greatly depends on what you want to accomplish.
jsonfile is, ignoring the horrible hacks (pkg/jsonlog) to get there, a fairly efficient logger today.


Commit access / move repo?

I don't need to do anything here right this instant, but since PackageEvaluator depends on this to generate its plots the site data, I might at some point. Probably better if multiple people have commit access, or have this in an org, just in case?

JuliaPackaging sounds fine by me!

Thanks!


Release for CUDA 11.2

The available CUDA version I have is 11.2 and I would like to install jax on it.
Currently there is now release for jaxlib-0.1.60+cuda112.
Is it possible to build a release that supports CUDA 11.2?
Is there another way to install jax with CUDA 11.2 other than running
pip install --upgrade jax jaxlib==0.1.60+cuda112 -f https://storage.googleapis.com/jax-releases/jax_releases.html
Thanks!

I am trying to get it working with cuda11.2 as well. I tried to build from source, but I could not manage to do it.
OS: Ubuntu 20.04.1
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Nov_30_19:08:53_PST_2020
Cuda compilation tools, release 11.2, V11.2.67
Build cuda_11.2.r11.2/compiler.29373293_0

cudnn version: <IP_ADDRESS>
python version: 3.8.5
After following the instructions for building from source, I got the following error.
$ python build/build.py --enable_cuda

     _   _  __  __
    | | / \ \ \/ /
 _  | |/ _ \ \  /
| |_| / ___ \/  \
 \___/_/   \/_/\_\

Another command (pid=8235) is running.  Waiting for it to complete on the server...
Bazel binary path: ./bazel-3.1.0-linux-x86_64
Python binary path: /home/fer/miniconda3/envs/jax-gpu/bin/python
Python version: 3.8
MKL-DNN enabled: yes
-march=native: no
CUDA enabled: yes
CUDA compute capabilities: 3.5,5.2,6.0,6.1,7.0
ROCm enabled: no

Building XLA and installing it in the jaxlib source tree...
./bazel-3.1.0-linux-x86_64 run --verbose_failures=true --config=short_logs --config=mkl_open_source_only --config=cuda --define=xla_python_enable_gpu=true :build_wheel -- --output_path=/home/fer/jax/dist
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'run' from /home/fer/jax/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'run' from /home/fer/jax/.bazelrc:
  Inherited 'build' options: --repo_env PYTHON_BIN_PATH=/home/fer/miniconda3/envs/jax-gpu/bin/python --action_env=PYENV_ROOT --python_path=/home/fer/miniconda3/envs/jax-gpu/bin/python --repo_env TF_NEED_CUDA=1 --action_env TF_CUDA_COMPUTE_CAPABILITIES=3.5,5.2,6.0,6.1,7.0 --repo_env TF_NEED_ROCM=0 --action_env TF_ROCM_AMDGPU_TARGETS=gfx803,gfx900,gfx906,gfx1010 --distinct_host_configuration=false -c opt --apple_platform_type=macos --macos_minimum_os=10.9 --announce_rc --define open_source_build=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true --spawn_strategy=standalone --strategy=Genrule=standalone --enable_platform_specific_config
INFO: Found applicable config definition build:short_logs in file /home/fer/jax/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:mkl_open_source_only in file /home/fer/jax/.bazelrc: --define=tensorflow_mkldnn_contraction_kernel=1
INFO: Found applicable config definition build:cuda in file /home/fer/jax/.bazelrc: --crosstool_top=@local_config_cuda//crosstool:toolchain --define=using_cuda=true --define=using_cuda_nvcc=true
INFO: Found applicable config definition build:linux in file /home/fer/jax/.bazelrc: --copt=-Wno-sign-compare --define=no_aws_support=true --define=no_gcp_support=true --define=no_hdfs_support=true --cxxopt=-std=c++14 --host_cxxopt=-std=c++14 --copt=-Wno-stringop-truncation
Loading:
Loading: 0 packages loaded
DEBUG: /home/fer/.cache/bazel/_bazel_fer/762940ba9c9dd4b483569ce1647fce9c/external/bazel_tools/tools/cpp/lib_cc_configure.bzl:118:5:
Auto-Configuration Warning: 'TMP' environment variable is not set, using 'C:\Windows\Temp' as default
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (0 packages loaded, 0 targets configured)
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/e635feb15a91e6eeb77876031be2811e63d542f3.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
DEBUG: Rule 'io_bazel_rules_docker' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1556410077 -0400"
DEBUG: Repository io_bazel_rules_docker instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule git_repository defined at:
  /home/fer/.cache/bazel/_bazel_fer/762940ba9c9dd4b483569ce1647fce9c/external/bazel_tools/tools/build_defs/repo/git.bzl:195:18: in <toplevel>
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/nvidia/nccl/archive/v2.8.3-1.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 404 Not Found
WARNING: Download from https://gitlab.mpcdf.mpg.de/mtr/pocketfft/-/archive/53e9dd4d12f986207c96d97c5183f5a72239c76e/pocketfft-53e9dd4d12f986207c96d97c5183f5a72239c76e.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 406 Not Acceptable
INFO: Analyzed target //build:build_wheel (14 packages loaded, 6829 targets configured).
INFO: Found 1 target...
[0 / 199] [Prepa] Expanding template build/build_wheel ... (3 actions, 0 running)
[1,613 / 2,315] Compiling external/com_google_protobuf/src/google/protobuf/descriptor.pb.cc; 3s local ... (8 actions, 7 running)
[1,830 / 2,315] Compiling jaxlib/cuda_prng_kernels.cu.cc; 5s local ... (8 actions, 7 running)
[1,942 / 2,315] Compiling external/com_google_protobuf/src/google/protobuf/extension_set_heavy.cc; 2s local ... (8 actions, 7 running)
[2,121 / 2,349] Compiling external/com_google_protobuf/src/google/protobuf/generated_message_table_driven_lite.cc; 5s local ... (8 actions, 7 running)
[2,285 / 2,363] Compiling external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_stub.cc; 8s local ... (8 actions, 7 running)
[2,645 / 2,877] Compiling external/org_tensorflow/tensorflow/core/framework/function.pb.cc; 4s local ... (8 actions, 7 running)
[3,199 / 3,375] Compiling external/org_tensorflow/tensorflow/core/framework/common_shape_fns.cc; 5s local ... (8 actions, 7 running)
[3,622 / 3,735] Compiling external/org_tensorflow/tensorflow/compiler/xla/service/hlo_instruction.cc; 19s local ... (8 actions, 7 running)
[3,789 / 3,878] Compiling external/org_tensorflow/tensorflow/core/util/batch_util.cc; 45s local ... (8 actions, 7 running)
[5,569 / 6,664] Compiling external/flatbuffers/src/idl_parser.cpp; 16s local ... (8 actions, 7 running)
[5,868 / 6,664] Compiling external/llvm-project/llvm/utils/TableGen/GlobalISelEmitter.cpp; 12s local ... (8 actions, 7 running)
[6,657 / 7,380] Compiling external/llvm-project/mlir/lib/IR/BuiltinDialect.cpp; 6s local ... (8 actions, 7 running)
[6,826 / 7,551] Compiling external/llvm-project/llvm/lib/Target/AMDGPU/AMDGPUTargetMachine.cpp; 10s local ... (8 actions, 7 running)
[6,870 / 7,570] Compiling external/org_tensorflow/tensorflow/compiler/xla/service/hlo_evaluator.cc; 100s local ... (8 actions running)
[6,870 / 7,570] Compiling external/org_tensorflow/tensorflow/compiler/xla/service/hlo_evaluator.cc; 280s local ... (8 actions running)
[6,870 / 7,570] Compiling external/org_tensorflow/tensorflow/compiler/xla/service/hlo_evaluator.cc; 466s local ... (8 actions running)
[6,870 / 7,570] Compiling external/org_tensorflow/tensorflow/compiler/xla/service/hlo_evaluator.cc; 680s local ... (8 actions running)
[6,870 / 7,570] Compiling external/org_tensorflow/tensorflow/compiler/xla/service/hlo_evaluator.cc; 927s local ... (8 actions running)
[6,870 / 7,570] Compiling external/org_tensorflow/tensorflow/compiler/xla/service/hlo_evaluator.cc; 1210s local ... (8 actions running)
[6,870 / 7,570] Compiling external/org_tensorflow/tensorflow/compiler/xla/service/hlo_evaluator.cc; 1535s local ... (8 actions running)
[6,870 / 7,570] Compiling external/org_tensorflow/tensorflow/compiler/xla/service/hlo_evaluator.cc; 1915s local ... (8 actions running)
ERROR: /home/fer/.cache/bazel/_bazel_fer/762940ba9c9dd4b483569ce1647fce9c/external/org_tensorflow/tensorflow/compiler/xla/service/BUILD:266:1: C++ compilation of rule '@org_tensorflow//tensorflow/compiler/xla/service:hlo_evaluator' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command
  (cd /home/fer/.cache/bazel/_bazel_fer/762940ba9c9dd4b483569ce1647fce9c/execroot/__main__ && \
  exec env - \
    LD_LIBRARY_PATH=/usr/local/cuda-11.2/lib64 \
    PATH=/usr/local/cuda-11.2/bin:/home/fer/miniconda3/envs/jax-gpu/bin:/home/fer/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
    PWD=/proc/self/cwd \
    TF_CUDA_COMPUTE_CAPABILITIES=3.5,5.2,6.0,6.1,7.0 \
    TF_ROCM_AMDGPU_TARGETS=gfx803,gfx900,gfx906,gfx1010 \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/compiler/xla/service/_objs/hlo_evaluator/hlo_evaluator.pic.d '-frandom-seed=bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/compiler/xla/service/_objs/hlo_evaluator/hlo_evaluator.pic.o' -DTF_USE_SNAPPY -DHAVE_SYS_UIO_H -DTENSORFLOW_USE_CUSTOM_CONTRACTION_KERNEL -DTENSORFLOW_USE_MKLDNN_CONTRACTION_KERNEL -DEIGEN_MPL2_ONLY '-DEIGEN_MAX_ALIGN_BYTES=64' '-DEIGEN_HAS_TYPE_TRAITS=0' -D__CLANG_SUPPORT_DYN_ANNOTATION__ -iquote external/org_tensorflow -iquote bazel-out/k8-opt/bin/external/org_tensorflow -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -iquote external/nsync -iquote bazel-out/k8-opt/bin/external/nsync -iquote external/eigen_archive -iquote bazel-out/k8-opt/bin/external/eigen_archive -iquote external/gif -iquote bazel-out/k8-opt/bin/external/gif -iquote external/libjpeg_turbo -iquote bazel-out/k8-opt/bin/external/libjpeg_turbo -iquote external/com_google_protobuf -iquote bazel-out/k8-opt/bin/external/com_google_protobuf -iquote external/zlib -iquote bazel-out/k8-opt/bin/external/zlib -iquote external/com_googlesource_code_re2 -iquote bazel-out/k8-opt/bin/external/com_googlesource_code_re2 -iquote external/farmhash_archive -iquote bazel-out/k8-opt/bin/external/farmhash_archive -iquote external/fft2d -iquote bazel-out/k8-opt/bin/external/fft2d -iquote external/highwayhash -iquote bazel-out/k8-opt/bin/external/highwayhash -iquote external/double_conversion -iquote bazel-out/k8-opt/bin/external/double_conversion -iquote external/snappy -iquote bazel-out/k8-opt/bin/external/snappy -iquote external/local_config_cuda -iquote bazel-out/k8-opt/bin/external/local_config_cuda -iquote external/local_config_tensorrt -iquote bazel-out/k8-opt/bin/external/local_config_tensorrt -iquote external/mkl_dnn -iquote bazel-out/k8-opt/bin/external/mkl_dnn -Ibazel-out/k8-opt/bin/external/local_config_cuda/cuda/_virtual_includes/cuda_headers_virtual -Ibazel-out/k8-opt/bin/external/local_config_tensorrt/_virtual_includes/tensorrt_headers -isystem external/nsync/public -isystem bazel-out/k8-opt/bin/external/nsync/public -isystem external/eigen_archive -isystem bazel-out/k8-opt/bin/external/eigen_archive -isystem external/gif -isystem bazel-out/k8-opt/bin/external/gif -isystem external/com_google_protobuf/src -isystem bazel-out/k8-opt/bin/external/com_google_protobuf/src -isystem external/zlib -isystem bazel-out/k8-opt/bin/external/zlib -isystem external/farmhash_archive/src -isystem bazel-out/k8-opt/bin/external/farmhash_archive/src -isystem external/double_conversion -isystem bazel-out/k8-opt/bin/external/double_conversion -isystem external/local_config_cuda/cuda -isystem bazel-out/k8-opt/bin/external/local_config_cuda/cuda -isystem external/local_config_cuda/cuda/cuda/include -isystem bazel-out/k8-opt/bin/external/local_config_cuda/cuda/cuda/include -isystem external/mkl_dnn/include -isystem bazel-out/k8-opt/bin/external/mkl_dnn/include -isystem external/mkl_dnn/src -isystem bazel-out/k8-opt/bin/external/mkl_dnn/src -isystem external/mkl_dnn/src/common -isystem bazel-out/k8-opt/bin/external/mkl_dnn/src/common -isystem external/mkl_dnn/src/cpu -isystem bazel-out/k8-opt/bin/external/mkl_dnn/src/cpu -isystem external/mkl_dnn/src/cpu/gemm -isystem bazel-out/k8-opt/bin/external/mkl_dnn/src/cpu/gemm -isystem external/mkl_dnn/src/cpu/xbyak -isystem bazel-out/k8-opt/bin/external/mkl_dnn/src/cpu/xbyak -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIC -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -fno-omit-frame-pointer -no-canonical-prefixes -fno-canonical-system-headers -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections -Wno-sign-compare -Wno-stringop-truncation '-std=c++14' -c external/org_tensorflow/tensorflow/compiler/xla/service/hlo_evaluator.cc -o bazel-out/k8-opt/bin/external/org_tensorflow/tensorflow/compiler/xla/service/_objs/hlo_evaluator/hlo_evaluator.pic.o)
Execution platform: @local_execution_config_platform//:platform
In file included from external/org_tensorflow/tensorflow/compiler/xla/service/hlo_evaluator.cc:15:
external/org_tensorflow/tensorflow/compiler/xla/service/hlo_evaluator.h:77:3: warning: multi-line comment [-Wcomment]
   77 |   //            /       \
      |   ^
external/org_tensorflow/tensorflow/compiler/xla/service/hlo_evaluator.h:79:3: warning: multi-line comment [-Wcomment]
   79 |   //        /      \
      |   ^
In file included from external/org_tensorflow/tensorflow/compiler/xla/service/hlo_evaluator.cc:39:
external/org_tensorflow/tensorflow/compiler/xla/service/hlo_evaluator_typed_visitor.h: In member function ‘double xla::HloEvaluatorTypedVisitor<ReturnT, ElementwiseT>::GetAsDouble(const xla::Literal&, absl::lts_2020_02_25::Span<const long int>)’:
external/org_tensorflow/tensorflow/compiler/xla/service/hlo_evaluator_typed_visitor.h:149:3: warning: no return statement in function returning non-void [-Wreturn-type]
  149 |   }
      |   ^
gcc: fatal error: Killed signal terminated program cc1plus
compilation terminated.
Target //build:build_wheel failed to build
INFO: Elapsed time: 3087.628s, Critical Path: 2160.17s
INFO: 3380 processes: 3380 local.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully
Traceback (most recent call last):
  File "build/build.py", line 506, in <module>
    main()
  File "build/build.py", line 501, in main
    shell(command)
  File "build/build.py", line 51, in shell
    output = subprocess.check_output(cmd)
  File "/home/fer/miniconda3/envs/jax-gpu/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/home/fer/miniconda3/envs/jax-gpu/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./bazel-3.1.0-linux-x86_64', 'run', '--verbose_failures=true', '--config=short_logs', '--config=mkl_open_source_only', '--config=cuda', '--define=xla_python_enable_gpu=true', ':build_wheel', '--', '--output_path=/home/fer/jax/dist']' returned non-zero exit status 1.

We can add CUDA 11.2 to our next wheel release, which will probably be in the next couple of weeks.
@fcossio That looks like your copy of gcc or clang crashed. Try a different (newer?) version.

The just-released jaxlib 0.1.61 now includes CUDA 11.2 wheels!
pip install --upgrade jax jaxlib==0.1.61+cuda112 -f https://storage.googleapis.com/jax-releases/jax_releases.html

I manually download and installed it. However, I think the following command also works:
pip install --upgrade pip
pip install --upgrade "jax[cuda111]" -f https://storage.googleapis.com/jax-releases/jax_releases.html  # Note: wheels only available on linux.

Great! I'm not sure what might have gone wrong initially (the command from https://github.com/google/jax/issues/5668#issuecomment-778536474 should work on supported systems) but I'm glad you found an approach that worked for you.

@fcossio did you attempt to update your gcc version? I have the same error!

This is a very old issue for me and I don't remember well. I think I solved this issue by expanding the swap memory and then restarting the computer after the build was successful. Sorry that I have no more details.


vector map conversion

Is there a way to convert a vector map from the carla simulator to apollo?

@SanduIrina unfortunately this is not something we have tried at our end. If it worked out for you, we would love if you could share your experience with us.

@SanduIrina Currently roadrunner supports opendrive to apollo, also there're other tools can do the same job.


feat: support automatic compatibility issue fixing for maven project

Hi,
Currently EMT4J already has the capability of scanning for compatibility issues within Java projects. This patch provides the ability of fixing some of common compatibility issue in Maven projects, like Base64 encoding and Maven plugin version upgrades. To implement this functionality, the OpenRewrite dependency has been introduced.
I would be very grateful if you could review this PR.

Hi，
When will this PR be merged ?

I'll review this patch this week and it'll be merged once it is well reviewed.


Help: How do you check for new item

I am using the AngularFireDatabase and I am subscribing for a value change to a list and I want to know if there is a new item added to list. My problem is I get notified by subscription even if there is only a change in field of some item.

Ok, I get it now. I'll add ['child_added'] for event.