Tag Component

A tag component should:

render a bg color and corresponding text color from Radix colors
Take in a simple radix color string as props (enum)
render children with these styles

Would likely need to store Radix color values in object enums to avoid complex patterns to navigate their data structure.

Perhaps start with 6 or so colors then add as we need.

Is #255 a duplicate?
I've built this out. with 4 colors that I chose. Blue, green, pink, and violet. I'd like to get some feedback from design on the colors and can add another 2.

Closed by https://github.com/HausDAO/daohaus-monorepo/pull/349


mission bio cellbarcode assignment and error correction feature enhancement request

Hello quan:
have you ever consider the capability of processing mission bio cell barcode and implement the official error correction algorithms? their cell barcode structure is somehow complex and the official error correction algorithm is somehow more error-prone.
 mission bio cell barcode structure information link
mission bio cell barcode error correction algorithm description
thank you~

To my understanding mission bio use a similar strategy to Dropseq but no UMIs. Am I right?
It contains two segments of cell barcodes. Check parse2 usage.
PISA parse2 -rule 'CR,R1:1-10,bc1.txt,CB,1;CR,R1:11-20,bc2.txt,CB,1;R1,R2:1-100' -1 read_1.fq raw_read_1.fq raw_read_2.fq
This command parses the first segment of cell barcode (CB) from reading 1 base 1-10, and the second segment from base 11 to 20. Each of these segments is corrected with a white list and concate into a CB tag.

not that easy, the positions of two cell barcodes are not the same for each read/library especially for their V2 chemistry

There is some dynamic offset sequence between cell barcode1 and the first const sequence


[Task] Refactor brpc plugin

Description
Refactor shenyu brpc plugin to remove startlight common package
Task List

remove the startlight common module
dependency startlight-api and startlight-brpc

assigned to me pls

It seems that dependency startlight api and startlight brpc are not found in the code

It seems that dependency startlight api and startlight brpc are not found in the code

brpc has been removed in the latest version of shenyu.


Improve tokenizer

PR types
Bug fixes
PR changes
APIs
Description
try to fix issue: https://github.com/PaddlePaddle/PaddleNLP/issues/2520
I believe there are some more potential bugs in from_pretrained, so more boundary tests are required, I have add the basic one.

Here are some testing status about the above unit test:

So, there may be some other problems in large pretrained model.

After using the with style to handle file, all of unit test pass.

Should I add the improvement in this pr ?

It seems there are some expired commits

When I pull the latest code from remote master branch or update branch in github pr webpage, it will create a new commit in this pr. But if this pr is merged, it will not appear in the develop history commits.

When I pull the latest code from remote master branch or update branch in github pr webpage, it will create a new commit in this pr. But if this pr is merged, it will not appear in the develop history commits.

Sure it is, however these expired commit messages would also be merged in which results to redundant messages, and that we want to avoid.


feat(feature-flags): adds feature flags example

:tada: This PR is included in version 1.24.0 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


On/off button styling

The default and active states should look a bit different in dark mode

figma:
https://www.figma.com/file/JXNtryaNYhY0XJNArzvPQx/Handoff?type=design&node-id=1287%3A18959&mode=design&t=CHyw8GVwoKn7OfQx-1
the white version is off as well

plus one thing is that the circle part moves beyond the component

its over the component in the on state

👋

so are we changing how all the toggle buttons look?

@prachi00 it was already like this, but somehow it broke


Allow disabling NotFoundAction via WebConfig

At Faire we want to be able to bind our own 404 action, and forward requests to another service. As discussed on Slack, neither the existing NotFoundAction or the WebProxyAction works for this case. Adding an option to not install the default action allows us to bind our own without having to worry about overlapping paths.

You might need gradle apiDump

@adrw it says the change is breaking, but should be safe with the default parameter?

End of the logs in the broken step it has the steps to solve, a bit annoying but it has caught actual breaking changes before.
gradle :misk:revapiAcceptAllBreaks --justification "{why this break is ok}"


Changing send amount after too high requires starting over

Have an amountless invoice or keysend to a node
pick an amount over your balance
get the error that amount is too high
change the amount to 1 sat (singular, not plural....)
the error still doesn't let you send

found my bug. thanks for opening the ticket @TonyGiorgio

Are we able to make an amountless invoice from within mutinywallet or the signet?

Are we able to make an amountless invoice from within mutinywallet or the signet to test?

No

Ok,well  I hacked together a quick test and figured it out.


Remove experimental status of custom collections

This PR removes the "experimental" status of the custom collections interface. cc @jcrist
xref https://github.com/dask/community/issues/2

LGTM, merging now to free up our test queue (no need to test a docs only change).

Thanks @jcrist

Thanks @jcrist, forgot to add '[skip ci]'


[#3617] Update README in generators folder to point to new locations

Fixes # 3617
Proposed Changes
This PR updates the README file in the generators folder to point to the generators' new location in each of the SDK repositories.
Testing
This image shows a preview of the updated README.

Promoted to MS in PR# 3630


How to get the selected annotation?

extension KiosksViewController: MapViewPlusDelegate {
    func mapView(_ mapView: MapViewPlus, calloutViewFor annotationView: AnnotationViewPlus) -> CalloutViewPlus {
        return EmptyView()
    }
}

As per my need, I don't want to show a CalloutView instead I will show up a View which I will add and display on self.view. So when I tap on any of the annotations, how would I get to know which annotation is tapped so that in my view I can update the label values from the selected annotation.

Let's continue in #4 about this.


Solve a warning when building in Release mode.

When building the iceoryx_hoofs package from the release_2.0 branch in Release mode (-DCMAKE_BUILD_TYPE=Release), with gcc 13, we get warnings like the following:
In file included from /home/ubuntu/rolling_ws/src/eclipse-iceoryx/iceoryx/iceoryx_posh/source/popo/subscriber_options.cpp:17:
In constructor ‘iox::popo::SubscriberOptions::SubscriberOptions(iox::popo::SubscriberOptions&&)’,
    inlined from ‘bool iox::cxx::variant<Types>::emplace_at_index(CTorArguments&& ...) [with long unsigned int TypeIndex = 0; CTorArguments = {iox::popo::SubscriberOptions}; Types = {iox::popo::SubscriberOptions, iox::cxx::Serialization::Error}]’ at /home/ubuntu/rolling_ws/install/iceoryx_hoofs/include/iceoryx/v2.0.6/iceoryx_hoofs/internal/cxx/variant.inl:174:5,
    inlined from ‘constexpr iox::cxx::variant<Types>::variant(const iox::cxx::in_place_index<N>&, CTorArguments&& ...) [with long unsigned int N = 0; CTorArguments = {iox::popo::SubscriberOptions}; Types = {iox::popo::SubscriberOptions, iox::cxx::Serialization::Error}]’ at /home/ubuntu/rolling_ws/install/iceoryx_hoofs/include/iceoryx/v2.0.6/iceoryx_hoofs/internal/cxx/variant.inl:42:24,
    inlined from ‘iox::cxx::expected<ValueType, ErrorType>::expected(iox::cxx::success<ValueType>&&) [with ValueType = iox::popo::SubscriberOptions; ErrorType = iox::cxx::Serialization::Error]’ at /home/ubuntu/rolling_ws/install/iceoryx_hoofs/include/iceoryx/v2.0.6/iceoryx_hoofs/internal/cxx/expected.inl:93:7,
    inlined from ‘static iox::cxx::expected<iox::popo::SubscriberOptions, iox::cxx::Serialization::Error> iox::popo::SubscriberOptions::deserialize(const iox::cxx::Serialization&)’ at /home/ubuntu/rolling_ws/src/eclipse-iceoryx/iceoryx/iceoryx_posh/source/popo/subscriber_options.cpp:56:61:
/home/ubuntu/rolling_ws/src/eclipse-iceoryx/iceoryx/iceoryx_posh/include/iceoryx_posh/popo/subscriber_options.hpp:32:8: warning: ‘queueFullPolicy’ may be used uninitialized [-Wmaybe-uninitialized]
   32 | struct SubscriberOptions
      |        ^~~~~~~~~~~~~~~~~
/home/ubuntu/rolling_ws/src/eclipse-iceoryx/iceoryx/iceoryx_posh/source/popo/subscriber_options.cpp: In static member function ‘static iox::cxx::expected<iox::popo::SubscriberOptions, iox::cxx::Serialization::Error> iox::popo::SubscriberOptions::deserialize(const iox::cxx::Serialization&)’:
/home/ubuntu/rolling_ws/src/eclipse-iceoryx/iceoryx/iceoryx_posh/source/popo/subscriber_options.cpp:40:23: note: ‘queueFullPolicy’ was declared here
   40 |     QueueFullPolicyUT queueFullPolicy;
      |

This change avoids the warning by just always assigning a zero.  However, this isn't really a serious attempt at a fix, but more of a way to start a discussion.  I noticed that the main branch compiles without warnings, but does not have this zero in place.  That said, I do not understand the mechanism that is being used there, so I'm having trouble tracking down the difference between the main branch and this one.  Any thoughts on the "correct" way to fix this?
(skipping the rest of the checklist for now until we decide what to do here)
Notes for Reviewer

Pre-Review Checklist for the PR Author

[ ] Code follows the coding style of CONTRIBUTING.md
[ ] Tests follow the best practice for testing
[ ] Changelog updated in the unreleased section including API breaking changes
[ ] Branch follows the naming format (iox-123-this-is-a-branch)
[ ] Commits messages are according to this guideline

[ ] Commit messages have the issue ID (iox-#123 commit text)
[ ] Commit author matches Eclipse Contributor Agreement (and ECA is signed)

[ ] Update the PR title

Follow the same conventions as for commit messages
Link to the relevant issue

[ ] Relevant issues are linked
[ ] Add sensible notes for the reviewer
[ ] All checks have passed (except task-list-completed)
[ ] Assign PR to reviewer

Checklist for the PR Reviewer

[ ] Consider a second reviewer for complex new features or larger refactorings
[ ] Commits are properly organized and messages are according to the guideline
[ ] Code according to our coding style and naming conventions
[ ] Unit tests have been written for new behavior
[ ] Public API changes are documented via doxygen
[ ] Copyright owner are updated in the changed files
[ ] All touched (C/C++) source code files from iceoryx_hoofs have been added to ./clang-tidy-diff-scans.txt
[ ] PR title describes the changes

Post-review Checklist for the PR Author

[ ] All open points are addressed and tracked via issues

References

Closes TBD

Once again, thanks for fixing the warnings. Could you please add iox-#210 to the commit message ... and push ;)

Once again, thanks for fixing the warnings. Could you please add iox-#210 to the commit message ... and push ;)

Sorry for the delay here.  I've now done that.


Slow to retrieve results with job.result()

What is the expected enhancement?
I am working with circuits of 15 to 25 qubits, with a depth of around 3-4. I want to send the simulations (w/o noise) to the IBMQ simulator_statevector and retrieve the state vector at the end of the circuits (200 different circuits).
While the actual job takes a short period of time to simulate (~2.5 seg) (job = backend_hpc.run(circuits_exact, shots=1)), for retrieve the results (job.result()) takes significantly longer (~15 min).
To solve this issue I can download the results in a .zip from the IBMQ webpage, uncompress it to obtain a .json file with the results and load it in python much faster than it takes with the intended solution of job.result(). I tried to automate this process by downloading the job results using wget https://quantum-computing.ibm.com/jobs/JobID, however, a 401 Unauthorized error occurs with the following text Username/Password Authentication Failed.
The same behavior is reproduced on Debian 11.3, Ubuntu 20.4 (on WSL2), and Windows 10.
I have not seen any other workaround on the internet or in the opened or closed Issues on Github.

Hi @kt474. The original code to reproduce these results is rather long and difficult. It involves computing the mapping for a molecule and grouping some Pauli labels. However, I was able to obtain a similar result with the following code
from qiskit import IBMQ
from qiskit.circuit.random import random_circuit
from qiskit.providers.aer.extensions.snapshot_statevector import *

IBMQ.load_account()
provider = IBMQ.get_provider(hub='ibm-q', group='open', project='main')
simulator = provider.get_backend('simulator_statevector')

n_circuits = 25
n_qubits = 22
depth = 4

circuits = []
for i in range(n_circuits):
    circuit = random_circuit(n_qubits, depth, seed=i, max_operands=1)
    circuit.snapshot_statevector('snapshot')
    circuits.append(circuit)

job = simulator.run(circuits, shots=1)
job.wait_for_final_state()

job.result().to_dict()['results']

Note that before the last line the job has been already sent to ibmq simulator and waited until completed. Measuring the time for the last line of code takes significantly longer than getting the job_id, download the .zip file and uncompress it, and finally load the data. It seems that this behavior is only reproducible for large circuits with lots of qubits.

@Davtax I can replicate your issue, it does take a couple minutes to retrieve the results after the job is finished - this is due to the large size of the results. I assume you have a very powerful computer that allows you to download, unzip and load the results from the web.
I'll look into see if we can add a method that would allow you to download the results directly from the endpoint

Thanks, @kt474. For the moment I was able to overcome the problem by downloading the data with
import webbrowser
import zipfile
import json

webbrowser.open('https://quantum-computing.ibm.com/api/jobs/download?id='+ job_id)
with zipfile.ZipFile(name_file, 'r') as zip_ref:
    zip_ref.extractall(download_path)
data = json.load(open(download_path + job_id + '-output.json'))['results']

However, this solution needs a browser to download the data and it could be challenging to implement it in an HPC server or cluster. It would be nice to have a method to directly download the data.


The Bazel target was removed because it was assumed to only be there for CW340 use, but it was also used by some DVSIM tests.
Re-adding until we work out how DVSIM can use the opentitan_test version.
See https://github.com/lowRISC/opentitan/issues/22733

CI failure is an unrelated i2c test.


In particular, corrected the code formatting in the kubernetes/operator/quickstart-operator.html page.

Thank you for your ongoing contributions 🙏

Thank you for your ongoing contributions 🙏


How to validate name-based and hashed

How can I validate a name-based hashed has the name that we assign it to the uuid?
    // Generate a version 3 (name-based and hashed with MD5) UUID object
    $uuid3 = Uuid::uuid3(Uuid::NAMESPACE_DNS, 'php.net');
    $uuid = $uuid3->toString();

For instance:
if ($uuid3->getName($uuid) === 'php.net') {
// do something
}

Is this possible?

The only way to validate it successfuly (due to the nature of hashes) would be to re-generate a new UUID of the same version from the same namespace and name. If you don;t know the name that was used then you are out of luck as the concept of the md5 hash (version 3) or the sha1 hash (version 5) is that they are irreversible hashes, that is the same data will always produce the same hash, but you cannot determine the data that was used to produce the hash.


-Onone support for access markers, fixes to SIL passes, options and pipeline config.

This enables underlying compiler support for the proposed exclusivity
language feature for testing and evaluation.
Static enforcement:
-Onone: static access markers present through IRGen
-O: all access markers stripped before optimization
-Onone -enforce-exclusivity=unchecked: diagnostics active, no SIL change.
-O -enforce-exclusivity=unchecked: diagnostics active, markers stripped before optimization
Dynamic enforcement:
dynamic markers are only present in SILGen and throughout SIL passes as long as necessary.
i.e. Once a marker is determined to be dynamic, it is stripped if it's inactive.
-Onone -enforce-exclusivity=checked: Runtime diagnostics active. No
changes to the SIL other than the access markers themselves except in
extreme unexpected cases.
-Onone -enforce-exclusivity=checked: Unsupported. Output a
warning. Dynamic markers will be stripped. Runtime diagnostics will
not be active.

@swift-ci test.

Build failed
Jenkins build - Swift Test Linux Platform
Git Commit - b2afe73d68911a097ee62516751e1e7714927b68
Test requested by - @atrick

Build failed
Jenkins build - Swift Test OS X Platform
Git Commit - b2afe73d68911a097ee62516751e1e7714927b68
Test requested by - @atrick

Build failed:
Migrator/wrap_optional.swift:7:8: error: module file was created for incompatible target x86_64-apple-macosx10.9

@swift-ci smoke test and merge.

@swift-ci smoke test linux.


GD-47: gdunit4.test.adapter Timeout when attempting to run tests on addon project using .NET 8.0

The used GdUnit4 version:

The used Godot version:

OS including version:
Windows 10
Describe the bug
When attempting to run or view unit tests on my addon project, I receive the following error:
Failed to negotiate protocol, waiting for response timed out after 90 seconds. This may occur due to machine slowness, please set environment variable VSTEST_CONNECTION_TIMEOUT to increase timeout.
Steps to Reproduce

Create a new Addons Project in Godot 4.2.1
Upgrade your solution to .NET 8.0
Add the following to the csproj:

    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="$(MicrosoftSdkVersion)" />
    <PackageReference Include="gdUnit4.api" Version="4.2.1" />
    <PackageReference Include="gdUnit4.test.adapter" Version="1.0.0" />

Create a unit test.
Build the solution
Error, and no tests appear in the test sln

Minimal reproduction project:
csproj:
<Project Sdk="Godot.NET.Sdk/4.2.1">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <EnableDynamicLoading>true</EnableDynamicLoading>
    <Nullable>enable</Nullable>
  </PropertyGroup>
  <PropertyGroup>
    <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
  </PropertyGroup>
  <ItemGroup>
    <Content Include="icon.png" />
  </ItemGroup>
  <ItemGroup>
    <None Include=".runsettings" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="$(MicrosoftSdkVersion)" />
    <PackageReference Include="gdUnit4.api" Version="4.2.1" />
    <PackageReference Include="gdUnit4.test.adapter" Version="1.0.0" />
  </ItemGroup>
</Project>

.runsettings:
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
    <RunConfiguration>
        <ResultsDirectory>./TestResults</ResultsDirectory>
        <TargetFrameworkVersion>net8.0</TargetFrameworkVersion>
        <TestSessionTimeout>180000</TestSessionTimeout>
        <TreatNoTestsAsError>true</TreatNoTestsAsError>
    </RunConfiguration>

    <LoggerRunSettings>
        <Loggers>
            <Logger friendlyName="console" enabled="True">
                <Configuration>
                    <Verbosity>detailed</Verbosity>
                </Configuration>
            </Logger>
            <Logger friendlyName="html" enabled="True">
                <Configuration>
                    <LogFileName>test-result.html</LogFileName>
                </Configuration>
            </Logger>
            <Logger friendlyName="trx" enabled="True">
                <Configuration>
                    <LogFileName>test-result.trx</LogFileName>
                </Configuration>
            </Logger>
        </Loggers>
    </LoggerRunSettings>

    <GdUnit4>
        <Parameters></Parameters>
        <DisplayName>FullyQualifiedName</DisplayName>
    </GdUnit4>
</RunSettings>

FileStructure:

HelloWorld.cs:
using GdUnit4;
using static GdUnit4.Assertions;

namespace GodotVoipNet.Tests;

[TestSuite]
public class HelloWorld
{
    [TestCase]
    public void Test()
    {
        AssertBool(true).IsTrue();
    }
}

To my knowledge, I have followed the steps required to correctly install the api and the test explorer. Additionally, I have taken the .runtime file, and set my csproj according to the example. If I missed something here I would be happy to try it, but currently my tests are unable to run.

Thanks for reporting this issue.
Please attach the 'C# Dev Kit Test Explorer' logs you can find on Output.

The API and test adapter are build in .NET 7 maybe this could be the issue.
Can you please add your project to reproduce please.

Thanks for reporting this issue. Please attach the 'C# Dev Kit Test Explorer' logs you can find on Output.
Have you set the MicrosoftSdkVersion if not you need to specify a version e.g. <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.9.0" />
Which Godot Version you have used? The API and test adapter are build in .NET 7 maybe this could be the issue. Can you please add your project to reproduce please.

https://github.com/Pieeer1/GodotVoipNet/tree/upgrade-version <-- Here is a branch with the tests.
Godot 4.2.1
Eh i would doubt the .NET 7 -> 8 transition would cause issues there, I have had little to no compatibility issues in the past but maybe.

just add the logs please, i need more details

The preconditions are specifically set in the readme for vscode development. Is there a method of configuring it for Visual Studio?

The preconditions are specifically set in the readme for vscode development. Is there a method of configuring it for Visual Studio?

I have no Visual Studio installed, I suggest asking @GeorgeS2019 he was able to run tests inside Visual Studio. You can reach out on my discord server.
I'm also interested in how to setup Visual Studio to finally document it.

The preconditions are specifically set in the readme for vscode development. Is there a method of configuring it for Visual Studio?

I have no Visual Studio installed, I suggest asking @GeorgeS2019 he was able to run tests inside Visual Studio. You can reach out on my discord server. I'm also interested in how to setup Visual Studio to finally document it.

I can reach out on discord if you can DM me a link, once we get it figured out I can document that process here.

Open the exampleProject.csproj

Thanks for the quick response,
I may be misunderstanding the environment variable value to be set, after adding the path in my env vars I can get it to appear in the test explorer, but running the tests from that window has another error:
An exception occurred while invoking executor 'executor://gdunit4.testadapter/': Godot runtime is not set! Set env 'GODOT_BIN' is missing!

I attempted to do so with just the name of the exe, as well as the raw path.

@Pieeer1
Have you seen this. I ran your project and it works!
I simply clone the branch
I already define GODOT_BIN
I build the project and the test appear.

Ok I got it to work. For anyone looking to do this in the future, adding GODOT_BIN to the environment variables of the .runsettings file did the trick.

Nice, Good to hear that you got it up and running. I will add the info to the documentation.
Close it as clarified


Unable run demo on remote machine without desktop environment

Hi @jiupinjia, good job! the result of your repo seems fantastic!
I try to run your demo on my remote GPU server which doesn't have a desktop environment(no x11 installed). and get the following error
initialize network with normal
loading renderer from pre-trained checkpoint...
begin to draw...
iteration step 0, G_loss: 0.00000, step_psnr: 3.38946, strokes: 25 / 500
qt.qpa.xcb: could not connect to display
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/home/shir/miniconda3/envs/tf1.15/lib/python3.6/site-packages/cv2/qt/plugins" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: xcb.

Aborted (core dumped)

Is there any way to solve this problem?

Hi, thanks for your valuable feedback! Since you are running on a remote machine, I believe the problem is in the image display during the runtime. Can you try "ctrl + f" and remove all the "cv2.imshow" code lines in the painter.py? Please tell me whether it works fine. Thank you!

Thanks for your fast response.
I comment all lines contain cv2.imshow and cv2.waitkey in painter.py, now I can run normally and get the correct output. it might be a good idea to comment these functions  by default since most people doing research or development on the remote server, Already created a PR to your repo, you may merge to the main branch if it's helpful.

Maybe it's better to add a parameter show_output which, when set to true, displays the images, instead of just switching off image display functionality?

Yes, I will be adding a parameter in the config file to control the displays. Thank Stocastico for the suggestion.

Yes, I will be adding a parameter in the config file to control the displays. Thank Stocastico for the suggestion.

Sorry, no config file here... I mixed up this project with another one. This parameter will be added to the arg parser.

Hi @zerollzeng and @Stocastico, the above feature has been added. Please check "Running through SSH" in the README file for details. I'm closing this issue. Please feel free the reach out to me if you have further questions. Thanks again for your valuable feedback.

Thank you for the quick fix!
I will check and let you know if I spot any issues :-)


new(sinsp): rework sinsp-example output

This PR all but drops the custom output code we have in sinsp-example and switches to using the same formatters as JSON output. This makes sinsp-example output actually useful to look at events (making it a usable replacement for sysdig in some scenarios)
It turns out our category->name mapping was incomplete (which we didn't really notice if we used our own in sinsp-example) so I updated that too.
I also added a raw output mode to almost literally dump the raw events in hex.

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area API-version

/area build

/area CI

/area driver-kmod

/area driver-bpf

/area driver-modern-bpf

/area libscap-engine-bpf

/area libscap-engine-gvisor

/area libscap-engine-kmod

/area libscap-engine-modern-bpf

/area libscap-engine-nodriver

/area libscap-engine-noop

/area libscap-engine-source-plugin

/area libscap-engine-savefile

/area libscap-engine-udig

/area libscap

/area libpman

/area libsinsp

/area tests

/area proposals

Does this PR require a change in the driver versions?

/version driver-API-version-major

/version driver-API-version-minor

/version driver-API-version-patch

/version driver-SCHEMA-version-major

/version driver-SCHEMA-version-minor

/version driver-SCHEMA-version-patch

Special notes for your reviewer:
Does this PR introduce a user-facing change?:

feat(sinsp)!: sinsp-example output is now different

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: gnosek
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [gnosek]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

I just pushed an unrelated commit to this PR to try to make the e2e failures more readable. Let's see how it goes

Hey all! Sorry I haven't been able to come around to this, I've been swamped with work and I'll be OOO next week.
We basically need to ensure the new output isn't missing any fields the tests are looking for, you can do this by checking the dictionaries defining events on the failing tests and using the logs provided in the summary.
Promise I'll try to come up with a better way to test this, we can keep the containerized version as an option, but is probably better if we can directly run the tests with the binaries and drivers compiled directly, (maybe pointing out how to use a virtual environment so we don't need to install Python dependencies system wide).

Could we revert those sinsp.py changes

Sure.

and only make the adjustments needed for the tests to pass, (read more below)?
From my perspective the only changes needed to e2e would be explicitly adding the -j flag, e.g. here
libs/test/e2e/tests/test_process/test_container.py

OMG thanks for spotting this! Just pushed an updated version, fingers crossed

Hi @gnosek and @incertum,
I finally found some time to test these changes locally and find why they are failing, sorry again it took so long. I've pushed the changes needed from my point of view in this commit, feel free to cherry-pick it and/or modify it as needed. I also run the e2e tests in this PR so we can double check that they are passing.
Feel free to ask any questions on why those changes are needed, I'll do my best to answer them 🙂

and only make the adjustments needed for the tests to pass, (read more below)?
From my perspective the only changes needed to e2e would be explicitly adding the -j flag, e.g. here
libs/test/e2e/tests/test_process/test_container.py

OMG thanks for spotting this! Just pushed an updated version, fingers crossed

I only now saw this, so for further visibility, the -j flag isn't needed because the sinsp-example container itself already runs with -j and -a, the flags passed from the tests are effectively added after those 2, see here:
https://github.com/falcosecurity/libs/blob/533739880f5fd183c02180cbd6175baffc4354ee/test/e2e/containers/sinsp.Dockerfile#L21

Checking in on status, this could be rebased and wrapped up.

Agree with Melissa. @gnosek ping :)

Eh, no good deed goes unpunished :D
@Molter73, do you know why you had to remove some of the checks? I mean this chunk: https://github.com/Molter73/falco-libs/pull/3/commits/1dfb6fe7da0e3a4e348118be9d0a6ecf37a18f95#diff-1f7d09b2e04a396a2d7e13d8f3f520032f8b5c54243025238437b71babc1cc0dL66 which looks like a valuable check

/milestone 0.12.0

Hey @gnosek could you rebase this, pls?

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale

are we still interested in this? @gnosek

I would vote for yes, I like the changes a lot and we use sinsp-example quite a lot for dev and testing. @gnosek? Would you be up to rebase or should we help?

no rush at all since we will probably merge this in the next libs tag :) it was just to understand what are the intention here :)

I've been swamped with other things (and seem to be for the foreseeable future) but I'll try to refresh this.

/remove-lifecycle stale

Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.
:memo: Please follow instructions in the contributing guide to update your commits with the DCO
Full details of the Developer Certificate of Origin can be found at developercertificate.org.
The list of commits missing DCO signoff:

d7469b0 hack: do not hardcode build dir path

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

This LGTM! @Molter73 can you PTAL? Thank you!
Also, this contains https://github.com/falcosecurity/libs/pull/1333, right?

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: gnosek, incertum
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [gnosek,incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LGTM label has been added.  Git tree hash: 11bca3c96a40c9222c52a922767e828f307c151d

/unhold


Adds multiple languages check

In Chrome (not sure about FF), navigator has both language and languages fields, so check for languages and use the first if it exists.

LGTM!


Automatically generated doc is not correct

In some points, there is some documentation generated automatically by some macro, that is not perfectly correct. A human should be able to see the meaning, but it can generate confusion at a first reading.
See for example https://nalgebra.org/rustdoc/nalgebra/base/struct.Matrix.html#method.sub_to. The doc says that the operation is equal to self + rhs while in fact it should perform self - rhs.

That's right, the generated documentation is misleading. The macro generating this code here should take an additional parameter for the documentation.


Accesss marker deserialization.


Add healthcheck route

Pull Request Test Coverage Report for Build 477234555

-1 of 15 (93.33%)  changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 93.151%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/server.js
4
5
80.0%

Totals

Change from base Build 475859515:
0.2%

Covered Lines:
204

Relevant Lines:
215

💛  - Coveralls

Pull Request Test Coverage Report for Build 477234555

-1 of 15 (93.33%)  changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 93.151%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/server.js
4
5
80.0%

Totals

Change from base Build 475859515:
0.2%

Covered Lines:
204

Relevant Lines:
215

💛  - Coveralls

Pull Request Test Coverage Report for Build 477234555

14 of 15 (93.33%)  changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 93.151%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/server.js
4
5
80.0%

Totals

Change from base Build 475859515:
0.2%

Covered Lines:
204

Relevant Lines:
215

💛  - Coveralls

Pull Request Test Coverage Report for Build 477234555

14 of 15 (93.33%)  changed or added relevant lines in 4 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.2%) to 93.151%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/server.js
4
5
80.0%

Totals

Change from base Build 475859515:
0.2%

Covered Lines:
204

Relevant Lines:
215

💛  - Coveralls


Change 'registries' to 'container registries' in man

Signed-off-by: TomSweeneyRedHat<EMAIL_ADDRESS>Per a suggestion form @pixdrift, added 'container' before registries in some man pages.

@rh-atomic-bot r+


consider changing size of empty enum to usize::MAX

spawned off of https://github.com/rust-lang/rust/issues/24590
Empty enums such as:
enum Empty { }

are a hazard in a number of ways; in particular, odd people like myself keep wanting to investigate what happens if you unsafely inject a value into Empty, e.g. via mem::transmute (or by derefing an unsafe pointer *const Empty, etc).
The resulting code you get when doing a match on such a value is pretty much guaranteed to be undefined behavior.  The project developers have stated that the memory layout of an empty enum (in fact, of all non C-like enums) is undefined, and thus undefined behavior is justifiable here.
Still, it seems to easy to expose this case, and it seems like it would be really easy to prevent it, e.g. by changing the representation size of an empty enum to something semi-absurd.  Like usize::MAX.

(another option that may be more palatable for various reasons would be to give empty enums zero size rather than usize::MAX size...)

As argued (not by me) in this thread the logically sensible size for uninhabited types would actually be negative infinity, because this makes the sizes of product and sum types with uninhabited types as members work out properly. sizeof Pair<A, B> = sizeof(A) + sizeof(B);1 then sizeof Pair<Foo, Uninhabited> is negative infinity as well, because a struct with an uninhabited type as a member is also uninhabited. And sizeof Either<A, B> = max(sizeof A, sizeof B);1 then sizeof Either<Foo, Uninhabited> = sizeof<Foo>, because an enum with an uninhabited type as one of the variants is equivalent to an enum without that variant.
Of course, this doesn't help much in practice because the type of sizes of types is usize which is not inhabited by negative infinity. (We'd probably want Option<usize> or something isomorphic if we wanted to be ale to represent things that way.) So I dunno.
Might this also be related to DST, e.g. the "truly unsized types" stuff?
1 Roughly, of course there's padding, the tag field, etc.

It's not enough for it to have a huge/zero/negative infinity size, because even a reference to an uninhabited type should itself be an uninhabited type (since references are guaranteed to be valid). It could be implemented using a more general version of DSTs/truly unsized types, where types get to freely choose the representation of references to themselves (there could be a built-in uninhabited type which all uninhabited types use as the representation for their references).
More realistically, shouldn't the compiler detect that code which actually uses uninhabited types, either directly or through a reference, is invalid? (similar, but not quite the same as how you can't use incomplete types in C++)

odd people like myself keep wanting to investigate what happens if you unsafely inject a value into Empty, e.g. via mem::transmute (or by derefing an unsafe pointer *const Empty, etc)

Given that uninhabited types currently have a size of 0, how can you inject a value into it? The only legal mem::transmutes should be transmutations of zero-sized values (such as ()), which doesn't seem like a problem. Coercing a pointer to some other value to a pointer to an uninhabited type should be a safe operation. The only thing that I think should change is that it should be illegal to dereference a pointer to an uninhabited type. That would rule out your match *x {} (from rust-lang/rust#24590) and thus avoid the undefined behavior.

I have been looking into using an uninhabited type in some changes to rust-postgres to avoid hardcoding support for specific SSL backends: https://github.com/sfackler/rust-postgres/commit/49c09c367a2ce00ab6a4e4f21b2975c1f8e6e18e#diff-9a0fb54f94cd0bef81a2ada2e9fb0c57R46. It involves having a type that stores the uninhabited type by value, so setting its size to usize::MAX would probably be bad news.
The idea is that the default for the N type parameter on SslMode means you can pass &mut SslMode::None directly to the functions that expect one without needing a "real" SSL implementation around. This doesn't actually work right now since default parameters don't seem to be sufficiently respected unfortunately. Note the use of match *x {} to create a dummy implementation of a trait here: https://github.com/sfackler/rust-postgres/commit/49c09c367a2ce00ab6a4e4f21b2975c1f8e6e18e#diff-9a0fb54f94cd0bef81a2ada2e9fb0c57R60


postgresql server isn't starting

Summary
Tried to follow this tutorial from @hbagdi but it looks like the postgres is not working.
I'm in a local kubernetes cluster provided by docker desktop on windows 10.
What happened

And this is the log msg:

Welcome to the Bitnami postgresql container
Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-postgresql
Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-postgresql/issues

nami    INFO  Initializing postgresql
postgre INFO  ==> No injected postgresql.conf file found. Creating default postgresql.conf file...
postgre INFO  ==> No injected pg_hba.conf file found. Creating default pg_hba.conf file...
postgre INFO  ==> Deploying PostgreSQL with persisted data...
postgre INFO  ==> Configuring PostgreSQL...
postgre INFO  ==> Configuring replication parameters...
postgre INFO  ==> Configuring permissions for config files...
postgre INFO
postgre INFO  ########################################################################
postgre INFO   Installation parameters for postgresql:
postgre INFO     Persisted data and properties have been restored.
postgre INFO     Any input specified will not take effect.
postgre INFO   This installation requires no credentials.
postgre INFO  ########################################################################
postgre INFO
nami    INFO  postgresql successfully initialized
INFO  ==> Starting postgresql...
nami    ERROR Unable to start com.bitnami.postgresql: pg_ctl: could not start server
Examine the log output.

This doesn't look specific to the Kong helm chart but an issue running the postgresql helm chart with your environment / setup.
Can you try and run the postgresql helm chart on it's own per https://github.com/helm/charts/tree/master/stable/postgresql and see if you can reproduce the issue there?

As you said, this was merely a setup issue with the postgresql helm chart, but it has already been fixed. Thanks!
That tutorial is great, but it can't cover all the details, so novice people may have different issues.


Fixed blur for Safari (Issue 9)

Resolving issue with the blurring for Mac Safari.
Safari (Version 17.0 (196<IP_ADDRESS>.1))

Chrome (Version 120.0.6099.129):

Firefox (121.0.1 (64-bit):

Formatting is all a mess and I didn't properly branch. Will re-commit.


fix: id is too long, must be no longer than 512 bytes

Fix https://github.com/apache/skywalking/issues/8878 <ElasticSearch doc id is too long, must be no longer than 512 bytes>

[x] Update the CHANGES log.

index:sw89_metrics-percent-20220607, docId:endpoint_relation_sla_202206071458_VXNlcg==.0.581-VXNlcg==-cHJvZC1zZWN1cml0eS1jZW50ZXItd2Vi.1.581-e0dFVH0vYXBpL2F1dGh6L2F1dGh6L2xpc3Qve3Jlc291cmNlVHlwZX0=, exception:java.lang.RuntimeException: {"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 514;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: id is too long, must be no longer than 512 bytes but was: 514;"},"status":400}, cost:150ms

By the way, how to become skywaling commiter, I want to contribute my part to skywalking！

I modified NamingControl: serviceNameMaxLength, instanceNameMaxLength, endpointNameMaxLength, the problem has been solved

So, should we close this?

So, should we close this?

ok,plus the model name should be just over a little bit, you can adjust the default value of endpointNameMaxLength to 100~120

The metric name could be shorter. We may consider a rule for metric name to avoid this case.

@chenhaipeng If you want to talk about committer, ping me on slack, we could talk more directly.

@chenhaipeng If you want to talk about committer, ping me on slack, we could talk more directly.

I'm not familiar with slack, please invite me to the group，my email<EMAIL_ADDRESS>

@chenhaipeng @wu-sheng I encountered the same issue on skywalking v9.3.0, and I tried to modify NamingControl as following, but no luck. Could you please suggest how to solve this issue? Thanks
serviceNameMaxLength: 50
instanceNameMaxLength: 50
endpointNameMaxLength: 100

I'm having the same issue.  Closing this is frustrating because this is an issue!!

@wu-sheng

oap version: 8.9.1
Same problem

serviceNameMaxLength

https://github.com/apache/skywalking/blob/master/oap-server/server-starter/src/main/resources/application.yml
serviceNameMaxLength: ${SW_SERVICE_NAME_MAX_LENGTH:70}
instanceNameMaxLength: ${SW_INSTANCE_NAME_MAX_LENGTH:70}
# The max length of service + endpoint names should be less than 240
endpointNameMaxLength: ${SW_ENDPOINT_NAME_MAX_LENGTH:150}

oap version: 8.9.1 Same problem
Caused by: java.lang.RuntimeException: {"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: id [endpoint_relation_cpm_202401121826_c2hhcmUuZWNzZWxlY3Rpb246OnBoLXNlbGVjdGlvbi11cGRhdGUtc2VydmljZQ==.1-UmFiYml0TVEvVG9waWMvcGhocy5zZWxlY3Rpb24uZGVsaXZlcnkudXBkYXRlLnN0b3JlLmRlbGF5LmV4Y2hhbmdlUXVldWUvcGhocy5zZWxlY3Rpb24uZGVsaXZlcnkudXBkYXRlLnN0b3JlLmRlbGF5LnJvdXRpbmcua2V5L1Byb2R1Y2Vy-c2hhcmUuZWNzZWxlY3Rpb246OnBoLXNlbGVjdGlvbi11cGRhdGUtc2VydmljZQ==.1-UmFiYml0TVEvVG9waWMvcGhocy5zZWxlY3Rpb24uZGVsaXZlcnkudXBkYXRlLnN0b3JlLmV4Y2hhbmdlUXVldWUvcGhocy5zZWxlY3Rpb24uZGVsaXZlcnkudXBkYXRlLnN0b3JlLnJvdXRpbmcua2V5L0NvbnN1bWVy] is too long, must be no longer than 512 bytes but was: 514

see here:
https://github.com/apache/skywalking/blob/master/oap-server/server-starter/src/main/resources/application.yml
serviceNameMaxLength: ${SW_SERVICE_NAME_MAX_LENGTH:70}
instanceNameMaxLength: ${SW_INSTANCE_NAME_MAX_LENGTH:70}
The max length of service + endpoint names should be less than 240
endpointNameMaxLength: ${SW_ENDPOINT_NAME_MAX_LENGTH:150}


Blas SGEMM launch failed

when I run "python yolo_video.py --image", it loads model successfully. But then I input the name of test picture, error occurs as follows:

Input image filename:1.jpg
(416, 416, 3)
2019-03-07 14:23:30.749652: E tensorflow/stream_executor/cuda/cuda_blas.cc:647] failed to run cuBLAS routine cublasSgemm_v2: CUBLAS_STATUS_EXECUTION_FAILED
Traceback (most recent call last):
File "/home/jmh/.conda/envs/tfgpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/jmh/.conda/envs/tfgpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/jmh/.conda/envs/tfgpu/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=43264, n=32, k=64
[[Node: conv2d_3/convolution = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_3/cond/FusedBatchNorm/Switch"], data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](leaky_re_lu_2/LeakyRelu, conv2d_3/kernel/read)]]
[[Node: boolean_mask_29/GatherV2/_3363 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_6054_boolean_mask_29/GatherV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

My GPU is RTX2080Ti 8G*2, tensorflow-gpu 1.9.0, keras 2.2.4.
I feel so annoyed by this problem, who can tell me what happend and how to solve it?

The same problem as you, can you tell me what happend and how to solve it?

I have met it many times...how to solve it?

I noticed some bugs in the original cuda runfile. I fixed it by installing the four patches of CUDA in https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal. It does work and hope it works for you.

I noticed some bugs in the original cuda runfile. I fixed it by installing the four patches of CUDA in https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal. It does work and hope it works for you.

Thank you so much. A few days ago, I run the some code with two 1080Ti GPUs, it works. So I guess there might be some problems in CUDA support in 2080Ti GPU as you said :)

The same problem as you, can you tell me what happend and how to solve it?

The comments above may help you~

you are right! Thanks

Thank u so much!

With those patches installed, my RTX2070 on win10 is still having the same issue.

With those patches installed, my RTX2070 on win10 is still having the same issue.

I have the same GPU RTX2070 and having the same problem in Ubunto 16.04!
Cuda 9 + 4 extra patches
Tensorflow 1.8

I noticed some bugs in the original cuda runfile. I fixed it by installing the four patches of CUDA in https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal. It does work and hope it works for you.

I noticed some bugs in the original cuda runfile. I fixed it by installing the four patches of CUDA in https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal. It does work and hope it works for you.

你好，安装四个补丁的时候执行完run sudo sh cuda_9.0.176_384.81_linux.run
为什么最后报错Run: command not found

I noticed some bugs in the original cuda runfile. I fixed it by installing the four patches of CUDA in https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal. It does work and hope it works for you.
I noticed some bugs in the original cuda runfile. I fixed it by installing the four patches of CUDA in https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal. It does work and hope it works for you.
你好，安装四个补丁的时候执行完run sudo sh cuda_9.0.176_384.81_linux.run
为什么最后报错Run: command not found

直接sudo sh cuda_9.0.176_384.81_linux.run会报错，run不在的

你好，安装四个补丁的时候执行完run sudo sh cuda_9.0.176_384.81_linux.run
为什么最后报错Run: command not found
直接sudo sh cuda_9.0.176_384.81_linux.run会报错？run不需要的

你好，安装四个补丁的时候执行完run sudo sh cuda_9.0.176_384.81_linux.run
为什么最后报错Run: command not found

直接sudo sh cuda_9.0.176_384.81_linux.run会报错？run不需要的

I noticed some bugs in the original cuda runfile. I fixed it by installing the four patches of CUDA in https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal. It does work and hope it works for you.

Thank you so much. A few days ago, I run the some code with two 1080Ti GPUs, it works. So I guess there might be some problems in CUDA support in 2080Ti GPU as you said :)
hi，when i execute the command：Run sudo sh cuda_9.0.176_384.81_linux.run,it shows an error

你好，安装四个补丁的时候执行完run sudo sh cuda_9.0.176_384.81_linux.run
为什么最后报错Run: command not found
直接sudo sh cuda_9.0.176_384.81_linux.run会报错？run不需要的

安装cuda的时候应该补丁一起装上了，为啥还要再装这个。。。

安装cuda的时候应该补丁一起装上了，为啥还要再装这个。。。

The base installer is available for download below.
There are 4 patches available. These patches require the base installer to be installed first.
CUDA下载界面的提示，这是额外的，可能是为了兼容最新出的显卡开发的

安装cuda的时候应该补丁一起装上了，为啥还要再装这个。。。

The base installer is available for download below.
There are 4 patches available. These patches require the base installer to be installed first.
CUDA下载界面的提示，这是额外的，可能是为了兼容最新出的显卡开发的

怎么知道这四个补丁是不是安装成功呢，有没有测试命令

怎么知道这四个补丁是不是安装成功呢，有没有测试命令

安装过程显示的和cuda安装时一样的，一般也不会报啥错的，再跑一下原来的程序看会不会继续报错嘛

怎么知道这四个补丁是不是安装成功呢，有没有测试命令

安装过程显示的和cuda安装时一样的，一般也不会报啥错的，再跑一下原来的程序看会不会继续报错嘛
Input image filename:dog.jpg
(416, 416, 3)
2019-07-12 11:28:11.879346: E tensorflow/stream_executor/cuda/cuda_blas.cc:462] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-07-12 11:28:11.879368: W tensorflow/stream_executor/stream.cc:2001] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=43264, n=32, k=64
[[Node: conv2d_3/convolution = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_3/cond/FusedBatchNorm/Switch"], data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](leaky_re_lu_2/LeakyRelu/Maximum, conv2d_3/kernel/read)]]
[[Node: boolean_mask_122/GatherV2/_3305 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4456_boolean_mask_122/GatherV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "yolo_video.py", line 75, in
detect_img(YOLO(**vars(FLAGS)))
File "yolo_video.py", line 16, in detect_img
r_image = yolo.detect_image(image)
File "/home/qian/Downloads/YOLO3/keras-yolo3-master/yolo.py", line 124, in detect_image
K.learning_phase(): 0
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=43264, n=32, k=64
[[Node: conv2d_3/convolution = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_3/cond/FusedBatchNorm/Switch"], data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](leaky_re_lu_2/LeakyRelu/Maximum, conv2d_3/kernel/read)]]
[[Node: boolean_mask_122/GatherV2/_3305 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4456_boolean_mask_122/GatherV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Caused by op 'conv2d_3/convolution', defined at:
File "yolo_video.py", line 75, in
detect_img(YOLO(**vars(FLAGS)))
File "/home/qian/Downloads/YOLO3/keras-yolo3-master/yolo.py", line 45, in init
self.boxes, self.scores, self.classes = self.generate()
File "/home/qian/Downloads/YOLO3/keras-yolo3-master/yolo.py", line 70, in generate
self.yolo_model = load_model(model_path, compile=False)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 419, in load_model
model = _deserialize_model(f, custom_objects, compile)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 225, in _deserialize_model
model = model_from_config(model_config, custom_objects=custom_objects)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 458, in model_from_config
return deserialize(config, custom_objects=custom_objects)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/layers/init.py", line 55, in deserialize
printable_module_name='layer')
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
list(custom_objects.items())))
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/network.py", line 1032, in from_config
process_node(layer, node_data)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/network.py", line 991, in process_node
layer(unpack_singleton(input_tensors), **kwargs)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py", line 457, in call
output = self.call(inputs, **kwargs)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/layers/convolutional.py", line 171, in call
dilation_rate=self.dilation_rate)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 3650, in conv2d
data_format=tf_data_format)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 780, in convolution
return op(input, filter)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 868, in call
return self.conv_op(inp, filter)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 520, in call
return self.call(inp, filter)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 204, in call
name=self.name)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 956, in conv2d
data_format=data_format, dilations=dilations, name=name)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
op_def=op_def)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1718, in init
self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access
InternalError (see above for traceback): Blas SGEMM launch failed : m=43264, n=32, k=64
[[Node: conv2d_3/convolution = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_3/cond/FusedBatchNorm/Switch"], data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](leaky_re_lu_2/LeakyRelu/Maximum, conv2d_3/kernel/read)]]
[[Node: boolean_mask_122/GatherV2/_3305 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4456_boolean_mask_122/GatherV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
安装完补丁之后还是这个错，知道是啥错吗？装了一遍补丁貌似没用，搞一天啦，这啥问题呢

可以加个qq吗，973616491，谢谢~

怎么知道这四个补丁是不是安装成功呢，有没有测试命令

安装过程显示的和cuda安装时一样的，一般也不会报啥错的，再跑一下原来的程序看会不会继续报错嘛
Input image filename:dog.jpg
(416, 416, 3)
2019-07-12 11:28:11.879346: E tensorflow/stream_executor/cuda/cuda_blas.cc:462] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2019-07-12 11:28:11.879368: W tensorflow/stream_executor/stream.cc:2001] attempting to perform BLAS operation using StreamExecutor without BLAS support
Traceback (most recent call last):
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=43264, n=32, k=64
[[Node: conv2d_3/convolution = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_3/cond/FusedBatchNorm/Switch"], data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](leaky_re_lu_2/LeakyRelu/Maximum, conv2d_3/kernel/read)]]
[[Node: boolean_mask_122/GatherV2/_3305 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4456_boolean_mask_122/GatherV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "yolo_video.py", line 75, in
detect_img(YOLO(**vars(FLAGS)))
File "yolo_video.py", line 16, in detect_img
r_image = yolo.detect_image(image)
File "/home/qian/Downloads/YOLO3/keras-yolo3-master/yolo.py", line 124, in detect_image
K.learning_phase(): 0
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Blas SGEMM launch failed : m=43264, n=32, k=64
[[Node: conv2d_3/convolution = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_3/cond/FusedBatchNorm/Switch"], data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](leaky_re_lu_2/LeakyRelu/Maximum, conv2d_3/kernel/read)]]
[[Node: boolean_mask_122/GatherV2/_3305 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4456_boolean_mask_122/GatherV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Caused by op 'conv2d_3/convolution', defined at:
File "yolo_video.py", line 75, in
detect_img(YOLO(**vars(FLAGS)))
File "/home/qian/Downloads/YOLO3/keras-yolo3-master/yolo.py", line 45, in init
self.boxes, self.scores, self.classes = self.generate()
File "/home/qian/Downloads/YOLO3/keras-yolo3-master/yolo.py", line 70, in generate
self.yolo_model = load_model(model_path, compile=False)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 419, in load_model
model = _deserialize_model(f, custom_objects, compile)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 225, in _deserialize_model
model = model_from_config(model_config, custom_objects=custom_objects)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 458, in model_from_config
return deserialize(config, custom_objects=custom_objects)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/layers/init.py", line 55, in deserialize
printable_module_name='layer')
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
list(custom_objects.items())))
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/network.py", line 1032, in from_config
process_node(layer, node_data)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/network.py", line 991, in process_node
layer(unpack_singleton(input_tensors), **kwargs)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py", line 457, in call
output = self.call(inputs, **kwargs)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/layers/convolutional.py", line 171, in call
dilation_rate=self.dilation_rate)
File "/home/qian/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 3650, in conv2d
data_format=tf_data_format)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 780, in convolution
return op(input, filter)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 868, in call
return self.conv_op(inp, filter)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 520, in call
return self.call(inp, filter)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/nn_ops.py", line 204, in call
name=self.name)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 956, in conv2d
data_format=data_format, dilations=dilations, name=name)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
op_def=op_def)
File "/home/qian/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1718, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
InternalError (see above for traceback): Blas SGEMM launch failed : m=43264, n=32, k=64
[[Node: conv2d_3/convolution = Conv2D[T=DT_FLOAT, _class=["loc:@batch_normalization_3/cond/FusedBatchNorm/Switch"], data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](leaky_re_lu_2/LeakyRelu/Maximum, conv2d_3/kernel/read)]]
[[Node: boolean_mask_122/GatherV2/_3305 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_4456_boolean_mask_122/GatherV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
安装完补丁之后还是这个错，知道是啥错吗？装了一遍补丁貌似没用，搞一天啦，这啥问题呢

有解决吗 ？跟你一样的问题  docker环境TensorFlow1.8.0 cuda9.0 keras2.2.4

I used cude9.0 with tf1.8 at first, and then I fixed this problem by changing it to cuda9.2 .
If you use anaconda, just type in conda install cudatoolkit=9.2
Hope this would help you.

I noticed some bugs in the original cuda runfile. I fixed it by installing the four patches of CUDA in https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal. It does work and hope it works for you.

It does work for me, thanks a lot

Input image filename:images/img00030.jpg
(416, 416, 3)
Traceback (most recent call last):
File "yolo_video.py", line 73, in
detect_img(YOLO(**vars(FLAGS)))
File "yolo_video.py", line 15, in detect_img
r_image = yolo.detect_image(image)
File "E:\YOLOV3\K-yolo3\yolo.py", line 125, in detect_image
self.input_image_shape: [image.size[1], image.size[0]],
File "D:\anaconda3\lib\site-packages\tensorflow_core\python\framework\ops.py", line 705, in hash
raise TypeError("Tensor is unhashable if Tensor equality is enabled. "
TypeError: Tensor is unhashable if Tensor equality is enabled. Instead, use tensor.experimental_ref() as the key.
help me

Try reinstalling TensorFlow to 1.6.0 version mentioned in README.md?
pip install tensorflow==1.6.0

I noticed some bugs in the original cuda runfile. I fixed it by installing the four patches of CUDA in https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal. It does work and hope it works for you.

thank you so much

I noticed some bugs in the original cuda runfile. I fixed it by installing the four patches of CUDA in https://developer.nvidia.com/cuda-90-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=runfilelocal. It does work and hope it works for you.

thank you so much

I've tried tensorflow==1.12.0 (cpu version). It's ok.


Add xyz data support

Add xyz data support to https://github.com/OpenBioML/chemnlp/blob/text_sampling/data/text_sampling/extend_tabular.py

duplicated in #393


Support for Firebase Cloud Messaging

Firebase Cloud Messaging (FCM) is the new version of GCM
It inherits the reliable GCM infrastructure.
What changes needs to be done to allow phonegap-plugin-push plugin to support FCM.

@devshisiyani please look at #929


Airpower rework

Closes #74

[ ] Document exported command
[ ] Add markdown documentation of CLI with examples
[x] Replace use of intermediate tar file with gzip custom seek
[x] Extraction progress async
[x] Add more tests
[x] Add coverage to workflow
[x] Rework db to use the filesystem
[x] Enable exec command to use local packages for testing/shipyard
[x] Perform an HTTP HEAD on */manifests/* endpoints before pulling to better avoid hitting the docker api request limit.

A couple things I've noticed using this version (not sure any of these are blockers for #74, but maybe issues after v0.6 is available:

Install seems a lot slower than it used to be
Noticed that after downloading, the tar.gz is decompressed into another temp file, which could result in excessive disk usage with larger packages (e.g., vs-buildtools would use ~4GB for tar.gz, ~15GB for .tar, ~15GB for content)

A couple things I've noticed using this version (not sure any of these are blockers for #74, but maybe issues after v0.6 is available:

Install seems a lot slower than it used to be
Noticed that after downloading, the tar.gz is decompressed into another temp file, which could result in excessive disk usage with larger packages (e.g., vs-buildtools would use ~4GB for tar.gz, ~15GB for .tar, ~15GB for content)

.NET added a TAR class in version 7, but since we're trying to be 5.1 compatible without calling system tar, we're left with implementing it directly in PowerShell. We can remove the intermediate tar file by using the gzip stream directly; however, I noticed that interrupting the extraction corrupts the gzip base stream, so if we use the tgz directly, it will result in a redownload. Also, an asynchronous untar will need to have random-ish access to the underlying tar bytes, otherwise we're left decompressing our way to the right position. Alternatively, we can use an internal memory buffer to avoid multiple passes and spawn multiple threads to extract files in small batches.
I'm weighing the options. It is slow, and I agree it needs to be faster.

Seeing some strange issues on AFNet. Install works fine, but when I try to pull I get strange results:
PS C:\Users\1379838568C> Airpower version

Major  Minor  Build  Revision
-----  -----  -----  --------
0      6      0      9

PS C:\Users\1379838568C> Airpower pull vs-buildtools:17.4.3
17.4.3: Pulling vs-buildtools
Digest: sha256:b8958be9f66a63e705501ceec1bb2fe7f6f267ec947ddc1016994df66e7841cf

HttpSend : Exception calling "GetResult" with "0" argument(s): "A task was canceled."
At C:\Users\1379838568C\Documents\WindowsPowerShell\Modules\Airpower\<IP_ADDRESS>\Airpower.psm1:490 char:31
+     return HttpRequest @params | HttpSend
+                                  ~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [HttpSend], MethodInvocationException
    + FullyQualifiedErrorId : TaskCanceledException,HttpSend

PS C:\Users\1379838568C> Airpower pull python
latest: Pulling python
Digest: sha256:2f59a0baa4215044bea41a569baba96ad575ff9af75409584233a784d520cbc8
095b3609a830: Downloading █████████████▌                   17.50 MB / 38.2 MB
removed C:\Users\1379838568C\AppData\Local\Airpower\temp\095b3609a830900f1853c74f05445f78a7deb66b2622814ecdccd785f9506f5e.tar.gz because it had
corrupted data
At C:\Users\1379838568C\Documents\WindowsPowerShell\Modules\Airpower\<IP_ADDRESS>\Airpower.psm1:372 char:3
+         throw "removed $Path because it had corrupted data"
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (removed C:\User... corrupted data:String) [], RuntimeException
    + FullyQualifiedErrorId : removed C:\Users\1379838568C\AppData\Local\Airpower\temp\095b3609a830900f1853c74f05445f78a7deb66b2622814ecdccd785f9506
   f5e.tar.gz because it had corrupted data

When I try to pull vs-buildtools:17.4.3, it just stalls for a couple minutes and then throws the error without attempting to download. (I tried multiple times, all with the same result.) I'm thinking this could be either a timeout because the size of the download is just too big, or it could be an issue with some of the variables not being able to hold a size larger than 2 GB / 4 GB.
When I try to pull python (python:3.11.1), the download starts but is interrupted part of the way through with a corruption error. (I tried multiple times, all with the same result at close to the same spot: ~17.6 MB downloaded.) This one seems like it might just be either a corrupted file or a bug in the downloader, since it was the only package I had trouble with.
On a good note, I was able to pull git and python:3.11.0 successfully.

I'd like to get your thoughts on the Airpower database, @troyaf and @tbateson. I've noticed that the database can get corrupted pretty easily and the only way to fix it is to re-pull and manually delete any leftover directories. This corruption can happen for several reasons, including an exception that gets thrown or running multiple commands simultaneously. I know with oldpwr we had the lock file to deal with some of these issues, which I'm assuming we'd like to avoid.
Ideally, I'd like to consolidate the database to only contain necessary information and use the filesystem to look up most of the information. Or maybe even distribute the database to the individual SHA directories stored in something like $AirpowerHome\content\[SHA]\.pwr or $AirpowerHome\content\[SHA]\.pwr_db. (This would at least allow Airpower pull or Airpower rm to happen simultaneously as long as the packages were different.)
We might be able to eliminate the database entirely. It seems like most of the information in the database is either redundant or unused:

Size doesn't seem to be used
RefCount only matters if it is 0, which can be checked using the file system: (Get-Item "$AirpowerHome/ref/*").Target
Link between Tag and Hash can be determined using (Get-Item "$AirpowerHome/ref/*").Target

Thoughts?

I'd like to get your thoughts on the Airpower database, @troyaf and @tbateson. I've noticed that the database can get corrupted pretty easily and the only way to fix it is to re-pull and manually delete any leftover directories. This corruption can happen for several reasons, including an exception that gets thrown or running multiple commands simultaneously. I know with oldpwr we had the lock file to deal with some of these issues, which I'm assuming we'd like to avoid.
Ideally, I'd like to consolidate the database to only contain necessary information and use the filesystem to look up most of the information. Or maybe even distribute the database to the individual SHA directories stored in something like $AirpowerHome\content\[SHA]\.pwr or $AirpowerHome\content\[SHA]\.pwr_db. (This would at least allow Airpower pull or Airpower rm to happen simultaneously as long as the packages were different.)
We might be able to eliminate the database entirely. It seems like most of the information in the database is either redundant or unused:

Size doesn't seem to be used
RefCount only matters if it is 0, which can be checked using the file system: (Get-Item "$AirpowerHome/ref/*").Target
Link between Tag and Hash can be determined using (Get-Item "$AirpowerHome/ref/*").Target

Thoughts?

Here's kind of what I was thinking: https://github.com/nicklauslittle/airpwr/compare/74-pwr..74-pwr-rework-db. It's not complete, but the general idea is there. (Needs quite a bit of cleanup and many tests need to be updated to support it.)
It basically stores a .pwrdb file in each content directory when it is untargzed (e.g. {Digest:"sha256:2f59a0baa4215044bea41a569baba96ad575ff9af75409584233a784d520cbc8","Size":40055955}). (We could probably store additional information in there as well, like maybe unpacked size in addition to the targzed size.) This information is only looked up when it is needed (Airpower list, etc.)
The main database keeps roughly the same structure, except that you don't have to save it or worry about it getting out-of-sync with the filesystem, since it is generated from the filesystem metadata. I changed RefCount to just Refs, which is now a list of directory junctions from the ref folder. I also changed the tag map ({"latest":"sha256:..."}) so the values are metadata objects with the Refs list and the Directory.
I wanted to get your thoughts before making any more updates, so please let me know if you have any comments or recommendations.

I was trying to use the new Airpower in a CI environment. While trying to pull I was seeing this:
> pwr pull cmake-converter
latest: Pulling cmake-converter
Digest: sha256:cfe7a7b5612f1221c32c50e56256eb20b14456ef6f91ff31398[60](https://github.com/76-eddge/build-images/actions/runs/4105220691/jobs/7081772368#step:3:61)2a89f557a1c
pwr : Exception setting "CursorVisible": "The handle is invalid."
At line:1 char:1
+ pwr pull cmake-converter
+ ~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-Airpower

I'm guessing it's related to the custom progress bar not working properly when running inside CI. Maybe we could have an option to disable the progress bar or not display it when [console]::CursorVisible = $false?

I'm consistently getting an error when running the following command with the latest Airpower on Powershell 7 (5.1 seems to work):
PS > Airpower pull cmake
latest: Pulling cmake
Digest: sha256:e6a11d495297f09149972be6c1adf262a16b7f5347cde434bb5dbe605330444a
6094ffdcbc5d: Extracting █████████████████████████████▌   34.79 MB / 35.18 MB
Invoke-Airpower: Cannot bind argument to parameter 'Buffer' because it is an empty array.

I was trying to use the new Airpower in a CI environment. While trying to pull I was seeing this:
> pwr pull cmake-converter
latest: Pulling cmake-converter
Digest: sha256:cfe7a7b5612f1221c32c50e56256eb20b14456ef6f91ff31398[60](https://github.com/76-eddge/build-images/actions/runs/4105220691/jobs/7081772368#step:3:61)2a89f557a1c
pwr : Exception setting "CursorVisible": "The handle is invalid."
At line:1 char:1
+ pwr pull cmake-converter
+ ~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-Airpower

I'm guessing it's related to the custom progress bar not working properly when running inside CI. Maybe we could have an option to disable the progress bar or not display it when [console]::CursorVisible = $false?
Update
Looks like the error is actually thrown when the line [Console]::CursorVisible = $false is hit. Maybe, these lines just need to be disabled when running in CI?

Added a fix but will need to consider whether to allow the progress writing in CI.

I'd like to get your thoughts on the Airpower database, @troyaf and @tbateson. I've noticed that the database can get corrupted pretty easily and the only way to fix it is to re-pull and manually delete any leftover directories. This corruption can happen for several reasons, including an exception that gets thrown or running multiple commands simultaneously. I know with oldpwr we had the lock file to deal with some of these issues, which I'm assuming we'd like to avoid.
Ideally, I'd like to consolidate the database to only contain necessary information and use the filesystem to look up most of the information. Or maybe even distribute the database to the individual SHA directories stored in something like $AirpowerHome\content\[SHA]\.pwr or $AirpowerHome\content\[SHA]\.pwr_db. (This would at least allow Airpower pull or Airpower rm to happen simultaneously as long as the packages were different.)
We might be able to eliminate the database entirely. It seems like most of the information in the database is either redundant or unused:

Size doesn't seem to be used
RefCount only matters if it is 0, which can be checked using the file system: (Get-Item "$AirpowerHome/ref/*").Target
Link between Tag and Hash can be determined using (Get-Item "$AirpowerHome/ref/*").Target

Thoughts?

Here's kind of what I was thinking: https://github.com/nicklauslittle/airpwr/compare/74-pwr..74-pwr-rework-db. It's not complete, but the general idea is there. (Needs quite a bit of cleanup and many tests need to be updated to support it.)
It basically stores a .pwrdb file in each content directory when it is untargzed (e.g. {Digest:"sha256:2f59a0baa4215044bea41a569baba96ad575ff9af75409584233a784d520cbc8","Size":40055955}). (We could probably store additional information in there as well, like maybe unpacked size in addition to the targzed size.) This information is only looked up when it is needed (Airpower list, etc.)
The main database keeps roughly the same structure, except that you don't have to save it or worry about it getting out-of-sync with the filesystem, since it is generated from the filesystem metadata. I changed RefCount to just Refs, which is now a list of directory junctions from the ref folder. I also changed the tag map ({"latest":"sha256:..."}) so the values are metadata objects with the Refs list and the Directory.
I wanted to get your thoughts before making any more updates, so please let me know if you have any comments or recommendations.

I think you're completely right about how to move forward with the db. The only way to keep it consistent without enforcing crippling synchronicity through a global lock is to use the file system for fine-grained locks. I'll whip up a prototype and take a look at what you wrote.

I was trying to use the new Airpower in a CI environment. While trying to pull I was seeing this:
> pwr pull cmake-converter
latest: Pulling cmake-converter
Digest: sha256:cfe7a7b5612f1221c32c50e56256eb20b14456ef6f91ff31398[60](https://github.com/76-eddge/build-images/actions/runs/4105220691/jobs/7081772368#step:3:61)2a89f557a1c
pwr : Exception setting "CursorVisible": "The handle is invalid."
At line:1 char:1
+ pwr pull cmake-converter
+ ~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-Airpower

I'm guessing it's related to the custom progress bar not working properly when running inside CI. Maybe we could have an option to disable the progress bar or not display it when [console]::CursorVisible = $false?
Update
Looks like the error is actually thrown when the line [Console]::CursorVisible = $false is hit. Maybe, these lines just need to be disabled when running in CI?

Added a fix but will need to consider whether to allow the progress writing in CI.

Yeah, it spits out quite a few lines (see https://github.com/76-eddge/build-images/actions/runs/4212235192/jobs/7311113939 for a what it looks like with my hacked Airpower version). I had thought about printing out progress at a slower rate when in CI, but it's not really a necessity. Just maybe a nice-to-have. But, disabling progress completely in CI would also work.

I'd like to get your thoughts on the Airpower database, @troyaf and @tbateson. I've noticed that the database can get corrupted pretty easily and the only way to fix it is to re-pull and manually delete any leftover directories. This corruption can happen for several reasons, including an exception that gets thrown or running multiple commands simultaneously. I know with oldpwr we had the lock file to deal with some of these issues, which I'm assuming we'd like to avoid.
Ideally, I'd like to consolidate the database to only contain necessary information and use the filesystem to look up most of the information. Or maybe even distribute the database to the individual SHA directories stored in something like $AirpowerHome\content\[SHA]\.pwr or $AirpowerHome\content\[SHA]\.pwr_db. (This would at least allow Airpower pull or Airpower rm to happen simultaneously as long as the packages were different.)
We might be able to eliminate the database entirely. It seems like most of the information in the database is either redundant or unused:

Size doesn't seem to be used
RefCount only matters if it is 0, which can be checked using the file system: (Get-Item "$AirpowerHome/ref/*").Target
Link between Tag and Hash can be determined using (Get-Item "$AirpowerHome/ref/*").Target

Thoughts?

Here's kind of what I was thinking: https://github.com/nicklauslittle/airpwr/compare/74-pwr..74-pwr-rework-db. It's not complete, but the general idea is there. (Needs quite a bit of cleanup and many tests need to be updated to support it.)
It basically stores a .pwrdb file in each content directory when it is untargzed (e.g. {Digest:"sha256:2f59a0baa4215044bea41a569baba96ad575ff9af75409584233a784d520cbc8","Size":40055955}). (We could probably store additional information in there as well, like maybe unpacked size in addition to the targzed size.) This information is only looked up when it is needed (Airpower list, etc.)
The main database keeps roughly the same structure, except that you don't have to save it or worry about it getting out-of-sync with the filesystem, since it is generated from the filesystem metadata. I changed RefCount to just Refs, which is now a list of directory junctions from the ref folder. I also changed the tag map ({"latest":"sha256:..."}) so the values are metadata objects with the Refs list and the Directory.
I wanted to get your thoughts before making any more updates, so please let me know if you have any comments or recommendations.

I think you're completely right about how to move forward with the db. The only way to keep it consistent without enforcing crippling synchronicity through a global lock is to use the file system for fine-grained locks. I'll whip up a prototype and take a look at what you wrote.

Yeah, I think fine-grained locking is probably the best idea. We could lock on sha256 during download and untar. Then we could lock anytime a DB write is needed, re-read and then re-write the updated DB. Not sure how best to handle multiple concurrent attempts to access the lock. (Maybe print a message and just wait until it's available?)

I noticed an issue with the locking. I was repulling all the packages with the latest Airpower, while accidentally using the jdk with vscode. (That explains the first error, but I would expect it to recover for the remaining commands.)
Pulling jdk:latest
Digest: sha256:9e662865b2bac015cdb78809aefb5f8901802c674f7eaa9bdc7e9ff87c9de4d8
28041ebac51b: Extracting                                 128.00 kB / 349.89 MB
pwr : Exception calling "Open" with "3" argument(s): "The process cannot access the file
'\\?\C:\Users\...\AppData\Local\Airpower\content\9e662865b2ba\jdk-17.0.6+10\bin\instrument.dll' because it is being used by another process."
At line:1 char:1
+ pwr exec
+ ~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-Airpower

PS C:\Users\...\project> pwr exec
pwr : Exception calling "ReadAllText" with "1" argument(s): "The process cannot access the file
'C:\Users\...\AppData\Local\Airpower\cache\cGtnZGIKamRrCmxhdGVzdA==' because it is being used by another process."
At line:1 char:1
+ pwr exec
+ ~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-Airpower

PS C:\Users\...\project> pwr list
pwr : a package pkgdb jdk latest is being used by another airpower process
At line:1 char:1
+ pwr list
+ ~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-Airpower

Deleting the zero length files under the cache directory seemed to fix the issue, but I had to close out of PowerShell before I could delete them. I'm not real familiar with the locking implementation, but I'm guessing we need some try { } finally { } blocks?

Another issue I've run into with version <IP_ADDRESS> is that I can't pull several different packages (vs-buildtools, perl, python) even working around the issue above. It downloads and untars the packages just fine, but then I get the following error before the database gets updated. It's always the same directory, which seems odd to me.
PS C:\Users\...\project> pwr pull vs-buildtools:17.4.5
Pulling vs-buildtools:17.4.5
Digest: sha256:238cb9dd2f0e045dff5c2635c9e90d91742d44916dad02636f1d7189fdf2c001
92a8fb0baa17: Pull complete
pwr : Exception calling "Delete" with "1" argument(s): "Access to the path 'C:\windows\system32\1033' is denied."
At line:1 char:1
+ pwr pull vs-buildtools:17.4.5
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-Airpower

Another issue I've run into with version <IP_ADDRESS> is that I can't pull several different packages (vs-buildtools, perl, python) even working around the issue above. It downloads and untars the packages just fine, but then I get the following error before the database gets updated. It's always the same directory, which seems odd to me.
PS C:\Users\...\project> pwr pull vs-buildtools:17.4.5
Pulling vs-buildtools:17.4.5
Digest: sha256:238cb9dd2f0e045dff5c2635c9e90d91742d44916dad02636f1d7189fdf2c001
92a8fb0baa17: Pull complete
pwr : Exception calling "Delete" with "1" argument(s): "Access to the path 'C:\windows\system32\1033' is denied."
At line:1 char:1
+ pwr pull vs-buildtools:17.4.5
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-Airpower

I have discovered the reason for this. It appears that any return statement is actually equivalent to Write-Output (https://devblogs.microsoft.com/powershell/suppressing-return-values-in-powershell-functions/). Since the GzipRead function returns the number of bytes read, the return value of ExtractTarGz was an array of thousands of ints plus the path at the end. So, $temp in SavePackage ended up with the int 1033 in it, which happens to exist in C:\Windows\System32, which is apparently the default directory for relative paths.
I ended up casting all the uses of GzipRead to void (e.g. [void]([Util]::GzipRead($Source, $buf, $Header.Size))), which solved the problem. I'm putting together a PR.

Experiencing a strange issue while trying to rm a package:
PS C:\Users\Me\Code\76cal-test> pwr list

Package         Tag    Digest       Size
-------         ---    ------       ----
jom             latest 19ca3c0e2361 1.27 MB
jdk             11     4cb1aa4abd99 348.79 MB
jdk             8      e39f16178524 193.25 MB
jdk             latest ac9d3f20576f 350 MB
llvm            latest fa4206020c93 815.37 MB
llvm                   abf1ba02c2c4 815.39 MB
node            latest 4debe28e4b76 29.28 MB
nasm            latest db2a58b317e9 509.15 kB
ninja           latest 2f13f35935ab 298.21 kB
powershell      latest 90de93e1481c 103.81 MB
perl            latest a705772c0f18 147.13 MB
python          latest 20fbc35deef2 38.59 MB
python                 520fbdfca6f4 38.38 MB
vs-buildtools   17.4.7 10e709d1a40e 3.99 GB
vs-buildtools   17.6.2 00020d7bf7ac 4.38 GB
vs-buildtools   latest 00020d7bf7ac 4.38 GB
vscode          latest 84a469aefc7f 129.89 MB
vscode                 820ac1ad081c 129.8 MB
vscode                 0de0e64ef725 129.79 MB
7-zip           latest 75fdb92c78cf 2.48 MB
cmake           latest 165323c05980 37.15 MB
cmake                  c62c7aa9b118 35.28 MB
cmake-converter latest b7d550f89cd8 1.39 kB
git             latest b9a240772d1c 143.04 MB
git                    875a59bd9cc7 124.44 MB
gradle          latest<PHONE_NUMBER>50 118.94 MB
doxygen         latest 94d31b816455 42.46 MB
doxygen                6b70d8abc471 42.08 MB
erlang          latest 9b61e382ef88 128.98 MB

PS C:\Users\Me\Code\76cal-test> pwr rm vs-buildtools:17.4.7
Untagged: vs-buildtools:17.4.7
pwr : Exception calling "Delete" with "2" argument(s): "Could not find a part of the path
'Microsoft.CodeAnalysis.VisualBasic.ExpressionEvaluator.ExpressionCompiler.resources.dll'."
At line:1 char:1
+ pwr rm vs-buildtools:17.4.7
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-Airpower

PS C:\Users\Me\Code\76cal-test> pwr list
pwr : a package pkgdb vs-buildtools 17.4.7 is being used by another airpower process
At line:1 char:1
+ pwr list
+ ~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-Airpower

I ended up having to close out of PowerShell, open a new PowerShell, completely remove everything in C:\Users\Me\AppData\Local\Airpower\ref\vs-buildtools-17.4.7, and then rerun pwr rm vs-buildtools:17.4.7.
I think there might be 2 separate issues here:

The delete call is not working correctly.
The locks are not released when an exception occurs. They might need to be wrapped in try { } finally { } blocks.

I'll double check the try finallys. I think there is also an outstanding issue with long file paths, so we may have to revert to robocopy again.
I'll see if I can reproduce this in a test first.