GSOC 2019: Make sure top 50 PyPI packages are supported

Nuitka works with most software. The aim of this project is to make sure it's true for the top 50 packages on PyPI, by compiling and using their example codes.
In a first stage, you would identify and report the issues to the bug tracker, in a second stage develop tools that help to narrow down issues, e.g. what extension module fails to load precisely, even with a segfault happening, and put them to use and try to fix a few of the simpler issues.
Setting up these as automated tests would be the ultimate goal, so we can follow these top 50 packages with Nuitka over time and make sure they continue to work.
In the past it has happened e.g. that Jinja2 was breaking for Python3.7, and it would be cool to discover this immediately.
Skills: Python programming, pip installation, Linux and/or Windows installs of Python, one is good, both would be great.

Hey.
This is something  that sounds really interesting to me and I'd love to be able to contribute to this.
But I am completely new to Open source as a whole and have no idea how to even begin. I don't expect to be able to do something significant enough to qualify for GSoC or whatever but I'd love for this to be my first foray into open source and to gain experience.
Since this and #230 both are labelled as easy in terms of difficulty, I think I should start here.
But I don't know what "Linux and/or Windows installs of Python" means in terms of skills and I don't know how to begin.
Could you please guide me to some resources or tell me how I should go about finding my bearings? I am competent with Python and consider myself to be a quick learner, I am sure I will be able to get up to speed soon.

Welcome @aditya-hari and I agree this is not yet detailed enough. Actually today I was going to flesh this further out with pointers.
As for the skill, more of an asset. We would prefer you did this on those platforms, doing it on macOS would e.g. be too difficult for us to help you with. We do not care if you use Windows or Linux, however in my experience the later works better for the debugging that will be needed, i.e. in case of a crash of standalone compiled software.
As for more pointers, they are coming later today, or so I hope.

So, first of all, we need to define what we mean by top 50 PyPI packages. For Python downloading the packages via pip is very widespread. So the first task would be to look at the top 100 and identify 50 packages that are even relevant to Nuitka.
Of course pip itself, leading the list, we will skip that, nobody needs that compiled or uses it as a library in their software. But urllib3 is top 2, and very important piece of software that people rely on, so breaking it in Nuitka compilation would be bad.
What the student would be expected to do, is to go and read its documentation in search for a tutorial, small examples, that make it work. Distill a minimal test out of that and add it to the existing pool of tests in tests/standalone say as tests/standalone/Urllib3Using.py and get it to work in standalone mode, using the Nuitka argument --standalone.
Then, if possible and easy, the tests should be made to work. For py.test and nose2 there is a good chance to get them working against a compiled package. For that you would be using the Nuitka argument --package, throw in magic environment variables if necessary, if the tests live below the package name space, say urllib3.tests and then run the tests. Looking at the test results, which hopefully pass and fail equally well (never assume a released software passes all the tests when not compiled in your environment or any), you then try to identify the issue, or report it as a Nuitka issue, or sometimes as an issue of the software we are testing.
When those work, we should try and turn this into a re-usable test as well, so we can apply them in an automated fashion.
Then on to next package on the list. The main benefit to the student will be to get to know the 50 most important software packages of Python on at least a cursory level. Something the mentors won't even do. And that will teach you a lot and the mentors too. And it will prevent people using Nuitka from encountering things that our testing will then find before Nuitka releases.

A good example of a test for a standalone usage is this one:
https://github.com/Nuitka/Nuitka/blob/cb0e0beb72041adeab1065306d5413a6bd868611/tests/standalone/PyQt5Using.py#L29
Here PyQt is used to create an application and signal a slot, which as a compiled function could be problematic. And in fact it was, until we reported to upstream with a potential patch to their source. The later would be fantastic if you manage, but not required.

I hate to ask but I would really appreciate it a great deal if you could guide me through it. I am willing to learn and devote as much time as necessary.

So I'm definitely showing my inexperience here, but how do I start? Apologize for the constant banal questions, just really want to help

You would start by making a minimal urllib3 using program, compile it with nuitka, and run it standalone. You would then integrate it into the test suites mentioned above. Then, optionally try to package urllib3 as a compiled package, and run its tests against that. Then next package.

And obviously, urllib3 documentation is most definitely going to contain a tutorial example, some prime use case, of what people usually use it. In that case, it probably is going to be downloading files via http and/or https protocols.

@kayhayen I am interested in this project and the oppurtunity it gives to know top 50 packages.

Alright, I'll get right to that

And obviously, urllib3 documentation is most definitely going to contain a tutorial example, some prime use case, of what people usually use it. In that case, it probably is going to be downloading files via http and/or https protocols.

Hi @kayhayen I am Mohammed Sanan currently pursing my third year in B.tech. I am interested in this project. I can learning more than 50 libraries from this gsoc project.
Documentation helps. But error are occuring

After all fixing this error is occuring.

@sannanansari make sure to use the right kind of MinGW64, the one linked. This error is typical of using MinGW. The user manual has links to what is working.

Ok. I will get right their.

You would start by making a minimal urllib3 using program, compile it with nuitka, and run it standalone. You would then integrate it into the test suites mentioned above. Then, optionally try to package urllib3 as a compiled package, and run its tests against that. Then next package.

@kayhayen I have gone through this step. Nuitka compiler is compiling urllib3 code and generating.exe file also but --standalone is give an warning and not generating .exe file.

Explain about this part? How can I fix this error?

@sannanansari It means that six.py is using the import statement in line 82: https://github.com/urllib3/urllib3/blob/master/src/urllib3/packages/six.py#L82 .  You'll have to find out what name is this case can be and pass it in via --include-plugin-files or --include-plugin-directory to your Nuitka compile statement.

Sorry for uploading incomplete but no output shown after that

As you mention about finding python module it is builtins module. But it is giving an error when I am including --include-plugin-files.
please ignore line from "trackback".

@sannanansari I think this is drifting off-topic. I've just compiled the example script from https://urllib3.readthedocs.io/en/latest/ with python -m nuitka --standalone test.py and it compiles fine and generates the executable. The warning is just a warning, it may cause trouble during runtime. builtins is part of Python, I doubt that you'd need it here, and it is not a directory.

Just to give the necessary background @sannanansari beyond what @kamgha said.
The six module is a popular Python2/Python3 compatibility layer, that often is included in the package itself, therefore living in the urllib3 namespace here. It does abstract some of the moved imports of Python3, and uses __import__ to execute its import.
Nuitka has a hard time to see through tricks like this, which is why it gives a warning, as an indication to the user, that something might go astray here. But it definitely doesn't have to be the case.
Sometimes programs, libraries have folders that they scan and load on the fly, e.g. plugins. These are also done via __import__ and in that case, Nuitka wants you to tell about such directories.
This is one of the usability quirks. I think Nuitka ought to be changed to not warn about "*.six" modules doing __import__ as these are all going to be harmless if they are six code. Plugins can suppress warnings, and such a check could easily be added. However, this wouldn't be on your plate with this issue.
You most often will see this warning and be able to ignore it. And where that is not the case, an ImportError will occur at runtime in standalone mode, and will be best explained by looking at this kind of warning.
I hope it clarifies things a bit. It is kind of expected that you won't know these yet. This idea will also expose you to this kind of usability issues of Nuitka naturally, but they are easy to learn.

Ok.
I want you to see this two things.

In first part C:\Users\Ansari\Anaconda3\python -m nuitka  testing.py. It is giving me a warning but an executable is created and it is running.
But on second part >C:\Users\Ansari\Anaconda3\python -m nuitka --standalone testing.py. It is giving warning but no executable is created.
Being new I am not able to get concept of plugin. But as you say plugin suppressed warning here main problem is that it is not creating executable. Warning is coming in first part also but executable is created. I'm not understanding that.

@sannanansari Please give us the output of dir /S after you have run Nuitka with --standalone.

This Screenshot is of C:\Users\Ansari\Anaconda3\python -m nuitka testing.py but after seeing into directory I have seen new folder. In first testing.build and executable is created but testing.dist in not.

C:\Users\Ansari\Anaconda3\python -m nuitka --standalone testing.py here testing.build and testing.dist is created not executable.
sorry for that I was only concentrating on executable file.

Why standalone is important? When I can directly use Nuikta. Which is providing executable.

@sannanansari  - the executable produced in non-standalone mode only works in presence of a (fully!) compatible Python installation.
A stanalone compile can be deployed to any compatible operating system - with or without a Python installation.

@sannanansari - when I say "a standalone can be deployed ...", then this means that the dist folder containing the executable is the deployable item: copy it to any compatible OS, and you will find the exe file in it usable.
Please also refer to the discussion around "One-File" distribution in this context (#230):
There is a request to transform the dist folder to a (big) exe-file that can be used directly to execute the original compiled script within it on the deployed-to system.
This request can be viewed as solved since a few days - for platforms Windows and Linux (thanks to @akshai9899), still missing Mac OSX for this.

I have not seen it sorry for that. It is working. So what should I do next?

I can email any mentor directly if you all don't mind.

@kayhayen Can I do this for GSOC?

@LilJohny @sannanansari sure please send in the small questionaire I put in the other GSoC questions thread.

Hi @kayhayen I am Pranamika and I am interested in this project. But please can you explain me that what did you mean by identifying 50 packages that are even relevant to Nuitka.

@Pihu1998 There are download counters on PyPI, and sites which aggregate the information. I think I gave a link previously, but I don't find it right now. But a Google search is all that it takes.

@kayhayen  Thank you for the clarification. I will surely search it out.

@kayhayen I have gone through the above mentioned steps but received the following error:

I am not able to figure it out. I am new to Open Sourece, maybe I just have't been able to figure out how to use Nuitka. Please help.

@Pihu1998 need to install Nuitka first

@kayhayen Nuitka is already installed.

@kayhayen I am trying to move forward and try to run more_itertools library from python. But it was giving me error No module name more_itertools. What should I do next? My aim is to install more_itertools in Nuitka to understand  it better.

@sannanansari Nuitka can only compile something that is already executable in your normal Python environment. Into that environment, you then apply Nuitka. So first of all, you would have to install what you want to compile, get it to run uncompiled, and only then you should compile it.

Can you can suggest what to do now or any bugs?

@sannanansari Look at the issues tagged GSoC or help_wanted and try to solve them.

@kayhayen
Greetings My name is Tambe Tabitha Achere and I program in Python.I'm completely new to opensource and  I want to contribute to Nuitka in order to sharpen my python skills.
I already have a dual  boot of UBUNTU 16.04LTS and Windows 10( with Python 3 and pip installled already). I am really interested in this issue. Please is are there specific guidelines on how a proposal to Nuitka should look like?
Or any prerequisites before I can apply to GSOC under Nuitka?

Just the normal Google requirements and Guidelines apply, @tab1tha you just start and try to come with a framework of doing e.g. this issue, and detail that in your application.

Hello! I am Anubhav, a student of Software Engineering and I want to apply for GSOC 2019 under Nuitka. I went through the documentation and successfully compiled the hello world program.
I am familiar with open source, git, github and I primarily code in Python and have used some of the famous modules like PyQT5 (Check my repos for more).
What can I do to assure you that I can be helpful for this program?
Hope I'm not late!

@annubv please check out the thread about other GSoC questions and find the information there.

Hey! I tried to compile the cryptography module code with Nuitka although, it was successful but it's taking way too long (more than 5 minutes for me) to do so. What is the reason behind this?
I successfully compiled the urllib3 code without any issues though.
Below is the source code of file t1.py
from cryptography.fernet import Fernet
key = Fernet.generate_key()
f = Fernet(key)
token = f.encrypt(b"A really secret message. Not for prying eyes.")
print(token)
print(f.decrypt(token))

The command :
python -m nuitka --standalone --mingw64 --follow-import-to=cryptography.fernet t1.py

Hi,
let me step in with an answer: 5 minutes is nothing!
But seriously:
On Windows, a DLL dependency analyzer (dependency walker) is automatically started, which detects all shared libraries required. This takes the most time ... at least for the first compiles. Analysis results are stored in a cache, which is consulted in subsequent compiles. As this cache enriches, compiles become shorter, because only DLLs yet unknown to the cache need be searched for.
Currently, there is a much faster package available as an "experimental" feature (meanwhile it no longer is truly experimental).
Add --experimental=use_pefile to your command line, to see the difference.

the issue description says the first task would be to look at the top 100 and identify 50 packages that are even relevant to Nuitka.Is there any criteria to check?

This may come as a shock. But the top 50 packages are best identified by looking at the top 100 list that I linked and using the first 50. :-)

Hi @kayhayen
I was running the pytest for the package Six. But I am not able to understand how to use the argument --package

For that you would be using the Nuitka argument --package, throw in magic environment variables if necessary, if the tests live below the package name space, say urllib3.tests and then run the tests

And I didn't understand the above part. Could you please explain or provide some pointers if possible?

@kayhayen Nuitka is already installed from the link provided in the documentation.

If you want to use Nuitka from cmd, you need to install it in command line with pip: "pip3 install nuitka" for python3

@tommyli3318 Thankyou so much for the response :-). I somehow figured out the problem and fixed it.
But now the problem is that I was trying to run the py.test for the package Six. But I am not able to figure out how to use the argument --package.

For that you would be using the Nuitka argument --package, throw in magic environment variables if necessary, if the tests live below the package name space, say urllib3.tests and then run the tests

And didn't understand the above part.

Ok, this is actually important @Pihu1998 look here:
https://github.com/Nuitka/Nuitka/blob/69183f6567befd62bc1d768dd070fb99475b8cb4/tests/test-runners/run_all.py#L120
This is the test runner that checks that py.test is usable. And the example software, uses package as a name for the package, and then the runner sets NUITKA_PACKAGE_package to indicate to the compiled package that its __path__ should also contain that directory.

This is the magic I was talking about. That way uncompiled tests can be run inside a package name space, if they so require (they may not!), and still import the compiled package and use it.

@kayhayen Thankyou for the elaboration :-)

I am setting up a machine once provided with it from @deajan company, and it will run it on a regular basis.


Full Screen Map: Update visualizations to work with Full Screen Map

[ ] Include Option to disable/hide the faceting input.
[ ] Include Option to allow a provided overlay variable.

Based on this slide, it occurred to me that we might need to further modify existing visualization components to hide various statistics and plot controls.
Screenshot from pass app:

Screenshot from mock-up:


az acr login hangs

az feedback auto-generates most of the information requested below, as of CLI version 2.0.62

Describe the bug
» az acr login -n <snip> --debug
cli.knack.cli: Command arguments: ['acr', 'login', '-n', '<snip>', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x10fef2290>, <function OutputProducer.on_global_arguments at 0x10ff3d120>, <function CLIQuery.on_global_arguments at 0x10ff9a680>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'acr': ['azure.cli.command_modules.acr']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: acr                       0.007        28       116
cli.azure.cli.core: Total (1)                 0.007        28       116
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
cli.azure.cli.core: Total (0)                 0.000         0         0
cli.azure.cli.core: Loaded 28 groups, 116 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : acr login
cli.azure.cli.core: Command table: acr login
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x1105c9360>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/Users/roy/.azure/commands/2022-04-15.14-04-37.acr_login.54779.log'.
az_command_data_logger: command args: acr login -n {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x1105eab90>, <function register_global_query_examples_argument.<locals>.register_query_examples at 0x1106b0790>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x1106b0820>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x1106b0940>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x10ff3d1b0>, <function CLIQuery.handle_query_parameter at 0x10ff9a710>, <function register_global_query_examples_argument.<locals>.handle_example_parameter at 0x1106b0700>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x1106b08b0>]

And it hangs there.
To Reproduce
Other than running az, 🤷. It reproduces for me, presently.
Expected behavior
It doesn't hang.
Environment summary
macOS, installed via brew.
» az version
{
  "azure-cli": "2.32.0",
  "azure-cli-core": "2.32.0",
  "azure-cli-telemetry": "1.0.6",
  "extensions": {
    "automation": "0.1.1",
    "ssh": "1.0.0"
  }
}

Additional context

The output appears to indicate that this is a different hang than the one in #16312.

route to CXP team

@roy-work Apologies for the late reply. Thanks for reaching out to us and sharing this feedback. I was able to run the az acr login cmd on my box. Sharing my Az CLI version here:
az version
{
"azure-cli": "2.35.0",
"azure-cli-core": "2.35.0",
"azure-cli-telemetry": "1.0.6",
"extensions": {
"aks-preview": "0.5.60",
"attestation": "0.2.0",
"db-up": "0.2.7"
}
}
The az acr login cmd helps to Log in to an Azure Container Registry through the Docker CLI. So Docker must be installed on your machine. Could you please confirm if that is done ? Once done, use 'docker logout ' to log out.
Note: If you only need an access token and do not want to install Docker, specify '--expose-token' parameter. Please let me know if that works fine. Awaiting your reply.

Docker must be installed on your machine. Could you please confirm if that is done ?

… even if it weren't, hanging would hardly be the appropriate response. But, it is.
What I suspect happened here is that az was making a silent call to docker, and it was that that was hanging. I found today that any docker command on my machine was hanging, and a force-restart of docker & all its parts fixed that.
After that, az acr login appears to work again.
It would be helpful to have a log in --debug if that is what's going on; if I had known that az was trying (unsuccessfully) to run docker, I'd know who to try restarting.

@roy-work I am glad that post restarting of the docker the issue is resolved. Regarding your suggestion on the --debug logs, You could check the logs that are saved in this path: /Users/roy/.azure/commands/2022-04-15.14-04-37.acr_login.54779.log and check if that has the details about the call being done to docker CLI.
If you need any further assistance on this github thread, please let us know.

@roy-work I wanted to do quick follow-up to check if you had a chance to look at my above comment. Please let us know if you need any further assistance on this. Awaiting your reply.

@roy-work We will now proceed with closure of this github issue. If you need any further assistance on this issue in future, please feel free to reopen this thread. We would be happy to help.


Segmentation on range images

Hello, @TixiaoShan !
I am interested in the segmentation on range images.
In the function void labelComponents in the imageProjection.cpp, almost all steps originate from the ref. Fast Range Image-Based Segmentation of Sparse 3D Laser Scans for Online Operation.
However, in the final step, you do a confusing method to handle the case in which allPushedIndSize < 30. More in detail, I don't know what are the means of segmentValidPointNum and segmentValidLineNum.
Could you offer any help?
Thank you!

@zorosmith
That's just some hand-tuned params. For example, if a segment has number of points less than segmentValidPointNum, or segment consists of scan lines less than segmentValidLineNum, then I consider it not a segment. It's for noise removal.


[Feature] Add a way to stop the stack properly

Feature description
We are working on a matter controller device.
We need to stop the Matter stack properly to test our code with Valgrind to make sure that there is no memory leak.
Currently it's difficult because the Matter stack is not able to be stopped properly.
It should be good to implement something that permit to cleanup all the memory used by the stack when we stop the stack.
Also, what is the good way to stop the stack ?
    // chip::DeviceLayer::PlatformMgr().HandleServerShuttingDown();
    // DeviceControllerFactory::GetInstance().ReleaseSystemState();
    // DeviceControllerFactory::GetInstance().Shutdown();
    // ReturnLogErrorOnFailure(chip::DeviceLayer::PlatformMgr().StopEventLoopTask());

    // chip::Platform::MemoryShutdown();
    chip::DeviceLayer::PlatformMgr().StopEventLoopTask();
    // chip::DeviceLayer::PlatformMgr().Shutdown();

    // Do this after StopEventLoopTask() so we don't need to Lock/Unlock the stack
    for (auto it = mCommissioners.begin(); it != mCommissioners.end(); it++) {
        ShutdownCommissioner(it->first);
    }

Platform
other
Platform Version(s)
No response
Anything else?
No response

Our software is based on chip-tool, so all of my remarks can be tested with chip-tool and valgrind.
Currently I found that most of the uncleared memory is related to DBUS, maybe we initialise some pointers that are never cleared.
Disabling chip_enable_wifi=false helps, but I still have some leaks:
==16042== LEAK SUMMARY:
==16042==    definitely lost: 0 bytes in 0 blocks
==16042==    indirectly lost: 0 bytes in 0 blocks
==16042==      possibly lost: 1,352 bytes in 18 blocks
==16042==    still reachable: 35,508 bytes in 214 blocks
==16042==                       of which reachable via heuristic:
==16042==                         newarray           : 1,536 bytes in 16 blocks
==16042==         suppressed: 0 bytes in 0 blocks
==16042== Reachable blocks (those to which a pointer was found) are not shown.
==16042== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==16042==
==16042== For lists of detected and suppressed errors, rerun with: -s
==16042== ERROR SUMMARY: 18 errors from 18 contexts (suppressed: 0 from 0)

Stopping the stack properly on the controller side consists of:

Stop the event loop task.

This will shut down the platform manager, etc, etc.
It sounds like you are running into a specific bug in the Linux bits where some things don't get shut down properly.  If so, that should be fixed in the Linux platform impl.


 In FreeBSD they normally use 7-char tags. Length should be configurable per-user.
Also please add a button "Copy to clipboard" near some important tags, or next to all of them.

Thanks your feedback!

Gogs displays 10-char commit tags. In FreeBSD they normally use 7-char tags. Length should be configurable per-user.

What is the real effect for this?

Also please add a button "Copy to clipboard" near some important tags, or next to all of them.

What is the definition of important tags?

What is the real effect for this?

Ports are built around 7-char tags. This is the (maybe arbitrary) choice made long ago and encoded in regexps, etc. GitHub downloads are requested with tags in 7-chars too.

What is the definition of important tags?

At least the current one, and last in the branch. But really copy-tag button should be attached to each one displayed if possible,

Ports are built around 7-char tags. This is the (maybe arbitrary) choice made long ago and encoded in regexps, etc. GitHub downloads are requested with tags in 7-chars too.

Gogs just shows 10 chars, why does this break anything? And what is the regexps you are pointing to?

At least the current one, and last in the branch. But really copy-tag button should be attached to each one displayed if possible,

OK, I cann add copy-tag button for latest one in the repository view and each one in commits history page.

Gogs just shows 10 chars, why does this break anything? And what is the regexps you are pointing to?

Because when such git site is used as a distfile source, I have to manually count 7 chars, then select them, copy and paste.
Example: https://svnweb.freebsd.org/ports/head/net-im/qTox/Makefile?revision=402148&view=markup
GH_TAGNAME= used there is always 7 chars

Ok, make sense.
BTW, for the copy-tag button feature, I suggest you file another issue for that to keep things clean.

Done: https://github.com/gogits/gogs/issues/2182

Thanks!

@yurivict Would 10 char SHA1 break this script using GH_TAGNAME ?
Anyways this isn't about commit tag but about commit SHA1 abbreviation, so this issue subject is wrong.
See How short can Git abbreviate?. 10 characters is super safe number, 7 characters is also safe as long the project isn't big.
@Unknwon FYI GitHub & BitBucket use 7 characters abbrevs by default, GitLab uses 8. I don't think there's any reason why Gogs should just switch to these 7 characters and have some SHA1_ABBREV_SIZE ini setting. This should be easy, just make base.ShortSha use this setting, instead builtin 10. Thoughts?

Make it a choice 7..10. This should fit everybody. Too long becomes unaesthetic, too short becomes unsafe.

Yes, I think allow config for range 7-10 should be good.


can't copy tensorflow/python/pywrap_tensorflow_internal.py

porter@fattire:~/Projects/tensorflow$ bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
Thu Apr 6 17:10:48 PDT 2017 : === Using tmpdir: /tmp/tmp.kCYPVwxjU5
~/Projects/tensorflow/bazel-bin/tensorflow/tools/pip_package/build_pip_package.runfiles ~/Projects/tensorflow
~/Projects/tensorflow
/tmp/tmp.kCYPVwxjU5 ~/Projects/tensorflow
Thu Apr 6 17:10:49 PDT 2017 : === Building wheel
error: can't copy 'tensorflow/python/pywrap_tensorflow_internal.py': doesn't exist or not a regular file

This happens after building tensorflow with bazel and running the binary build by bazel:
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
pywrap_tensorflow_internal.py doesn't exist in the source directories:
Tensorflow commit f8dce81
Bazel 0.4.5
CUDA release 8.0, V8.0.26, cuDNN 5105
GPU: NVIDIA Titan 6GB

I met the same problem when install r1.1.
And I run find . -name "pywrap_tensorflow_internal.py" in the tf root, got nothing.

A workaround:
copy ./tensorflow/python/pywrap_tensorflow.py  to ./bazel-bin//tensorflow/python/pywrap_tensorflow_internal.py.
Then I got /tensorflow/python/gen_resource_variable_ops.py doesn't exist or not a regular file
I think it's likely because some bazel compiling rules may be missed in r1.1.

+1, experiencing the same issue
Tensorflow commit d48ef8f6b743c715e8209e7faf4d2c46ace57046
Bazel 0.4.5
CUDA release 8.0, V8.0.26, cuDNN 5105
GPU: NVIDIA GTX 1080
The file in question clearly exists:

[zamparol@server tensorflow]$ ls -lh tensorflow/python/pywrap_tensorflow.py
-rw-r--r-- 1 zamparol cllab 2.5K Apr 25 15:47 tensorflow/python/pywrap_tensorflow.py

Resolved my issue, I'm not sure what the problem is but I was able to build a wheel by doing the following from the root of the repo:
git submodule update --init
bazel clean --expunge
bazel build -c opt --config=cuda --verbose_failures //tensorflow/tools/pip_package:build_pip_package
bazel-bin/tensorflow/tools/pip_package/build_pip_package ~/utils/tensorflow_pkg

Maybe if one of the bazel build steps fails, you need to do bazel clean --expunge, and try again?

Does that solve your issue @CharlesShang?

Unfortunately, It failed again.
The same error mssg was prompted out.
I'm on commit 1ec6ed5.

I had a similar issue and was able to fix it by removing the VERBS support from the configure. This is what (I guess) happened:  bazel build with the VERBS support actually failed, but I didn't notice it and it still created the file of 'bazel-bin/tensorflow/tools/pip_package/build_pip_package'. And this file gives me the error.

@jiangxu87 Could you please give a more detailed howto?

@CharlesShang
Was your error solved?
I have the same error. please help

No.
@nishalpereira, I've installed the pre-built binary instead.

@CharlesShang
Thanks for your reply.
Can you tell me how to install the pre-built binary.

This problem is random.
On another machine (same OS, same cuda version, but different GPU), tf can be built successfully. So I just copy the built file to my machine, install it..

It has been 14 days with no activity and this issue has an assignee.Please update the label and/or status accordingly.

My 2 cents (no CUDA, with Cmake on windows 7 64 ) :
1>------ Début de la génération : Projet : tf_python_build_pip_package, Configuration : Release x64 ------
1>Error copying file "C:/MKTS/tensorflow/tensorflow/contrib/cmake/build/Release/pywrap_tensorflow_internal.dll" to "C:/MKTS/tensorflow/tensorflow/contrib/cmake/build/tf_python/tensorflow/python/_pywrap_tensorflow_internal.pyd".
looks like pretty much the same.
On Ubuntu 14.04, works like a charm on Ubuntu with bazel nevertheless


Test classes importing android packages are not found

From<EMAIL_ADDRESS>on September 28, 2011 21:59:41
What steps will reproduce the problem? 1.build test classes using ant from commandline 2. 3. What is the expected output? What do you see instead? No errors but a release apk. I see for example:
[javac] C:\MyTest.java:6: package android.app does not exist
[javac] import android.app.Activity; What version of the product are you using? On what operating system? Google API level 8, Windows 7 Please provide any additional information below. Android home is set, java home is set, android.jar is in android home etc but still all test classes that use robotium and import android librraies are failing to compile via ant
Original issue: http://code.google.com/p/robotium/issues/detail?id=168

From<EMAIL_ADDRESS>on September 29, 2011 05:18:49
Unfortunately I am not able to help you with this as this is an ant issue.
Status: Invalid


Merge SHA2 into single module

Current code is too modularized, highly regarded. The only thing changing are inputs/outputs.
Merge everything into sha2 module. Same for sha3.

Wondering how to go about handling the sha2-256 and sha2-512 modules... So early that I suspect it'll be ok to simply drop them.


Stac 3402- Allow https on agent6

What does this PR do?
A brief description of the change being made with this pull request.
Motivation
What inspired you to submit this pull request?
Additional Notes
Anything else we should know when reviewing?

I think, proposed ruins a bit original test.
More adequate seems extracting that in separate scenario
check
https://molecule.readthedocs.io/en/latest/


Allow apps to force lang via code.language and add special cased files

This makes some tweaks to language modes:

It adds a list of special-cased files that are mixed into the determination of the language, so you can return a language for things even if the file extension itself doesn't imply the language. The languages included aren't actually implemented in community, so I can keep the list empty if preferred, but I figured I'd keep it populated at first to give an example of why it's useful.
Adds a command to show the currently forced language
#1256 semi-recently went the route of removing language-specific tags. This has the side effect (afaict) that an app that doesn't expose the file extension in its title can no longer force a language (previously it would just tag: user.<lang>). Given we removed the tags I assume we don't want them back, so this change goes a different route and adds an action that can be overridden to force a language. It's done in a way that the language being forced by an app will still be overridden by a user manually forcing a language with force <lang>. I've included an obsidian skeleton to demonstrate how to use it. Obsidian is a markdown editor that doesn't include file extensions in it's title. I took a look at search.talonvoice.com and see that some people are using tag.markdown (so are their community is not up to date, or they custom implemented tags again) and some others are re-implementing markdown commands. Since I tried to use this app recently, and I don't like forcing a language across my entire system and have to remember to clear it afterwards, I figured I should add a way to do it.

I'm open to other ways to do this if there is a better idea.

I personally think an app that wants to force a language should override the code.language action.

I personally think an app that wants to force a language should override the code.language action.

Totally agree. If the problem is that forced language should take precedents over an application specific language then we just need to make the matcher of the forced context more specific. mode: all or something.

I personally think an app that wants to force a language should override the code.language action.

Hrm ya that makes sense, is simpler. I'll change it.
Will try the 'mode: all' thing as well, as ya I think we still want forced language to take precedence over overridden code.langauge

I will look into if we can make the forced override more specific and always win over application specific

To be clear we tried mode: all and a couple other things in the community backlog session and couldn't get it to work.
A temporary workaround if you've already got code.language overrides is to do something like this:
https://github.com/nriley/talon_community/commit/b2d584a04805ad65ad411f3ed2fde13f9ca9ae19
but @AndreasArvidsson wasn't too excited about having to do that everywhere and neither am I…

It appears that the app context has a really high priority. For now I would probably recommend putting not tag: user.code_language_forced on the context where you override code.language

Do you think creating another context for this is better than calling actions.next? :-) thought we were trying to avoid having to make changes every time you override code.language

I'm OK with the not tag solution! On the community backlog session apparently people are confused by actions.next...

Finally got time to circle back to this. I use the code.language override with a context that uses not forced language for obsidian, as suggested. I left in the filename special casing in language_modes.py.


Validation: date_format:c says incorrect format for 2019-03-20T15:23:20+05:30 or similar

The PHP date() man page says that the format 'c' is ISO8601. However, when passing a string like the man says "2004-02-12T15:19:21+00:00", the validation says that the format is incorrect.
I had to replace for 'Y-m-d\TH:i:sP'

Hi there,
Thanks for reporting but it looks like this is a question which can be asked on a support channel. Please only use this issue tracker for reporting bugs with the library itself. If you have a question on how to use functionality provided by this repo you can try one of the following channels:

Laracasts Forums
Laravel.io Forums
StackOverflow
Discord
Larachat
IRC

However, this issue will not be locked and everyone is still free to discuss solutions to your problem!
Thanks.

Hi @driesvints , it was not a question, it was a report, as the validation fails for standard input.

@JorgeSivil I don't think this is related to Laravel. Please try one of the support channels first.


net: context: Set target network interface in send if needed

If we are sending a network packet and if the remote address
is not set in the context (which means that bind() has not
been called), then we must set the target network interface
to a proper value. This is required if we have multiple network
interfaces in which case the packet would be always assigned to
first network interface if remote address is unspecified.
Noticed this while debugging #32750. This issue has been undetected earlier but after commit 157c11d6e7072b5cf1ae8816a5673147c0e587f5 we started to return -ENETDOWN if network interface is down. In my tests, I had several network interfaces, but sometimes when using the net udp command, the sending failed because the first interface was down. In that case, the packet was meant to be sent to the second interface so it was quite mysterious why that happened.
The fact that the network packet was sent to wrong interface might explain some weird routing issues with VLANs.

Now I'm thinking about potential functional change with this patch. Let's assume the device has two network interfaces A with IP address fd00::1/64 and B with IP address fd01::5/64. The application creates an UDP socket and binds it to address fd00::1, the context interface is set to A. The application receives a frame with destination address fd00::1 and source address fd01::2. The application sends a response with destination address fd01::2 and source address fd00::1.
I think that without this patch the response would be sent through interface A, but with this patch interface B would be selected.
I think that proposed this change is correct (the output interface for response should be selected by the destination address and routing rules, not matching the interface from which request was received). However I would like to verify if my understanding is correct and if this change intentional.

The application creates an UDP socket and binds it to address fd00::1

Note that the patch only handles the case where the application has not bound to specific src address. Your use case should work just fine. After some more testing, the issue seems to be more related to IPv4 which does not work properly without this patch.

Note that the patch only handles the case where the application has not bound to specific src address

Does it? This patch checks remote address of a net context, while bind() function sets local address. My understanding is that it applies to sockets that are bound, but not connected.

This patch checks remote address of a net context, while bind() function sets local address. My understanding is that it applies to contexts that are not connected, including bound contexts.

Sure, but we do a default bind if user has not done it already in order to select the source interface. If user has not bound, then context->remote is not set which then causes problems when network interface is selected later. Setting the interface here helps later in the code.

@jukkar, forgive me for digging deeper in this patch, but I would like to seize the opportunity to learn more about network stack while I'm reviewing.

If user has not bound, then context->remote is not set

But if user has bound, context->remote is not set as well. Or am I missing something?

Setting the interface here helps later in the code.

I agree and I have not marked -1 because of this. However, I'm interested in side effects of this change and I would like to verify if my understanding of network components, I've been gaining recently, is correct.
I don't plan to mark this PR as +1, because I don't feel competent enough. I think other reviewers should take a look as well.

But if user has bound, context->remote is not set as well. Or am I missing something?

No you are not missing anything. I seem to have used wrong wording, I will revisit this PR tomorrow, it has been a long day and need to some rest.

I clarified the comments and commit message. Basically when I was talking about binding, I should have said connecting. So as Hubert said, the context->remote is related to target address which could have been set by connect() call.
So if we have not called connect(), then when sending a packet (UDP in this case), the target network interface might be missing when we are figuring out the local address to use (default bind if user has not bound it, which is typically the case with UDP and sendto()).

Thanks for the explanation. Now I think I understand what's going on.
I'm thinking if this change should be broader. I mean, for each packet being sent the output interface should be selected when the packet is added to TX queue, not when the socket is connected. In current implementation, when the socket is connected one interface is selected for this socket based on some rules. However, the reasons to use particular interface to reach given destination may change (network addresses, routing topology) while socket is connected. If such reasons change, packet sent through given socket should start using other interface than previously, what seems missing in current code.
It's obviously out of scope of this PR. But maybe we could start thinking of network stack refactoring, to easily support devices which from network point of view are routers with dynamic routing tables.

I still miss one +1, anyone?


Fix #3, init DstFile and SrcFile to NULL, Validate DstFile and SrcFil…

Describe the contribution
Fix #3, initialize DstFile and SrcFile to NULL. Validate before closing.
Testing performed

Build
Execute with ddd
Forced Return status of AllocateSectionHeader() to -1
Verify that git dstfiledesc was NULL and did not cause program to crash.

Expected behavior changes
No impact to behavior
System(s) tested on:

Hardware
Ubuntu 18.04.03
cFE 6.7, elf2cfetbl 3.1

Contributor Info
Anh Van, NASA Goddard
Community contributors
You must attach a signed CLA (required for acceptance) or reference one already submitted

txt file removed


fix(website): document missing cli command

Seems the instructions are missing the cli command

$ npx @park-ui/cli add button card
park-ui <command>

Commands:
  park-ui components add [components..]  Add components to your project

Options:
  --version  Show version number                                       [boolean]
  --help     Show help                                                 [boolean]

Unknown arguments: add, button, card

@fabianmarz
Thanks mate <3


Backslashs in documentation

There are several backslashs in the highlighteds commands:

Section DOWNLOAD THE BLOB: "Download blobs to your local disk using the get_blob_to_path method."
Section CLEAN UP RESOURCES:  "... using the delete_container method. To delete individual files instead, use the delete_blob method."

I am viewing this site using Firefox

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 15591874-317e-ce4c-0dff-20d06b625e40
Version Independent ID: be7fed9a-343a-0ec7-a94d-02c354d1ae33
Content: Azure Quickstart - Create a blob in object storage with Python
Content Source: articles/storage/blobs/storage-quickstart-blobs-python.md
Service: storage
GitHub Login: @tamram
Microsoft Alias: tamram

@1-Peter, thanks for your feedback. I've updated the article and removed the backslashes. We'll now close this issue.
cc: @tamram,


Change the way how request body is sent out

At this moment post body is encoded and concatenated in url
see: https://github.com/heresydev/stripy/blob/1.2.1/lib/stripy.ex#L36
If the request contains a large body(>5000 characters), it will be rejected by Stripe with
{
  "body": {
    "error": {
      "message": "Invalid string: Lorem Ipsu...rem Ipsum.; must be at most 5000 characters",
      "param": "line_items[0][description]",
      "type": "invalid_request_error"
    }
  }
}

I belive it is better for https://github.com/heresydev/stripy/blob/1.2.1/lib/stripy.ex#L65 to send out the actual body, rather than just throw everything to the url.

I think this should be a matter of checking if the request is :post and passing the payload in the body, rather than encoding it in the URL.
I don't have much free time to work on this at the moment, so feel free (or anyone else) to send a PR with the relevant changes. I'd be happy to help if anyone needs guidance starting on this.


Feature request?: create sub task for stories

Expected Behavior
Be able to create sub tasks for a story.
Actual Behavior
Unable to find this feature and am not sure if it is there and I am just blind or whether this is a feature request.
Specifications

Chronos-timetracker version: (unable to find this)
OS: Windows 10

Not waiting any longer. Closing thread.


[CT-1013] [CT-989] Support insert_overwrite strategy with delta

Is this your first time submitting a feature request?

[X] I have read the expectations for open source contributors
[X] I have searched the existing issues, and I could not find an existing issue for this feature
[X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature
Previously, delta didn't have support for dynamic partition overwrite which prevented it to be used with the insert_overwrite strategy.
The recent 2.0 release of delta added support for dynamic partition overwrite mode for partitioned tables.
So, it would be great if dbt could allow insert_overwrite when file_format is delta.
Describe alternatives you've considered
No response
Who will this benefit?
No response
Are you interested in contributing this feature?
No response
Anything else?
No response

Hey @flvndh, welcome back! I remember you were working on this last year: #186

The recent 2.0 release of delta added support for dynamic partition overwrite mode for partitioned tables.

Huzzah! That's great news. The only change we'll need to make is removing the validation logic here:
https://github.com/dbt-labs/dbt-spark/blob/7f6cffecf38b7c41aa441eb020d464ba1e20bf9e/dbt/include/spark/macros/materializations/incremental/validate.sql#L50-L52
I'm going to transfer this one to dbt-spark, and mark it a good first issue :)

Fantastic! I'm gonna do that ASAP 😊

Hi there. I'm just wondering if there is any progress on this one? Would be super useful.

Per this:

It appears that you cannot do these Delta overwrite strategies for whatever reason at the same time the database is configured with access control list.
spark.databricks.acl.dfAclsEnabled true

When I flip this to false, desta-overwrites work!! But, grants STOP working.
org.apache.spark.SparkException: Trying to perform permission action on Hive Metastore

Likewise, if I turn it back on, I run into the really weird cryptic errors that I’ve been dealing with for a while on the database cluster for delta dynamic overwrites.
I only found this out by finding an obscure unanswered question on the databricks forum plus a user of dbt five months ago struggling to solve this in the community Slack with no real resolution.
This may be a bug on databricks' side or that something to do with the implementation of a databricks environment prevents the simultaneous presence of these options. Either way it's not documented as to what is happening as far as I can tell. If there ends up being a solution I'd love to push that in and get this merged with future
Note, Setting the partition overwrite option to "dynamic" in the cluster set up does not resolve this issue


⚠ SceneTime has degraded performance

In 2635a61, SceneTime ($ST) experienced degraded performance:

HTTP code: 200
Response time: 11241 ms

Resolved: SceneTime performance has improved in 0a71ab2 after 7 minutes.


Questsxdungeonsxl

**What is your Quests version / build number (do not say "latest")?**Quests 3.9.2 DungeonsXLquestsmodule 1.1
Describe the problem and how to replicate it. Any console errors to post using Pastebin.com? I’m trying to have a quest be put in the dungeons I’ve set the quest start with an npc but quests don’t show up when we test the dungeon out.
If applicable, can you provide an example quest from quests.yml _using pastebin.com yes

Thanks for the report. Issues with that module should be opened at https://github.com/PikaMug/DungeonsXLQuestsModule/issues


doesn't work with patch 1.61 anymore

Love the mod, thanks your hard work on it!
The mod used to work just fine for me before 1.6. Now with patch 1.61 the game refuses to start if I have the mod active in Vortex (updated everything else), but it starts with it deactivated (deploying after each change, of course). If enabled, the game exe starts foa second or two & then quietly crashes. There's no graphics mode switching, no dialog messages, nothing. It just disappears from the task manager a second or so after having launched the game.
I'll gladly provide more information if necessary; just tell me what & where to find it. Thanks!

Oops, I confused the error message situation with another mod.
in_world_navigation does cause an error message to be printed by the RED script compiler:

The newly released 0.0.5 works fine with patch 1.61 again (remember to also update its dependency "Mod Settings", which has also just been updated!). Thank you so much!


Auto submit bot can revert correctly due to merge queue

Type of Request
bug
Infrastructure Environment
github, autosubmit bot
What is happening?
https://github.com/flutter/flutter/pull/159354
The bot created the revert and tried to merge the revert with merge queue. The bot then close the pr instantly before waiting for the merge queue to merge the pr. I had to manually reopen the revert and merge myself.
Steps to reproduce
use revert label on a merged pr.
Expected results
should work correctly

cc @yjbanov

I think this might be fixed by https://github.com/flutter/cocoon/pull/4043

That's correct. Deduping into https://github.com/flutter/flutter/issues/159365


Any way to just check if local cache exists?

Description
I'm writing a photo collection app and at some point I'd to know if an image can be shown right away. Is there any fast (non-Task, non-queued) way to check this?
Placeholders and similar solutions can be a workaround but it doesn't integrate well with my current code flow.
Steps to Reproduce

Expected Behavior

Actual Behavior

Basic Information

Version with issue:
Last known good version:
Platform:

Screenshots

Reproduction Link / Code

Never mind. I went to source and found cache checking logic.


add formats that make sample (or other) ids explicit

@colinvwood and @Oddant1 noticed while working on https://github.com/bokulich-lab/q2-moshpit/issues/63 and https://github.com/bokulich-lab/q2-assembly/pull/46 that many of the formats in q2-types-genomics require some knowledge of the underlying format to identify the sample ids. This is requiring code like:
result = ContigSequencesDirFmt()

for sample_fp in samples:
    # These paths are defined in the ContigSequencesDirFmt class as
    # {sample_id}_contigs.(fa | fasta). This should get the id from a
    # name like that
    sample_id = sample_fp.name.rsplit("_contigs", 1)[0]

to crop up fairly frequently (that example is from https://github.com/bokulich-lab/q2-assembly/pull/46). If a developer misses this, or doesn't handle the stripping of _contigs correctly (e.g., replacing all occurrences, rather than just stripping the last occurrence) downstream results could have misnamed sample ids (e.g., sample-1 could easily become sample-1_contigs, or sample2_contigs [a terrible sample name] could become sample2).
We're thinking that, after the alpha release, it will make sense to define new formats that contain manifests which explicitly map sample ids to filenames, and then add transformers from these existing formats to the new manifest formats. That way we encapsulate all knowledge about how the filenames map to sample ids in those transformers, and actions that need these data can use the manifest formats and have explicit access to sample ids. This would be similar to how we handle the "demux formats" in q2-types (see an example of how this is used here).
We would want to keep the existing formats as-is so that existing artifacts (generated pre- or post-alpha) would continue to work.

@ebolyen pointed out that we can add a method on the format that gives us tuples or a dict mapping sample ids to relative filepaths in the Archive, so that will achieve the same goal without having to create new manifest formats. So we should do that for the next release (2023.11), and take a pass to use these throughout in places where we're parsing filenames to access the sample ids.

Sounds good - thanks @gregcaporaso and @ebolyen! 🙏

After looking at the places in q2-assembly where we could use this new approach, I think that things will be much more refactorable once we have corresponding functionality for the mag directory format(s).

Ok, makes sense. Do we have an issue for updating the MAG directory format(s)? Could you create one if not?

Changes in q2-assembly are (at least partially) blocked by bokulich-lab/q2-types-genomics#66


kopf subhandlers miss some events

Expected Behavior
when watched resource is updated several times, all subhandlers must run for at least the latest state of the watched resource
Actual Behavior
some subhandlers are not executed at all when the watched resource is updated quickly enough several times
Steps to Reproduce the Problem
(using kopf namespace and kopfexamples custom resource)

minimal controller

$ cat subhandlers.py
import functools
import kopf

N_HANDLERS = 5

@kopf.on.update("zalando.org", "v1", "kopfexamples")
@kopf.on.create("zalando.org", "v1", "kopfexamples")
async def ensure(body, logger, event, **kwargs):
    fns = {s: functools.partial(handle, s) for s in range(N_HANDLERS)}
    await kopf.execute(fns=fns)
    return {"message": f"{event}d"}

async def handle(s, logger, event, spec, **kwargs):
    logger.info(f"Handler {s} for event {event} with field {spec['field']}")

run it with kopf run --standalone -n kopf subhandlers.py
create example resource

apiVersion: zalando.org/v1
kind: KopfExample
metadata:
  name: kopf-example-1
  namespace: kopf
spec:
  field: value

quickly patch it with 2 different values for spec.field:

kubectl -n kopf patch kopfexamples.zalando.org kopf-example-1 --type json -p='[{"op": "replace", "path": "/spec/field", "value": "value1"}]' && kubectl -n kopf patch kopfexamples.zalando.org kopf-example-1 --type json -p='[{"op": "replace", "path": "/spec/field", "value": "value2"}]'

see controller logs:

[2019-12-23 17:38:58,993] kopf.reactor.activit [INFO    ] Initial authentication has been initiated.
[2019-12-23 17:38:59,003] kopf.activities.auth [INFO    ] Handler 'login_via_pykube' succeeded.
[2019-12-23 17:38:59,004] kopf.reactor.activit [INFO    ] Initial authentication has finished.
[2019-12-23 17:39:28,932] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 0 for event create with field value
[2019-12-23 17:39:28,933] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 'ensure/0' succeeded.
[2019-12-23 17:39:29,056] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 1 for event create with field value
[2019-12-23 17:39:29,057] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 'ensure/1' succeeded.
[2019-12-23 17:39:29,167] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 2 for event create with field value
[2019-12-23 17:39:29,169] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 'ensure/2' succeeded.
[2019-12-23 17:39:29,279] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 3 for event create with field value
[2019-12-23 17:39:29,280] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 'ensure/3' succeeded.
[2019-12-23 17:39:29,389] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 4 for event create with field value
[2019-12-23 17:39:29,392] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 'ensure/4' succeeded.
[2019-12-23 17:39:29,392] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 'ensure' succeeded.
[2019-12-23 17:39:29,393] kopf.objects         [INFO    ] [kopf/kopf-example-1] All handlers succeeded for creation.
[2019-12-23 17:39:45,963] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 0 for event update with field value1
[2019-12-23 17:39:45,970] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 'ensure/0' succeeded.
[2019-12-23 17:39:46,094] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 1 for event update with field value1
[2019-12-23 17:39:46,096] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 'ensure/1' succeeded.
[2019-12-23 17:39:46,245] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 2 for event update with field value2
[2019-12-23 17:39:46,247] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 'ensure/2' succeeded.
[2019-12-23 17:39:46,361] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 3 for event update with field value2
[2019-12-23 17:39:46,363] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 'ensure/3' succeeded.
[2019-12-23 17:39:46,474] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 4 for event update with field value2
[2019-12-23 17:39:46,475] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 'ensure/4' succeeded.
[2019-12-23 17:39:46,476] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 'ensure' succeeded.
[2019-12-23 17:39:46,477] kopf.objects         [INFO    ] [kopf/kopf-example-1] All handlers succeeded for update.

Note that the subhandlers 0 and 1 were never called with spec.field = value2 and missed that update.
That creates quite a trouble when actual subhandlers create / update other resources (derived from the spec of the watched resource) as they potentially miss updates.
Specifications

Platform: Ubuntu 18.04
Kubernetes version: v1.15.3
Python version: 3.7.3
Python packages installed: aiohttp==3.6.2,aiojobs==0.2.2,async-timeout==3.0.1,attrs==19.3.0,certifi==2019.11.28,chardet==3.0.4,Click==7.0,idna==2.8,iso8601==0.1.12,kopf==0.24,multidict==4.7.2,pykube-ng==19.12.0,PyYAML
==5.2,requests==2.22.0,typing-extensions==<IP_ADDRESS>,urllib3==1.25.7,yarl==1.4.2

when starting the 'subhandler' functions manually (w/o kopf.execute) still an interesting behaviour occurs:
it seems that each 'retry' of main handler is using the latest watched object state - and may be this is the reason why the behaviour described in the issue manifests.
Consider the following handler code (slight modification of the previous example):
import functools
import kopf

N_HANDLERS = 5

@kopf.on.update("zalando.org", "v1", "kopfexamples")
@kopf.on.create("zalando.org", "v1", "kopfexamples")
async def ensure(body, logger, event, **kwargs):
    fns = {s: functools.partial(handle, s) for s in range(N_HANDLERS)}
    for n, f in fns.items():
        await f(logger, event, body, **kwargs)
    return {"message": f"{event}d"}

async def handle(s, logger, event, body, spec, **kwargs):
    if s == 1 and kwargs['retry'] == 0:
        raise kopf.TemporaryError(f"Handler {s} failed, will retry")
    logger.info(f"Handler {s} for event {event} with field {spec['field']}")

Running with the same repro procedure as described already produces following logs:
2019-12-24 09:04:09,829] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 0 for event update with field value1
[2019-12-24 09:04:09,829] kopf.objects         [ERROR   ] [kopf/kopf-example-1] Handler 'ensure' failed temporarily: Handler 1 failed, will retry
[2019-12-24 09:05:09,994] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 0 for event update with field value2
[2019-12-24 09:05:09,994] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 1 for event update with field value2
[2019-12-24 09:05:09,994] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 2 for event update with field value2
[2019-12-24 09:05:09,995] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 3 for event update with field value2
[2019-12-24 09:05:09,995] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 4 for event update with field value2
[2019-12-24 09:05:09,996] kopf.objects         [INFO    ] [kopf/kopf-example-1] Handler 'ensure' succeeded.
[2019-12-24 09:05:09,996] kopf.objects         [INFO    ] [kopf/kopf-example-1] All handlers succeeded for update.

Since the exception was not raised second time (for value2 in 'subhandler' 1) it looks like it was already in 'retry' mode due to previous exception - but already uses new values from the latest spec of watched object.

Looks like a known problem (with no issue though, so this one will be it). I've made few attempts to solve this problem previously: #9, and then again in #163 — but it failed spectacularly after the merge (had to revert it in #182).
The fix should be implemented again sooner or later, but extra care is needed for the cases where it failed in the previous attempts.
Thanks for this detailed report and a sample scenario to reproduce — it will help when the fix will be implemented the next time.

It looks the bug doesn't allow to watch for changes of specific fields of object like status. As we might miss specific status update event, and latest change will not be handled at all.


Question about the meaning of guidance in model forward

I read the code about flux and found that there are two guidance parameters during model inference, one is guidance_vec and the other is true_gs.
def forward(
        self,
        img: Tensor,
        img_ids: Tensor,
        txt: Tensor,
        txt_ids: Tensor,
        timesteps: Tensor,
        y: Tensor,
        guidance: Tensor | None = None,    ## <---- this one
    ) -> Tensor:
        if img.ndim != 3 or txt.ndim != 3:
            raise ValueError("Input img and txt tensors must have 3 dimensions.")

        # running on sequences img
        img = self.img_in(img)
        vec = self.time_in(timestep_embedding(timesteps, 256))
        if self.params.guidance_embed:
            if guidance is None:
                raise ValueError("Didn't get guidance strength for guidance distilled model.")
            vec = vec + self.guidance_in(timestep_embedding(guidance, 256))
        vec = vec + self.vector_in(y)

guidance_vec = torch.full((img.shape[0],), guidance, device=img.device, dtype=img.dtype)
for t_curr, t_prev in zip(timesteps[:-1], timesteps[1:]):
    t_vec = torch.full((img.shape[0],), t_curr, dtype=img.dtype, device=img.device)
    pred = model(
        img=img,
        img_ids=img_ids,
        txt=txt,
        txt_ids=txt_ids,
        y=vec,
        timesteps=t_vec,
        guidance=guidance_vec,
        image_proj=image_proj,
        ip_scale=ip_scale,
    )
    if i >= timestep_to_start_cfg:
        neg_pred = model(
            img=img,
            img_ids=img_ids,
            txt=neg_txt,
            txt_ids=neg_txt_ids,
            y=neg_vec,
            timesteps=t_vec,
            guidance=guidance_vec,
            image_proj=neg_image_proj,
            ip_scale=neg_ip_scale,
        )
        pred = neg_pred + true_gs * (pred - neg_pred)

true_gs is used for denoising, which is the famous CFG, which I understand.
But guidance_vec, called guidance in the model forward function, seems to control the time step embedding. My question is what is the role of this guidance. I don’t seem to find a clear reference, and it’s hard for me to understand how this parameter works during training.
If anyone can answer, I’d be grateful!

Sorry, I checked the code again and found that the use of the two guidance are in x-flux codes. The true_gs parameter does not exist in the flux code of BFL.
I'll leave this question for people who have the same doubts.
In the BFL code, this guidance is actually CFG, but when distilling the model, it is turned into an embedding to learn the result of the teacher model adjusting the CFG parameters. Therefore, after the distillation is completed, that is, in the inference of flux-dev, it is not necessary to use CFG inferencing twice (conditional and unconditional) to get the result, thereby speeding up the entire inferencing process.
At the same time, the x-flux code changed true_gs from 4 to 1 in one submission, which should be the reason.

issue closed

Hi @huayecaibcc, I am also confused about guidance_vec. Why can it achieve the effect of distillation when it is added? I see it only adds to the timestep.


Add global cache key to invalidate all existing cache

Hi,
Context
Sometimes the cache can get into an non ideal state, and we want to nuke all of them. However that is not always feasible because there might be local cache on many hosts. It can be time consuming to manually do it, or potentially dangerous to automate for deleting the wrong thing.
Expected Behavior
Therefore I think it would be handy to add a global cache key, so when we bump it, it will invalidate all the existing cache, so we don't have to worry about the stale ones.
Same as --cache-key-gen-version in Pants, http://www.pantsbuild.org/options_reference.html
Thanks,
Yi

Hi @wisechengyi!
Do you have an example of an 'non ideal state' that would not be fixed by 1) fixing the task implementation, 2) adding inputs to the task, 3) adding/removing outputs of the task or changing the build script in some way?

Yes gladly :) As an occasional Pants developer, we ran into similar scenario before:
For example, it can get tricky when gradle's own code changes, for some task:
Before the change: input X, output Y, stores it in cache
After: input X should result output Y'
The cache key is the hash of input (+ others we assume remain constant here), so after an gradle upgrade, it sees the input is still X, then go ahead and fetch the cache for X which is Y, whereas Y is not correct anymore.
Yes this can be fixed by fingerprinting the task itself into the cache key. Although in practice, there can be here and there we can miss, and having this feature would be handy.

The version of Gradle is built into the cache key, so upgrading Gradle would generate a new cache key.
The implementation of the task and any of the task's actions are also a part of the cache key.  So if you change the code for the task, we would generate a new cache key. This happens automatically for all tasks.

Although in practice, there can be here and there we can miss

I think you meant "we can miss inputs"?  In that case, fixing the task (adding those inputs) would change the cache key.
Does that cover what you were thinking?

Yes in this particular case.

I think you meant "we can miss inputs"? In that case, fixing the task (adding those inputs) would change the cache key.

Correct, but fixing the tasks and deploy them takes a longer time than bump the cache key. Iterating in this mode isn't feasible from operation's standpoint. We want to deliver the workaround asap then work on the actual fix.
Given the gradle version is already in the cache key, it should not be hard to inject another user defined key, and it will give us the leeway to get out of sticky situations.

I don't yet understand the use of this global key yet. What would one-off-fixing the cache do for you? If your task is broken then the cache will go bad again. There is no way around fixing the broken task.
That being said, this requirement is trivial to implement without being a core feature. Just add a project property as an input to every task.

Thanks for the pointer, @oehme! Looks like I can use that for https://issuetracker.google.com/issues/62678772 :)
I agree it is generally not needed, although the point is that it would be handy in those sticky situations, as a plan B, from operation's point of view.
Another scenario is that cache could get corrupted, which does happen occasionally and it is hard to determine which one is bad, so the question is: does gradle explicitly verify the integrity of the cache?

which does happen occasionally

You mean you've seen Gradles cache get corrupted? If so please open an issue with more details, we'd be very interested in that.
I'm closing this issue as the requested feature is already there in the form of input properties.

tasks.all {
  inputs.property "globalKey", findProperty("globalKey")
}
gradle someTask -PglobalKey=12345

does not seem to invalidate the cache...i'll try to attach the repro in a bit.


ucfirst for correct filenames

Hey,
had some issues with failed includings.
i think line 69 of Validator.module must be using this
require_once(wire('config')->paths->Validator . 'lib/' . ucirst($validator) . 'Validator.php');

Hi, I just merged the depending PR. My local system is case insensitive so I didn't notice this. Thanks!


chore: bdd-spring-1574857502 to 0.0.1

chore: Promote bdd-spring-1574857502 to version 0.0.1

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approvers:
If they are not already assigned, you can assign the PR to them by writing /assign  in a comment when ready.
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment


Question - Proxy User Support?

Great library but I'm struggling to see how to connect using a Proxy User, it looks like it is implemented but could you include an example of how it should be used - it looks like I can't just pass the proxy user as part of the OracleConnection.Configuration?

Thanks, although Proxy User isn't part of an initializer on OracleConnection.Configuration you're able to set it after initialization, e.g.:
var config = OracleConnection.Configuration(
    host: "my_host",
    service: .serviceName("my_service_name"),
    username: "my_username",
    password: "my_password"
)
config.proxyUser = "my_proxy_user"
try await OracleConnection.connect(configuration: config, id: 1, logger: logger)

Let me know if this works for you.

Thanks for the quick response, unfortunately still can't connect (and I know the parameters are correct), I'm getting a OracleSQLError(code: uncleanShutdown) error.

Just as a follow-up, I'm trying to connect to a 19c database, but noticed in the Constants.swift file it only lists 11R1, 11R2 and 12C, is this the problem (i.e. 19c not supported yet?)
// MARK: Verifier types
static let TNS_VERIFIER_TYPE_11G_1: UInt32 = 0xb152
static let TNS_VERIFIER_TYPE_11G_2: UInt32 = 0x1b25
static let TNS_VERIFIER_TYPE_12C: UInt32 = 0x4815

Thanks for the quick response, unfortunately still can't connect (and I know the parameters are correct), I'm getting a OracleSQLError(code: uncleanShutdown) error.

You might have to disable OOB, I needed to do this in a 19c database once. So it might solve your problem, too.
Just add the following line after config.proxyUser = ... config.disableOOB = true (I should add this to the documentation)
If it still doesn't work, you can pass in the environment variable LOG_LEVEL=trace, it would help me to know where to problem is exactly.

Just as a follow-up, I'm trying to connect to a 19c database, but noticed in the Constants.swift file it only lists 11R1, 11R2 and 12C, is this the problem (i.e. 19c not supported yet?)
// MARK: Verifier types static let TNS_VERIFIER_TYPE_11G_1: UInt32 = 0xb152 static let TNS_VERIFIER_TYPE_11G_2: UInt32 = 0x1b25 static let TNS_VERIFIER_TYPE_12C: UInt32 = 0x4815

11G and upwards are supported, I'm using the driver with 19, 21 and 23 currently. The verifier is just an internal implementation detail, which is different in some db versions.

Adding disableOOB didn't seem to make any difference, here's the code i'm using (it's in a SwiftUI App within a Task section) -
do {
  // check LOG_LEVEL
  let logger = Logger(label: "oracle-logger")
  if let logLevel = ProcessInfo.processInfo.environment["LOG_LEVEL"] {
    logger.info("LOG_LEVEL = \(logLevel)")
  }

  logger.info("-- 1 --")
  var config = OracleConnection.Configuration(
    host: "...",
    port: 1521,
    service: .serviceName("..."),  // or .sid("sid")
    username: "...",
    password: "..."
  )

  // set Proxy user and disable OOB
  config.proxyUser = "..."
  config.disableOOB = true
  logger.info("-- 2 --")

  let connection = try await OracleConnection.connect(
    configuration: config,
    id: 1,
    logger: logger
  )
  logger.info("-- 3 --")
  try await connection.close()

  logger.info("-- 4 --")
} catch {
  print(String(reflecting: error))
  print(error.localizedDescription)
}

which shows the following in the console -
2024-03-01T15:30:14+0000 info oracle-logger : [Oracle_Testbed] LOG_LEVEL = trace
2024-03-01T15:30:14+0000 info oracle-logger : [Oracle_Testbed] -- 1 --
2024-03-01T15:30:14+0000 info oracle-logger : [Oracle_Testbed] -- 2 --

OracleSQLError(code: uncleanShutdown)
The operation couldn’t be completed. (OracleNIO.OracleSQLError error 1.)

That's odd, there should be more console output. Could you try setting the log level directly on the logger?
var logger = Logger(label: "oracle-logger")
logger.logLevel = .trace
// rest of your code...

On another note, you have to enable outgoing connections in Singing & Capabilities -> App Sandbox -> Network -> Outgoing Connections (Client), if this is a macOS app. The error should be a timeout in that case, so I don't think that's the problem. But it might be worth to check.

 (OracleNIO.OracleSQLError error 1.)

Seems like the database server is closing the connection before any authentication happens at all, so the issue shouldn't be related to using a proxy user. It might be something completely unrelated. Can you set the environment variable ORANIO_TRACE_PACKETS to 1 and send the log output again?

 (OracleNIO.OracleSQLError error 1.)

I'll see if I can replicate in a VM too, by the way I can successfully connect using the exact same parameters in Oracle SQL Developer, so I know that connectivity etc is fine.

Very interesting, I hoped to see a packet sent by the db before the close happens. It's very likely that the db is closing the connection without any notice after the client is sending the required TNS protocol (>= 8.1) version and driver name. Maybe the db is rejecting the driver/client (I don't know if that's even a thing). Can you verify that's not the case? The driver name is oracle-nio.
If it still doesn't work, I'll ask someone at Oracle for further advice, because I've never encountered such a strange issue.
Thanks for sticking with me so far.

Thanks for continuing to look into it, I'll see if I can get anything from the Oracle logs side and let you know if I find anything

Did you find anything?

I still have the same issue and been unable to get to the bottom of it.
Feel free to close this issue and I'll post again if I make progress and / or manage to create a reproducible test case for you

Alright, I'll close the issue, just reply to the issue and @ me if you have something I can work with :)

@lovetodream I think I got to the bottom of this one.
Long story short the server was configured to enforce NNE (Native Network Encryption), via the following sqlnet.ora settings -
SQLNET.ENCRYPTION_SERVER=REQUIRED
SQLNET.CRYPTO_CHECKSUM_SERVER=REQUIRED
SQLNET.ENCRYPTION_TYPES_SERVER=(AES256,AES192,AES128)
SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER=(SHA256,SHA384,SHA512,SHA1)
SQLNET.ENCRYPTION_CLIENT=REQUIRED
SQLNET.CRYPTO_CHECKSUM_CLIENT=REQUIRED
SQLNET.ENCRYPTION_TYPES_CLIENT=(AES256,AES192,AES128)

Note - I'm not expecting you to support this, I believe Oracle now recommends TLS over NNE and I note that other Thin clients (such as the Python oracledb driver) don't support NNE either.

Oh, thanks for letting me know. Actually I had the boilerplate for NNE in the driver at the start, but the Oracle guys told me I should get rid of it. That was around the same time when they removed it from the thin python driver. They told me to support and encourage users to use TLS instead, just as you mentioned. I didn't know that could be the reason for your errors, but now I do know, thanks for that again.
Let me know if you're facing any other issues in the future!

The database I was testing against was actually in Oracle Cloud, so it was provisioned with NNE by default (and no TLS) - so I think it's a mistake by Oracle to remove it from their own Python driver (as lots of people would hit the same issue and then give up).
However on a positive note - your driver is working great on DBs that don't use NNE!
Sorry for any confusion, took me a while to track down this was the actual root cause (and not oracle-nio).

No worries, I thought Oracle Cloud uses mutual TLS by default nowadays. But I'm only using the free tier, so maybe there's a difference


Request: Verify ipfs.get by hashing in client

My understanding is that after getting a file from a node, there is no additional check to verify the returned file matches the hash. It would be good to hash the file in the client to make sure the file is correct. That way, it would be possible to use untrusted nodes (for example https://ipfs.infura.io)

IPFS definitely verifies data from other nodes. See white paper, section 3.5. I think https://github.com/ipfs/go-ipfs/blob/master/exchange/bitswap/message/message.go#L105 is where blocks from peers are checked. Am I misunderstanding your question?

Ipfs nodes don't trust each other, but js-ipfs-api trusts the node. For example if I do
ipfsApi = IpfsApi(host, port})
ipfsApi.get('Qm...')

I'm trusting that the node at host:portis not lying to me. This is fine if we assume the user is using a local node on their machine, however users could be using remote nodes. For example https://infura.io/.

Oh I see. The simplest solution I can think of is to expose ipfs add --only-hash in the js api so you can compare the two hashes.


Polymer support

Error messsage while starting ember:
"Found "dom-module" element. Vulcanize 0.7 does not support Polymer > 0.5 File:"
Why does it not support it?

dom-modules are supported in vulcanize > 1.0.


Feat/types/generate better get schema return

Just like #700 this PR  aims to make getSchema more specific by inferring the properties of the entity.
This PR overrides the type definition in the global.d.ts  and globs the ./types/{name} imports into a single index.d.ts.

I made the change as small as possible so others can expand upon this as I am a bit lost with JSON schema

Could you please fix the DCO metadata?

Could you please fix the DCO metadata?

What do you mean?

^
Follow the instructions [here](https://github.com/platformatic/platformatic/pull/728/checks?check_run_id=11564246226 (you can get by clicking on the DCO details).


Create API Documentation

To make it easier to create plugins and develop ULS, it's required to create API Documentation.

CLOSED
There will be only ONE version of API Documentation - https://login.ds-software.xyz/docs
It won't be open-source.


Legacy address in the Windows 2.21.0 Desktop Wallet not working.

Actual behavior:
After creating a legacy address in the Windows 2.21.0 Desktop wallet, backing up and closing the wallet the address is not there when you re-open the backup, when you try to add it sometimes it says it's there already, but does not show in the receive addresses and the balance does not show. Even if it allows you to add it manually, the balance never shows up. Rescan does not fix it.  I replicated this on a second Windows PC.
Expected behavior:
To keep all the addresses associated with it when you open it and to show balances for the accounts associated to it.
How to reproduce it (as minimally and precisely as possible):
Install Fresh Windows 10 and all updates including 22H2, instal the Windows 2.21.0 Desktop wallet, createda legacy address, backup and close the wallet, Open/Restore wallet, no addresses are showing. No balances show when you add the addresses manually.
What are your environment parameters?

DeFiCh/app Version: Windows 2.21.0 Desktop wallet
Operating System (OS): Windows 10 22H2
– RAM (GB): 32GB
– CPU (Intel Core i5/i7/etc.): Ryzen 5800
Current free storage capacity: 1.2TB

I received some assistance from Bernd on Telegram and it seems to be my misunderstanding.
I was not aware you needed to add the addresses after restoring, I was under the impression that once you restored the .dat that everything would be contained within and restored automatically. Once I figured out the correct .dat and added the 2 addresses belonging to it manually, it seems to be working OK and the balance is showing.


feat: move NodeBuilderWithDatabase to reth bin

Moves the NodeBuilderWithDatabase, along with some other types and methods that are only used by NodeBuilderWithDatabase and command execution, to bin/reth.
Fixes #6211

fixing conflicts


[PKG-2247] meson-python 0.13.1

Changelog: https://github.com/mesonbuild/meson-python/blob/0.13.1/CHANGELOG.rst
License: https://github.com/mesonbuild/meson-python/blob/0.13.1/LICENSE
Requirements: https://github.com/mesonbuild/meson-python/blob/0.13.1/pyproject.toml
Actions:

Remove build with patch an m2-patch
Add strong pinnings and update dependencies in host
Update run dependencies
Reorder run dependencies
Add pyproject.toml to source_files
cython is required for Python 3.12 support in tests
Move typing-extensions >=3.7.4  # [py<310] from run to test/requires
Skip lints

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_13ppmvwiq1/clone/recipe/meta.yaml:25: missing_wheel: For pypi packages, wheel should be present in the host section
Errors were found


Download page

This PR eliminates awkward word breaks.
It also better aligns checkboxes and text for a cleaner look.
I also disabled hyphens for the owners of plugins because I don't want to see these names chopped into pieces.
I have tested it on Chrome and FireFox.
Before

After

Thanks!


fix(examples): clean up trusted sync logging

#263  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @refcell and the rest of your teammates on  Graphite


Size of (IPC) decompressed data can be larger then the original data

(Don't really know if this is a bug or a feature)
Describe the bug
We have a use-case where we want to store Arrow data (StructArray) compressed in memory. Currently the best way for this appears to be the IPC FileWriter with compression and then storing the compressed bytes. For our use-case it is important to know that the uncompressed data has the same size in memory as the original input.
Using the FileReader we noticed that our assumption, that the decompressed data has the same size as the original input, does not hold.  There is a case that can cause the memory footprint of the decompressed data to become ~ "decompressed bytes + compressed bytes".
If CompressionCodec decompress_to_buffer detects an uncompressed slice of data (decompressed_length == LENGTH_NO_COMPRESSED_DATA) it returns a new Buffer that uses a pointer into the Buffer that holds the raw bytes produced by the reader. This causes the raw Buffer to be kept in memory. Additionally the compressed data slices are decompressed into new Buffers that use newly allocated memory.
To Reproduce
The easiest way is to create a StructArray with one column that is good to compress and one that is not:
use std::{io::Cursor, sync::Arc};

use arrow::{
    array::{Array, ArrayRef, Int32Array, StringArray, StructArray},
    datatypes::{DataType, Field, Fields},
    ipc::{
        writer::{FileWriter, IpcWriteOptions},
        CompressionType,
    },
    record_batch::RecordBatch,
};

fn main() {
    // Setup StructArray with two columns
    let primes = Arc::new(Int32Array::from(vec![
        31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79,
    ]));
    let strings = Arc::new(StringArray::from(vec![
        "Hello world one",
        "Hello world two",
        "Hello world three",
        "Hello world four",
        "Hello world five",
        "Hello world six",
        "Hello world seven",
        "Hello world eight",
        "Hello world nine",
        "Hello world ten",
        "Hello world eleven",
        "Hello world twelve",
    ]));
    let table = StructArray::new(
        Fields::from(vec![
            Field::new("primes", DataType::Int32, false),
            Field::new("strings", DataType::Utf8, false),
        ]),
        vec![primes as ArrayRef, strings as ArrayRef],
        None,
    );
    // Print the size of the original data
    println!("Original size: {:?}", table.get_array_memory_size());

    // Write to file with compression
    let record_batch = RecordBatch::from(&table);
    let mut file_writer = FileWriter::try_new_with_options(
        Vec::new(),
        record_batch.schema().as_ref(),
        IpcWriteOptions::default()
            .try_with_compression(Some(CompressionType::LZ4_FRAME))
            .unwrap(),
    )
    .unwrap();
    file_writer.write(&record_batch).unwrap();
    file_writer.finish().unwrap();
    let bytes = file_writer.into_inner().unwrap();
    // Print the size of the compressed data
    println!("Compressed size: {:?}", bytes.len());

    // Read from file and decompress
    let mut reader = arrow::ipc::reader::FileReader::try_new(Cursor::new(bytes), None).unwrap();
    let record_batch = reader.next().unwrap().unwrap();
    let sr = StructArray::from(record_batch);
    // Print the size of the decompressed data
    println!("Decompressed size: {:?}", sr.get_array_memory_size());
}

The output will be something like this:
Original size: 688
Compressed size: 950
Decompressed size: 1104

Expected behavior
We expected the decompressed data to use the same amount of memory as the original data.
An easy way to fix this would be to change the "uncompressed in compressed" case in decompress_to_buffer to return a real copy of the uncompressed data slice.
The easy fix, might cause a lot of memcopys in cases where compression was preferred but  a lot of data could not be compressed. To make the behavior a user choice, we could add a "copy_uncompressed" parameter to the decompress_to_buffer method. Since the user only has access to readers, this should probably be set as a reader option that is passed down to the decompression step...
Additional context

Ran into the same problem here. What's worse is that the difference between decoded data size and real data size grows disproportionately to number of fields in record batch.

use arrow_array::{ArrayRef, PrimitiveArray, RecordBatchWriter};
use arrow_ipc::writer::FileWriter;
use arrow_schema::DataType;
use datatypes::types::Int64Type;

fn encode_and_decode(num_fields: usize) -> (usize, usize) {
    let fields = (0..num_fields)
        .map(|idx| {
            arrow::datatypes::Field::new(
                &format!("f_{idx}"),
                arrow::datatypes::DataType::Float64,
                false,
            )
        })
        .collect::<Vec<_>>();

    let schema = Arc::new(arrow::datatypes::Schema::new(fields));

    let mut buffer = vec![];
    let mut writer = FileWriter::try_new(&mut buffer, &*schema).unwrap();

    let col_values = (0..num_fields)
        .map(|idx| {
            Arc::new(PrimitiveArray::<arrow_array::types::Float64Type>::from(
                vec![1.0],
            )) as ArrayRef
        })
        .collect::<Vec<_>>();
    let rb = arrow::record_batch::RecordBatch::try_new(schema.clone(), col_values).unwrap();

    let write_mem_size = rb.get_array_memory_size();
    writer.write(&rb).unwrap();
    writer.close().unwrap();

    let mut reader =
        arrow::ipc::reader::FileReader::try_new(std::io::Cursor::new(&buffer), None).unwrap();
    let rb = reader.next().unwrap().unwrap();
    let read_mem_size = rb.get_array_memory_size();
    return (write_mem_size, read_mem_size);
}

fn main() {
    for num_cols in (0..51).step_by(10).skip(1) {
        let (write, read) = encode_and_decode(num_cols);
        println!("{},{},{}", num_cols, write, read)
    }
}

Have you tried running with a memory profiler like heaptrack, the code you've provided will count the same allocations multiple times

@tustvold could you point to the multiple counting of the example above?
I see the original view of not making it a default, but I still would love to have some feedback on the compaction kernel vs. adding an option approach. If there is some common understanding, maybe I can try to come up with something.

get_array_memory_size returns the total capacity of all the buffers backing the array, e.g. https://github.com/apache/arrow-rs/blob/master/arrow-array/src/array/boolean_array.rs#L305C44-L305C66
If those buffers are slices of the same underlying allocation, as they will be in the above example, you will count that allocation multiple times.
FWIW it is relatively trivial to rewrite an array if you so wish
let data = array.to_data();
let mutable = MutableArrayData::new(vec![&data], false, 0);
mutable.extend(0, 0, array.len());
let compacted = make_array(mutable.freeze());

But at least in the example provided by @v0y4g3r the issue might simply be one of accounting


Fix for builds on arm32, arm64, ppc, etc

Fixes this build error on platforms where c_char is defined as u8:
error[E0308]: mismatched types
   --> /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/extendr-api-0.2.0/src/wrapper.rs:368:25
    |
368 |     unsafe { Rf_install(bytes.as_ptr() as *const i8) }
    |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `u8`, found `i8`
    |
    = note: expected raw pointer `*const u8`
               found raw pointer `*const i8`

The underlying Rf_install binding defines this as a C char, see e.g.: https://github.com/extendr/libR-sys/search?q=rf_install
That C char type can either be an i8 or a u8, depending on platform: https://doc.rust-lang.org/src/std/os/raw/mod.rs.html#48-90

Thanks for this @nickbp
I should have written this more formally in the first place.

Side note:
Do we still have that "myvar" test in there? It should have gone long ago.
We can't test the base_env() using just libR.

Yes, the myvar test needs to go, but that has nothing to do with this PR. See #211.

Good to merge? I'm not able to merge myself


Unable to refer classes from extenal plugins in other *.gradle files

Expected Behavior
Referring to classes from already applied plugins should work fine in other *.gradle files applied from build.gradle.
Current Behavior
Gradle fails to resolve a class from a plugin with:
> Could not get unknown property 'org' for root project 'plugin-class-other-script'
of type org.gradle.api.Project.

for any class reference (from not build.gradle), e.g.:
task fromOther(type: org._10ne.gradle.rest.RestTask) { ... }

It works fine if used from build.gradle or if in a separate file the whole buildscript section is repeated.
Context
To improve readability of the Gradle build it is useful to put connected parts to a separate *.gradle files (applied from build.gradle). Not all code can be moved due to the aforementioned limitation.
Steps to Reproduce (for bugs)
//build.gradle
buildscript {
  repositories { ... }
  dependencies {
    classpath "org._10ne.gradle:rest-gradle-plugin:0.4.2" //any external plugin can be used
  }
}

apply plugin: "org.tenne.rest"

task fromMain(type: org._10ne.gradle.rest.RestTask) { //works fine
  //...
}

apply from: 'gradle/other.gradle'

//other.gradle
task fromOther(type: org._10ne.gradle.rest.RestTask) { //fails
  //...
}

FAILURE: Build failed with an exception.

* Where:
Script '/tmp/plugin-class-other-script/gradle/other.gradle' line: 1

* What went wrong:
A problem occurred evaluating script.
> Could not get unknown property 'org' for root project 'plugin-class-other-script' of type org.gradle.api.Project.

Your Environment
All modern Gradle versions (tested up to 3.5). That issue was reported years ago, but was lost in the migration.

This seems to be the same as #1262 as you said @szpak. Let's continue discussion there. I'll close this as a duplicate.

After you edited the original issue it seems to be a very good idea.


New card : Create Defi Compliance card

see Figma for card and discover
credential type is DefiCompliance

issuer
same flow as tezotopia card with https://issuer.talao.co/defi/card

for testing
add mode=test in the call as an argument to the GET
https://issuer.talao.co/defi/card?mode=test


Update isolate-machine.md

Udpated important note about When Full Isolation or selective isolation is available (Min requirement)
reference - https://docs.microsoft.com/en-us/microsoft-365/security/defender-endpoint/respond-machine-alerts?view=o365-worldwide#isolate-devices-from-the-network

Issue#https://github.com/MicrosoftDocs/microsoft-365-docs/issues/5807

@JohanFreelancer9 Please copy review


How to keep data after browser refresh?

Everytime the browser refreshes, redux data is dumped. How can I stop this behaviour?

You can achieve this by using browser's local storage. I did this with redux-persist-immutable (check also redux-persist for more information.
Btw, it's probably not a good idea to use this to save state, it's not quite meant to be used this way. I ended up removing all persist stuff and redesigning functionalities around this problem.
And also, not related to react-boilerplate, should have asked this on stackoverflow. ;)

Thank you!
And sorry for the asking this here instead, next time i'll go to stackoverflow.


feat: versioned feature states

Thanks for submitting a PR! Please check the boxes below:

[x] I have run pre-commit to check linting
[x] I have filled in the "Changes" section below?
[x] I have filled in the "How did you test this code" section below?
[x] I have used a Conventional Commit title for this Pull Request

Changes

Opt into feature state versioning via environment settings
Create versioned segment overrides, environment feature state updates and multivariate options

What this does not fully cover, this should follow in an upcoming PR

Change Requests workflow for versioning
Creating/Deleting Multiviariate options
When enabled,

Please describe.
How did you test this code?

Created and updated feature state remote config

Updated feature enabled state

Created a segment override

Update segment override
Note: there is currently an API issue @matthewelwell is looking into.

Update multivariate weightings
Note: this will probably change in the future, different feature versions could technically have totally different multivariate options, this currently only diffs the options that exist in the live feature.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (76cc152) 96.10% compared to head (9d45fee) 96.12%.
Report is 28 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2688      +/-   ##
==========================================
+ Coverage   96.10%   96.12%   +0.01%
==========================================
  Files        1058     1059       +1
  Lines       31882    32210     +328
==========================================
+ Hits        30641    30961     +320
- Misses       1241     1249       +8

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

This is blocked waiting for a full manual E2E test (currently sitting with me).

Some FE tweaks required here @kyle-ssg

Create a flag without a config value
Toggle the flag
Add a config text value
Hit this console error and entire app crashes/whitescreens.

Above issues are resolved.

OK yep my issues are fixed. Is there anything else blocking this going live?

OK yep my issues are fixed. Is there anything else blocking this going live?

Well, yes - this

@kyle-ssg

My bad, was just a duplicate data-test attribute added from latest feedback to split audit log links from versioning links. I think we're good to go when tests pass.