BuildCache unpack and CleanupStaleOutputsExecuter fails on Windows with UnrecoverableUnpackingException due to file deletion fails

The BuildCache unpack operation fails sporadically on Windows machines with an IOException.
The class org.gradle.caching.internal.packaging.impl.PackerDirectoryUtil tries to delete directories, which fails sporadically in windows machines.
See Stacktrace below.
The same issue happens in the class org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter which also uses the org.apache.commons.io.FileUtils.forceDelete method.
See Stacktrace below.
Expected Behavior
The BuildCache should retry to delete the files. The file delete on Windows does not block until the files where deleted.
The class org.gradle.api.internal.file.delete.Deleter has already a fix for that issue in the method handleFailedDelete(). It would be best if the same code is called when tried to delete the Directories in
org.gradle.caching.internal.packaging.impl.PackerDirectoryUtil.ensureDirectoryForTree()
The method org.gradle.util.GFileUtils.forceDelete should also be changed to use the same logic as the Deleter.
Current Behavior
The org.gradle.caching.internal.packaging.impl.PackerDirectoryUtil class uses org.apache.commons.io.FileUtils.cleanDirectory but this does not handle the odd windows delete case correctly.
Context
The JavaCompile task aborts when the issue occurs.
The expected Compile-Artifacts must be in a remote build cache, which are downloaded.
This does not fail every time due to the timing issue between Windows and the Gradle Java process.
Steps to Reproduce (for bugs)

Java-Project
Remote-Build cache
Publish the artifacts in the build cache
Change the built artifacts of the JavaCompile task that it tries to clean them up
Execute gradlew compileJava multiple times

Your Environment
Windows 7 6.1
Gradle 5.2.0
Java Project
Exception stacktrace
BuildCache:
org.gradle.caching.internal.packaging.UnrecoverableUnpackingException: Failed to unpack trees for task ':project:compileJava', and then failed to
        at org.gradle.caching.internal.command.BuildCacheCommandFactory$LoadCommand.load(BuildCacheCommandFactory.java:134)
        at org.gradle.caching.internal.controller.DefaultBuildCacheController$Unpack$1.run(DefaultBuildCacheController.java:179)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:402)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:394)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
        at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:92)
        at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
        at org.gradle.caching.internal.controller.DefaultBuildCacheController$Unpack.execute(DefaultBuildCacheController.java:175)
        at org.gradle.caching.internal.controller.DefaultBuildCacheController$1.execute(DefaultBuildCacheController.java:143)
        ... 152 more
Caused by: java.io.IOException: D:\path\to\folder
        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1497)
        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2223)
        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1595)
        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1488)
        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2223)
        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1595)
        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1488)
        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2223)
        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1595)
        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1488)
        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2223)
        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1595)
        at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1488)
        at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2223)
        at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1595)
        at org.gradle.caching.internal.packaging.impl.PackerDirectoryUtil.ensureDirectoryForTree(PackerDirectoryUtil.java:30)
        at org.gradle.caching.internal.packaging.impl.TarBuildCacheEntryPacker.unpackTree(TarBuildCacheEntryPacker.java:239)
        at org.gradle.caching.internal.packaging.impl.TarBuildCacheEntryPacker.unpack(TarBuildCacheEntryPacker.java:198)
        at org.gradle.caching.internal.packaging.impl.TarBuildCacheEntryPacker.unpack(TarBuildCacheEntryPacker.java:158)
        at org.gradle.caching.internal.packaging.impl.GZipBuildCacheEntryPacker.unpack(GZipBuildCacheEntryPacker.java:49)
        at org.gradle.caching.internal.command.BuildCacheCommandFactory$LoadCommand.load(BuildCacheCommandFactory.java:103)
        ... 162 more

Stale output:
java.io.IOException: Unable to delete directory D:\path\to\folder.
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1623)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2391)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1721)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1617)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2391)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1721)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1617)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2391)
at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:1721)
at org.apache.commons.io.FileUtils.deleteDirectory(FileUtils.java:1617)
at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:2391)
at org.gradle.util.GFileUtils.forceDelete(GFileUtils.java:266)
at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter$1.run(CleanupStaleOutputsExecuter.java:80)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:402)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$RunnableBuildOperationWorker.execute(DefaultBuildOperationExecutor.java:394)
at org.gradle.internal.operations.DefaultBuildOperationExecutor$1.execute(DefaultBuildOperationExecutor.java:165)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:250)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.execute(DefaultBuildOperationExecutor.java:158)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:92)
at org.gradle.internal.operations.DelegatingBuildOperationExecutor.run(DelegatingBuildOperationExecutor.java:31)
at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:74)

I got the following information from @AndreasTu:

We cannot reproduce the problem consistently, yet.
It happens only on some machines and even there only sporadically.
We know that it happens more often if there is more IO activity.
We saw this phenomenon already with other Java programs trying to delete deep order structures.
My suspicion is the following, see the Windows-API documentation:
https://docs.microsoft.com/de-de/windows/win32/api/fileapi/nf-fileapi-removedirectorya
Remarks
The RemoveDirectory function marks a directory for deletion on close. Therefore, the directory is not removed until the last handle to the directory is closed.
Java.io.File.delete() calls WinNTFileSystem.delete().
This is called from Java native code:
https://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/windows/native/java/io/WinNTFileSystem_md.c

JNIEXPORT jboolean JNICALL
Java_java_io_WinNTFileSystem_delete0(JNIEnv *env, jobject this, jobject file)
{
    jboolean rv = JNI_FALSE;
    WCHAR *pathbuf = fileToNTPath(env, file, ids.path);
    if (pathbuf == NULL) {
        return JNI_FALSE;
    }
    if (removeFileOrDirectory(pathbuf) == 0) {
        rv = JNI_TRUE;
    }
    free(pathbuf);
    return rv;
}

static int
removeFileOrDirectory(const jchar *path)
{
    /* Returns 0 on success */
    DWORD a;

    SetFileAttributesW(path, FILE_ATTRIBUTE_NORMAL);
    a = GetFileAttributesW(path);
    if (a == INVALID_FILE_ATTRIBUTES) {
        return 1;
    } else if (a & FILE_ATTRIBUTE_DIRECTORY) {
        return !RemoveDirectoryW(path);
    } else {
        return !DeleteFileW(path);
    }
}

In the code which fails, FileHandles are used to traverse the directory contents to delete child files.

Let's use Deleter when cleaning outputs for the cache.

We also need https://github.com/gradle/gradle/issues/10227 and https://github.com/gradle/gradle/issues/10226 to close this.

@AndreasTu this should now be fixed on master. Can you please try if you can still reproduce the problem using the latest nightly Gradle version 5.7<PHONE_NUMBER>0048+0000?

@lptr thank you for the fixes. We will test it.

@AndreasTu I'm closing this as fixed now. Please reopen if your tests indicate that something's still not working.


Stacked bar chart PlotOptionsBarDataLabels MaxItems property not working

Hi all,
First of all thanks for this wrapper, I like it so much!
I would like to have a stacked bar chart with labels JUST when i have less than X elements. According to the description
(Maximum limit of data-labels that can be displayed on a bar chart. If data-points exceed this number, data-labels won't be shown.)
of PlotOptionsBarDataLabels.MaxItems, it should do that.
But if I add this property to the example code like MaxItems=1:

It still shows the labels.
Am I wrong, or i dont use it well?
Thanks for your response in advance!

Hello,
I'm sorry but I'm not familiar with the MaxItems property. I think it's probably better to ask in apexcharts.js repo.


Upgrade k8s.io/* to v0.26, sigs.k8s.io/controller-runtime to v0.14

How to categorize this issue?

/area open-source dev-productivity
/kind enhancement
What would you like to be added:
We should upgrade to the latest versions of our go upstream dependencies:

<EMAIL_ADDRESS>https://github.com/kubernetes-sigs/controller-runtime/releases/tag/v0.14.0
<EMAIL_ADDRESS>
Important changes / Action items
Here is a list of a few upstream changes to look out for when vendoring g/g, k/* and c-r in any of our repos (e.g. extensions). Please consider the release notes of<EMAIL_ADDRESS>and upwards as well for a more complete list and more details.
⚠ breaking / most notable changes

There is a new subresource client (https://github.com/kubernetes-sigs/controller-runtime/blob/ddcb99d6be40ce4b69d97505e4ef222eb053b14e/pkg/client/client.go#L296-L304) - let's make use of this and drop remaining usages of the client-go kubernetes clientset where possible
Instead of configuring RecoverPanic again and again per controller, let's configure it globally: https://github.com/kubernetes-sigs/controller-runtime/pull/2093
There is now a Not predicate which comes out-of-the-box: https://github.com/kubernetes-sigs/controller-runtime/pull/2031. Let's drop https://github.com/gardener/gardener/blob/759c2641dab95cba6653924e3462aeec2e1fe56d/pkg/controllerutils/predicate/predicate.go#L48-L64 and use this instead.

ℹ other changes / good to know / for information only:

It is now possible to add/remove event handlers on the cache during runtime: https://github.com/kubernetes-sigs/controller-runtime/pull/2046
The controller builder does not require For() anymore (ref https://github.com/kubernetes-sigs/controller-runtime/pull/2091). However, this seems to be buggy with 0.14.0, ref https://github.com/kubernetes-sigs/controller-runtime/issues/2101, so we cannot use it yet.
There is a new cluster.ClientBuilderWithOptions function (https://github.com/kubernetes-sigs/controller-runtime/pull/2054) which allows caching unstructured.Unstructured. Might be beneficial for gardener-resource-manager in the future.
Logging on event source start-up was improved 🎉 https://github.com/kubernetes-sigs/controller-runtime/pull/2057

"Add indexes to fake client to mimic field selectors" (https://github.com/kubernetes-sigs/controller-runtime/pull/2025) is also included, which can probably replace https://github.com/gardener/gardener/blob/b6cb42087674acc702495f347de0c3e07cf0d787/pkg/utils/test/client.go#L29-L37

/assign

The controller builder does not require For() anymore (ref https://github.com/kubernetes-sigs/controller-runtime/pull/2091). However, this seems to be buggy with 0.14.0, ref https://github.com/kubernetes-sigs/controller-runtime/issues/2101, so we cannot use it yet.

https://github.com/kubernetes-sigs/controller-runtime/issues/2101 is fixed in https://github.com/kubernetes-sigs/controller-runtime/pull/2105 and released with v0.14.1


SCRUM-26 Results Endpoint

lgtm, maybe findByUser can be named findByUserId, or use "findOptimizationResultsByUser_Id" from jpa so you dont need custom query but for MVP tommorow its fine


Reduce register pubkey batch sizes

The current implementation allows batch register of 16pubkeys which means that the coordinator needs to wait for 16 new create2txs before he can register the batch on-chain and process the transaction.
This means that once the user uses create2transfer to send someone payment his normal transfers won't get processed till the create2transfer goes through.

Can we have multiple register batches for different sizes?

Yeah, that sounds like a good idea. I propose we

Restore the account tree batch register size back to 1024 pubkeys. The batch register was designed to onboard a lot of users.
Create a medium size solution for our usual use cases.

The medium size solution is a contract wrapper that call AccountTree.registerSingle serval times. So we can save the 21000 gas for base ETH1 tx.

Keep the status quo before the devnet


Actually test that the error message matches expectations

Current dependencies on/for this PR:

main

PR #27   👈

PR #28

This comment was auto-generated by Graphite.

Merge activity

Oct 28, 9:06 PM: Graphite rebased this pull request as part of a merge.


matrixdropdown does not keep values selected

The typical example matrixdropdown has a few rows and columns.
Each has a select box with the same choices.
Now when one of the selectboxes is changed, all the selectboxes change to the same value.
Second issue: none of these values are kept so none of these can be stored.
Expected result: being able to set a different selection for each select box.

Hello,
I am afraid that I do not get you. I have created an example of using the question: http://surveyjs.org/examples/questiontype-matrixdropdown.html
Could you please, check it out? It seems work as expected.
PS: I have fixed several bugs in QuestionMatrixdropdown yesterday.
Thank you,
Andrew

Hi Andrew,
Your example is very nice and helpful.
This also makes me realise the issue is in the visual editor.
In JSON you have supplied choices per column.
In editor one can only set choices on question level, not column level.
This is causing the question type to do the behaviour as described in this issue.

I have added a property editor for the MatrixDropdown.columns property.


Several WARNINGS and ERRORS about 'isHostComponent' after migration from v4.12.3 to v5

I'm getting several WARNINGS and ERRORS related to isHostComponent

[x] The issue is present in the latest release.
[x] I have searched the issues of this repository and believe that this is not a duplicate.

Current Behavior 😯
1. Webpack compiles with several WARNINGS of:
export 'isHostComponent' (imported as 'isHostComponent') was not found in '@mui/core'

in<EMAIL_ADDRESS>in<EMAIL_ADDRESS>in<EMAIL_ADDRESS>in<EMAIL_ADDRESS>in<EMAIL_ADDRESS>in<EMAIL_ADDRESS>
2. React renders with several ERRORS of:
Uncaught TypeError: (0 , _mui_core__WEBPACK_IMPORTED_MODULE_19__.isHostComponent) is not a function
Expected Behavior 🤔

No WARNINGS in compilation
No Errors in render

Steps to Reproduce 🕹

Steps:

npm start

Context 🔦

I'm trying to migrate from v4.12.3 to v5
Your Environment 🌎

`npx @mui/envinfo`
System:
    OS: macOS 11.5
    CPU: (12) x64 Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
    Memory: 612.88 MB / 16.00 GB
    Shell: 5.8 - /bin/zsh
Binaries:
    Node: 14.17.3 - ~/.nvm/versions/node/v14.17.3/bin/node
    npm: 6.14.13 - ~/.nvm/versions/node/v14.17.3/bin/npm
Managers:
    Homebrew: 3.2.13 - /usr/local/bin/brew
    Maven: 3.8.1 - ~/.jenv/shims/mvn
    pip3: 21.2.4 - /usr/local/bin/pip3
    RubyGems: 3.0.3 - /usr/bin/gem
Utilities:
    Make: 3.81 - /usr/bin/make
    GCC: 4.2.1 - /usr/bin/gcc
    Git: 2.30.1 - /usr/bin/git
    Clang: 12.0.5 - /usr/bin/clang
Servers:
    Apache: 2.4.46 - /usr/sbin/apachectl
    Nginx: 1.21.3 - /usr/local/bin/nginx
IDEs:
    Nano: 2.0.6 - /usr/bin/nano
    Vim: 8.2 - /usr/bin/vim
    Xcode: /undefined - /usr/bin/xcodebuild
Languages:
    Bash: 3.2.57 - /bin/bash
    Java: 14.0.2 - /Users/luissardon/.jenv/shims/javac
    Perl: 5.30.2 - /usr/bin/perl
    PHP: 7.3.24 - /usr/bin/php
    Python: 2.7.16 - /usr/bin/python
    Python3: 3.9.7 - /usr/local/bin/python3
    Ruby: 2.6.3 - /usr/bin/ruby
Databases:
    SQLite: 3.32.3 - /usr/bin/sqlite3
Browsers:
    Chrome: 94.0.4606.61
    Edge: 94.0.992.31
    Firefox: 92.0.1
    Safari: 14.1.2

It doesn't look like this bug report has enough info for one of us to reproduce it.
Please provide a CodeSandbox (https://material-ui.com/r/issue-template-latest), or a link to a repository on GitHub.
Here are some tips for providing a minimal example: https://stackoverflow.com/help/mcve

Closing due to inactivity.


Fix broken "browser builds" link

Broken link found while browsing through the docs.

Thanks @mgmcdermott!


Fix branch/tag notifications in mirror sync (#13855)

Fix: https://github.com/go-gitea/gitea/issues/13810
Backport to 1.13

🚀

Ping @bot

🚀


Exponential backoff for failed notifications

Summary of the Pull Request
What is this about?
#2553
Today, our file-changes-poison queue does not get consumed and we manually re-queue those messages. This PR automatically re-processes those messages using an exponential backoff. The backoff times are:
// 1 - 5 mins
// 2 - 25 mins
// 3 - 2 hours 5 mins
// 4 - 10 hours 25 mins
// 5 - 2 days 4 hours 5 mins
// >6 - 7 days +/- 1 day of random variance so all the messages don't dequeue at the same time

Once a message hits the 1 week backoff point (or 6 attempts), we log a warning (though we should consider adding a webhook as well).
Info on Pull Request
What does this include?

Consuming the file-changes-poison queue
Exponential backoff retry for poison queue messages
Logging when poison queue messages are really failing
Rename failTaskOnTransientError to isLastRetryAttempt to more accurately reflect the behavior of that flag (it was changed in a previous PR)

Validation Steps Performed
How does someone test & validate?

Enqueue a poison queue message
Run the C# code such that it will fail when processing the message (aka: always throw an exception)
Validate the message was consumed from the queue
Validate the message was put back on the queue but hidden (azure portal shows 0 of 1 messages)
Re-run the C# code but don't throw any exceptions
Validate the message was consumed from the queue after the visibility period and make sure it was created successfully and not put back in the poison queue

Codecov Report

Merging #2555 (f28a733) into main (cafba18) will decrease coverage by 5.31%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##             main    #2555      +/-   ##
==========================================
- Coverage   29.84%   24.52%   -5.32%
==========================================
  Files         288      119     -169
  Lines       35682    12446   -23236
==========================================
- Hits        10648     3052    -7596
+ Misses      25034     9394   -15640

Impacted Files
Coverage Δ

...piService/ApiService/Functions/QueueFileChanges.cs
0.00% <0.00%> (ø)

...rvice/TestHooks/NotificationOperationsTestHooks.cs
0.00% <0.00%> (ø)

...ce/ApiService/onefuzzlib/NotificationOperations.cs
0.00% <0.00%> (ø)

...Service/ApiService/onefuzzlib/notifications/Ado.cs
0.00% <0.00%> (ø)

...rc/agent/input-tester/src/test_result/fast_fail.rs

src/agent/onefuzz-agent/src/main.rs

src/agent/coverage/src/cache.rs

src/agent/coverage/src/block/pe_provider.rs

src/agent/dynamic-library/src/linux.rs

src/agent/srcview/src/lib.rs

... and 163 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Add to rootfs the required packages for raspberrypi4-superhub

@balena-ci I self-certify!

@resin-jenkins test this please

@resin-jenkins test this please


Does not open Visual Studio Solution Files

Repro Steps:

Open Visual Studio Code
Click File -> Open File
Select a Visual Studio Solution file from the file system.
Click Open

What happened

The solution file was opened as though it is a text file.

What should have happened:

The solution should've been loaded as though it is a "workspace".
All files referenced by the solution should have been displayed in the explorer window.

Note:
Why would you dub this a"C#" editor if you can't even support solution files?
Most developers use Visual Studio and solution files are the way to work with c# projects. If you provide a "first class" support for C# this kind of a prerequisite?!?
VS Code Version:
Version: 1.38.1 (system setup)
Commit: b37e54c98e1a74ba89e03073e5a3761284e3ffb0
Date: 2019-09-11T13:35:15.005Z
Electron: 4.2.10
Chrome: 69.0.3497.128
Node.js: 10.11.0
V8: 6.9.427.31-electron.0
OS: Windows_NT x64 10.0.17763
Snapshot

This isn't strictly something we support and will need to design an experience around, since VSCode is very different from VS, and typical VS-isms like opening a solution file don't quite fit in with how VSCode typically works. But I think there's room for a solution explorer view.
Separately, there is a selector UI where you can select a solution and it will load all of it in the editor.


While Importing Cluster getting error: Failed to communicate with cluster: Unauthorized

Hi Team,
I have setup rancher on my eks cluster. When I try to import other clusters from rancher portal I am getting error:
Failed to communicate with cluster: Unauthorized
I am able to create the cluster but not able to import the clusters.
Can u please let me know why it is happening?
Rancher Version : 2.6.0

Also having this issue, no problem when I was on 2.5 with the same process
Steps:
Set up amazon IAM user
Configured cloud credentials
imported eks cluster in
"Failed to communicate with cluster: Unauthorized"


Removing sort/filter

Is it possible to remove filters/sort programatically after they have been added to a jexcel table?
For example something like $('#myTable').jexcel('removeSort');
For example something like $('#myTable').jexcel('removeFilter');
On a large table with a lot of filters applied it would be ideal to have a quick way of removing them. Is this possible?

You know you can look inside jexcel.js to see how filters are implemented?
A quick grep shows resetFilters() which seems like it might do the filters...

resetFilters() will do the trick.


Support torch.compile

This issue aims to track the progress, bugs, and requests regarding the support of torch.compile.
@tjruwase
Tasks

[x] ZeRO3 support (#4878, we currently break the graph to make communication collectives work)
[x] Pipeline parallel support (#4677)
[ ] Activation checkpointing with Megatron-DeepSpeed (https://github.com/microsoft/Megatron-DeepSpeed/pull/322)
[ ] Pass HuggingFace CI tests

PRs

#4878: The initial proposal of the official torch.compile support

Issues

#4677
#3375
#3032

Thanks @tohtana!
@stas00, FYI

Hello, quick related question to #4878 : We can only compile certain parts of our model so just compiling the whole thing as the new option does wouldn't work. Is there a path for this sort of model?

Hi @zaptrem, thank you for your question!
You can pass a function to compile the model with set_compiler_fn. I think you can define a function that compiles only the necessary part of the model and pass it to this API.
You'll find an example of the API usage here:
https://github.com/microsoft/DeepSpeed/blob/master/tests/unit/runtime/compile/test_load_config.py

Is still actively being worked on?


AB#8413 Play Media Native - Edit Athlete details screen

On the edit screen I don't see a save/cancel buttons. Maybe we should add them.


CSS/JS injection

As a follow-up to our discussion during the last call, I created a new document to summarize requirements and start discussing how we could define configuration files and APIs for CSS/JS injection: https://github.com/readium/readium-2/blob/master/other/apis/CSS-JS Injection API.md

Quick question: do we want to manage writing modes in there?
For instance, loading the vertical-rl styles by default would not necessarily make a lot of sense for a french or german vendor, and although not critical, that could be an easy perf win (less styles to load and parse, etc.). On the opposite, a japanese or chinese vendor might not need a lot of styles for latin-based languages.

So yeah I can confirm we’ll need to manage vertical-rl, maybe even CJK in there.
On a related note, we’ll also probably need something to set attributes on the root element (html) because there are cases for which we’ll need dir, xml:lang, etc. Solving the flickering of undefined variables could probably benefit from this.
And another thing the app’s UI can use to load different fonts’ list in user settings, but also remove/add settings, depending on the main language of the publication.
I guess it could be a good idea discussing localization instead of Readium CSS per se on the next call, because the biggest impacts are quite definitely at the streamer, navigator, APIs and UI level. My gut feeling tells me that prioritizing needs/features will be critical there, and we’d better do it sooner than later or else we might have to redo a lot of stuff at some point, should we end up managing localization on our side.
CC’ing @rkwright since he pointed out localization a few weeks back, and it probably belongs in another issue.

@HadrienGardeur
I took your idea and came up with this model:
https://github.com/readium/readium-2/blob/master/other/injection.md
To start, I feel like it needs some streamlining.


Add a traits system

OK, here's my proposal for a new ColorTypes. Probably the easiest way to get started is to look at the README, which you can see here.
To your original, the main feature is a robust "traits" system that makes it much easier to write generic code without using @eval everywhere. I can post the upcoming (and working!) ColorVectorSpace package if you want to see what I mean. You have to be running a recent julia for the tests to pass (you need https://github.com/JuliaLang/julia/pull/12645).
I've also created a Colors package and a branch of Images that both work with this. So I'm reasonably confident this works pretty well.
One question I have is, which packages do you want to "host", and which should be put elsewhere? (JuliaLang?)

The change to Images is https://github.com/timholy/Images.jl/pull/340

This looks really great!
I see if I can integrate it before I go to bed!
First thing I did is to also export TransparentGray, which you define as
a new typealias in Images.jl. Why not just export it from ColorTypes?!
2015-08-17 23:05 GMT+02:00 Tim Holy<EMAIL_ADDRESS>
The change to Images is timholy/Images.jl#340
https://github.com/timholy/Images.jl/pull/340
—
Reply to this email directly or view it on GitHub
https://github.com/SimonDanisch/ColorTypes.jl/pull/1#issuecomment-131960882
.

Okay I think I have mostly integrated it.
What are your plans with ImageMagick!? Should I clean it up and then submit a pull request to Images?

Yeah, having the standalone ImageMagick package is a big part of the goal here. I almost finished that before I realize the "Color/ColorType" split, so I may be able to submit that momentarily. Rebasing Images on that is not something I've started, so if you'd like to take a crack at it, that would be awesome!

OK, here's ColorVectorSpace: https://github.com/timholy/ColorVectorSpace.jl (again, if you prefer I'm happy to host on JuliaLang or turn it over to you).

OK, see https://github.com/JuliaIO/ImageMagick.jl/pull/3

I'm going to be working on tests for the new Colors package, so I won't be touching Images for quite a few hours.

Great!
I'm happy to put it all under JuliaGraphics/JuliaLang/JuliaImages ;)
JuliacGraphics is a little deserted, so I'm not sure which one would be the
best.
2015-08-18 0:32 GMT+02:00 Tim Holy<EMAIL_ADDRESS>
OK, here's ColorVectorSpace:
https://github.com/timholy/ColorVectorSpace.jl (again, if you prefer I'm
happy to host on JuliaLang or turn it over to you).
—
Reply to this email directly or view it on GitHub
https://github.com/SimonDanisch/ColorTypes.jl/pull/1#issuecomment-131983001
.

Actually, I like the idea of hosting anything color-related on JuliaGraphics. I just moved ColorVectorSpace over there.

I'll post Colors.jl there too. Will you move ColorTypes, once you merge this?

Sure!
2015-08-18 0:40 GMT+02:00 Tim Holy<EMAIL_ADDRESS>
I'll post Colors.jl there too. Will you move ColorTypes, once you merge
this?
—
Reply to this email directly or view it on GitHub
https://github.com/SimonDanisch/ColorTypes.jl/pull/1#issuecomment-131984286
.

Some @inferred tests seem to not work on Ubuntu (9eC29dc, 7 day old master) and on windows ( 72c344c, 4 days old master)
Not working:
@infrd(eltype(RGB))
@test @infrd(colortype(RGBA)) == RGB
@test @infrd(basecolortype(HSVA)) == HSV

This is the problem:
julia> Base.return_types(colortype, Base.typesof(RGBA))
1-element Array{Any,1}:
 Any

The rest seems fine!

As no one besides me is using ColorTypes yet and I'm set on integrating this, I will just merge it!

That's what I meant above by

You have to be running a recent julia for the tests to pass (you need JuliaLang/julia#12645).

Before you switch to it, try this for fun:
julia> using ColorTypes

julia> c = RGB(rand(),rand(),rand())
RGB{Float64}(0.3187238017125491,0.7814248615656347,0.21993973242405107)

julia> @code_llvm convert(RGBA{Float32}, c)

Then upgrade your julia and do the same; watch what happens :smile:.

oh okay, I didn't realize that you mean super recent!
The problem is, that GLVisualize does not seem to work with the recent changes... Didn't have the patience to track down the segfault so far.

You could cherry-pick that commit.

@code_llvm convert(RGBA{Float32}, c) yields the same result for both versions.
julia> @code_llvm convert(RGBA{Float32}, c)

define %RGBA @julia_convert_21915(%jl_value_t*, %RGB*) {
top:
  %2 = load %RGB* %1, align 8
  %3 = extractvalue %RGB %2, 0
  %4 = extractvalue %RGB %2, 1
  %5 = extractvalue %RGB %2, 2
  %6 = fptrunc double %3 to float
  %7 = insertvalue %RGBA undef, float %6, 0
  %8 = fptrunc double %4 to float
  %9 = insertvalue %RGBA %7, float %8, 1
  %10 = fptrunc double %5 to float
  %11 = insertvalue %RGBA %9, float %10, 2
  %12 = insertvalue %RGBA %11, float 1.000000e+00, 3
  ret %RGBA %12
}

But Base.return_types(colortype, Base.typesof(RGBA)) works now =)

Ah, shoot, I gave you a version that didn't even use the traits (duh). Here's a better example:
julia> @code_llvm coloralpha(basecolortype(c)){eltype(c)}(c)

define %RGBA @julia_call_21452(%jl_value_t*, %RGB*) {
top:
  %2 = load %RGB* %1, align 8
  %3 = extractvalue %RGB %2, 0
  %4 = extractvalue %RGB %2, 1
  %5 = extractvalue %RGB %2, 2
  %6 = insertvalue %RGBA undef, double %3, 0
  %7 = insertvalue %RGBA %6, double %4, 1
  %8 = insertvalue %RGBA %7, double %5, 2
  %9 = insertvalue %RGBA %8, double 1.000000e+00, 3
  ret %RGBA %9

Compare julia 0.3:
julia> @code_llvm coloralpha(basecolortype(c)){eltype(c)}(c)

define %jl_value_t* @julia_RGBA_20946(%jl_value_t*, %jl_value_t**, i32) {
top:
  %3 = getelementptr inbounds %jl_value_t* %0, i64 2, i32 0, !dbg !2624
  %4 = load %jl_value_t** %3, align 8, !dbg !2624, !tbaa %jtbaa_func
  %5 = alloca [4 x %jl_value_t*], align 8
  %.sub = getelementptr inbounds [4 x %jl_value_t*]* %5, i64 0, i64 0
  %6 = getelementptr [4 x %jl_value_t*]* %5, i64 0, i64 2, !dbg !2624
  store %jl_value_t* inttoptr (i64 4 to %jl_value_t*), %jl_value_t** %.sub, align 8
  %7 = getelementptr [4 x %jl_value_t*]* %5, i64 0, i64 1, !dbg !2624
  %8 = load %jl_value_t*** @jl_pgcstack, align 8, !dbg !2624
  %.c = bitcast %jl_value_t** %8 to %jl_value_t*, !dbg !2624
  store %jl_value_t* %.c, %jl_value_t** %7, align 8, !dbg !2624
  store %jl_value_t** %.sub, %jl_value_t*** @jl_pgcstack, align 8, !dbg !2624
  store %jl_value_t* null, %jl_value_t** %6, align 8
  %9 = getelementptr [4 x %jl_value_t*]* %5, i64 0, i64 3
  store %jl_value_t* null, %jl_value_t** %9, align 8
  %10 = getelementptr inbounds %jl_value_t* %4, i64 2, i32 0, !dbg !2628
  %11 = load %jl_value_t** %10, align 8, !dbg !2628, !tbaa %jtbaa_tuplelen
  %12 = getelementptr inbounds %jl_value_t* %11, i64 1, i32 0, !dbg !2628
  %13 = load %jl_value_t** %12, align 8, !dbg !2628
  %14 = icmp eq %jl_value_t* %13, null, !dbg !2628
  br i1 %14, label %err, label %ok, !dbg !2628

err:                                              ; preds = %top
  call void @jl_undefined_var_error(%jl_value_t* inttoptr (i64<PHONE_NUMBER>27640 to %jl_value_t*)), !dbg !2628
  unreachable

ok:                                               ; preds = %top
  %15 = getelementptr inbounds %jl_value_t* %13, i64 0, i32 0, !dbg !2628
  %16 = load %jl_value_t** %15, align 8, !dbg !2628
  %magicptr = ptrtoint %jl_value_t* %16 to i64, !dbg !2628
  switch i64 %magicptr, label %notf [
    i64 19679504, label %isf
    i64 19547520, label %isf
  ], !dbg !2628

notf:                                             ; preds = %ok
  call void @jl_type_error_rt_line(i8* getelementptr inbounds ([5 x i8]* @_j_str51, i64 0, i64 0), i8* getelementptr inbounds ([6 x i8]* @_j_str10, i64 0, i64 0), %jl_value_t* inttoptr (i64 19679504 to %jl_value_t*), %jl_value_t* %13, i32 415), !dbg !2628
  unreachable

isf:                                              ; preds = %ok, %ok
  %17 = load %jl_value_t** %1, align 8, !dbg !2624
  %18 = getelementptr inbounds %jl_value_t* %13, i64 1, i32 0, !dbg !2628
  %19 = load %jl_value_t** %18, align 8, !dbg !2628, !tbaa %jtbaa_func
  %20 = bitcast %jl_value_t* %19 to %jl_value_t* (%jl_value_t*, %jl_value_t**, i32)*, !dbg !2628
  store %jl_value_t* %17, %jl_value_t** %6, align 8, !dbg !2628
  store %jl_value_t* inttoptr (i64 147557024 to %jl_value_t*), %jl_value_t** %9, align 8, !dbg !2628
  %21 = call %jl_value_t* %20(%jl_value_t* %13, %jl_value_t** %6, i32 2), !dbg !2628
  %22 = load %jl_value_t** %7, align 8, !dbg !2628
  %23 = getelementptr inbounds %jl_value_t* %22, i64 0, i32 0, !dbg !2628
  store %jl_value_t** %23, %jl_value_t*** @jl_pgcstack, align 8, !dbg !2628
  ret %jl_value_t* %21, !dbg !2628
}

OK, I just pushed https://github.com/JuliaGraphics/Colors.jl. I need to update the README and add some documentation. (I'm trying to add some value, as incentive for people to upgrade. Note the recent performance and testing improvements. But internal documentation will help, too.)
As soon as you push this to JuliaGraphics, then everything will be in its "real" home and we can ask for feedback.
BTW, if you're working on Images at all (to use ImageMagick), then definitely base it on the teh/colors branch (the one in PR 340).


added new proposaltxt

Hey @ManishOffi can you explain the purpose of this file? Right now folks can propose new terms as issues or PRs, or ideally both.

@ManishOffi can you help me understand what you're trying to accomplish by adding these text files?


run scripts: augment with env variables, and have runWPT.sh update ex…

…pectations
To make it easier to switch between headful and headless. For example:
HEADLESS=false ./runWPT.sh
Furthermore, tighten puppeteer types and argument parsing on bidi server.
Status

[ ] Add CHROMEDRIVER= argument to update wpt-metadata/chromedriver/headless/.
[ ] (?) Set PUPPETEER_EXECUTABLE_PATH. Same as WPT_BROWSER_PATH.

It would be nice to have chromium_path param as well for local testing.

executable_path would be WPT_BROWSER_PATH, correct?

Yes, but Mapper Server does not support that argument yet. Can be out of scope.

I see. Let's do that in a follow up then.


Allow for freeze-thaw of configurations where possible

To allow for a wider multi-fidelity scope, it would be nice to allow configurations to be optionally restarted from a model checkpoint. This would be applicable for the tree-based search spaces (RandomForest, XGB) and the neural network spaces. That is if a call to the objective_function is made for a configuration that has already been evaluated on a particular fidelity, but on a higher unseen fidelity, the function should load a model for the configuration at the lower fidelity and continue training till the higher fidelity (add more trees or training more epochs). In such a case, the costs returned would indicate the cost involved in continuing the training.
Implementation of this into HPOBench would require considerations around the best API design that doesn't break existing APIs. While also checking how model loading and saving can be managed with the Docker interface. Also, it needs to be decided if the function evaluation costs should account for the model I/O to disk. Since ignoring them might affect the true cost of benchmark querying.

Yes, this would be a useful addition and in principle, this could even work for all benchmarks including the tabular ones.
Here are a few things we should keep in mind when designing the API: This will make the benchmarks stateful, more complex, and requires additional memory since the benchmarks need to store which configs/seeds have been trained on which budget. There are several options for this, e.g. via a file, database, or in memory, each with pros and cons.
Additionally, this will interfere with optimizers evaluating configurations in parallel and raises further questions, e.g. whether and how to share the state among evaluations.

Hey guy,
This sounds like a very useful addition.
I know this might be a special scenario of freeze-thaw, but there are some scenarios in which "continue training" could simply mean loading some weights. E.g. the weights of a neural network, etc. This particular use-case/scenario is potentially easy to tackle. And might be a good starting point.
We could solve this case by implementing some hooks that are called before or after the objective_function call: on_train_start, on_train_finish, ...
These functions could be used to load or save the model weights to a directory defined in the hpobenchconfig.
We integrate these functions in the "workflow" of calling the objective function. I mean the user only calls the objective_function and hpobench automatically starts to look for a potential savepoint and loads that if present. After returning the result, it calls then a second function to save the current state of the model.
We could also create a second "baseclass" that makes clear that the benchmark does not support freeze-thaw.
However, @KEggensperger and I have already discussed that there are some cases for which that easy solution does not work.
But as mentioned first, it could be a first starting point. Freeze-Thaw as "moving a process in the background" is pretty hard for containers. Or at least it is not clear to me how to do it.
What do you think?

Thank you for your valuable input and for raising important points. Given you both have thought about this much longer, I would definitely like to have a more detailed conversation once I have a basic design or a prototype ready.

@KEggensperger

requires additional memory...

For the first iteration, I was thinking of going with some in-memory data structure of unique configurations evaluated, and their corresponding latest fidelity evaluated. One immediate issue would be how to handle duplicate queries. Also, we could include a reset() function to clear the memory and kind of bring the object to the initialization state again.

could even work for all benchmarks including the tabular ones

But we won't need this feature for tabular benchmarks, right?

interfere with optimizers evaluating configurations

Definitely a concern. However, just like we say that we don't support freeze-thaw now, can we later claim that we support freeze-thaw only for the 1-worker setup?

@PhMueller

might be a good starting point

Indeed, and I agree. For the first iteration, I want to take the sklearn MLP space we added, and allow freeze-thaw for that. That way we can test without changing the base class and affecting other spaces.

by implementing some hooks

Not too familiar with its implementation, but having used them, sounds apt.

save the model weights to a directory defined in the hpobenchconfig

Again, sounds perfect as a starting point. Just like the directory for the data for the tabular benchmarks are managed. In the long run, I wonder if it might be useful to allow a more flexible option for the user to edit the directory. Since I am not sure how the memory starts bloating with long runs of an instantiation of HPOBench. Secondly, given we don't allow checkpointing and resuming of HPOBench runs, we might want to define some operations in __del__ (or something better) to clean the directory storing the models from the objective_function calls. Or just clean the directory at __init__.

Freeze-Thaw as "moving a process in the background" is pretty hard for containers

This is something I have heard a few times now and I need to probably work with containers more to fully understand why. Can a process that runs on a container, not access files saved locally outside the container? Isn't that what the TabularBenchmark kind of does?

But as mentioned first, it could be a first starting point.t

I totally agree.

@KEggensperger @PhMueller
All the above points I mentioned are w.r.t. the first iteration of Freeze-Thaw for the MLP space, that we could try first.
Would be great to hear your thoughts on the same!


Scrolling Object Collection causes TextMeshPro MissingReferenceExceptions outside of example scene

Describe the bug
When copying any of the Scrolling Object Collection sample objects (with text) outside of the Scrolling Object Collection example scene, scrolling the list would cause the following error to appear:
MissingReferenceException: The object of type 'Material' has been destroyed but you are still trying to access it.
Your script should either check if it is null or you should not destroy the object.
TMPro.TextMeshPro.GenerateTextMesh () (at Library/PackageCache/com.unity.textmeshpro@2.0.1/Scripts/Runtime/TMPro_Private.cs:1969)
TMPro.TextMeshPro.OnPreRenderObject () (at Library/PackageCache/com.unity.textmeshpro@2.0.1/Scripts/Runtime/TMPro_Private.cs:1549)
TMPro.TextMeshPro.Rebuild (UnityEngine.UI.CanvasUpdate update) (at Library/PackageCache/com.unity.textmeshpro@2.0.1/Scripts/Runtime/TextMeshPro.cs:241)

To reproduce

Create a new Unity project
Import MRTK and MRTK Examples packages
Open ScrollingObjectCollection.unity
Press play and verify that scrolling any of the Scrolling Object Collection objects has no errors
Copy any of the Scrolling Object Collection objects that has text on it
Create a new scene
Select Mixed Reality Toolkit/Add to Scene and Configure...
Paste the Scrolling Object Collection objects that we have copied
Press play and try scrolling any of the Scrolling Object Collection objects
The error indicated above will appear

Expected behavior
Users should be able to copy the sample over and modify it for my own use without running into any errors
Your setup

Unity Version 2019.3.14f1
MRTK Version 2.3.0
TextMeshPro Package Version 2.0.1

Target platform

HoloLens
HoloLens 2
Android
iOS

I've also encountered the same problem in 2019.3.12f (macOS) with TextMeshPro 2.0.1.
change line 2094 in<EMAIL_ADDRESS>to
if (m_currentMaterial != null && m_currentMaterial.HasProperty(ShaderUtilities.ID_GradientScale))
seems like to fix the problem.


Does the algorithm use multiple threads in fitting

Hi
I've come to a point where i want to fit multiple MARS models independently at the same time. However I am finding that when I try to fit multiple models with Multi-processing module I am not getting much of a performance increase. i am struggling to get 2x speeds with an i7 (4 cores 8 threads).
I wonder if this is due to MARS already fitting the models using multiple threads and such i cant get much performance boost?

@Fish-Soup While py-earth itself doesn't do any parallelization, it makes a lot of calls to BLAS and Lapack.  Depending on what implementations of those libraries you have installed, your system may be using multiple CPUs simultaneously for a single fit.  If you want to fit multiple models simultaneously, there may be benefits to limiting BLAS to a single thread (for each process).  However, I haven't experimented with doing this at all.  I'm not even sure exactly how you would, although I believe it's possible to do.  It may be that doing so actually ends up slowing things down, however.  It's something you'd have to test out to be sure.  Another way to get speed might be just to get a faster BLAS, depending on your system.
Also, setting use_fast=True will use the "fast MARS" algorithm, which might perform slightly worse in terms of accuracy but is significantly faster.  You can fine-tune it with the fast_k and fast_h parameters.

Hi
After posting I came to the same conclusion. I'm now using the use_fast flag, at least while I am prototyping, which is as you say significantly faster. thanks


[R] Calling distinct() then compute() on a large dataset pulls everything into memory and crashes my R session

Describe the bug, including details regarding any error messages, version, and platform.
I'll flesh this out with a proper reprex later; the dataset in question is the 40Gb data from the Voltron Data S3 bucket
nyc_taxi <- open_dataset(here::here("data/nyc-taxi"))

nyc_taxi %>%
  distinct(pickup_longitude, pickup_latitude) %>%
  compute() %>%
  nrow()

Component(s)
R

Not a bug


Implement HTTPOnly cookies

As mentioned by @jfresco at #836.
The disscussion here is that to do this we should change the logout functionality to make it do a request to the server, because right now we only unset the 'token' cookie to logout.
_More info: http://blog.codinghorror.com/protecting-your-cookies-httponly/_

:+1: to a request to the sever and marking the cookie. If something goes wrong with the request you can always note that the state is "logging out" and just re-call the logout endpoint later. Also, this would allow to navigate to a specific page after logout.


"Change all occurrences" is missing from context menus

Version: 1.41.0-insider (system setup)
Commit: bf7d03bf88732af0b46e473c663c49e18e839f9c
Date: 2019-11-15T05:40:12.201Z
Electron: 6.1.4
Chrome: 76.0.3809.146
Node.js: 12.4.0
V8: 7.6.303.31-electron.0
OS: Windows_NT x64 10.0.18363
There used to be a "Change all occurrences" option in the context menu that would activate cursors at every instance of the selection in the current file. Now it seems to be missing as of recently.

I see there's still a Selection -> Select All Occurrences option, but is there any way to get it back in the context menu?

Apparently done by @jrieken in https://github.com/microsoft/vscode/commit/5c27cdccf8621e212d0113b18f4ef07714c28d3f

We did that on purpose and only when there is a rename provider - which we think is the better alternative. @iansan5653 What the use case for "change all" over "rename"?

For renaming symbols alone,  "change all" sometimes is useful when the language provider is hanging (happens often with Python) and I know the symbol is only used in the current file.
But it's also useful for tons of other cases:
For example, in this snippet I might want to add : void to all the functions where I didn't define a return type. I can highlight one instance of () { and then use "Change all occurrences" and then change them to (): void {.
function example0(): number {
  // ...
}

function example1(): string {
  // ...
}

function example2() {
  // ...
}

function example3(): string {
  // ...
}

function example4() {
  // ...
}

The above could also be done with find and replace. However, having multiple cursors in use is compatible with things like the "Increment Selection" extension, which is exactly how I generated the above example - I wrote one example() function, copied it a couple times, selected example(), change all, used the arrow keys to put the cursor before the parentheses and then put the numbers with Increment Selection.
Here's another example. Say I'm fixing some old code. I want to change var to const, add semicolons, and change example to sample. I can select the first var example, change all occurrences, type const sample, press End, type ; and I'm done.
var exampleString = "sample"
// This stops me from just using column select
var exampleBoolean = true
var exampleNumber = 45

Makes sense - I'll restore it


Deploy backend to fly-io

ead that when you set DEBUG to False you have to setup an actual webserver to serve the static files so the browsable API still

Ah OK :) Yep, I can fix that as well. I was just trying to use the JSON API part and not necessarily the UI parts (I hope I got you there). Will fix that in the next days.

Fixed now: https://polybot-mission-db-backend.fly.dev/restapi/missions/create/

The browsable API now works. Thank you.
But I found another issue: When trying to delete a mission I get a Server Error (500) response. This happens when I try to make a DELETE request using the browsable API and with curl. This only happens with missions. A Tag can be deleted using a DELETE request. When I test it locally, it also works.

I suspect that's more an issue of the API implementation than the deployment.
Can you please share the curl command that you used?

The browsable API now works. Thank you.
But I found another issue: When trying to delete a mission I get a Server Error (500) response. This happens when I try to make a DELETE request using the browsable API and with curl. This only happens with missions. A Tag can be deleted using a DELETE request. When I test it locally, it also works.

The curl command I used:
curl -X DELETE https://polybot-mission-db-backend.fly.dev/restapi/missions/2

I don't know if it's an issue of the deployment or the implementation, because I can't test it locally.

For me this does not even to work locally. I'll try to debug it.

Now everything works. Thank you.


power prompt lora prompt found and  loaded problem

There are the problem when I load some lora ( It found the another lora even have right path):

🛈  rgthree Power Prompt:  Found "Cloth\Rich.safetensors" for "XQ\ziedRichDota2d" in prompt
✓  rgthree Power Prompt: Loaded "Cloth\Rich.safetensors" from prompt
🛈  rgthree Power Prompt:  1 Loras processed; stripping tags for TEXT output.

Yea, that doesn't seem right. Made the matching a bit stricter, and also checked for more exact matches before iterating to find a loose match.

Thx, it work!
have a nice day


CypressError: cy.its() only accepts a single argument.

I've just setup using this package, and am receiving the above error message when I try to invoke one of the Laravel integration commands. The source of the error is in the laravel-commands.js, when the CSRF Token is retrieved. It currently reads:
Cypress.Commands.add('csrfToken', () => {
    return cy
        .request({
            method: 'GET',
            url: '/__cypress__/csrf_token',
            log: false,
        })
        .its('body', { log: false });
});

If I remove the { log: false } argument to its the command works successfully.
Of course I can get it to work this way, as I can simply remove this option specification in my published copy of the commands (I am assuming I will have this issue across the board where this pattern is used).
I am posting this as information for others, but also to see if anyone else has seen this and if there's a better solution to the problem?
I am using the official docker containers for cypress for actually invoking my tests (rather than a local install of cypress), but I don't believe that should be having an impact.
Bit new to cypress though, so very aware that I could be missing a subtlety in the setup.

What version of Cypress are you using? I think updating will resolve this, we run the same code without any issues.
Also see here: https://github.com/cypress-io/cypress/issues/1450

oh dear, I hang my head in shame.
It's absolutely the docker container I am using, it's very out of date.
Thanks for taking the time to respond @Krisell much appreciated.


Add security context to enable ssh on Openshift

Deploy a multi nodes cluster on Openshift fail because nodes cannot ssh each other.
This security context is added to enable that.

Looks good.  You hit an error in the azb test while it was running the unit test.  Can you rerun that one?

Yeah I am manually rerunning it on the forked branch.

I reran it. looks fine to me:
https://github.com/roypaulin/vertica-kubernetes/actions/workflows/e2e-azb.yml?query=branch%3Aadd-security-context++

It should have been rerun by following the Details link.  That way the PR is updated with its run result.  I have rerun it now.  So I'll monitor the run and push when it is clean.

How do you do that? Can I do that too?


adding support for stratified partition, closes #113

this:
y = ones(Int, 1000)
y[end-500:end-200] .= 2
y[end-200+1:end] .= 3
p1 = sum(y .== 1) / length(y) # 0.5
p2 = sum(y .== 2) / length(y) # 0.3
p3 = sum(y .== 3) / length(y) # 0.2

s1, s2, s3 = partition(eachindex(y), 0.3, 0.6, stratify=y, rng=111)

now works.
Also some basic refactoring of the partition function.
closes:  #113

Codecov Report

Merging #138 into dev will increase coverage by 0.38%.
The diff coverage is 93.02%.

@@            Coverage Diff            @@
##             dev     #138      +/-   ##
=========================================
+ Coverage   76.3%   76.69%   +0.38%
=========================================
  Files         21       21
  Lines       1110     1137      +27
=========================================
+ Hits         847      872      +25
- Misses       263      265       +2

Impacted Files
Coverage Δ

src/MLJBase.jl
75% <ø> (ø)
:arrow_up:

src/data.jl
94.2% <93.02%> (-0.4%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 403e37c...db7bcf2. Read the comment docs.

Ok:
julia> y = [1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2];

julia> s1, s2 = partition(eachindex(y), 0.7, stratify=y)
([1, 2, 3, 4, 5, 6, 9, 10], [7, 8, 11])

julia> p = sum(y .== 1) / length(y)
0.7272727272727273

julia> sum(y[s1] .== 1) / length(s1)
0.75

whereas
julia> s1, s2 = partition(eachindex(y), 0.7, stratify=y, rng=11)
([2, 5, 1, 4, 11, 3, 6, 9], [8, 10, 7])

julia> sum(y[s1] .== 1) / length(s1)
0.75


(Rust) Inconsistent type returned for actual and expected mismatches

Context
When calling mock_server_mismatches the types of actual and expected keys in mismatches array are inconsistent.
Using Rust pact_mock_server_ffi for use with Swift/iOS.
Changed the rust/pact_matching/Cargo.toml and set onig = 4.3.3 to be able to build on my system. Onig 5.0 fails to compile. 🤷♂
macOS 10.15.4
clang and other needed tools to compile were installed using homebrew.
Example:

when expected query parameter is not present in the actual request, then actual and expected values are returned as a String.
when expected body does not match actual body, then the actual and expected values are returned as an Array<Int>.

Steps to reproduce
Set Mock Server expectation:

POST /path
body: {"foo": "bar"}

Run API client that sends the following request:

POST /path
headers: "Content-Type": "application/json"
body: {\n\n}

Expectation:
When fetching mismatches by calling mock_server_mismatches(socketAddress)
Then Mock Server should respond with:
[
... // key:values
        "mismatches": [
            "mismatch": "Expected entry foo=bar but was missing",
            "type": "BodyMismatch",
            "path": "$"
        ]
... // key:values
]

Observed behaviour:
When fetching mismatches by calling mock_server_mismatches(socketAddress)
Then Mock Server responds with:
[
	{
    "method": "POST",
    "mismatches": [
      {
        "actual": [123,125],
        "expected": [123,34,102,111,111,34,58,34,98,97,114,34,125],
        "mismatch": "Expected a Map with 1 elements but received 0 elements",
        "path": "$",
        "type": "BodyMismatch"
      },
      {
        "actual": [123,125],
        "expected": [123,34,102,111,111,34,58,34,98,97,114,34,125],
        "mismatch": "Expected entry foo=bar but was missing",
        "path": "$",
        "type": "BodyMismatch"
      }
    ],
    "path": "/user",
    "type": "request-mismatch"
  }
]

JSON will now look like
{
  "method": "POST",
  "mismatches": [
    {
      "actual": "\"no-very-bar\"",
      "expected": "\"bar\"",
      "mismatch": "Expected 'bar' to be equal to 'no-very-bar'",
      "path": "$.foo",
      "type": "BodyMismatch"
    }
  ],
  "path": "/path",
  "type": "request-mismatch"
}

Just tested with FFI v0.0.2 and it works as expected. This issue can be closed.


add rippleColor prop to IconButton

@danleavitt0 Thanks for this PR - although, I think the spread attribute ...other should allow a developer to pass in focusRippleColor and touchRippleColor here:
https://github.com/callemall/material-ui/blob/master/src/icon-button.jsx#L132


g.part4 fails with "object 'undef' not found" error

Describe the bug
g.part4 fails on some datasets with the following error:
Error in rainbow(length(undef), start = 0.7, end = 1) :
  object 'undef' not found

The error happens on line 898 because variable undef is undefined.
undef normally should get defined on line 585 if the condition on line 584 holds true. However, I have some data files where I end up with calendar_date[j] == "" and so this condition fails and undef never gets set.
(for this data file, I end up with calendar_date == ["", "2/5/2017", "", "", "", "", "", "" ], and then at j==6 the process crashes.
To be fair, the data files that are giving me trouble aren't actually good data -- it looks like the subject probably wasn't wearing the device much, if at all. So I get weird-looking results with one single 7-day sleep period across 8 days. So it's not useful data by any stretch. But I think even with not-so-nice data, GGIR should exit gracefully and not error out.
I think the solution should be similar to this fix, and the if (addlegend == TRUE) block needs to be brought in one more level, inside the if (nrow(spocum) > 1 & ncol(spocum) >= 5 & calendar_date[j] != "") conditional block
To Reproduce
Steps to reproduce the behavior.

Sensor brand: Actigraph
Data format: gt3x file, parsed into a CSV using our own parser.
Approximate recording duration: 8 days
Are you using a sleep diary to guide the sleep detection: NO
Copy of R command used:

g.shell.GGIR(datadir="<...>/data", outputdir="<...>/results", do.report=1:5, visualreport=TRUE, do.visual=TRUE, do.enmo = FALSE, do.enmoa = TRUE, acc.metric="ENMOa", threshold.lig = c(61.84), threshold.mod = c(100.4), mvpathreshold =c(100.4), rmc.firstrow.acc = 2, rmc.col.acc = 2:4, rmc.col.time=1, rmc.unit.time = "UNIXsec", configtz = "UTC", desiredtz = "UTC", rmc.desiredtz = "UTC", rmc.sf = 80, rmc.check4timegaps = TRUE, rmc.doresample=FALSE, interpolationType=1)

Have you tried processing your data based on GGIR's default argument values? Does the issue you report still appear?

Not entirely. Since the data fed into GGIR is in a csv file, I had to specify some additional parameters, so that the csv file can be read in:
g.shell.GGIR(datadir="<...>/data", outputdir="<...>/results", do.report=1:5, visualreport=TRUE, do.visual=TRUE, rmc.firstrow.acc = 2, rmc.col.acc = 2:4, rmc.col.time=1, rmc.unit.time = "UNIXsec", rmc.sf = 80)
Expected behavior
I was expecting GGIR to exit gracefully, without throwing an error.
Screenshots
Desktop (please complete the following information):

OS: MacOS 10.15.7
GGIR Version: 2.4-2

Additional context
I'm going to send a PR for the fix, since it's just one line, similar to this fix
Before submitting

[] Have you tried the steps to reproduce? Do they include all relevant data and configuration? Does the issue you report still appear there?
[X] Have you tried this on the latest master branch from GitHub?

Now closing this issue, see my comment in #456. Sorry about the miscommunication.

@vincentvanhees I left a comment on #456  -- I think this issue still exists in your latest release. You had fixed a similar issue in the latest release, and I saw that, but your fix didn't go far enough.

thanks @l-k- I agree, I have now merged your PR.

@vincentvanhees awesome, thank you for taking another look!


Remove setAppCacheEnabled

If applied this commit will prune the setAppCacheEnabled to make library compatible with Android target SDK level 33

Partially Solves https://github.com/aiincorporated/bobMobileAppAndroid/issues/1242
Prune deprecated setAppCacheEnabled  from the code, setAppCacheEnabled enables/disables application cache API As the application cache api is depreceated this method is no longer supported. https://android.googlesource.com/platform/frameworks/base/+/master/core/java/android/webkit/WebSettings.java#1171

This should have merged to master. Please recreate another PR.


bugfix: emissive

emissive surfaces  need to use full lighting values, not no lighting values.

Emissive means the pixel is a light source. Therefore no lighting should be applied to it. This should be detected in the lighting combine pass and the original color of that pixel should be output without additional lighting calculation.
The reason your current approach won't work is because lights are additively blended into the lighting buffer, so if you output (1.0, 1.0, 1.0) into the lighting buffer for emissive surfaces than 2 overlapping lights on an emissive surface will add up to (2.0, 2.0, 2.0) in the lighting buffer. This will double the intensity of the emissive pixel which isn't how an emissive pixel should behave.


flag permissions issues

Nucleus-1.14.7-S7.1-MC1.12.2-plugin /  LuckPerms-Sponge-5.2.53 /  griefdefender-sponge-1.2.4-B15 /  spongeforge-1.12.2-2838-7.3.0
MC eternal mod pack <IP_ADDRESS> .. updated minecolonies mod to its last version for mc 1.12 (minecolonies-1.12.2-0.11.841-ALPHA-universal)
i seem to be unable to use flags either by in game commands or the gui from clicking claim with a stick
thought i had altered the config file some how.. so deleted it and generated a new one with no change
its giving me the response of "you do not have permission to use this command"
i used verbose with luckperms and keeps showing the flag syntax check as failed or unidentified.
so i am wondering if maybe i got the permission node entered wrong or if its actually broke
seems permissions in wiki seems might be missing one or something.. griefdefender.user.* allows the flags to work but to set each node without the * seems broken

griefdefender-sponge-1.2.4-B15 is a very old build and not supported.

been looking for updates to it.. but the discord link that was the last one posted. oddly not seen a link in this github for latest version .. im probably over looking it tho

seems im on old discord not the private one.. guessing im unable to get update from what im sifting threw so far.


Support ES 2.x

There's a nearly finished WIP in #22 with a few small kinks to be ironed out.

I really hope this would be finished soon - thank you for the effort!
If I was a developer I would love to help :)

any news on supporting recent ES versions?

@digitalkaoz Unfortunately, I haven't made any progress on the WIP in #22 since I created it because I'm not very familiar with the Java ecosystem. If you're willing to take a stab at jumping the few remaining hurdles, go for it!
By the way, there's a $15 bounty on this issue: https://www.bountysource.com/issues/32043561-support-es-2-x

Looks like these guys got it working: https://github.com/crate/crate/tree/master/dns-discovery

Thanks, @falencastro! I submitted a proposal to combine efforts with Crate https://github.com/crate/crate/issues/3670

Btw with some minor adjustments I got the plugin to run on ES 2.2.2 and 2.3.3, but either with this code or crate's code, it apparently gets stuck while instantiating a new org.xbill.DNS.Lookup object, as the second logger doesn't get printed  and I receive this message:
[2016-06-24 15:23:49,360][WARN ][discovery                ] [hostname] waited for 30s and no initial state was set by the discovery

At src/main/java/org/elasticsearch/discovery/srv/SrvUnicastHostsProvider.java
protected List<Record> lookupRecords(String query, int type) throws TextParseException {
	logger.trace ("lookupRecords, query: {} type {}", query, type);
        Lookup lookup = new Lookup(query, type);
	logger.trace ("lookup succesfull");

        if (this.resolver != null) {
            lookup.setResolver(this.resolver);
        }

        Record[] records = lookup.run();
        return records == null ? new ArrayList<Record>() : Arrays.asList(records);
    }

@falencastro Wonderful! Would you like to submit a PR that we could iterate on?
Could you try stepping into the Lookup constructor in a debugger to determine which line it gets stuck on?

Hi Folks,
any progress on this stuff?
BTW, I'm currently running into an error complaining about a missing file which is needed in ES 2.x:
{code}
Verifying https://github.com/github/elasticsearch-srv-discovery/archive/1.5.1.zip checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)
ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in plugin zip
Error installing SRV plugin. Maybe it is allready installed.
{code}


Bug: Errors with nil value trigger nil pointer dereference panic in JSON formatter

First of all, thanks for the amazing library! Unfortunately, below bug is triggering a nil pointer dereference panic.
Considering code:
package main

import (
	log "github.com/sirupsen/logrus"
)

type GenericError struct{}

// Error converts the error into a string
func (e GenericError) Error() string {
	return "test"
}

func main() {
	var genErr *GenericError
	log.SetFormatter(&log.JSONFormatter{})
	log.WithField("error", genErr).Info("Test") // Same for .WithError(genErr)
}

Will return:
panic: value method main.GenericError.Error called using nil *GenericError pointer

goroutine 1 [running]:
main.(*GenericError).Error(0x0)
	<autogenerated>:1 +0x39
github.com/sirupsen/logrus.(*JSONFormatter).Format(0xc00001a080, 0xc00007c150)
	/tmp/gopath383329124/pkg/mod/github.com/sirupsen/logrus@v1.8.1/json_formatter.go:70 +0x174
github.com/sirupsen/logrus.(*Entry).write(0xc00007c150)
	/tmp/gopath383329124/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:279 +0x56
github.com/sirupsen/logrus.(*Entry).log(0xc000068f60, 0x4, {0xc000018068, 0x4})
	/tmp/gopath383329124/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:251 +0x1f7
github.com/sirupsen/logrus.(*Entry).Log(0xc00007c0e0, 0x4, {0xc000068f60, 0x4b6060, 0x0})
	/tmp/gopath383329124/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:293 +0x4f
github.com/sirupsen/logrus.(*Entry).Info(...)
	/tmp/gopath383329124/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:310
main.main()
	/tmp/sandbox3248861311/prog.go:19 +0x9b

Program exited.

Go Playground example: https://go.dev/play/p/6DFobtugSz2
Seems issue got introduced by fix for https://github.com/sirupsen/logrus/issues/137

Hi @JensSkipr
It's not a bug. Let me explain for you !
at
var genErr *GenericError

You were created a "Pointer of GenericError struct type". So follow golang document !

The default value or zero-value of a pointer is always nil. Or you can say that an uninitialized pointer will always have a nil value.

at line
log.WithField("error", genErr).Info("Test") // Same for .WithError(genErr)

You passed genErr to second parameter of WithField method and expected to call method Error(). But previous, you were created a pointer variable and no initialize it and genErr still always nil. And you can not pass a nil pointer to WithField method !
But
If you pass genErr as "Pointer of Pointer of GenericError" like this
log.WithField("error", &genErr).Info("Test") // Same for .WithError(genErr)

it will work as you expect !!!

@scila1996 thanks for checking my bug report. But I don't really agree it isn't a bug (unless the documentation strictly states this is not supported).
The provided code was a simplified version to be easy to replicate. But we actually have following situation:
package main

import (
	log "github.com/sirupsen/logrus"
)

type GenericError struct{}

// Error converts the error into a string
func (e GenericError) Error() string {
	return "test"
}

func main() {
	log.SetFormatter(&log.JSONFormatter{})
	result, genErr := doSomething()
	if genErr != nil || !result.OK {
		log.WithError(genErr).WithField("result", result).Info("Test")
	}
}

type Result struct {
	OK    bool
	Error string
        // Data fields omitted
}

func doSomething() (*Result, *GenericError) { // We're aware it is against Golang guidelines to return a specific error type
	return &Result{false, "test error"}, nil
}

So, expected result is error is just logged as nil. But since https://github.com/sirupsen/logrus/blob/79c5ab66aa2ce7d9ff7b3c437ebc22fcc519a967/json_formatter.go#L65-L73 is forcing the .Error call, it will throw a panic instead.
Therefore, I would like to suggest adding a if v == nil before the switch or inside case error to shield from this situation.

nil are mean variable is not initialized, does not to treat is as a value. Panic occurred can be significant for debug or unit testing before build or run app.

Fix implemented in our code using:
// Error converts the error into a string
func (e GenericError) Error() string
	if e == nil {
		return ""
	}
	return "test"
}


[Enhancement][n/a] Use Merge commits instead of rebasing PRs

Is your feature request related to a problem? Please describe.
I have difficulty following the git history of the repository because merged PRs are rebased on top of main. This means that the commit IDs change and git doesn't know that two commits with different IDs represent the same change.
Describe the solution you'd like
Please use merge commits and avoid rebasing/rewriting the git commits in PRs.
Describe alternatives you've considered
n/a
Additional context
n/a

My reason for raising this is that it breaks git branch --merged, but if no-one else cares about that it's not a huge deal :slightly_smiling_face:

@smlx as of now I think all opensearch project repos is doing that.
If no concerns please close this issue :)


Support the Llam2 model

Set up to select the llama2 model. First, connect via the API.

given the architecture is nextjs and client-side, might best be solved using an API to a hosted solution. #140


MacOS installation is outdated

Describe the bug
The mac install instructions are outdated. cask is no longer a brew command. I think you just do brew install xquartz or brew install --cask xquartz  instead of brew cask install xquartz.
To Reproduce
Steps to reproduce the behavior:
(.venv) (base) peter@Peters-MBP14 % brew config
HOMEBREW_VERSION: 3.5.2
ORIGIN: https://github.com/Homebrew/brew
HEAD: fba051f98f609634ae4392d0053ddfc627f8cdb3
Last commit: 3 months ago

(.venv) (base) peter@Peters-MBP14 % brew cask install xquartz
Error: `brew cask` is no longer a `brew` command. Use `brew <command> --cask` instead.

Desktop (please complete the following information):

OS: Mac 12.5.1
Textract textract-1.6.5-py3-none-any.whl
Python 3.9
Virtual environment yes

Although it looks like they're correct in github


Privacy note for EU/APAC Customers

Need to send a note to customers (Specially EU) to let them know that their data might be hosted in the US for a temporary period of time.  We'll have to delete this line after the dev-teams make the changes necessary to prevent this data transfer to the US
Summarize the change in the pull request title
Describe your change, specifically why you think it's needed.
Fixes #Issue_Number (if necessary)

@TheAviPrasad : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Replaced by a PR on the private repo. Closing.


Running "WWBD: Create Environment" in vscode repo results in error "6"

The extension doesn't use the terminal for anything; I create the virtual environment via Python code.
https://github.com/brettcannon/WWBD/blob/fc0cd8b67f0226764b1cd9161493cd0452f3094a/python-src/wwbd/venv.py#L24
Did you paste that command into the terminal and are saying the created virtual environment is broken? Otherwise it looks like your Python install may be busted (the dyld means the linker can't find a file it needs).

I put that into terminal based on the error I saw, which said the executing that failed.

I take it the command was listed in a traceback in the WWBD output channel? If so then it means when your install of Python tried to create a virtual environment it was unable to install pip into it.
How did you install your copy of Python? And is it able to create a virtual environment somewhere else (python -m venv <name for environment>)?
Regardless, this isn't due to the extension itself but your installation of Python and how it created the virtual environment.

➜  Desktop python -m venv test-venv
zsh: command not found: python

I have to use python3, I believe this may be the standard for recent MacOS versions.

Actually, Python has been removed from macOS starting in 12.3, so now there's no standard 😅.
But yes, Python 2 is not supported (see the requirements section of the README). It's also not supported by the Python extension either. 😁
Thanks for figuring out the cause!


Where can I get this checkpoint?

https://mmdetection.readthedocs.io/en/latest/2_new_data_model.html
load_from = 'checkpoints/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth'
Where can I get this checkpoint?
It seems not mentioned in this doc

https://download.openmmlab.com/mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco/mask_rcnn_r50_caffe_fpn_mstrain-poly_3x_coco_bbox_mAP-0.408__segm_mAP-0.37_20200504_163245-42aa3d00.pth


Sample SettingsPage.OnNavigatedTo() defends against calling ToString …

…on a null e.Parameters, but then calls ToString() on the result of Deserialize(null) which is also null.
A recent change removed the parameter from the navigate in Shell.xaml

Some of these changes already snuck in with my other pull request. Still getting the hang of this git thing.


bump ilastik version

update to new tag
see https://github.com/ome/omero-guide-ilastik/pull/21

Looks alright, the link works 👍


add delete function

I have added a delete function that allows you to remove any individual task.
I adjusted the styling a little bit.

everything looks good. good work.
maybe a request if you really want to delete ?


Dockerfile

Would it make sense to include an official Dockerfile for this project?

Based on https://hub.docker.com/_/redis/
Dockerfile
FROM debian:jessie

# add our user and group first to make sure their IDs get assigned consistently, regardless of whatever dependencies get added
RUN groupadd -r disque && useradd -r -g disque disque

RUN apt-get update && apt-get install -y --no-install-recommends \
		ca-certificates \
		curl \
	&& rm -rf /var/lib/apt/lists/*

# grab gosu for easy step-down from root
RUN gpg --keyserver pool.sks-keyservers.net --recv-keys B42F6819007F00F88E364FD4036A9C25BF357DD4
RUN curl -o /usr/local/bin/gosu -fSL "https://github.com/tianon/gosu/releases/download/1.7/gosu-$(dpkg --print-architecture)" \
	&& curl -o /usr/local/bin/gosu.asc -fSL "https://github.com/tianon/gosu/releases/download/1.7/gosu-$(dpkg --print-architecture).asc" \
	&& gpg --verify /usr/local/bin/gosu.asc \
	&& rm /usr/local/bin/gosu.asc \
	&& chmod +x /usr/local/bin/gosu

ENV DISQUE_VERSION 1.0-rc1
ENV DISQUE_DOWNLOAD_URL https://github.com/antirez/disque/archive/1.0-rc1.tar.gz
ENV DISQUE_DOWNLOAD_SHA1 b82a588a12994a14d5a2817b485be2f21808ad6a

RUN buildDeps='gcc libc6-dev make' \
	&& set -x \
	&& apt-get update && apt-get install -y $buildDeps --no-install-recommends \
	&& rm -rf /var/lib/apt/lists/* \
	&& mkdir -p /usr/src/disque \
	&& curl -sSL "$DISQUE_DOWNLOAD_URL" -o disque.tar.gz \
	&& echo "$DISQUE_DOWNLOAD_SHA1 *disque.tar.gz" | sha1sum -c - \
	&& tar -xzf disque.tar.gz -C /usr/src/disque --strip-components=1 \
	&& rm disque.tar.gz \
	&& make -C /usr/src/disque \
	&& make -C /usr/src/disque install \
	&& rm -r /usr/src/disque \
	&& apt-get purge -y --auto-remove $buildDeps

RUN mkdir /data && chown disque:disque /data
VOLUME /data
WORKDIR /data

COPY docker-entrypoint.sh /entrypoint.sh
COPY disque.conf /usr/local/etc/disque/disque.conf
ENTRYPOINT ["/entrypoint.sh"]

EXPOSE 7711 17711
CMD [ "disque-server", "/usr/local/etc/disque/disque.conf" ]

docker-entrypoint.sh
#!/bin/bash
set -e

if [ "$1" = 'disque-server' ]; then
	chown -R disque .
	exec gosu disque "$@"
fi

exec "$@"

Thanks @benbro. I've pushed a working Dockerfile on Docker Hub here if anyone is interested.
@antirez Would love any suggestions / improvements on the Dockerfile if you have any. I also am not completely sure about the best way to spin up > 1 node and getting them to meet in the docker world.

@swarajban Perhaps a Compose file? https://docs.docker.com/compose/compose-file/

@swarajban I'd go with a much smaller base image and not do the building in the docker image but rather assume the binary has already been built.
For example:
FROM base/archlinux:latest
ADD disque.conf /etc/disque/disque.conf
ADD src/disque-server /disque-server
ADD src/disque /disque

EXPOSE 7711
ENTRYPOINT [ "/disque-server", "/etc/disque/disque.conf" ]

Works quite nicely and is nice and small. Personally I'd prefer a Dockerfile for disque-server and one for the client. That way testing/use of the Disque CLI in a Docker setup doesn't need it locally installed. Gliderlabs did this quite well for Consul.  For and example of how that works out so nicely see https://getcarina.com/docs/tutorials/data-stores-redis/ where it is the redis-client. Having a client-only Docker image would be quite nice. Failing that adding it into the server image would not be the end of the world.
Exposing the cluster bus port doesn't actually do any good here. More on that follows. As for spinning up multiple, Compose can do that part, but the assumption that the cluster bus port is base+10000 is broken by using standard docker networking.  You can only do it if you're running in a docker cluster and use host networking. See issue #164 for details. As such until that is fixed a compose file will have to use the host networking option and thus only work when your docker setup points to a Docker Swarm.
Cheers,
Bill

Thanks @therealbill. Agree with on all your points:

I want to build disque and check it into the repo
Switch to a smaller linux distribution (looks like the Docker team is trying to move everything to alpine linux so I think I'll do the same)
Have a separate client & server Dockerfile

I'm currently using Kubernetes instead of compose & swarm, so I have related, but separate issues w.r.t. the cluster set up. Would definitely appreciate any advice / suggestions for recommended compose configurations

@swarajban I started working a docker-compose.yml file up. Until #164 is addressed (see Salvatore's latest comment on it) there are two cases - meaning two compose files.
Case 1:  A docker swarm
In this case the compose file is fairly simple and straightforward. It can do everything but the closet meet commands as that isn't something compose does. But I can get one here (or in a gist) which which sets up the containers at least.
Case 2: Single Docker host
This is commonly the way developers and experimenters try it out. The catch here is having to use different ports for each host AND using host networking. Doable, but a tad more complex because you have to combine configuring disc with different ports and use host networking. Doable, but I'm not sure the same file can do either. I'll work on it this weekend to validate and past/gist one in. Or I suppose I could fork-and-PR. ;)
I either case Docker-Compose can't configure the cluster. I could write a tool which does it (and likely will)  but that gets outside the context of a docker or compose file.

While this may be slightly off topic, I do have Disque clusters working in Vagrant if anyone is in need of a working dev cluster now:
https://github.com/canardleteer/vagrant-disque
(@swarajban glad to hear that Alpine is getting some traction, I've been using it for years, and I'll probably convert the above to use Alpine instead of Ubuntu when I get a chance. It's an ideal slim distro.)


Refactor to use herdstat binary release instead of docker image

This should also include the ability to set the used herdstat version as an action input (see here).

Adressed by version v8.1.0.


pipeline()...exec() returns Promise<[error: Error | null, result: unknown][] | null>

Hey there,
while migrating to 5.x I noticed that our pipelines have ts errors now. Consider this example:
cacheAndExpireValue(value: string): Promise<string | null> {
    return cluster
        .pipeline()
        .getset("someKey", value)
        .expire("someKey", 42)
        .exec()
        .then((result) => {
            if (result[0][0]) {
                throw result[0][0];
            }

            return result[0][1];
        });
}

I'd expect that result[0][1] should always be a string (correct me if I am wrong please) but exec() returns Promise<[error: Error | null, result: unknown][] | null> thus the need to cast the result manually, otherwise we get a compiler error since we collide with the return type of cacheAndExpireValue():
TS2322: Type 'Promise<unknown>' is not assignable to type 'Promise<string | null>'.
   Type 'unknown' is not assignable to type 'string | null'.
     Type 'unknown' is not assignable to type 'string'.

If I am right and we can only ever expect string here, why not let exec() return Promise<[error: Error | null, result: string][] | null> or, if I am wrong, at least be able to use a generic like exec<T>(): Promise<[error: Error | null, result: T][] | null> (or similar)?

Yes, result[0][1] should be of type string, and result[1][1] should be of type number, but because of TypeScript restrictions, we have to make them unknown. This makes pipeline() a bit inconvenient but I haven't thought of any solutions to improve this.

Could you elaborate a bit on those typescript restrictions you're referring to?

@schw4rzlicht since the pipeline has a dynamic list of commands in it, and each command has a different return type, it is not possible to correctly type the response array from the pipeline execution.  Hence the need for the pipeline creator (your code) to interpret the results based on what is in the pipeline.

Should be possible to type correctly with the pipeline array style, i think it's also possible with chaining if each call modified the context


Less frequent reporting of potentially stuck records.

Once the threshold has been reached, don't report on every sample, but instead in a capped exponentially decaying manner.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

R: @priyansndesai


Add Google Vertex AI (i.e. Gemini) in PROVIDER_MODELS of config.py

Thank you for the great work and it is prominent! Previously I used Google Vertex AI (i.e. Gemini) for doing something similar to yours but this repository is way better than mine.
It would be great if you can add Google Vertex AI (i.e. Gemini) in PROVIDER_MODELS of config.py (and add some codes to handle Vertex AI) becoz it is more intelligent on parsing and handling different languages rather than others who can only work well with English content

Also openrouter.ai aswell.

Thank you everyone, and thank you, @huibrian , for your nice words. I hope this can be really helpful. For your information, we are using the Litellm library for LLM extraction, which allows us to support almost any of the 100-plus LLM providers, including Google Vortex AI. If you refer to our code, when you are passing the name of the provider, you can pass any name that the Litellm library supports, including Google Vortex. Then, you can pass your instructions and get the result. You're not bound to one or two providers, or anything else. You can even pass Ollama to have a local LLM on your machine or Hugging Face, for example. Open Rotor Light LLM is also supported. If you look at the litellm document for all the providers (https://docs.litellm.ai/docs/providers), we also cover the same things in our library. Here an example of using different provider:
async def extract_structured_data_using_llm(provider: str, api_token: str = None, extra_headers: Dict[str, str] = None):
    print(f"\n--- Extracting Structured Data with {provider} ---")

    if api_token is None and provider != "ollama":
        print(f"API token is required for {provider}. Skipping this example.")
        return

    extra_args = {}
    if extra_headers:
        extra_args["extra_headers"] = extra_headers

    async with AsyncWebCrawler(verbose=True) as crawler:
        result = await crawler.arun(
            url="https://openai.com/api/pricing/",
            word_count_threshold=1,
            extraction_strategy=LLMExtractionStrategy(
                provider=provider,
                api_token=api_token,
                schema=OpenAIModelFee.schema(),
                extraction_type="schema",
                instruction="""From the crawled content, extract all mentioned model names along with their fees for input and output tokens.
                Do not miss any models in the entire content. One extracted model JSON format should look like this:
                {"model_name": "GPT-4", "input_fee": "US$10.00 / 1M tokens", "output_fee": "US$30.00 / 1M tokens"}.""",
                extra_args=extra_args
            ),
            bypass_cache=True,
        )
        print(result.extracted_content)

async main():
    await extract_structured_data_using_llm("gemini/gemini-1.5-pro", os.getenv("GEMINI_API_KEY"))

if __name__ == "__main__":
    asyncio.run(main())


UI of the footer

We can change the UI of the footer to make it look more professional and collaborative.

What are the changes you would like to do??

@PiyushKalyanpy we can add the logo of the website at the bottom too, at the left side and keep the other data on the right side of the page

like this screen shot, we can do some changes

That's great, @ShivangRawat30 I am assigning this issue to you.

please assign me this issue

What are the updates @ShivangRawat30

We are closing this issue because it has been inactive for a long time.


[Icon Request] Support for Kubernetes files

[x] I'm sure this issue is not a duplicate.
[x] I'm submitting an icon request:

Type: filename
Icon Name: kubernetes
Sample original Icon:
Filenames: kubernetes-deployment.yml, kubernetes-deployment.yaml, kubernetes-service.yml, kubernetes-service.yaml

Any documentation?

What kind of documentation are you looking for? On the naming convention or on Kubernetes in general?

The naming.

Here is an example of a deployment file.  https://kubernetes.io/docs/concepts/workloads/controllers/deployment/.

From what I'm seeing the pattern is *-deployment.yaml. Unfortunately, we can't support glob patterns simply because vscode does not support that either.
So I have to decline your request. But you can use our customization feature to assign the icon to the namings your use in your projects.

That's too bad! Thanks for checking!


fixed js / css options source - now accepting arrays

fixed issues when providing arrays of files ... these were rendered with comma as the whole array was assigned to the head / body - not the single elements itself.

still have to add the related tests

Thanks! I went ahead and corrected this bug with 372fe1fc184c8254c76a2f0a312638c76f7d31fa and published a new release. I did not add any tests around this, but will gladly accept if you still want to write them.


ParentedTree breaks deepcopy

Can deepcopy Tree but not ParentedTree, not sure if may be related to this: https://github.com/nltk/nltk/issues/130

from nltk.tree import Tree,ParentedTree
t1 = Tree.fromstring("(TOP (S (NP (NNP Bell,)) (NP (NP (DT a) (NN company)) (SBAR (WHNP (WDT which)) (S (VP (VBZ is) (VP (VBN based) (PP (IN in) (NP (NNP LA,)))))))) (VP (VBZ makes) (CC and) (VBZ distributes) (NP (NN computer))) (. products.)))")
t2 = copy.deepcopy(t1)
t3 = ParentedTree.fromstring("(TOP (S (NP (NNP Bell,)) (NP (NP (DT a) (NN company)) (SBAR (WHNP (WDT which)) (S (VP (VBZ is) (VP (VBN based) (PP (IN in) (NP (NNP LA,)))))))) (VP (VBZ makes) (CC and) (VBZ distributes) (NP (NN computer))) (. products.)))")
import copy
t4 = copy.deepcopy(t3)
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 351, in _reconstruct
item = deepcopy(item, memo)
File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 351, in _reconstruct
item = deepcopy(item, memo)
File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 352, in _reconstruct
y.append(item)
File "/usr/local/lib/python2.7/site-packages/nltk/tree.py", line 1061, in append
self._setparent(child, len(self))
File "/usr/local/lib/python2.7/site-packages/nltk/tree.py", line 1221, in _setparent
raise ValueError('Can not insert a subtree that already '
ValueError: Can not insert a subtree that already has a parent.

I'm not sure what's the use of copying the Tree object but it seems like ParentedTree can't be deeply copied.
>>> import copy
>>> from nltk.tree import Tree,ParentedTree

>>> t1 = Tree.fromstring("(TOP (S (NP (NNP Bell,)) (NP (NP (DT a) (NN company)) (SBAR (WHNP (WDT which)) (S (VP (VBZ is) (VP (VBN based) (PP (IN in) (NP (NNP LA,)))))))) (VP (VBZ makes) (CC and) (VBZ distributes) (NP (NN computer))) (. products.)))")

>>> t2 = copy.deepcopy(t1)

>>> t3 = ParentedTree.fromstring("(TOP (S (NP (NNP Bell,)) (NP (NP (DT a) (NN company)) (SBAR (WHNP (WDT which)) (S (VP (VBZ is) (VP (VBN based) (PP (IN in) (NP (NNP LA,)))))))) (VP (VBZ makes) (CC and) (VBZ distributes) (NP (NN computer))) (. products.)))")

>>> t4 = copy.deepcopy(t3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 351, in _reconstruct
    item = deepcopy(item, memo)
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 351, in _reconstruct
    item = deepcopy(item, memo)
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 352, in _reconstruct
    y.append(item)
  File "/usr/local/lib/python2.7/dist-packages/nltk/tree.py", line 1064, in append
    self._setparent(child, len(self))
  File "/usr/local/lib/python2.7/dist-packages/nltk/tree.py", line 1224, in _setparent
    raise ValueError('Can not insert a subtree that already '
ValueError: Can not insert a subtree that already has a parent.

And having NLTK object breaking native Python copy isn't a good thing...
We should look into this, most probably the error is coming from how ParentedTree are made up of ParentedTree:
>>> t3
ParentedTree('TOP', [ParentedTree('S', [ParentedTree('NP', [ParentedTree('NNP', ['Bell,'])]), ParentedTree('NP', [ParentedTree('NP', [ParentedTree('DT', ['a']), ParentedTree('NN', ['company'])]), ParentedTree('SBAR', [ParentedTree('WHNP', [ParentedTree('WDT', ['which'])]), ParentedTree('S', [ParentedTree('VP', [ParentedTree('VBZ', ['is']), ParentedTree('VP', [ParentedTree('VBN', ['based']), ParentedTree('PP', [ParentedTree('IN', ['in']), ParentedTree('NP', [ParentedTree('NNP', ['LA,'])])])])])])])]), ParentedTree('VP', [ParentedTree('VBZ', ['makes']), ParentedTree('CC', ['and']), ParentedTree('VBZ', ['distributes']), ParentedTree('NP', [ParentedTree('NN', ['computer'])])]), ParentedTree('.', ['products.'])])])

@ssantichaivekin and I have found a fix to this, which also extends to the ImmutableTree object.
Native Python copy first looks for __copy__ and __deepcopy__ (which don't exist in Tree), then if it doesn't find it looks for __reduce_ex__ and __reduce__ instead. Since all trees (Tree, ParentedTree, ImmutableTree) derive from the list base class, the trees inherit __reduce_ex__ from it. However, this doesn't necessarily apply to produce the intended behavior. For example, here for list-like objects we will try to call append in deepcopy, which will break both ParentedTree and ImmutableTree objects and raise an exception.
The easiest way to fix this is to define __copy__ and __deepcopy__ methods that call NLTK tree's copy. NLTK tree objects have their own copy, with an option to make it a deep copy, so no other code has to be modified. We are slightly concerned that this fix may not support pickle, but the one or two examples we checked seemed to work, since pickle seems to use append and/or extend and both of those are implemented for trees.

We will finish testing and submit a pull request soon!


@BusyJay
By using atomic crate, we can

remove nightly feature,
support atomic f64,
support atomic u64,
less locks,
less code.

I think we have support f64 and u64 already when nightly feature is enabled. Depends on the std library seems more reliable to me.

The atomic lib uses standard AtomicUsize in 64 bit system, and uses spinlock if falls back, it is ok here.

@BusyJay
As you can see in the header. RwLock<u64> drops the performance when the nightly feature is disabled.
More details in commit https://github.com/pingcap/rust-prometheus/pull/67/commits/22ca0841f2135a485facb50ff477f7d6ae1c12a4

LGTM

LGTM
Please add a document about nightly use.