CosineAnnealingWithWarmupScheduler breaks when using torchdata with a .cycle

Look at some other issues I've filed (https://github.com/mosaicml/composer/issues/1334).
** To reproduce
Steps to reproduce the behavior:

Use the CosineAnnealWithWarmupScheduler with a small warmup phase (e.g 10ba)
Use a torchdata datapipe with a .cycle (causes dataloader to have infinite length)

Some example code:
def overfit_dataloader(num_batches, batch_size, dir):
    # 3. Using print(item[indexes]) inside of the dataloader
    # loop, we see the same samples are printed everytime
    datapipe = dp.iter.FSSpecFileLister(dir)
    datapipe = datapipe.map(load_parquet)
    datapipe = datapipe.flatmap(identity)
    datapipe = datapipe.header(num_batches * batch_size)
    datapipe = datapipe.shuffle()
    # datapipe = datapipe.cycle()
    datapipe = datapipe.sharding_filter()
    # We want no parallelism b/c this might lead to batches
    # that are smaller than batch_size, which will break grad_accum
    return DataLoader(datapipe, batch_size=batch_size, num_workers=1)

When the warmup phase is finished the scheduler will crash

Expected behavior
Scheduler doesn't crash.
Additional context
Stacktrace:
  File "/home/vedant/Desktop/improved-ddpm-pytorch/train.py", line 61, in <module>
    run()
  File "/home/vedant/Desktop/improved-ddpm-pytorch/train.py", line 51, in run
    trainer.fit()
  File "/home/vedant/miniconda3/envs/imagen/lib/python3.9/site-packages/composer/trainer/trainer.py", line 1348, in fit
    self._train_loop()
  File "/home/vedant/miniconda3/envs/imagen/lib/python3.9/site-packages/composer/trainer/trainer.py", line 1555, in _train_loop
    scheduler.step()
  File "/home/vedant/miniconda3/envs/imagen/lib/python3.9/site-packages/torch/optim/lr_scheduler.py", line 154, in step
    values = self.get_lr()
  File "/home/vedant/miniconda3/envs/imagen/lib/python3.9/site-packages/torch/optim/lr_scheduler.py", line 252, in get_lr
    return [base_lr * lmbda(self.last_epoch)
  File "/home/vedant/miniconda3/envs/imagen/lib/python3.9/site-packages/torch/optim/lr_scheduler.py", line 252, in <listcomp>
    return [base_lr * lmbda(self.last_epoch)
  File "/home/vedant/miniconda3/envs/imagen/lib/python3.9/site-packages/composer/optim/scheduler.py", line 187, in scheduler_fn
    return scheduler(state, ssr)
  File "/home/vedant/miniconda3/envs/imagen/lib/python3.9/site-packages/composer/optim/scheduler.py", line 761, in __call__
    t_max = _convert_time(self.t_max, state, ssr=ssr)
  File "/home/vedant/miniconda3/envs/imagen/lib/python3.9/site-packages/composer/optim/scheduler.py", line 136, in _convert_time
    raise RuntimeError('Cannot convert time, as state.dataloader_len is None.')
RuntimeError: Cannot convert time, as state.dataloader_len is None.

Can fix the issue by setting t_max=100ba, which makes sense since the scheduler cannot know what 1dur corresponds to in an infinite dataset. Closing.


Compare a file in a dir with the similar file in a different dir recursively.

Hi,
I am trying to use snapshot testing to compare two yaml files in different dirs. One of the file is created by a test class and the other one is present in our repo.
var file1 = File.ReadAllText("../../../../../file1.yaml");
Directory.SetCurrentDirectory("../../../../../");
var file2 = File.ReadAllText("file2.yaml");
await Verifier.Verify(file1).UseTextForParameters(file2).
The above few lines code is the simple code I am trying with first. File2 is a nested deep yaml file.
I am learning how to use verify now and trying to use  to compare two deeply nested yaml files. Is this even possible?

I was able to use verify.diffplex plugin to compare the files.


Fix copy-paste typo: "CONFIG_" prefix in structuredExtraConfig patches

Description of changes
Fix copy-paste typo: "CONFIG_" prefix in structuredExtraConfig patches

Unintentional typo, due to copy-paste from linux-surface upstream config

Things done

[ ] Tested the changes in your own NixOS Configuration
[ ] Tested the changes end-to-end by using your fork of nixos-hardware and
importing it via <nixos-hardware> or Flake input

bors merge


Translate content/pt/auto_scaling.md to portuguese

Hi Matheus, thanks for submitting this! I've left some suggestions. I think would be nice if you could split this PR into other ones (keep the auto_scaling.md translation file in this PR and open another ones for abstraction.md and agile_software_development.md.

I've just seen you already opened https://github.com/cncf/glossary/pull/312 - so would be missing just other PR to abstraction.md :)

I think regarding the links to the other pages (scale, virtualizing and containerizing as in the english doc) maybe would be better actually to keep the links to the english pages? (as we did here)
Then once we have them, we'd need to update with the link to the pt page. For now I opened a ticket in our board cc @edsoncelio @brunoguidone

Hi @MatheusPaes do you need some help with the requested changes? If yes, just ping one of the reviewers on the slack :D

@matheuspaes could you change the target branch from main to dev-pt?

alterado :) obrigado edson!!


add nvidia Dockfile

Hi, contributing an nvidia Dockerfile for users who have a need
I have tested this on Truenas Scale and it works as expected
Also I selected a lower version of ffmpeg n5.1.3 for nvidia driver compatibility
Cheers! 🍻

This works for me without any issues. Thank you.

Perfect, thanks @Markvis!


Convert "Invalid operation while connection is closing" to retriable ServiceBusException.

Convert "Invalid operation while connection is closing" to retriable ServiceBusException so the SDK retry logic could kick in automatically.
Reference PR: https://github.com/Azure/azure-sdk-for-net/pull/15984
Github Issues: https://github.com/Azure/azure-sdk-for-net/issues/9416, https://github.com/Azure/azure-sdk-for-net/issues/13637

@JoshLove-msft Can you check if a similar change would be needed for Azure.Messaging.ServiceBus?

I believe this is being handled here - https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/servicebus/Azure.Messaging.ServiceBus/src/Amqp/AmqpConnectionScope.cs#L1109


Sanitizer internal functions not found

Steps to reproduce (using rustc 1.27.0-nightly (acd3871ba 2018-05-10)):

cargo install cargo-fuzz (installed 0.5.2 for me)
git clone https://github.com/hsivonen/encoding_rs
git clone https://github.com/hsivonen/safe_encoding_rs_mem
cd encoding_rs
cargo fuzz run fuzz_encodings

Actual results:
WARNING: Failed to find function "__sanitizer_get_total_unique_coverage".
WARNING: Failed to find function "__sanitizer_reset_coverage".
INFO: Seed:<PHONE_NUMBER>
ERROR: __sanitizer_reset_coverage is not defined. Exiting.
Did you use -fsanitize-coverage=... to build your code?

Expected results:
Expected fuzzing to start like with older versions.

I followed your steps and I'm able to fuzz without any issues. I'm using rustc 1.28.0-nightly (a3085756e 2018-05-19). Which platform are you on?

(Updated steps to reproduce to include safe_encoding_rs_mem.)
I see this with two computers that run x86_64 version of Ubuntu 16.04. I tried updating to the latest nightly.
I'll try Ubuntu 18.04 next.

Clean install of Ubuntu 18.04 plus make, gcc and clang from system repos works.
Could this be a conflict with the version of system clang libs? Do you want to keep this open to investigate further, or shall we just close this as "upgrade Ubuntu and move on"?

I'm not sure to be honest, I also don't know what actionable thing there is here so I'd lean towards closing it.

OK. Let's close this. Thanks.


Bug bounty test - PLEASE DO NOT APPROVE. dbgoji

Bug bounty test - PLEASE DO NOT APPROVE. dbgoji

Hello @pastramahodu,
Thanks for your pull request! A Core Committer will review your pull request soon. For code contributions, you can learn more about the review process here.
Per the Mattermost Contribution Guide, we need to add you to the list of approved contributors for the Mattermost project.
Please help complete the Mattermost contribution license agreement?
Once you have signed the CLA, please comment with /check-cla and confirm that the CLA check is green.
This is a standard procedure for many open source projects.
Please let us know if you have any questions.
We are very happy to have you join our growing community! If you're not yet a member, please consider joining our Contributors community channel to meet other contributors and discuss new opportunities with the core team.

This PR has been automatically labelled "stale" because it hasn't had recent activity.
A core team member will check in on the status of the PR to help with questions.
Thank you for your contribution!


Many design bugs with SharePoint JSON

Hey,
there are many bugs with SharePoint JSON currently in the new lists version.
The action "SetValue" seems to not work anymore. We need a fix for that asap - were using it quite a bit in our solutions
Code example:

Additionally, could you please update the schema to allow for disabling the feature where clicking on an item selects it, and enable hiding the menu as seen in the screenshot when right-clicking on an item:

Also there is quite often different SharePoint JSON code from other lists in the List Formatting.
Thanks a lot!

Another bug is that when you use "Group by" with the new Lists Update, the list item stretches as shown in the screenshot
Screenshot without the new Update:

Screenshot with the new Update:

@gc-LMA : Sorry for responding late here, but i hope these issues are fixed for you now.


bump up image size to prevent bottom clipoff

@mambocab lots we can improve here, but for now, I think this should be safe to try, and roll back if we don't like the results

Lgtm. @mshuler, you good to merge and deploy?

Yep, looks good!


Different typing for SafeParseReturnType

Behavior description
Nowadays, the safeParse and safeParseAsync - functions for parsing Zod schemas - return types are based on SafeParseReturnType, this type is an union containing the following types:

SafeParseSuccess<Output>
export declare type SafeParseSuccessOutput<Output> = {
  success: true;
  data: output;
}

SafeParseError<Input>
export declare type SafeParseError<Input> = {
  success: false;
  error: ZodError<Input>;
}

Behavior problem
This kind of behavior, besids applying bad practices by design - here I'm talking about having const values true and false for success instead of a boolean - also needs an explicit declaration on what type it should be. This situation causes more complexity in the user code - complexity that should be handled by the library itself and it's against the TypeScript idea of good inference and usability.

Solution suggestion
My solution, for that case, would be implementing a single typing for SafeParseReturnType without unions - in a way that will not break existing code and will correctly adapt to any situation that the previously implemented unions covered.
Code:
export declare type SafeParseResult<InputOrOutput> = {
  success: boolean,
  data?: InputOrOutput,
  error?: ZodError<InputOrOutput>
}

As far as I see, the suggestion is exactly the opposite of TypeScript good practices - it incorrectly describes the return type and does not enable the standard practice of type narrowing based on eg if (result.success) { use(result.data); }
You are just using TypeScript wrong.
The correct expression to use in your screenshot is error: item.success === false ? item.error : undefined. Otherwise the code really is trying to access an undefined property (error  actually does really not exist in that case) for which TypeScript correctly raises an error.
If you object to that, I suggest not using TypeScript with the default strictness options.  Yes, it is ergonomically slightly unwieldy. Yes, every application or library WILL have the same problem and will need to use the same workaround. This is just standard TypeScript,

I think there's a middle path here, where we keep the safety of the current type (which is the main reason I use zod) but improve the ergonomics so that it can be used more flexibly.
export declare type SafeParseSuccessOutput<Output> = {
    success: true;
    data: output;
    error? : never
}

export declare type SafeParseError<Input> = {
    success: false;
    error: ZodError<Input>;
    data? : never
}

The never fields do not affect safety; this is still a tagged union that can be refined by checking success. However, it can also immediately be used as result?.data ?? default, or in other patterns where we legitimately don't care which side of the union we have because we can handle the undefined case. This avoids writing a lot of intermediate variables that are currently necessary to address such cases.

@scarletquasar
The type Zod implements is a disjoint union type. That's how disjoint union types are implemented. It's the only correct type in this situation.
Proposed type has 8 possible values instead of 2, and is just wrong.
Please consult TS documentation or any other literature on how to properly apply bad practices in your code.

Again, not the only correct type; the version with never branches also has only 2 distinct values (ignoring degenerate cases of explicit vs implicit undefineds), and solves the usability problem @scarletquasar is having.

As others have said, this is intended to be a discriminated union and it dramatically improves DX due to TypeScript's automatic type narrowing.

@colinhacks the suggestion I made in the PR is still a discriminated union and in addition to the DX benefits of that also allows for better DX when the user needs to respond with something even if parsing failed. This is a fairly common case and even came up in the tests for safeParse. Does that seem like a reasonable change?


[BUG] No OperatorHub Items Found

General information

OS: Linux
Hypervisor: KVM
Did you run crc setup before starting it (Yes/No)/? Yes

CRC version
crc version: 1.8.0+0a318dc
OpenShift version: 4.3.8 (embedded in binary)

CRC status
# Put `crc status` output here

CRC config
CRC VM:          Running
OpenShift:       Running (v4.3.8)
Disk Usage:      12.86GB of 32.72GB (Inside the CRC VM)
Cache Usage:     11.8GB
Cache Directory: /home/uebele/.crc/cache

Host Operating System
NAME=Fedora
VERSION="30 (Thirty)"
ID=fedora
VERSION_ID=30
VERSION_CODENAME=""
PLATFORM_ID="platform:f30"
PRETTY_NAME="Fedora 30 (Thirty)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:30"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f30/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=30
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=30
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"

Steps to reproduce

Login to OpenShift console as kube:admin
Goto Operators -> OperatorHub
or using CLI:
oc login as kubeadmin
oc get opsrc -n openshift-marketplace

Expected
I expect to see the catalog, used to work in previous versions of CRC
Actual
Web Console: No OperatorHub Items Found
CLI:
$ oc get opsrc -n openshift-marketplace
NAME                  TYPE          ENDPOINT              REGISTRY              DISPLAYNAME           PUBLISHER   STATUS        MESSAGE                                                                                AGE
certified-operators   appregistry   https://quay.io/cnr   certified-operators   Certified Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=certified-operators: context deadline exceeded   19d
community-operators   appregistry   https://quay.io/cnr   community-operators   Community Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=community-operators: context deadline exceeded   19d
redhat-operators      appregistry   https://quay.io/cnr   redhat-operators      Red Hat Operators     Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=redhat-operators: context deadline exceeded      19d

Logs
You can start crc with crc start --log-level debug to collect logs.
Please consider posting this on http://gist.github.com/ and post the link in the issue.

@Harald-U Can you try to describe one of the operator and see what events it have or what causing them as context deadline exceeded this looks like the operator not able to fetch the details over the internet. Also can you try to deploy a sample application on this cluster?
$ oc describe opsrc certified-operators  -n openshift-marketplace

Are you running behind a proxy?

I ran the "oc describe opsrc certified-operators  -n openshift-marketplace" about an hour ago, then went into a web meeting, and just ran it again. It is now giving results (showing packages), and the OperatorHub catalog in the console is now displaying the expected list.
Seems like it takes about an hour now to populate ... I never experienced this before in previous versions of CRC.
I guess you can close this issue. Thanks so much for your extremely quick responses!!!

I hit the same issue, after a fresh install of crc no operatorhub content appears:
$ crc version
crc version: 1.8.0+0a318dc
OpenShift version: 4.3.8 (embedded in binary)

The opsrc shows:
$ oc get opsrc -n openshift-marketplace
NAME                  TYPE          ENDPOINT              REGISTRY              DISPLAYNAME           PUBLISHER   STATUS        MESSAGE                                                                                AGE
certified-operators   appregistry   https://quay.io/cnr   certified-operators   Certified Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=certified-operators: context deadline exceeded   20d
community-operators   appregistry   https://quay.io/cnr   community-operators   Community Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=community-operators: context deadline exceeded   20d
redhat-operators      appregistry   https://quay.io/cnr   redhat-operators      Red Hat Operators     Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=redhat-operators: context deadline exceeded      20d

The packagemanifests are neither showing any result:
$ oc get packagemanifests -n openshift-marketplace
No resources found in openshift-marketplace namespace.

I'll check after a while for reproduce the behaviour of @Harald-U.

Seems like it takes about an hour now to populate ... I never experienced this before in previous versions of CRC.

@Harald-U Are you able to reproduce this each single time, we do have integration tests around marketplace where we deploy an etcd operator and we didn't observe any kind of issue during those tests.

@praveenkumar I tried it three times this morning (crc stop, crc delete, crc start) and the OperatorHub displayed its entry every time immediately after startup of CRC. Demo Effect? I'll try it again this afternoon, maybe its a overloaded server or network somewhere?

I tried it again in the afternoon (European time) and it still works ...

I am seeing similar problems ( running crc 1.8.0 on MacOS)
I have deleted my cluster and recreated.  The MESSAGE field seems to flip between dial tcp .....: i/o timeout and context deadline exceeded
i.e...
NAMESPACE               NAME                  TYPE          ENDPOINT              REGISTRY              DISPLAYNAME           PUBLISHER   STATUS        MESSAGE                                                                                AGE
openshift-marketplace   certified-operators   appregistry   https://quay.io/cnr   certified-operators   Certified Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=certified-operators: dial tcp <IP_ADDRESS>:443: i/o timeout   21d
openshift-marketplace   community-operators   appregistry   https://quay.io/cnr   community-operators   Community Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=community-operators: dial tcp <IP_ADDRESS>:443: i/o timeout   21d
openshift-marketplace   redhat-operators      appregistry   https://quay.io/cnr   redhat-operators      Red Hat Operators     Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=redhat-operators: context deadline exceeded                   21d

And after a few minutes, I see the
NAMESPACE               NAME                  TYPE          ENDPOINT              REGISTRY              DISPLAYNAME           PUBLISHER   STATUS        MESSAGE                                                                                AGE
openshift-marketplace   certified-operators   appregistry   https://quay.io/cnr   certified-operators   Certified Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=certified-operators: context deadline exceeded   21d
openshift-marketplace   community-operators   appregistry   https://quay.io/cnr   community-operators   Community Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=community-operators: context deadline exceeded   21d
openshift-marketplace   redhat-operators      appregistry   https://quay.io/cnr   redhat-operators      Red Hat Operators     Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=redhat-operators: context deadline exceeded      21d

and after another few minutes, I see
NAMESPACE               NAME                  TYPE          ENDPOINT              REGISTRY              DISPLAYNAME           PUBLISHER   STATUS        MESSAGE                                                                                AGE
openshift-marketplace   certified-operators   appregistry   https://quay.io/cnr   certified-operators   Certified Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=certified-operators: dial tcp <IP_ADDRESS>:443: i/o timeout   21d
openshift-marketplace   community-operators   appregistry   https://quay.io/cnr   community-operators   Community Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=community-operators: context deadline exceeded                21d
openshift-marketplace   redhat-operators      appregistry   https://quay.io/cnr   redhat-operators      Red Hat Operators     Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=redhat-operators: dial tcp <IP_ADDRESS>:443: i/o timeout    21d

I will leave it for a while to see if it recovers itself like it did for @Harald-U

@hosie same here. In my case, after some time, stills failing... How about your installation?

2 hours later and it is still failing.

After deleting the cluster and creating it again, it seems to be working now
NAMESPACE               NAME                  TYPE          ENDPOINT              REGISTRY              DISPLAYNAME           PUBLISHER   STATUS      MESSAGE                                       AGE
openshift-marketplace   certified-operators   appregistry   https://quay.io/cnr   certified-operators   Certified Operators   Red Hat     Succeeded   The object has been successfully reconciled   22d
openshift-marketplace   community-operators   appregistry   https://quay.io/cnr   community-operators   Community Operators   Red Hat     Succeeded   The object has been successfully reconciled   22d
openshift-marketplace   redhat-operators      appregistry   https://quay.io/cnr   redhat-operators      Red Hat Operators     Red Hat     Succeeded   The object has been successfully reconciled   22d

and the operator hub page on the console UI looks correct.
The only thing that might be different from before was that I was connected to my company's VPN when I created the cluster before.  I disconnected from the VPN, waited about an hour and the cluster was still broken.  When I deleted it and recreated while disconnected from the VPN, it seems to have worked.  I am not sure if this is a coincidence or whether the VPN was causing networking issues.

I'm running in to this every time I launch CRC. Running behind proxy.
Looking at the deployments for the operators they don't have any *_PROXY environment variables set. When I set them manually everything kicks off as expected and operatorhub items are populated.
crc version: 1.9.0+a68b5e0
OpenShift version: 4.3.10 (embedded in binary)

> oc get opsrc
NAME                  TYPE          ENDPOINT              REGISTRY        	DISPLAYNAME           PUBLISHER   STATUS 	MESSAGE                                                                                AGE
certified-operators   appregistry   https://quay.io/cnr   certified-operators   Certified Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=certified-operators: dial tcp: lookup quay.io on <IP_ADDRESS>:53: no such host   12d
community-operators   appregistry   https://quay.io/cnr   community-operators   Community Operators   Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=community-operators: dial tcp: lookup quay.io on <IP_ADDRESS>:53: no such host   12d
redhat-operators      appregistry   https://quay.io/cnr   redhat-operators	Red Hat Operators     Red Hat     Configuring   Get https://quay.io/cnr/api/v1/packages?namespace=redhat-operators: dial tcp: lookup quay.io on <IP_ADDRESS>:53: no such host	  12d

@opuk I'd prefer that you create a new issue for this, thanks!


the python debugger is going faster than conda activation

Environment data

VS Code version: 1.48.1 (user setup)
Extension version (available under the Extensions sidebar): v2020.8.105369
OS and version: Windows 10 v1909
Python version (& distribution if applicable, e.g. Anaconda):  Python 3.7.6 (default, Jan  8 2020, 20:23:39) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): conda
Value of the python.languageServer setting:  Microsoft

[NOTE: If you suspect that your issue is related to the Microsoft Python Language Server (python.languageServer: 'Microsoft'), please download our new language server Pylance from the VS Code marketplace to see if that fixes your issue]
Expected behaviour

Actual behaviour

XXX
Steps to reproduce:
[NOTE: Self-contained, minimal reproducing code samples are extremely helpful and will expedite addressing your issue]

Python programming with conda virtual env, for example, I have a conda env named tf
the debugger is going faster than conda activation which caused when I start the debugger at the first time, "Import Error" will be reported, since I install the module in env "tf" instead of env "base" but the debugger goes faster than conda activation, means that my python program is running in env "base" so that it can't find the target module to import.
this issue will reproduce every time after I terminate the Python Debug Console

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

This is a known issue with activating conda. This occurs due to some limitation we have with the VS Code. Here is a workaround for this:

Turn of auto activation. To do this, disable these two settings, ideally only for the project you are working on:

Exit VS Code.
Open Anaconda Prompt, navigate to your project.
Activate the environment that you want.
Run code . or code project.code-workspace to launch VS Code.

This will give you an activate environment, and the extension will not have to try and activate every time a terminal is created.

@karthiknadig Is this issue a bug? If it is, hope it can be fixed as soon as possible.

@ztibeike This is a dupe of #4203, #5664. We are trying get a common solution to various distributions and their activation mechanisms. In the case of conda, we tried conda run to avoid activating each time. But we had to revert it due to the limitations of conda run that broke debugging in other ways. For now, the workaround i mentioned above is the best way to get around this problem.


Theme Editing

Description

Let us select and edit numbers without having to guess the color in the color box.

Add HEX as a color option.

Ah, that seems to be a bug in the color picker. Is there any change when rotating the device to landscape/portrait? What is the device resolution? In fact, the interface should look like this:

Ah, that seems to be a bug in the color picker. Is there any change when rotating the device to landscape/portrait? What is the device resolution?

Sorry I should have included the device information.
I was using Latest Nox emulator, Android 9. The resolution is 1366x768.

I can't reproduce the problem, but there is now a native Windows version.  Maybe the emulator is no longer needed.


xss-lock.service: Use XDG_SESSION_ID and give hint about setting it.

I don't use this myself but this is based on discussion in https://bugs.debian.org/994762 and https://bugs.debian.org/1029513.
$GRAPHICAL_SESSION_ID doesn't appear to have ever been a thing.

Never noticed GRAPHICAL_SESSION_ID either.
One thing, systemctl --user show-environment shows XDG_SESSION_ID on my system, and I'm using a vanilla xsession (not i3, but spectrwm which shouldn't matter here).
I have use-session-bus in /etc/X11/Xsession.options.
Not sure if that's maybe also to include as a possibly nicer alternative?

I too have use-session-bus in /etc/X11/Xsession.options so I don't know if that is what has made it work for you.
Stuart in the Debian issue suggested clarifying the comment to say you need to do the import in ".xsessionrc or similar " to have an effect. I'll do that later today I hope.
But I found https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986100, I wonder if there is some recent Debian change which means this envvar isn't set in the user session anymore. I confess I'm not really sure how this stuff is supposed to fit together.

So I have restarted my session with a true reset, and now I only have XDG_RUNTIME_DIR set. I have no clue what set XDG_SESSION_ID in the first place.
Reading through Xsession.d/20dbus_xdg-runtime, there's no mention of XDG_SESSION_ID either, most of the uplifting is done though dbus-update-activation-environment --systemd which explicitly says it doesn't set XDG_* (only DBUS_*).
Grepping through the system I see:
src/other/systemd/src/login/pam_systemd.c
433:        r = pam_misc_setenv(handle, "XDG_SESSION_ID", id, 0);

pam_systemd says:
        2. The $XDG_SESSION_ID environment variable is initialized. If auditing is available and pam_loginuid.so was
           run before this module (which is highly recommended), the variable is initialized from the auditing
           session id (/proc/self/sessionid). Otherwise, an independent session counter is used.

and yes, I have libpam-systemd installed and active, but why XDG_SESSION_ID isn't set now?
Honestly, it's the usual mess of who-does-what in systemd land complicated by dbus activation, and I'm sure things are different when using a DE.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986100

I'm actually not using a service for xss-lock, but run it old-style from the start of the session itself and letting it die when the display connection goes off. This means xss-lock inherits the current (and true) session.
I have no idea how starting xss-lock with a service can work if you have two graphical sessions running? Looks like to me that importing XDG_SESSION_ID globally could resulting in the same bug of one session not locking correctly and thus as an argument to not import XDG_SESSION_ID.

I suspect the people using a service are not starting multiple graphical sessions, but I assume the reason XDG_SESSION_ID is not exported (any more?) is exactly because of that possibility?

I suspect the people using a service are not starting multiple graphical sessions, but I assume the reason XDG_SESSION_ID is not exported (any more?) is exactly because of that possibility?

It is my guess, since I don't see any way to start the same service multiple times. In that sense, import-environment seems to be always per-user, assumes you'll have just one graphical session.
And while saying this, I notice I have $DISPLAY set into show-environment... which would break spectacularly too in such a scenario. That is set by Xsession.d/20dbus_xdg-runtime.
I honestly don't know. I think suggesting import-environment is ok, but I see potential issues with services surrounding graphical user sessions if you think about remote login. All of that would seem broken.
Let me know if you want to reword some stuff or when you want to merge.

I've updating the wording along the lines Stuart proposed. From my PoV this can be merged now, thanks!


Mark the crate #[no_std]

as it currently stands I don't see why we shouldn't do this

That was always my plan. I looked into it before your pull request and the only thing prevent it was the Error if I recall correctly. Something which was being addressed in rust 1.8.1
Might not be relevant anymore though. Will investigate.

Yeah I pulled the latest version and moved as much as could to no_std using core instead of std, but error_in_core is still an unstable feature :( At least it is meant to be in the next update for rust

you can add a std feature 🙃


Remove guid in staged-director-config

Here is a yamldiff in the payload returned by staged-director-config
-   guid: "d0d8a581761ad2f9b9a7",
-   guid: "e05565e0f0d6b9bbad49",
-   guid: "972870601b177cc825d1",
-    guid: "7ed1a39e060aebb9a4ee",
-      guid: "0f6c7c24eed38802172a",
-    guid: "073ed694903051e8c17e",
-      guid: "16e06eb88f6f57e271f3",
-    guid: "5435d51a1252e56151df",
-      guid: "a5693ddc72c48f1e2fa4",
-    guid: "85b62bee9a0e5f6cca10",
-      guid: "08b7920ea01cc9ee57e5",

Basically remove all the guid fields as they are incresing noise in the configuration.
The roundtrip om configure-director <(om staged-director-config) is still working.
Caveat:

roundtrip only works if the iaas-configuration section is removed (regardless of this PR), as the section contains obscure creds fields (e.g. auth_json: '****' for gcp)
roundtrip doesn't work for vsphere until https://github.com/pivotal-cf/om/issues/170 is done, as that story adds necessary cluster guid back to the payload.

TODO: when I write the caveat, it reminds me that I didn't implement the logic to remove cluster guid for vsphere, will do that... Done
Diff for vsphere az-configuration part:
Before:
az-configuration:
  - name: AZ01
    guid: ed9fd3e85cc3851406e4
    clusters:
    - guid: a3eb20363a2fda7211cd
      cluster: canada
      resource_pool: oshawa

After:
az-configuration:
- name: AZ01
  clusters:
  - cluster: canada
    resource_pool: oshawa

@heycait: We just took a look. This parameter only exists in 2.2.

@jtarchie That sounds right. I know this is a new OpsMan feature with the long term goal of setting up and running on multiple IAASes. So far this is only applicable to vsphere (as mentioned during our call).


Sync with generator-jhipster, because of postgresql by default

Related to https://github.com/jhipster/generator-jhipster/issues/11736
and following https://github.com/jhipster/generator-jhipster/pull/12564

Merging since webflux is failing since some weeks


Does this operator support second order differentiation? (hashgrid)

Hi, excellent work;
I try use it in our program, and find it can not support second order differentiation;
Do you implement it?
Thank you very much;

No, it doesn't support higher order derivitives for now. I think the original implementation doesn't have a use case for this?

Yes, i also don't find this in the original implenmentation.
Thank you very much.
Do you have any plans to develop optimizations for higher order derivatives?

As the aim of this repo is to reproduce instant-ngp (and is unfinished yet), I don't have a plan to develop new functionality. Anyway, contributions are welcome.

Ok，thank you very much.


Feature/@mention

New feature to support @mention in sendMessage.

Thanks for your PR's, I'll take a look at them, and try to incorporate them into #202 :)


merge from arlac77/npm-package-template-esm-only

.npmignore

chore(npm): merge from template .npmignore

README.md

docs(README): update from template

Pull Request Test Coverage Report for Build 3070

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 83.871%

Totals

Change from base Build 3066:
0.0%

Covered Lines:
72

Relevant Lines:
87

💛  - Coveralls


Issues with getting started in custom deployment

I'm recently trying out activitypods deployment on my machine. Deploying-wise it is fine.
However, when using it, I encounter several issues. Some are minor, while some are major. I'm listing them here both for seeking help (if there is solution already), and also for reference by others.

My host has port 5000 taken already, so the frontend UI is launched with a different port (5001). However, many links has hardcoded port 5000. Manually changing it from address bar seems to work fine.
Sending a friend request to an existing mastodon account won't work. The UI says request is sent, but I see nothing from my mastodon account. Maybe this is related to localhost deployment, as I'm just trying it out. (p.s. Adding a friend on the [same] acitvipods instance works fine.)
When using the app (mastopod) and sending a new post, the friend will not receive it. (But sending direct messages works.)
Mastopod has a "compose and send" box in the "inbox" section. This is a bit unintuitive.
No followers and following is listed in Mastopod, although I have a friend from the default profile mangement UI.
Log-in uses the local username (e.g. test) instead of a full-form of username (e.g. @test@localhost:3000). This is incorret for a decentralised App.
I cannot access my Pod (of the activitypods account) from any regular Solid App, due to

Data are private, so cannot access without credential (I managed to access through LDP protocol with auth header [following the sign-up example but to log-in])
(The supposed) WebID Profile is private, so cannot be accessed outside
(The supposed) WebID Profile is incomplete as a Solid WebID Profile (#122), so actually Solid Apps won't be able to use it
Solid-OIDC is not supported (e.g. #229?), so cannot log-in from Solid Apps

Actually unable to use regular Solid Apps is a major issue for us. We would like to explore the Pod content through a databrowser App, but that is not possible for the moment. Any suggestions?

Hey @renyuneyun thanks for opening the issue and detailing your problems. And cool to see, you are interested in the project!
I'll try to answer what I can right now and review the others later :)

Regarding port 5000 for your frontend app, have you set the PORT in the frontend's .env  file? We might have to make the frontend create an informative warning or error, if that port is already busy.
Regarding communication with mastodon, I think this is, as you say, related to the localhost environment. It's been a long while since I last tried but what worked for me was using a service like https://loophole.site
Will investigate that
Thanks for the feedback!
Will investigate
I'm not sure if I understand you correctly. Currently, the mastopod app only shows a couple of preset pod providers from a list (this one + custom ones). But once you are on your pod provider, the username is tied to the domain.
You're right. Solid OIDC is not fully supported yet so login with compliant solid apps is not possible right now. Support for this is coming some time after August probably, when we received the new NLnet grant. You can find more infos on solid compliance here: https://activitypods.org/specs/solid/
If you want to make a resource or container public, you can do so by changing its WAC permission though.

I hope that helps a bit.
Also tagging @srosset81

Hi @Laurin-W, thanks for the quick reply!
$1. No, but when launching the frontend App, it warns me about port 5000, and directly offers to automatically switch to port 5001, with just my confirmation. I presume that should have the same effect?
$2-5. Thanks for that!
$6. I'm talking about the front-end App(s) I launched on my machine -- sure, one of them is mastopod (although the story is a bit complicated...). In particular, I'm trying to log-in from the profile management App. It's on localhost:5001, and the UI asks for username and password. However, putting @test@localhost:3000 as username will not work ("wrong username/password"), but putting test works. This is incorrect, because localhost:5001 (the frontend App) is not the same as localhost:3000 (the activitypods service), and it should in no way assume the user exists on localhost:3000 (the best it can assume, however, is localhost:5001, which is meaningless), let aside unable to interpret @test@localhost:3000 as a user test on localhost:3000.
$7. Thanks for sharing an estimated time. That's great to know. This seems like a chicken-and-egg issue, as normally you would use a data browser App for managing data and permission... (Yes, I know I can use HTTP requests directly for that, which is what I have been doing.)
(Personally, I have got interested in this project a year ago after seeing the Solid World presentation. But lacking usable interoperability with Solid is a pain for me -- which is the main point of this project instead of yet-another Mastodon-like system. Although, of course, I'm having opinions on Solid's current design, from research perspective.)

Regardign 1 -- Created an issue #249 for that :)
Regarding 6 -- Ah, I see what you mean. Right now the APods frontend app is a special app, we might change that in the future to make it all more spec-compliant. The management frontend app is coupled to the backend so you are not supposed to use it for managing accounts on other instances. The frontend's backend URI is specified in the .env file's REACT_APP_POD_PROVIDER_URL variable.
Regarding 7 -- I agree, using a solid data browser to manage this would be great and getting APods closer to solid interop :)

On 7 yes looking forward very much to being able to log in with Solid.
But also good for profiles to be public (for many people) and then people can read then with no authentication being necessary.  And follow the social graph.  When we test the way the public bits link together we don't have to worry about security!  Blogged about value of personal dat awhich is public in https://www.w3.org/DesignIssues/PersonalPublic.html


libmodel_android.a error when build android sdk

I'm trying build android sdk but got an error with error: (nm: invalid argument to -U/--unicode: dist/lib/libmodel_android.a)
when excuting: mlc_llm package --device android --output dist mlc-package-config.json
Whats wrong and how to fix it?
colab code:
!source $HOME/.cargo/env &&
export PATH=$PATH:/content/androidSdk/cmake/3.22.1/bin &&
export ANDROID_NDK=/content/androidSdk/ndk/26.2.11394342 &&
export TVM_NDK_CC=/content/androidSdk/ndk/26.2.11394342/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android24-clang &&
export TVM_HOME=/content/mlc-llm/3rdparty/tvm &&
export MLC_LLM_HOME=/content/mlc-llm &&
mlc_llm package --device android --output dist mlc-package-config.json
error:
[2024-05-10 09:21:39] INFO package.py:142: Clean up all directories under "dist/bundle"
[2024-05-10 09:21:39] INFO jit.py:43: MLC_JIT_POLICY = ON. Can be one of: ON, OFF, REDO, READONLY
[2024-05-10 09:21:39] INFO chat_module.py:379: Using model folder: /content/mlc-llm/dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC
[2024-05-10 09:21:39] INFO chat_module.py:380: Using mlc chat config: /content/mlc-llm/dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json
[2024-05-10 09:21:39] INFO package.py:198: Model lib is not specified for model "RedPajama-INCITE-Chat-3B-v1-q4f16_1". Now jit compile the model library.
[2024-05-10 09:21:39] INFO jit.py:120: Compiling using commands below:
[2024-05-10 09:21:39] INFO jit.py:121: /usr/bin/python3 -m mlc_llm compile /content/mlc-llm/dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC --opt 'flashinfer=1;cublas_gemm=1;faster_transformer=1;cudagraph=0;cutlass=1;ipc_allreduce_strategy=NONE' --overrides 'context_window_size=768;prefill_chunk_size=768;tensor_parallel_shards=1' --device android --output /tmp/tmpr7nwf6ja/lib.tar --system-lib-prefix gpt_neox_q4f16_1_74594e3964d8e2a2df0222b930ea2162_
[2024-05-10 09:21:43] INFO auto_config.py:69: Found model configuration: /content/mlc-llm/dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC/mlc-chat-config.json
[2024-05-10 09:21:43] INFO auto_config.py:153: Found model type: gpt_neox. Use --model-type to override.
Compiling with arguments:
--config          GPTNeoXConfig(use_parallel_residual=False, hidden_size=2560, intermediate_size=10240, num_attention_heads=32, num_hidden_layers=32, layer_norm_eps=1e-05, vocab_size=50432, rotary_pct=1.0, position_embedding_base=10000, context_window_size=768, head_dim=80, prefill_chunk_size=768, tensor_parallel_shards=1, ffn_out_dtype='float32', max_batch_size=80, kwargs={})
--quantization    GroupQuantize(name='q4f16_1', kind='group-quant', group_size=32, quantize_dtype='int4', storage_dtype='uint32', model_dtype='float16', linear_weight_layout='NK', quantize_embedding=True, quantize_final_fc=True, num_elem_per_storage=8, num_storage_per_group=4, max_int_value=7)
--model-type      gpt_neox
--target          {"thread_warp_size": 1, "host": {"kind": "llvm", "tag": "", "keys": ["arm_cpu", "cpu"], "mtriple": "aarch64-linux-android"}, "texture_spatial_limit": 16384, "max_threads_per_block": 256, "max_function_args": 128, "max_num_threads": 256, "kind": "opencl", "max_shared_memory_per_block": 16384, "tag": "", "keys": ["opencl", "gpu"]}
--opt             flashinfer=0;cublas_gemm=0;faster_transformer=0;cudagraph=0;cutlass=0;ipc_allreduce_strategy=NONE
--system-lib-prefix "gpt_neox_q4f16_1_74594e3964d8e2a2df0222b930ea2162_"
--output          /tmp/tmpr7nwf6ja/lib.tar
--overrides       context_window_size=768;sliding_window_size=None;prefill_chunk_size=768;attention_sink_size=None;max_batch_size=None;tensor_parallel_shards=1
[2024-05-10 09:21:43] INFO config.py:106: Overriding context_window_size from 768 to 768
[2024-05-10 09:21:43] INFO config.py:106: Overriding prefill_chunk_size from 768 to 768
[2024-05-10 09:21:43] INFO config.py:106: Overriding tensor_parallel_shards from 1 to 1
[2024-05-10 09:21:43] INFO compile.py:138: Creating model from: GPTNeoXConfig(use_parallel_residual=False, hidden_size=2560, intermediate_size=10240, num_attention_heads=32, num_hidden_layers=32, layer_norm_eps=1e-05, vocab_size=50432, rotary_pct=1.0, position_embedding_base=10000, context_window_size=768, head_dim=80, prefill_chunk_size=768, tensor_parallel_shards=1, ffn_out_dtype='float32', max_batch_size=80, kwargs={})
[2024-05-10 09:21:43] INFO compile.py:157: Exporting the model to TVM Unity compiler
[2024-05-10 09:21:47] INFO compile.py:163: Running optimizations using TVM Unity
[2024-05-10 09:21:47] INFO compile.py:181: Registering metadata: {'model_type': 'gpt_neox', 'quantization': 'q4f16_1', 'context_window_size': 768, 'sliding_window_size': -1, 'attention_sink_size': -1, 'prefill_chunk_size': 768, 'tensor_parallel_shards': 1, 'kv_cache_bytes': 0}
[2024-05-10 09:21:50] INFO pipeline.py:52: Running TVM Relax graph-level optimizations
[2024-05-10 09:21:58] INFO pipeline.py:52: Lowering to TVM TIR kernels
[2024-05-10 09:22:18] INFO pipeline.py:52: Running TVM TIR-level optimizations
[2024-05-10 09:22:59] INFO pipeline.py:52: Running TVM Dlight low-level optimizations
[2024-05-10 09:23:03] INFO pipeline.py:52: Lowering to VM bytecode
[2024-05-10 09:23:09] INFO estimate_memory_usage.py:57: [Memory usage] Function alloc_embedding_tensor: 3.75 MB
[2024-05-10 09:23:10] INFO estimate_memory_usage.py:57: [Memory usage] Function batch_decode: 3.52 MB
[2024-05-10 09:23:10] INFO estimate_memory_usage.py:57: [Memory usage] Function batch_prefill: 33.75 MB
[2024-05-10 09:23:10] INFO estimate_memory_usage.py:57: [Memory usage] Function batch_verify: 33.75 MB
[2024-05-10 09:23:10] INFO estimate_memory_usage.py:57: [Memory usage] Function create_tir_paged_kv_cache: 0.00 MB
[2024-05-10 09:23:10] INFO estimate_memory_usage.py:57: [Memory usage] Function decode: 0.04 MB
[2024-05-10 09:23:10] INFO estimate_memory_usage.py:57: [Memory usage] Function embed: 3.75 MB
[2024-05-10 09:23:10] INFO estimate_memory_usage.py:57: [Memory usage] Function prefill: 33.75 MB
[2024-05-10 09:23:10] INFO estimate_memory_usage.py:57: [Memory usage] Function softmax_with_temperature: 0.00 MB
[2024-05-10 09:23:13] INFO pipeline.py:52: Compiling external modules
[2024-05-10 09:23:13] INFO pipeline.py:52: Compilation complete! Exporting to disk
[2024-05-10 09:23:24] INFO model_metadata.py:96: Total memory usage: 1525.09 MB (Parameters: 1491.34 MB. KVCache: 0.00 MB. Temporary buffer: 33.75 MB)
[2024-05-10 09:23:24] INFO model_metadata.py:105: To reduce memory usage, tweak prefill_chunk_size, context_window_size and sliding_window_size
[2024-05-10 09:23:24] INFO compile.py:203: Generated: /tmp/tmpr7nwf6ja/lib.tar
[2024-05-10 09:23:25] INFO jit.py:128: Using compiled model lib: /root/.cache/mlc_llm/model_lib/b48ad1fec028c53c0957452e4c5f9694.tar
[2024-05-10 09:23:25] INFO package.py:246: Bundle weight for RedPajama-INCITE-Chat-3B-v1-q4f16_1, copy into dist/bundle/RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC
[2024-05-10 09:23:39] INFO package.py:271: Dump the app config below to "dist/bundle/mlc-app-config.json":
{
"model_list": [
{
"model_id": "RedPajama-INCITE-Chat-3B-v1-q4f16_1",
"model_lib": "gpt_neox_q4f16_1_74594e3964d8e2a2df0222b930ea2162",
"model_path": "RedPajama-INCITE-Chat-3B-v1-q4f16_1-MLC",
"estimated_vram_bytes":<PHONE_NUMBER>
}
]
}
Traceback (most recent call last):
File "/usr/local/bin/mlc_llm", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/mlc_llm/main.py", line 49, in main
cli.main(sys.argv[2:])
File "/usr/local/lib/python3.10/dist-packages/mlc_llm/cli/package.py", line 51, in main
package(
File "/usr/local/lib/python3.10/dist-packages/mlc_llm/interface/package.py", line 277, in package
validate_model_lib(
File "/usr/local/lib/python3.10/dist-packages/mlc_llm/interface/package.py", line 71, in validate_model_lib
available_model_libs = _get_model_libs(lib_path)
File "/usr/local/lib/python3.10/dist-packages/mlc_llm/interface/package.py", line 24, in _get_model_libs
global_symbol_map = cc.get_global_symbol_section_map(lib_path)
File "/usr/local/lib/python3.10/dist-packages/tvm/contrib/cc.py", line 215, in get_global_symbol_section_map
raise RuntimeError(msg)
RuntimeError: Runtime error:
nm: invalid argument to -U/--unicode: dist/lib/libmodel_android.a

Can you cross check the nm command in your system, this might due to some differencs in behavior of nm

https://www.man7.org/linux/man-pages/man1/nm.1.html

You can try confirm with which nm

You can try to run conda install nm

Hi @lengjing606, we've updated the android flow again. The latest documentation introduces the new flow https://llm.mlc.ai/docs/deploy/android.html. Could you please update the mlc-llm pip package and try again?
To summarize the new flow, first we need to make sure that "device": "android" exists in mlc-package-config.json, and then the command
mlc_llm package

does everything, without needing additional arguments.

@MasterJH5574 it work fine now,thanks

@MasterJH5574 it work fine now,thanks
I have updated the latest code, but I'm still getting the same error. How did you solve it? Did you install mlc_llm and TVM based on source code, or  install pre-built binaries? The mlc_llm I installed is from: mlc_llm_nightly_cu122-0.1.dev1213-cp311-cp311-manylinux_2_28_x86_64.whl and mlc_ai_nightly_cu122-0.15.dev315-cp311-cp311-manylinux_2_28_x86_64.whl.

@zw0241 Please update your mlc_llm package and mlc_ai package by following https://llm.mlc.ai/docs/install/mlc_llm.html and try again. The latest wheel is dev1224 for mlc_llm.

@zw0241 Please update your mlc_llm package and mlc_ai package by following https://llm.mlc.ai/docs/install/mlc_llm.html and try again. The latest wheel (as of this moment) is dev1224 for mlc_llm.

yes. It's resolved now. Thank you.

@lengjing606 @zw0241 thank you for getting back. Glad that the new flow works smoothly.


Add open existing project functionality for VS Code

Since we don't currently support the remote git option (due to lack of the clone API being exposed) the button will just open up the folder browse dialog directly.
Also cleaning up an unused string and fixing a missing await

Pull Request Test Coverage Report for Build 987380614

12 of 13 (92.31%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.003%) to 42.752%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

extensions/data-workspace/src/services/workspaceService.ts
0
1
0.0%

Totals

Change from base Build 987295558:
-0.003%

Covered Lines:
26167

Relevant Lines:
55572

💛  - Coveralls


Memory deallocation bug in Drawables

When running the drawing example I see a problem with memory comsumption; the process crashes after a few minutes. Same behavior under Ubuntu or Centos. This seems the buggy memory allocation:
amsoares@ubuntu:drawing$ pmap -x 4964 | grep renderD128 | wc -l # <-- Value increases until it crashes
64488
amsoares@ubuntu:drawing$ pmap -x 4964 | grep renderD128 | wc -l # <-- Already crashing
48439
amsoares@ubuntu:drawing$ pmap -x 4964 | grep renderD128 | wc -l # <-- Process dead
0
amsoares@ubuntu:drawing$
I'm not a developer, I'm just learning Rust and ggez/gwg.
Originally posted by @ccie18473 in https://github.com/ggez/good-web-game/issues/57#issuecomment-991799606

This is really bad.
I haven't yet had the time to look much into it, but for some reason memory leaks have started to come up in all of our examples where Meshes are created inside the draw loop and then dropped, meaning there is probably still something wrong with them not deallocating their ressources correctly.
I assume it's the textures not being released correctly, due to the hacky system we currently have in place for that, but I could be wrong.
I'll try to look into it as soon as I can.

Fixed in cf3a50b


Finding the installed version

Where on the git repo and on the src/ folder I can argue the actual version of the library?

That really depends on how you are installing the library. If you are using composer, your composer.lock file will show you, and your project's composer.json can limit it to what you want..
If you are manually downloading, you can choose one of the tagged versions here on GitHub by going to the Code tab, click the Branch: button, and select a Tagged version you would like to download. If you are downloading the master branch, then it doesn't really have a "version" as it's still in development.
The composer.json file does have a branch-alias describing the master branch as 1.1.x-dev, so that's the closest the branch has to a "version" per se.

@fpiraneo: Was @moderndeveloperllc's response sufficent? If you are not using are tagged release, the version would be inferred from branch-alias. That only denotes the alias of the master branch, as other branches can be inferred from there name. For instance,  v1.0 would be "1.0.x-dev".
@derickr did create PHPLIB-131 a while back to add a VERSION constant to the library, but I'm not sure it's necessary and would only add an extra step to the release process.

@jmikola I agree that a constant is a solution in need of a problem. Unlike a driver that could be variable depending on environment (like the PHP version), the library is always the version you want it to be for any deployment. Therefore no good reason I can think of to have version-checking code, so no need for a constant.

Sorry for late answer.
@jmikola the issue is that you really don't know which version of the library you have in use if you take the development of the code back in hand after some time; yes, I can create the "version" constant by myself... but why don't incorporate such constant directly on code and directly controlled by the library developers?

@fpiraneo isn't that a dependency management problem? If your project is managed by composer, you'll see the installed version by running composer show mongodb/mongodb. If you're using something else to track dependencies, it's their job to track your installed verions.

@alcaeus I don't use composer.

In my opinion it's up to the project developers to manage the libraries and which versions are in use. If you don't use composer you'll have to do it in whichever way suits you, and not rely on the libraries providing this information as a constant or a function. I have no idea how you manage your external libraries today, but I were to do it without composer I'd make sure to manually note it in my include or autoload file and then make sure to update it when updating the libraries. It's unrealistic to expect all libraries to provide this information in a way that it can be generalized and accessed the same way.
I do think, however, that it'd be good to have a @version in the documentation for each class reflecting the version it was last changed for.

Closing this due to inactivity. We ultimately decided not to introduce a version constant in PHPLIB-131.


Support faraday error

We could support faraday.
Maybe it should be opt-in?

The notified exception looks like below.

Hey @dtaniwaki! Thanks for suggesting this! For now, I don't think we want to specifically support Faraday. This can be added using metadata outside of the gem.


[SUPPORT][FLINK SQL] Can not insert join result into hudi table

Problem
I attempted to insert the join result of two Hudi tables into another Hudi table by using Flink SQL, the task executes normally and there are no errors reported on Flink Web UI, but there are no join results in wide table.
After checking Flink Web UI, i find something wrong with [stream_write] phase, it received join result but can not send to downstream operator.

To Reproduce
Steps to reproduce the behavior:
# create two source tables and insert some data into them
CREATE TABLE source_t1 (
id BIGINT PRIMARY KEY NOT ENFORCED,
name varchar(10),
t2_id BIGINT,
ts timestamp(3)
) WITH (
'connector' = 'hudi',
'path' = '/hudi/source_t1',
'table.type' = 'MERGE_ON_READ',
'hoodie.datasource.write.recordkey.field'='id',
'hoodie.datasource.write.operation'='insert',
'table.type'='MERGE_ON_READ',
'write.bucket_assign.tasks'='1',
'write.tasks' = '1',
'read.streaming.enabled' = 'true'
);

CREATE TABLE source_t2 (
id BIGINT PRIMARY KEY NOT ENFORCED,
name varchar(10),
ts timestamp(3)
) WITH (
'connector' = 'hudi',
'path' = '/hudi/source_t2',
'table.type' = 'MERGE_ON_READ',
'hoodie.datasource.write.recordkey.field'='id',
'hoodie.datasource.write.operation'='insert',
'table.type'='MERGE_ON_READ',
'write.bucket_assign.tasks'='1',
'write.tasks' = '1',
'read.streaming.enabled' = 'true'
);

# create wide table
create table inner_table (
id BIGINT PRIMARY KEY NOT ENFORCED,
name varchar(10),
name2 varchar(10),
ts timestamp(3)
) WITH (
'connector' = 'hudi',
'path' = '/hudi/inner_table',
'table.type' = 'MERGE_ON_READ',
'hoodie.datasource.write.recordkey.field'='id',
'hoodie.datasource.write.operation'='insert',
'table.type'='MERGE_ON_READ',
'write.bucket_assign.tasks'='1',
'write.tasks' = '1'
);

# insert join result into inner_table
insert into inner_table
select
	a.id as id,
	a.name as name,
	b.name as name2,
	LOCALTIMESTAMP as ts
from
source_t1 a JOIN source_t2 b
ON a.id = b.id;

Environment Description

Hudi version : 1.13.0

Hadoop version : 3.1.3

Flink version: 1.16.2

Running on Docker? (yes/no) : yes

Additional context
Insert data into hudi table by using insert into table_name values (...) works fine, but insert join result into hudi table failed. I don't know this is something wrong with Hudi or Flink.

i check the log of taskmanager, is this issue related to timeline service?
2023-05-31 18:24:50,106 INFO  org.apache.hudi.client.BaseHoodieClient                      [] - Embedded Timeline Server is disabled. Not starting timeline service
2023-05-31 18:24:50,106 INFO  org.apache.hudi.client.BaseHoodieClient                      [] - Embedded Timeline Server is disabled. Not starting timeline service

The stream writer has no outputs, but it would flush the records to Hudi table anyway. Does your job checkpoint proceeds normally?

it seems like it doesn't trigger a checkpoint.

Intreasting, can you share us the flink checkpoint configuration params?

Intreasting, can you share us the flink checkpoint configuration params?

ah oh, your concern about checkpoint is right.
Originally, i don't have any customized settings for checkpoints, so it looks like this in flink-conf.yml, everything uses default value.

Then because of your reminder, i add some configurations to checkpoints, and join result can be successfully inserted into  wide table.
set execution.checkpointing.interval=1000;
set state.checkpoints.dir=hdfs:///ckp/;
set execution.checkpointing.externalized-checkpoint-retention=RETAIN_ON_CANCELLATION;

Really thx for your reminder, but i still wonder why under default checkpoint configuration, join result can not be inserted into wide table, however data can be inserted one by one.

Flink Hudi relies on the checkpoints to keep exactly-once semantics, one checkpoint equals to one commit in Hudi table.


Small bug (isNotNew)

Small bug here
https://github.com/dekeysoft/yii2-domain/blob/master/src/db/Record.php#L63

Hello @Enkey,
I deeply sorry I haven't noticed your issue when you posted it.
I had a lot of PRs in April so notification for this issue just got lost in tons of other notifications.
The issue is already fixed.
Thanks for the participation!
With best regards,
Dmitry Kolodko


Fix for #1034

Ok, understand what you trying to do. I will do a merge as soon as possible. Thanks for that.

Thanks

Thanks


Resolving error; wry links to a different version of webkit2gtk-sys than tauri-plugin-dialog

Describe the bug
You can't use wry and tauri-plugin-dialog in the same project. They both require webkit2gtk-sys on my machine, but  wry requires version 0.18 and the dialog plugin requires 2.0.1.
Steps To Reproduce
Use wry and tauri-plugin-dialog on a mac or another machine using webkit
Expected behavior
They should have matching dependencies.
Screenshots
Here the cargo error:
error: failed to select a version for `webkit2gtk-sys`.
    ... required by package `wry v0.18.3`
    ... which satisfies dependency `wry = "^0.18.3"` of package `tauri-runtime-wry v0.9.0`
    ... which satisfies dependency `tauri-runtime-wry = "^0.9.0"` of package `tauri v1.0.0`
    ... which satisfies dependency `tauri = "^1"` of package `manager v0.0.0 (/Users/tom/indev/Schulverwaltung/manager/src-tauri)`
versions that meet the requirements `^0.18` are: 0.18.0

the package `webkit2gtk-sys` links to the native library `web_kit2`, but it conflicts with a previous package which links to `web_kit2` as well:
package `webkit2gtk-sys v2.0.1`
    ... which satisfies dependency `ffi = "^2.0.1"` of package `webkit2gtk v2.0.1`
    ... which satisfies dependency `webkit2gtk = "=2.0.1"` of package `tauri v2.0.0-rc.3`
    ... which satisfies dependency `tauri = "^2.0.0-rc.3"` of package `tauri-plugin-dialog v2.0.0-rc.2`
    ... which satisfies dependency `tauri-plugin-dialog = "^2.0.0-rc.2"` of package `manager v0.0.0 (/Users/tom/indev/Schulverwaltung/manager/src-tauri)`
Only one package in the dependency graph may specify the same links value. This helps ensure that only one copy of a native library is linked in the final binary. Try to adjust your dependencies so that only one package uses the links ='webkit2gtk-sys' value. For more information, see https://doc.rust-lang.org/cargo/reference/resolver.html#links.

failed to select a version for `webkit2gtk-sys` which could resolve this conflict

Platform and Versions (please complete the following information):
OS: macOS Sequoia 15.1
Rustc: 1.74.0
PC: Mac Mini M1 2020

I think you're using tauri@v1 which is not supported tauri-plugin-dialog, you need to use tauri@v2


Download page is considered as a deceptive site

When I followed the download link, Chrome (also Safari reported similarly) reported the site was a deceptive site.
You may need to request a site review for being marked as dangerous.
https://developers.google.com/webmasters/hacked/docs/request_review
Pasted the sentences shown in the download page.

Deceptive site ahead
Attackers on github-production-release-asset-2e65be.s3.amazonaws.com may trick you into doing something dangerous like installing software or revealing your personal information (for example, passwords, phone numbers, or credit cards).

Thanks for opening this. I'll follow those instructions and see if we can get this cleared up.

Looks like it's still there. This is for me major problem, when I recommend it to someone else, there is low chance of using it at all.

It sounds like this should be resolved. I never got the warning page, so I can't confirm this. Can someone please try again? Thanks.

@mastahyeti works for me now. Downloaded package.
3265818 Jul 25 15:56 SoftU2F.pkg
⫸  shasum -a 512 SoftU2F.pkg
3e626b7eaa6895e89198ce1a3840d925fe69629fd909adbde0018491d4d87b4709283845a446b7fbdeec41992c36461fdf3860e43759241d7b05cea475d2f14a  SoftU2F.pkg

However, how do I know what is proper checksum of package? I think I'll put it in another issue.

Thanks for verifying that this is fixed.


Conditional package references

In a mixed solution where some of the projects use new VS17 csproj format and target multiple frameworks (net462/nestandard) conditional references are required for some scenarios, such as referencing System.AppDomain for netstandard but not when building for net46.
Example:
<ItemGroup>
  <PackageReference Include="System.AppDomain" Version="2.0.11"
                    Condition="'$(TargetFrameworkIdentifier)' == '.NETStandard'" />
</ItemGroup>

Would be great to have this supported in paket.dependencies and/or paket.references

Similar to https://github.com/fsprojects/Paket/issues/2394


Add AWLoader

Project URL
https://github.com/Aymenworks/AWLoader
Category
Activity Indicator
Description
AWLoader is a UI Component that allows you to integrate loader that fits your needs within your app.
Why it should be included to awesome-ios (mandatory)
It is an open source Carthage and cocoapods allowing users to have not just a hud progress, but a more friendly and customizable loader. I am regularly updating it. Easy to use as well.
Checklist

[ ] Has 50 Github stargazers or more
[x] Only one project/change is in this pull request
[x] Isn't an archived project
[x] Has more than one contributor
[ ] Has unit tests, integration tests or UI tests
[x] Addition in chronological order (bottom of category)
[x] Supports iOS 9 / tvOS 10 or later
[x] Supports Swift 4 or later
[x] Has a commit from less than 2 years ago
[x] Has a clear README in English

          1 Warning

:warning:
Found 3 link issues, a project collaborator will take care of these, thanks :)

Link issues by awesome_bot

Line
Status
Link

371
301
https://github.com/YR/Cachyr redirects tohttps://github.com/nrkno/yr-cachyr

1178
301
https://www.photoeditorsdk.com redirects tohttps://photoeditorsdk.com/

1385
301
https://www.airship.com/products/mobile-app-engagement redirects tohttps://www.airship.com/platform/channels/mobile-app/

  Generated by :no_entry_sign: Danger

@Aymenworks. Some items contribution guidelines aren't cover in this project. Once it is ready, you can reopen this PR.


Removed the ++ and -- operators

Removed the ++ and -- operators as they will no longer work with
upcoming changes
https://github.com/apple/swift-evolution/blob/master/proposals/0004-remove-pre-post-inc-decrement.md

Ah, I was wondering why you weren't using arithmetic operators; I didn't realize Index was its own class. Anyway, :+1:

Thank you!  There are a lot more in the tests.

Please remove unrelated whitespace changes from the patch.

Please compile the code and run tests before submitting -- utils/build-script -RT.

Would it be better if this change was merged in at the same time as the removal of the post/pre operators?

No, for two reasons: incremental development, and shaking out possible performance issues.

@gribozavr Just to clarify, is that reasoning for accepted proposals? I understand this isn't the case here, but I would imagine it to be counterproductive to merge in changes when the decision has yet to be finalized.

@katzenbaer Yes, just for accepted proposals.

@divadretlaw I'm sorry for the delays, would you mind rebasing the PR and resolving the conflicts?

Oh, I'm sorry, I didn't realize that there was an outstanding patch to help with this.

Sorry, but this change causes test failures:
Failing Tests (3):
    Swift :: SILGen/coverage_while.swift
    Swift :: SILGen/sil_locations.swift
    Swift :: Sema/immutability.swift

I'll revert.

@divadretlaw Could you take a look at the test failures, fix and resubmit a new PR?


Is there some document for specific one GPU on my system to  use in WSLG？

for example:
I have a intel XE ( within i7 - 1165g7 ) and a nvidia rtx2060
I want to use nvidia 2060 in windows and XE in wsl2-ubuntu with wslG
is this we can custom?

This is not currently possible, but we're working on a change to Mesa that will allow you to select which GPU it selects by default.


zookeeper authorization

Added authorization for zookeeper, where user can opt for world:anyone permission or username:password authentication. On opting username:password, it will have world readable permissions too.

Can one of the admins verify this patch?

@mesosphere-ci: retest this please

Closing this request.
Started new one. Due to merge conflict locally.
https://github.com/mesosphere/dcos-commons/pull/481


Speed up LED driver updates when using NumPy

Addresses #1152
Profiling revealed large time spent in DriverBase._render() method.

Add a numpy path for DriverBase._render() to avoid Python-based looping.
Add Gamma.np_table and Gamma.batch_correct.
Add Palette.np_table and Palette.batch_apply_palette.

One concern: do we care that the package runs without numpy installed? Because importing numpy prevents non-numpy users from running Bibliopixel.

One question: I'm not sure how threading is supposed to work - looks like there's a lock on brightness but not on _buf? Is threading actually used in the drivers or is it WIP?


New version: Ryujinx.Ryujinx version 1.1.1207

Pull request has been created with komac v2.1.1 :rocket:
Microsoft Reviewers: Open in CodeFlow

Automatic Validation ended with:

No errors to post.

(Automated response - build 795.)


Improve CEP representation of l4-only allow

BPF map state which allows all peers to access traffic on an L4 port
encodes the identity as "0", which in this case means "allow all", not
the "reserved:unknown" identity. To properly reflect this in the API,
don't serialize the labels in this case. API users should interpret the
lack of Identity and IdentityLabels as allowing all peers on the L4 port.
Fixes: #7511

This change is

test-me-please

test-me-please

Hit #7495 . Also fixing unit-test issue, rebasing.

test-me-please

Coverage increased (+0.08%) to 42.769% when pulling 8f20076f0559bfee07d2d05be54df575d491810d on joestringer:submit/fix-cep-l4-only into 1d6f46b74074e40d61e8a09fcc3b09ad3445349a on cilium:master.


Private to Protected fields

Due to its flourishing tweaking nature, Processing's community loves to modify classes in their own sketches.
Both to add new functionality and hack existing 1s. And private members go hard against this philosophy.
Protected members prohibits directly tampering like private 1s. But at least allow subclasses to modify such members!
Besides the access level switch, I've made some further tweaks, so the class is shorter, but keeping the same functionality. ;-)

Thanks for the patch. The toString() was planned already, I had decided to add it in my next release.  Some of the other changes you made (using the smaller conditional expression or whatever its called) were also planned for the next release.
I didn't know that you can use underscores in numbers though.


Fix for gt_ignore

Fix for issue #1236. If list gt_ignore was empty, it should become None after the following if statement:
if not gt_ignore:
        gt_ignore = gt_ignore

otherwise, the eval_map function throws an AssertionError.
Changed it to gt_ignore = None.

Thanks! It is definitely a typo.

Thanks! It is definitely a typo.

Not sure why all the checks haven't completed yet. Is that an issue?


Getting "can't dup NilClass" with Reform 2.2.0

Hi there,
After updating from v2.1 to v2.2 I now get the following error:
can't dup NilClass
…/.gem/ruby/2.3.1/gems/reform-2.2.0/lib/reform/contract.rb:33:in `dup'
…/.gem/ruby/2.3.1/gems/reform-2.2.0/lib/reform/contract.rb:33:in `block in properties'
…/.gem/ruby/2.3.1/gems/reform-2.2.0/lib/reform/contract.rb:33:in `each'
…/.gem/ruby/2.3.1/gems/reform-2.2.0/lib/reform/contract.rb:33:in `properties'
…/Projects/MyProject/app/forms/some_form.rb:2:in `<class:SomeForm>'

The error seems to occur because of the following code:
def self.properties(*args)
  options = args.pop if args.last.is_a?(Hash)
  args.each { |name| property(name, options.dup) }
end

When no option is provided it tries to call dup on nil options
Maybe it should be replaced with
options = args.last.is_a?(Hash) ? args.pop : {}

I can do a PR if no-one is already on it :smile:

Are you doing this? http://trailblazer.to/gems/reform/#rails-support

@apotonick the same error and reform-rails didn't solve it.
Gemfile.lock
    reform (2.2.0)
      disposable (>= 0.3.0, < 0.4.0)
      representable (>= 2.4.0, < 3.1.0)
      uber (~> 0.0.11)
    reform-rails (0.1.2)
      activemodel (>= 3.2)
      reform (>= 2.2.0.rc1)

Backtrace reference to this concept:
class User
  class Profile < Reform::Form
    properties :username, :password
    property :password_confirmation, virtual: true

    validates :username, :password, presence: true
    validates_uniqueness_of :username, case_sensitive: false
  end
end

Trace:
/home/bulai/.rvm/gems/ruby-2.3.1/gems/reform-2.2.0/lib/reform/contract.rb:33:in `dup': can't dup NilClass (TypeError)
	from /home/bulai/.rvm/gems/ruby-2.3.1/gems/reform-2.2.0/lib/reform/contract.rb:33:in `block in properties'
	from /home/bulai/.rvm/gems/ruby-2.3.1/gems/reform-2.2.0/lib/reform/contract.rb:33:in `each'
	from /home/bulai/.rvm/gems/ruby-2.3.1/gems/reform-2.2.0/lib/reform/contract.rb:33:in `properties'
	from /home/bulai/projects/sd2labs/rap_shared_services/app/concepts/user/contracts/profile.rb:3:in `<class:Profile>'

I confirm, reform-rails doesn't change anything :wink:

I could swear I fixed this a few weeks back! I'll look into it tomorrow morning.
The RC has been out for 4 days, but as always, no one tests it, so my strategy for the last 2 years is to release the stable version to force people to test it. 😜

Well in the meantime I replaced all my calls to properties by a list of calls to property as a workaround.

AAaaaaah, I understand now, @simonc ! Send the PR! <3

Let's go then! :wink:

It took me some time to get the specs right but here it is :wink:

2.2.1 released, thanks so much! :heart:

And, awesome work on the specs! :+1:

Glad I could help :wink:


Extra tokens in input file

Hi, I'm confused about the <extra_id_0> and <extra_id_1> in json files under the data/text2tree folder.
In Huggingface's example, these extra_ids represent to their replaced token, such as
The <extra_id_0> walks in <extra_id_1> park and <extra_id_0> cute dog <extra_id_1> the <extra_id_2>.
But in json file, most event label contains multiple <extra_id_0> and <extra_id_1>, such as
"text": "He will blow a city off the earth in a minute if he can get the hold of the means to do it .", "event": "<extra_id_0> <extra_id_0> Attack blow <extra_id_0> Place earth <extra_id_1> <extra_id_1> <extra_id_1>"
<extra_id_0> and <extra_id_1> also appears when there is no event label:
"text": "I am shook over the aftermath .", "event": "<extra_id_0>  <extra_id_1>"
so what are the <extra_id_0> and <extra_id_1> mean?

Hi,
In the T5 pre-training, <extra_id_0> and <extra_id_1> are used as mask symbol for mask language modeling.
And in our fine-tuning stage, we use them as the structure symbols to combine the event schema and spans from the text.
<extra_id_0>  is ‘（’  in our paper, and <extra_id_1>  is ')'.
<extra_id_0> <extra_id_0> Attack blow <extra_id_0> Place earth <extra_id_1> <extra_id_1> <extra_id_1>
->
((Attack blow (Place earth) )
In this way, '<extra_id_0> <extra_id_1>' represents "()", which means there is no event in the sentence.

Thanks for your clear explanation!


Is there a way to get the token metadata info?

I am starting a project recently and I was using node-js until I found out about this library. In the metaplex library I have been using (JS) I have the NFT model that is described as follows https://github.com/metaplex-foundation/js#the-nft-model - my question is: Is there a way to get all this info such as URI, collection details, symbol etc by just using these methods provided by Solana-py, or the Metaplex team worked in a more advanced way to provide access to this information?

There's a Python API for Metaplex but I've never tried it: https://github.com/metaplex-foundation/python-api
Either way it's probably out of scope for solana-py


#260: updated logos and site links

Closes #260 .

You need to check in the logo, assets/bmgf-logo.png.

Fixed!


Version Detection Partially Broken - Zoom Client

AppGet cannot see Zoom when already installed.  When removed and then installed using AppGet it can then see the Zoom client in appget list

Sorry I don't have the old installer as it was pushed out a few weeks back using Ninite.

ok, that's a good start, I'll try and see what gets installed using Ninite

I may be wrong about this, but the latest version of Zoom writes its Uninstall information to the path of HKLM\SOFTWARE\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall, with WOW64 being the main difference. In your WindowsInstallerClient, this path is not listed.
I believe it may work if this was added.

Disregard my message. I mocked it up and it did not work.


set_intra_op_parallelism_threads and set_inter_op_parallelism_threads has no impact on thread usage

System information

I am using a custom implementation of DepthwiseConv3D, extends the Conv3D class and is partially based on code from the following repository, https://github.com/alexandrosstergiou/keras-DepthwiseConv3D adapted to work in tensorflow 2.4.1
CentOS Linux Version 7
Installed from source
Tensorflow version 2.4.1
Python 3.8.8
Cuda 11.0, CuDNN 8
Happens on GTX1080, RTX2080

Describe the current behavior
I am running code on a compute cluster, hence the different GPUs. I am required by the compute cluster admin to restrict thread usage when possible and had been referred to use the functions from tf.config.threading in the tensorflow documentation. I set both intra and inter thread parallelism to 2 and used an interactive session on the node to monitor thread usage with top, however the usage of the thread parameters seems to have no impact on thread usage. I still observe the python process using all available threads.
My understanding from the documentation for these functions is that all is required is to call them with the parameters wanted, no errors related to threading have been raised.
My github repository is as follows, I use the mult.csh file under the EEGNet folder to execute the code, which runs the runMB3D.py file, using the network model from MB3DEEGNet.py.
https://github.com/matt-houk/MB3DCNN
I have attached both the stderr and stdout output, I canceled the code when I noticed it using excessive threads.
err-mult.txt
out-mult.txt

I just wanted to include some followup info from efforts I have made to debug the issue over the past week.
I had attempted to set OMP_NUM_THREADS prior to running the python file, this also appears to have no impact.
I have also set the environment variables TF_NUM_INTEROP_THREADS and TF_NUM_INTRAOP_THREADS prior to application runtime, checking them using os.environ from within python both before and after tensorflow is imported, as well as the tf.config.threading.set_intra/inter_op_parallelism_threads calls, the environment variables remain unchanged at 1, however, when observing the python process using top, I observe that it is using 13 thread using the nTH column in top.
Please let me know if it seems I am making an incorrect assumption on anything here. Thank you.

If there are any other issues that are relevant that I have been unable to unearth during my searches that might be able to help, I would love if somebody could link them! While this issue doesn't prevent me from being able to do work, it dramatically slows the rate at which I can do so and none of my attempts to resolve it have been successful.

Can someone clarify for me whether the functions
tf.config.set_intra_op_parallelism_threads(NUM_THREADS) tf.config.set_inter_op_parallelism_threads(NUM_THREADS)
are meant to control the threads used by the python process or the nvidia-cuda-mps process?
Adding
tf.config.set_soft_device_threading(True)
Was able to restrict nvidia-cuda-mps to 4 threads when observed using top, which would be in line with my case where NUM_THREADS = 4, if this is the case, I simply need to determine why Python is choosing to use as many threads as it can. If that is the case, while I would appreciate any advice anyone might have, that might be outside of TensorFlow and instead of a Python issue. Thanks!

Is there a possibility of getting a response on this issue? Because otherwise, it is probably best to close it. The issue is still unresolved but there has been no activity outside of my own updates. Any additional insight would be appreciated.

Apologies for the delayed response. Would it be possible to provide some sample D_cropped.npy and K_cropped.npy so that I can try and reproduce the issue? Generally the inter op and intra op threads are meant for the C++ tensorflow executor runtime. Usually we rarely do any multi-threading in python but I'd like to reproduce the issue to figure out what might be going on.

Here is a sample of the files,
Data files.zip
Sorry for the delayed response, I will also look into the profiler on my end!

Thanks... I was able to reproduce the issue and yeah I do see a large number of threads. A hypothesis is that there are some tf.data threads that you're seeing. But it'll be a lot clearer with the profile.

Just got the opportunity to try the profiler, sorry for the further delay, was working to meet some deadlines. Any attempt I make to use profiling appears to immediately cause a segfault. I will update the GitHub (I've been behind on that anyway) and the relevant files can be found under the Groups directory, I made some modifications to the code in an attempt to improve runtime, using a different method of implementing a DepthwiseConv3D layer I found.
Below is the stack trace from the GDB debugger, everything else should be about the same. The Segmentation Fault consistently occurs at the model.fit call on line 149 in the runMotor.py file.
#0  0x00002aaaaaf70c3c in free () from /lib64/libc.so.6 #1  0x00002aaab742b65d in Eigen::internal::TensorExecutor<Eigen::TensorAssignOp<Eigen::TensorMap<Eigen::Tensor<float, 5, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorReverseOp<Eigen::array<long, 5ul> const, Eigen::TensorShufflingOp<Eigen::array<long, 5ul> const, Eigen::TensorReshapingOp<Eigen::DSizes<long, 5> const, Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorReshapingOp<Eigen::DSizes<long, 2> const, Eigen::TensorVolumePatchOp<-1l, -1l, -1l, Eigen::TensorForcedEvalOp<Eigen::TensorShufflingOp<Eigen::array<long, 5ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 5, 1, long>, 16, Eigen::MakePointer> const> const> const> const> const, Eigen::TensorForcedEvalOp<Eigen::TensorReshapingOp<Eigen::DSizes<long, 2> const, Eigen::TensorShufflingOp<Eigen::array<long, 5ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 5, 1, long>, 16, Eigen::MakePointer> const> const> const> const, Eigen::NoOpOutputKernel const> const> const> const> const> const, Eigen::ThreadPoolDevice, true, (Eigen::internal::TiledEvaluation)1>::run(Eigen::TensorAssignOp<Eigen::TensorMap<Eigen::Tensor<float, 5, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorReverseOp<Eigen::array<long, 5ul> const, Eigen::TensorShufflingOp<Eigen::array<long, 5ul> const, Eigen::TensorReshapingOp<Eigen::DSizes<long, 5> const, Eigen::TensorContractionOp<Eigen::array<Eigen::IndexPair<long>, 1ul> const, Eigen::TensorReshapingOp<Eigen::DSizes<long, 2> const, Eigen::TensorVolumePatchOp<-1l, -1l, -1l, Eigen::TensorForcedEvalOp<Eigen::TensorShufflingOp<Eigen::array<long, 5ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 5, 1, long>, 16, Eigen::MakePointer> const> const> const> const> const, Eigen::TensorForcedEvalOp<Eigen::TensorReshapingOp<Eigen::DSizes<long, 2> const, Eigen::TensorShufflingOp<Eigen::array<long, 5ul> const, Eigen::TensorMap<Eigen::Tensor<float const, 5, 1, long>, 16, Eigen::MakePointer> const> const> const> const, Eigen::NoOpOutputKernel const> const> const> const> const> const&, Eigen::ThreadPoolDevice const&) ()    from /usr/local/usrapps/multibranch/mjhouk/tf241_py377/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so #2  0x00002aaab742c42e in tensorflow::functor::CuboidConvolutionBackwardFilter<Eigen::ThreadPoolDevice, float>::operator()(Eigen::ThreadPoolDevice const&, Eigen::TensorMap<Eigen::Tensor<float, 5, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorMap<Eigen::Tensor<float const, 5, 1, long>, 16, Eigen::MakePointer>, Eigen::TensorMap<Eigen::Tensor<float const, 5, 1, long>, 16, Eigen::MakePointer>, int, int, int) ()    from /usr/local/usrapps/multibranch/mjhouk/tf241_py377/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so #3  0x00002aaab7435294 in tensorflow::Conv3DCustomBackpropFilterOp<Eigen::ThreadPoolDevice, float>::Compute(tensorflow::OpKernelContext*) ()    from /usr/local/usrapps/multibranch/mjhouk/tf241_py377/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so #4  0x00002aaae1004707 in tensorflow::(anonymous namespace)::ExecutorState<tensorflow::SimplePropagatorState>::Process(tensorflow::SimplePropagatorState::TaggedNode, long long) ()    from /usr/local/usrapps/multibranch/mjhouk/tf241_py377/lib/python3.7/site-packages/tensorflow/python/../libtensorflow_framework.so.2 #5  0x00002aaab4f65d32 in Eigen::ThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)     ()    from /usr/local/usrapps/multibranch/mjhouk/tf241_py377/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so #6  0x00002aaab4f62047 in std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()    from /usr/local/usrapps/multibranch/mjhouk/tf241_py377/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so #7  0x00002aaae10a3d3c in tensorflow::(anonymous namespace)::PThread::ThreadFn(void*) ()    from /usr/local/usrapps/multibranch/mjhouk/tf241_py377/lib/python3.7/site-packages/tensorflow/python/../libtensorflow_framework.so.2 #8  0x00002aaaaacd6e25 in start_thread () from /lib64/libpthread.so.0 #9  0x00002aaaaafe9bad in clone () from /lib64/libc.so.6
This might require a separate issue post, if so let me know and I will proceed. I am approaching concluding the project I was working on, but am fully willing to continue with efforts to debug this issue as I am able.
Let me know if you have any other questions!

Hi,
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.


[TASK] Script to build packages without conda

Issue Description
Is your feature request related to a problem? Please describe.
Conda built pip packages can link in libraries that are not available on a different computer. This resulted in errors like:
Traceback (most recent call last):
  File "/Users/jack/miniconda3/envs/test_raven/lib/python3.10/site-packages/ravenframework/utils/utils.py", line 659, in find_crow
    import crow_modules.distribution1D
  File "/Users/jack/miniconda3/envs/test_raven/lib/python3.10/site-packages/crow_modules/distribution1D.py", line 13, in <module>
    from . import _distribution1D
ImportError: dlopen(/Users/jack/miniconda3/envs/test_raven/lib/python3.10/site-packages/crow_modules/_distribution1D.cpython-310-darwin.so, 0x0002): Library not loaded: @rpath/libc++.1.dylib
  Referenced from: <E3888BAA-1459-3DCE-B489-9E19A724980B> /Users/jack/miniconda3/envs/test_raven/lib/python3.10/site-packages/crow_modules/_distribution1D.cpython-310-darwin.so
  Reason: tried: '/Users/fred/miniconda3_arm64/envs/python310_pip/lib/libc++.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/fred/miniconda3_arm64/envs/python310_pip/lib/libc++.1.dylib' (no such file), '/Users/fred/miniconda3_arm64/envs/python310_pip/lib/libc++.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/fred/miniconda3_arm64/envs/python310_pip/lib/libc++.1.dylib' (no such file), '/Users/jack/miniconda3/envs/test_raven/bin/../lib/libc++.1.dylib' (no such file), '/Users/jack/miniconda3/envs/test_raven/bin/../lib/libc++.1.dylib' (no such file)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jack/miniconda3/envs/test_raven/bin/raven_framework", line 8, in <module>
    sys.exit(wheelMain())
  File "/Users/jack/miniconda3/envs/test_raven/lib/python3.10/site-packages/ravenframework/Driver.py", line 34, in wheelMain
    main(False)
  File "/Users/jack/miniconda3/envs/test_raven/lib/python3.10/site-packages/ravenframework/Driver.py", line 43, in main
    dutils.doSetup(checkLibraries)
  File "/Users/jack/miniconda3/envs/test_raven/lib/python3.10/site-packages/ravenframework/CustomDrivers/DriverUtils.py", line 39, in doSetup
    setupCpp()
  File "/Users/jack/miniconda3/envs/test_raven/lib/python3.10/site-packages/ravenframework/CustomDrivers/DriverUtils.py", line 105, in setupCpp
    utils.find_crow(frameworkDir)
  File "/Users/jack/miniconda3/envs/test_raven/lib/python3.10/site-packages/ravenframework/utils/utils.py", line 683, in find_crow
    raise IOError(UreturnPrintTag('UTILS') + ': '+UreturnPrintPostTag('ERROR')+ ' -> Crow has not been found. It location is supposed to be one of '+str(crowDirs)+'. Has RAVEN been built?')
OSError: UTILS                    : ERROR      -> Crow has not been found. It location is supposed to be one of ['/Users/jack/miniconda3/envs/test_raven/lib/python3.10/site-packages/crow', '/Users/jack/miniconda3/envs/test_raven/lib/python3.10/crow', '/Users/jack/miniconda3/envs/test_raven/lib/python3.10/site-packages']. Has RAVEN been built?

Describe the solution you'd like
Building the pip packages without using conda would be useful.  This might fix the linking errors.
Describe alternatives you've considered
Building the libraries with static linking might be possible, but there doesn't seem to be a documented way to do this.

For Change Control Board: Issue Review
This review should occur before any development is performed as a response to this issue.

[x] 1. Is it tagged with a type: defect or task?
[x] 2. Is it tagged with a priority: critical, normal or minor?
[x] 3. If it will impact requirements or requirements tests, is it tagged with requirements?
[x] 4. If it is a defect, can it cause wrong results for users? If so an email needs to be sent to the users.
[x] 5. Is a rationale provided? (Such as explaining why the improvement is needed or why current code is wrong.)

For Change Control Board: Issue Closure
This review should occur when the issue is imminently going to be closed.

[x] 1. If the issue is a defect, is the defect fixed?
[x] 2. If the issue is a defect, is the defect tested for in the regression test system?  (If not explain why not.)
[x] 3. If the issue can impact users, has an email to the users group been written (the email should specify if the defect impacts stable or master)?
[x] 4. If the issue is a defect, does it impact the latest release branch? If yes, is there any issue tagged with release (create if needed)?
[x] 5. If the issue is being closed without a pull request, has an explanation of why it is being closed been provided?

checklist is good.


Example results

Hi, I tried to reproduce some of your examples and this is what I got:

While I understand that errors may occur, it looks to me that something is broken in my installation, am I correct? I have loaded LLaMA in bfloat16, could it be caused by that in your opinion?

Yeah. It looks like the text-to-panel results are different with the same seed, which might be caused by the Llama dtype.


Port scheduled sampling HOWTO from old diff-based system

And instead, use a standalone doc with tests like in #771
Here is the old (pre-Linen) HOWTO diff, for reference:
https://github.com/google/flax/blob/master/howtos/diffs/scheduled-sampling.diff

It appears the old howto branches are removed, and given that these HOWTOs we written pre-Linen, I don't see an easy way to add these. Also since we haven't acted on this in 1.5 I'm closing this issue.


Fix template params to match final C++17 iterator categories

https://wg21.link/n4700 changed most of the parallel algorithms from InputIterator to ForwardIterator requirements. This change renames the template parameters to match this change.

Hi, rodgert.
Thank you for your request.
One question regarding the patch.
How important the meta-information in the beginning of files?
(For example, //  -- C++ --)
Why it's needed here? Can it be removed?
The reason is that we usually haven't any symbols before "copyright information".


fix(la): use correct gender for ordinal numbers

Previously, we used ordinal numbers in masculine gender for Sundays in Ordinary Time, Lent and Easter Time.

@TobiTenno, could you review this PR please? :pray:

i like the point of the change, the clarity of fixtures is testy, so i might submit a PR after this to converse/propose some more expanded names for fixtures variables

Go for it! :wink:

planning to merge at midnight UTC tonight

I want to rebase this after merging #477 so that we import the Fixture type.

@tukusejssirs #477 is merged now, so feel free to rebase

But the la tests pass in GitLab Actions. :thinking:

yeah, i'm trying to repro it failing, as I'm not seeing failures on my end

Okay, it seems like some caches caused all these issues. After ditching them all and starting from scratch, everything works as expected.
@TobiTenno, the PR is ready to be merged. Note that I have added another to fix some lint and format issues. I know it is out of the PR scope, so if you want to do it in a separate PR, just say so. :wink:

nah, i like fixing format & lint issues in progress, so this is great 👍

But the changed files are unrelated to this PR. :man_shrugging:

yeah, but sometimes this happens when we update lint/prettier cause we don't have a format check in our CI... which i'm probably gonna change shortly

yeah, but sometimes this happens when we update lint/prettier cause we don't have a format check in our CI... which i'm probably gonna change shortly

I am for manual checking of the lint/format changes, as sometimes they mess some stuff up.
On the other hand, I agree to add lint/format checks to the CI and when they fail, the PR checks would fail.
WDYT? :thinking:

technically we already have lint checks, as they're part of the prebuild, so npm run build triggers npm run lint.
the format checks (in #493) don't fix the problems, they just let you know about them.

technically we already have lint checks, as they're part of the prebuild, so npm run build triggers npm run lint.

Yeah, however, I disagree with having them there. IMHO we should be able to build (locally) an unlinted and unformatted code. IMHO prebuild should be used for build preparation (like cleanup or some files preparation), not for linting or formatting.
Lint and format checks could be part of the pre-commit checks (e.g. using Husky) though.

hmmm.... i'll do this pr first to move them to just on ci, then i'll do a followup for setting up husky. how does that sound?

It sounds perfect to me, thanks! :pray:


chore(main): release 0.2.7

:robot: I have created a release beep boop
0.2.7 (2023-08-27)
chore

ignore unnecessary files and directories in git (77130f3)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/cloverdefa/Whosis-Sayings/releases/tag/v0.2.7 :sunflower:


show marker select with images

This branch lets the user select the markers by images instead of the text list.
To make the marker select field behave like a <select> i used a button and divs. Images in the background of select options seem to only work in firefox. To hide the box when the map is clicked I needed to add an eventlistener to the map. I didn't want to create multiple eventlisteners (for every form opening) so i rewrote the app to use only one StyleForm.
Reselecting a marker / line / polygon loads the current data of that element, so for example a change  of the icon will not change the size.
If you prefer another solution, would like some changes let me know.
Another question: which tool do you use to minify the javascript?

woops sorry this commit should not have endet here

Thanks for the pull request.
In order to minify the code you can simply execute:
npm install grunt
The necessary tools are included in the package.json and Gruntfile.js

thank you - it is now included


bump kube_version to v1.29

What type of PR is this?
/kind enhancement
What this PR does / why we need it:
Update the current default kube_version of kubean to v1.29, and also update the upstream commit of kubespray accordingly.
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
https://github.com/kubean-io/kubean/actions/runs/9405638560/job/25909680441

/lgtm