Memory leak on ARM (Raspbian) with .NET Core 3.1

Hi,
I have a memory leak since migration my application to .NET Core 3.1.
Environment

Raspbian (9)
Raspberry Pi 3 B+
.NET Core 3.1 Self contained application in Release Build (all latest SDK and nugets)
IronPython 2.7.10-candiate1 (but problems also present in 2.7.9).

Problem
The software is a smart home system (https://github.com/chkr1011/Wirehome.Core) and runs 24/7 on the Raspberry Pi. With .NET Core 2.1 and 2.2 I had no issues with memory usage. It stays at 300 MB for several weeks or even months. But since migrating to .NET Core 3.1 and latest IronPython the system consumes all memory after roughly 3 days so that it stops working and I have to reboot the system.
Sample
The following code shows basically what I am doing. It calls some Python methods and Python methods can register callbacks which are invoked from my host from different threads (Tasks). This sample code runs without problems on my PC but it seems that the ARM version has a leak somehow.
I the live project the Python scripts are also called within ASP.MVC requests, MQTT messages etc. So many different threads. I also have around 600 active ScriptScopes in memory which are invoked via previously registered callbacks or have predefined methods which are called from some APIs etc. Also data is send via some PythonDictionary across these ScriptScopes.
List<Action<PythonDictionary>> _callbacks = new List<Action<PythonDictionary>>();

void Main()
{
	var engine = IronPython.Hosting.Python.CreateEngine();
	var scriptScope = engine.CreateScope();
	var source = scriptScope.Engine.CreateScriptSourceFromString(GetPythonCode(), SourceCodeKind.File);
	var compiledCode = source.Compile();
	compiledCode.Execute(scriptScope);

	IDictionary<string, object> wirehomeWrapper = new ExpandoObject();
	wirehomeWrapper["ping"] = new Action<PythonDictionary>(Ping);
	wirehomeWrapper["register_callback"] = new Action<Action<PythonDictionary>>(RegisterCallback);

	scriptScope.SetVariable("wirehome", wirehomeWrapper);

	var testPythonFunction = (PythonFunction)compiledCode.Engine.Operations.GetMember(scriptScope, "test");
	var initPythonFunction = (PythonFunction)compiledCode.Engine.Operations.GetMember(scriptScope, "init");

	compiledCode.Engine.Operations.Invoke(initPythonFunction, new PythonDictionary
	{
		["x"] = 1,
		["y"] = "Test string"
	});

	while(true)
	{
		Task.Run(() =>
		{
			compiledCode.Engine.Operations.Invoke(testPythonFunction, new PythonDictionary
			{
				["x"] = 1,
				["y"] = "Test string"
			});

			foreach (var callback in _callbacks)
			{
				callback(new PythonDictionary
				{
					["x"] = 1,
					["y"] = "Test string"
				});
			}
		}).Wait();

		Thread.Sleep(10);
	}
}

void Ping(PythonDictionary parameters)
{
	Console.Write(".");
}

void RegisterCallback(Action<PythonDictionary> callback)
{
	_callbacks.Add(callback);
}

string GetPythonCode()
{
	return @"
call_count = 0

def init(params):
	wirehome.register_callback(test2)

def test(params):
	global call_count
	call_count = call_count + 1

	#wirehome.ping({'x': 1})

def test2(params):
	test(params)
";
}

Memory dump
I tried to create a memory dump using dotnet-dump but I did not find anything useful. Maybe the following screenshot contains something useful.

I also tried to use LightweightScopes or GC.Collect() but with no difference. What bothers me is that it was running with .NET Core 2.2. Nothing special has changed in the migration to .NET Core 3.1 (at least from my perspective).
This issue is probably related to #688
Please let me know if I can provide some more details or test some workarounds etc. At the moment I have to restart my entire house every 3 days 😅.
Best regards Christian

Thanks for the report! The only difference I can think of between the 2.1 and 3.0 DLLs is COM support (which should change nothing in your case). Have you tried using the 2.1 DLLs instead of the 3.0 DLLs (e.g. download the nuget packages, unzip and reference the 2.1 DLLs)?

You mean reverting to .NET Core 2.2 or 2.1?

No, I mean just for the IronPython (and DynamicLanguageRuntime) assemblies. Seeing if the leak is present with the DLLs compiled for 2.1 would give me an idea of if it's something that changed on our end of if the leak is caused by .NET Core itself.

OK now I got it. I will test a build with the binaries from the directory netcoreapp2.1 from latest 2.7.10-candidate1 ZIP package. If you prefer another version please let me know. I may need a couple of days to see if something has changed...

In the meantime I found something strange. Maybe not that important for the initial problem but still interesting I guess. When installing the Python STD Lib nuget I get the following output:
Restoring packages for D:\SourceCode\Wirehome.Core\Wirehome.Core\Wirehome.Core.csproj...
Installing NuGet package IronPython.StdLib 2.7.10-candidate1.
Committing restore...
Generating MSBuild file D:\SourceCode\Wirehome.Core\Wirehome.Core\obj\Wirehome.Core.csproj.nuget.g.props.
Writing assets file to disk. Path: D:\SourceCode\Wirehome.Core\Wirehome.Core\obj\project.assets.json
Successfully uninstalled 'System.Text.Encoding.CodePages 4.3.0' from Wirehome.Core
Successfully installed 'IronPython 2.7.8' to Wirehome.Core
Successfully installed 'IronPython.StdLib 2.7.10-candidate1' to Wirehome.Core
Successfully installed 'System.Text.Encoding.CodePages 4.4.0' to Wirehome.Core
Committing restore...
Committing restore...
Committing restore...
Assets file has not changed. Skipping assets file writing. Path: D:\SourceCode\Wirehome.Core\Wirehome.Cloud\obj\project.assets.json
Assets file has not changed. Skipping assets file writing. Path: D:\SourceCode\Wirehome.Core\Wirehome.Tests\obj\project.assets.json
Assets file has not changed. Skipping assets file writing. Path: D:\SourceCode\Wirehome.Core\Wirehome.Core.Hosts.Console\obj\project.assets.json
The strange part to me is "Successfully installed 'IronPython 2.7.8' to Wirehome.Core".
Does the latest STD Lib candidate 2.7.10 comes with IronPython 2.7.8? I would assume that it is bundled together with 2.7.10 of the runtime!? Or is this only some wrong output?

Hmm, the StdLib package has the following dependency: IronPython (>= 2.7.8 && < 3.0.0). I guess NuGet resolves to the lowest version?
Anyway, if you have the Lib folder already you don't need to reference the StdLib package.

OK makes sense to use at least the lowest version...

No lock. Running the app with .NET Core 3.1 and a direct reference to IronPython netcoreapp2.1 does not help. The memory is after some days at roughly 600 MB. I will now try to reference the .NET 4.5 library of IronPython. If this does not help I will downgrade my app to .NET Core 2.2 (or 2.1). I will also try to get a memory dump.

I tried net45 but it seems that I need several more libraries then and that the compatibility mode in netstandard is not enough.
I will now try to revert my app to .NET Core 2.2 and IronPython 2.7.9.

@chkr1011 Any luck with this? Just got my hands on a Raspberry Pi to try and reproduce.

It seems that the issue is coming from somewhere else in my code. I am not 100 % percent sure but after deactivating some parts which are starting processes through C# (Process.Start) the memory leak is gone. I checked the code a hundred times but saw now potential issue like missing Dispose etc.
At the moment I am rewriting the code to use the Popen method from the Python standard library. I will see what happens. The code is actually running the app "coap-client" every two seconds. That lead to the memory leak. I will let you know as soon as I got some results.

It turns out that the memory leak was caused by a process start I made in a loop in a Python script. It uses the regular Process.Start API from .NET and properly disposed all things like the process itself and all used streams. So I have no clue why this turns into a memory leak but in the end my problem is solved by using a different approach instead of starting a new process.
Thank you for your support 👍


Select kernel in test and ignore output count

Codecov Report

Merging #12135 (9684466) into main (0b76195) will increase coverage by 0%.
 Consider uploading reports for the commit ad30346 to get more accurate results

@@           Coverage Diff           @@
##            main   #12135    +/-   ##
=======================================
  Coverage     69%      70%
=======================================
  Files        495      495
  Lines      31114    31131    +17
  Branches    4886     4891     +5
=======================================
+ Hits       21629    21844   +215
+ Misses      7656     7438   -218
- Partials    1829     1849    +20

Impacted Files
Coverage Δ

...lPythonAndRelatedNonPythonKernelSpecFinder.node.ts
71% <90%> (+<1%)
:arrow_up:

...-side/variablesView/variableViewMessageListener.ts
75% <0%> (-25%)
:arrow_down:

src/notebooks/notebookEditorProvider.ts
74% <0%> (-20%)
:arrow_down:

src/platform/webviews/webviewViewProvider.ts
75% <0%> (-9%)
:arrow_down:

src/standalone/api/kernelApi.ts
72% <0%> (-4%)
:arrow_down:

src/kernels/helpers.ts
70% <0%> (-3%)
:arrow_down:

...ractive-window/editor-integration/codeGenerator.ts
89% <0%> (-3%)
:arrow_down:

...raw/finder/contributedLocalPythonEnvFinder.node.ts
87% <0%> (-2%)
:arrow_down:

src/interactive-window/commands/commandRegistry.ts
35% <0%> (-2%)
:arrow_down:

src/interactive-window/interactiveWindow.ts
69% <0%> (-2%)
:arrow_down:

... and 28 more


[nrf noup] ci: replace jenkins pipeline in jenkinsfile with a stub

load pipeline from internal config repo
Signed-off-by: Thomas Stilwell<EMAIL_ADDRESS>

cherry-picked to  #136

cherry-picked to  #136

I'm guessing since the upmerge just started it is going to be a few days.
since this is only 1 [noup] commit, can we merge it to master so compliance works on other PR's?
@carlescufi

I'm guessing since the upmerge just started it is going to be a few days.
since this is only 1 [noup] commit, can we merge it to master so compliance works on other PR's?
@carlescufi

@thst-nordic This patch removes patch which allows to skip part of test for [nrf merge] up sauce code.
Can this be introduced back?

@thst-nordic This patch removes patch which allows to skip part of test for [nrf merge] up sauce code.
Can this be introduced back?

I'll take a look

I'll take a look


Dependencies not up to date

Where in the docs did you come across this?
I am new to Rust and I am coding along tutorial on Game of Life.
Describe what about it does not make sense
Cargo.lock is not up to date as dependencies have bumped up their versions.
Why does it not make sense?
Rust fails to compile
How could we improve it?
update proc-macro2 to latest v0.4.9 cargo update -p proc-macro2
update wasm-bindgen to latest v0.2.13 cargo update -p wasm-bindgen

Fixed in https://github.com/rustwasm/wasm_game_of_life/pull/24 -- thanks for filing an issue!


Request for adding Hindusthan Educational Institutions

Request Submitted for adding (group) Hindusthan Educational Institutions, coimbatore

@arasu-hindusthan Pull request merged. Thank you!


feat: API for updating open call applications (funding/not_funding)

This PR adds three new endpoints:

PUT /api/v1/account/open_call_applications/{id} -- available to PD, allows to change message of application
POST /api/v1/account/open_call_applications/{id}/funding -- available to Investor, allows to mark application as funded
POST /api/v1/account/open_call_applications/{id}/not_funding -- available to Investor, allows to mark application as not funded

Testing instructions
All new endpoints can be found at rswag documentation
Tracking
https://vizzuality.atlassian.net/browse/LET-897

I would probably do it just by one update endpoint and permit different parameters, but I read in the ticket description that this was the recommended approach. I would not argue, both ways are good.

I liked that forbidden error is returned when investor/pd tries to edit what he is not allowed so I have preferred this approach, but for sure just allowing some params for different roles would work as well ;)


Allow for silent execution

Veeam Backup & Replication version
12.1
Description
I am trying to configure this to run as a periodically scheduled task, however I cannot get the module to skip prompting for information like the Author name, even with the sample config that I generated beforehand.
Additional Context
No response
Before submitting

[X] I have read the documentation, and referred to the known issues before submitting this change request.
[X] I have checked for previously opened & closed issues before submitting this change request.

Hello,
These are the parameters I use to create the scheduling task:
-NoProfile -ExecutionPolicy Bypass -Command "New-AsBuiltReport -Report Veeam.VBR -Target veeam-vbr.pharmax.local -AsBuiltConfigFilePath C:\AsBuiltReport\AsBuiltReport.json -OutputFolderPath C:\AsBuiltReport\ -Username 'domain\User' -Password 'p@ssw0rd' -Format HTML -ReportConfigFilePath C:\AsBuiltReport\AsBuiltReport.Veeam.VBR.json -EnableHealthCheck -Timestamp"

It is required to generate both JSON files so that it can be referenced within the task.
https://github.com/AsBuiltReport/AsBuiltReport.Core?tab=readme-ov-file#new-asbuiltconfig
-AsBuiltConfigFilePath C:\AsBuiltReport\AsBuiltReport.json
https://github.com/AsBuiltReport/AsBuiltReport.Veeam.VBR?tab=readme-ov-file#pencil2-configuration
-ReportConfigFilePath C:\AsBuiltReport\AsBuiltReport.Veeam.VBR.json
It would be a good idea to create a cmdlet for this purpose but it would be necessary to add it to the AsBuiltReport.Core.


[SOP Document] Central Processing

Cloud Layout
Date Shifting

@rkamaleswaran - I have created quite a bit of material related to this SOP - let me know if you'd like to setup a brief call where we outline what it should contain.

@rkamaleswaran - I've created a branch in the repo where you can find the SOP document template and diagram:


Register account on editor website does not works

On this page:
https://editor.apexcharts.com/login
I am not able to create account because of CORS problem:

I created account using curl xD, login works correctly.

Same issue. Doesn't work as of today

curl -X POST \
  https://apexcharts.com/wp-json/wp/v2/users/register \
  -H "Content-Type: application/json" \
  -d '{"username":"anything","email":"you@example.com","password":"yourStrongP@ssw0rd"}'

this do work.


Reproducing office31 results

Good morning, which hyperparameters should I use to reproduce your Office31 results? I tried with the same of Office-Home but I get results way lower than the ones in the paper.

Hi, set K to 2 and KK to 3 (for a2w you can set them to 3/2). You need to make sure the pytorch version is under 1.4 (we are using 1.3 with cuda 10.0) to reproduce the results.

Thank you very much! And how many epochs on source, still 20? or 100?

We are using 50 but it seems to be not that important.

And I thought (not sure) I use the lr_scheduler (which is commented) just as office-home, maybe you should use it.

Alright, thank you very much!

I report the results that I was successfully able to reproduce (40 epochs on source without scheduling, torch 1.3, cuda 10, K=2, KK=3).

AW
AD
WA
WD
DA
DW
average

reprod.
94.09
95.78
75.43
99.8
75.65
98.36
89.85

paper
90.8
96.0
75.0
100
75.3
99.0
89.35

Thanks, good work! XD


fix management of nontrivial dynamic flavours of libHSrts

Where non trivial means e.g "threaded dynamic", i.e not just "dynamic".
This requires a Cabal patch: https://github.com/haskell/cabal/pull/5606
and a GHC patch that bumps Cabal and also adds:
diff --git a/rts/rts.cabal.in b/rts/rts.cabal.in
index 76fd353134..9c95fbb342 100644
--- a/rts/rts.cabal.in
+++ b/rts/rts.cabal.in
@@ -39,6 +39,8 @@ flag debug
   default: False
 flag logging
   default: False
+flag dynamic
+  default: False

 library
     -- rts is a wired in package and
@@ -65,8 +67,14 @@ library
         extra-library-flavours: _debug_p _thr_debug_p
     if flag(debug)
       extra-library-flavours: _debug _thr_debug
+      if flag(dynamic)
+        extra-dynamic-library-flavours: _debug _thr_debug
     if flag(logging)
       extra-library-flavours: _l _thr_l
+      if flag(dynamic)
+        extra-dynamic-library-flavours: _l _thr_l
+    if flag(dynamic)
+      extra-dynamic-library-flavours: _thr

     exposed: True
     exposed-modules:

I will submit it once the Cabal patch is merged. Until then, I thought it would be a good idea to have this PR ready. As reported in #695 I can build & install all the dynamic RTS flavours I want with all those patches. This effectively fixes #695.

Note that after rebasing against latest hadrian & ghc, my Cabal patch doesn't work anymore. The source of the problem has not been identified yet, but the symptom is very simple to describe.
1/ hadrian still builds all the fancy flavours that we want, it puts them under /stage1/rts/build/
2/ when we Hadrian.Haskell.Cabal.configurePackage, we do get all 4 flags (profiling, logging, dynamic, debug) from the Cabal Flags builder and pass that to Cabal
3/ much further down the road, when comes the time to install the static and dynamic libraries for libHSrts, it looks as if extra-dynamic-library-flavours is empty, except that with the 4 flags that we do pass when configuring, we should get the entire menu of dynamic flavours listed in there! So my Cabal patch ends up installing just the "vanilla shared" RTS library...

Many thanks @alpmestan and @DavidEichmann!

@bgamari A lot :-) See https://ghc.haskell.org/trac/ghc/ticket/15837 which sums everything up. In particular, the cabal PR I made got merged yesteday, so we're finally going to be able to proceed with the rest of the plan outline at #15837.

@alpmestan Thanks! I guess we should close this PR here then?

Yes we can, the ticket I mentionned above "took over" :-)


Typescript error when 'noImplicitAny' in set to TRUE in tsconfig

node_modules/@polymer/lit-element/lit-element.d.ts(14,46): error TS7016: Could not find a declaration file for module<EMAIL_ADDRESS><EMAIL_ADDRESS>implicitly has an 'any' type.   Trynpm install @types/polymer__polymerif it exists or add a new declaration (.d.ts) file containingdeclare module 'polymer__polymer';node_modules/@polymer/lit-element/lit-element.d.ts(15,44): error TS7016: Could not find a declaration file for module<EMAIL_ADDRESS><EMAIL_ADDRESS>implicitly has an 'any' type.   Trynpm install @types/polymer__polymerif it exists or add a new declaration (.d.ts) file containingdeclare module 'polymer__polymer';node_modules/@polymer/lit-element/lit-element.d.ts(17,46): error TS7016: Could not find a declaration file for module<EMAIL_ADDRESS><EMAIL_ADDRESS>implicitly has an 'any' type.   Trynpm install @types/polymer__polymerif it exists or add a new declaration (.d.ts) file containingdeclare module 'polymer__polymer';node_modules/@polymer/lit-element/lit-element.d.ts(18,44): error TS7016: Could not find a declaration file for module<EMAIL_ADDRESS><EMAIL_ADDRESS>implicitly has an 'any' type.
Goes away if noImplicitAny is FALSE. I'd like it to be true in my projects - because typed

This is actually an issue from the untypedness in Polymer, as the error message explains. Therefore, to be able to resolve this we would need to publish new types for Polymer 3. That is being tracked in https://github.com/Polymer/gen-typescript-declarations/issues/18 which we are working on per  https://www.polymer-project.org/blog/2018-01-25-polymer-2-4.html Please subscribe to that issue to be notified when we have resolved it.


OTT-1758-Update device.feature for group scope

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.nandana seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Chart.dependencies is nil for chart with several known dependencies

dependencies field of helm Chart (https://github.com/helm/helm/blob/master/pkg/chart/chart.go#L52) is nil when it has several known dependencies.
Chart object was created using NewList constructor (https://github.com/helm/helm/blob/master/pkg/action/list.go#L132) and Run (https://github.com/helm/helm/blob/master/pkg/action/list.go#L140).
Then releases[0].Chart.dependencies was nil (where releases, _ := lister.Run()).
Output of helm version:
version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13.8"}

Output of kubectl version:
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-13T11:52:32Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.): Minikube

As implied by the JSON meta tags present in the Chart struct, the Dependencies is not saved when the chart is saved in the release object. It is therefore impossible to fetch the dependency tree using helm list.
This was intended as a way to save space in the release object. Kubernetes only allows 1MB of metadata to be stored in each object, and Helm requires all of the rendered output as part of the Manifest to perform its three-way diff functionality as part of helm upgrade. Adding the dependency tree to the release metadata would break several cases where the chart templates go well above the 1MB limit.
To determine a chart's dependency tree, you will need to load the chart from disk. That is the only time where the dependency tree is resolved and loaded into memory.
Hope this helps.

closing as a wontfix. This is an intentional design decision to work around Kubernetes' limitation of 1MB storage space. Thanks.


bump up pcidevices controller to v0.2.4

IMPORTANT: Please do not create a Pull Request without creating an issue first.
Problem:

PR introduces new version of pcidevices controller for 1.1.2
Solution:

Related Issue:
Test plan:

@mergify backport v1.1


top-level links for collection question

I want to be sure I understand how top-level links should be generated for a request for a collection.
My current understanding:

the self link should not include query parameters related to pagination, filtering, sorting, sparse field sets, or compound documents.
the pagination links should include generated parameters for pagination, and pass along parameters for filtering, sorting, sparse field sets and compound documents as received.

Does this correspond with the intent of the specification authors?  Or is the client responsible for mixing in parameters related to any of those aspects for pagination links?

The self link at the top-level should include whatever params are needed to return an identical result. Top-level self should be considered a link to the current document. So if the current document contains a collection that is filtered or paginated, those params should be included in the self link.
Similarly, paginated links (next, prev, etc.) should step through the current collection and thus should also include whatever params are present on self.

Thanks, @dgeb!  This is very helpful.
Was this covered in the specification in some way that I missed, or will clarifications be added (at least for 1.1)?

Glad to help! I'll reopen so we can review / discuss if clarification is needed in the spec.

Reviewing the 1.0 spec, the description of the top-level self link here:

https://jsonapi.org/format/#document-links

does not spell out that the link refers specifically to the document, and should be constructed to reproduce the current rendering.  The Top Level section does indicate the self link is for the generated response; perhaps that should be considered sufficient, though this portion of the spec doesn't include the exact term "self link", which does yield the other description of the top-level self link.
Perhaps the difficulty is that the top-level self link is described in two places?

Two months and no response to my analysis?
@dgeb, when you reopened for discussion, I was hoping not to be talking to myself.

@freddrake in 1.1 isn't the top-level self link only specifically described in https://jsonapi.org/format/1.1/#document-top-level ? That looks pretty accurate and sufficient imo.
I'm not clear how it contradicts any of the general language about links contained in https://jsonapi.org/format/1.1/#document-links

@dgeb: I don't think there's a contradiction.  The problem I was trying to point out is that it's fairly difficult to be sure how these things need to be handled in an implementation.  To some degree, that's inherent in interpreting a specification, but... JSON:API isn't exactly a complicated spec.
Part of the difficulty may be mine rather than a fault in the specification.  I want to think of resources, collections, and relationships as being "equal citizens", but the specification doesn't actually suggest that at all.

I just reviewed v1.1 and it seems to be clear to me.

The meaning of top-level self link is defined in Top Level chapter. It explicitly states that it must be "the link that generated the current response document". I think the additional clarification for profiles and extensions makes that even more clear.

self: the link that generated the current response document. If a document has extensions or profiles applied to it, this link SHOULD be represented by a link object with the type target attribute specifying the JSON:API media type with all applicable parameters.

The Links chapter points back to the different places, in which a links object may be used.

A link’s context is the top-level object, resource object, or relationship object in which it appears.

I just reviewed v1.1 and it seems to be clear to me.

The meaning of top-level self link is defined in Top Level chapter. It explicitly states that it must be "the link that generated the current response document". I think the additional clarification for profiles and extensions makes that even more clear.

self: the link that generated the current response document. If a document has extensions or profiles applied to it, this link SHOULD be represented by a link object with the type target attribute specifying the JSON:API media type with all applicable parameters.

The Links chapter points back to the different places, in which a links object may be used.

A link’s context is the top-level object, resource object, or relationship object in which it appears.

Let's maybe better wait until #1604 lands. Reopening for now.

#1604 has been merged.


chore(deps): update module github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper to v1.5.0

This PR contains the following updates:

Package
Type
Update
Change

github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper
require
minor
v1.3.0 -> v1.5.0

Release Notes

terraform-ibm-modules/ibmcloud-terratest-wrapper
v1.5.0
Compare Source
Features

expose functions as public (#363) (dc57f6f)

v1.4.6
Compare Source
Bug Fixes

deps: update gomod (#362) (a94a6a2)

v1.4.5
Compare Source
Bug Fixes

deps: update module github.com/ibm/platform-services-go-sdk to v0.31.4 (#351) (3eb0257)

v1.4.4
Compare Source
Bug Fixes

changed InitAndApply() to Apply() for the optional last step of Upgrade test, fixes kubecontrol missing issues (#350) (9430e46)
deps: update module github.com/ibm/platform-services-go-sdk to v0.31.4 (#351) (f0e67ba)

v1.4.3
Compare Source
Bug Fixes

deps: update github.com/ibm-cloud/bluemix-go digest to afb4811 (#334) (efee9af)

v1.4.2
Compare Source
Bug Fixes

deps: update github.com/ibm-cloud/bluemix-go digest to 9a11ac0 (#325) (cb84653)

v1.4.1
Compare Source
Bug Fixes

deps: update gomod (#324) (f7e090e)

v1.4.0
Compare Source
Features

added schematics unit test support (#307) (4af1675)

v1.3.8
Compare Source
Bug Fixes

deps: update module github.com/gruntwork-io/terratest to v0.41.9 (#317) (3d22c84)

v1.3.7
Compare Source
Bug Fixes

only output diff on update (#315) (43f68e5)

v1.3.6
Compare Source
Bug Fixes

deps: update module github.com/gruntwork-io/terratest to v0.41.8 (#314) (7c9354f)

v1.3.5
Compare Source
Bug Fixes

deps: update github.com/ibm-cloud/bluemix-go digest to 28d386b (#312) (7536a70)

v1.3.4
Compare Source
Bug Fixes

set API_DATA_IS_SENSITIVE=true which is required by restapi provider (#310) (2afe543)

v1.3.3
Compare Source
Bug Fixes

deps: update module github.com/ibm/go-sdk-core/v5 to v5.12.0 (#309) (6aad3c8)

v1.3.2
Compare Source
Bug Fixes

deps: update module github.com/ibm/go-sdk-core/v5 to v5.11.0 (#305) (34a135c)

v1.3.1
Compare Source
Bug Fixes

deps: update module github.com/go-git/go-git/v5 to v5.5.2 (#297) (e69b2bf)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, click this checkbox.

This PR has been generated by Renovate Bot.

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Copy of Wireframe: News article (UP)

User story
As a ReportStream prospective/current user, I want the website content to be up-to-date, accurate and timely so I can  trust the information to inform my decisions in my relationship with ReportStream.
Background & context
This ticket captures the work of putting together the News article (UP) using the detail page template for dev ready handoff.
Working links

Figma: Working file
Content doc: News article

Acceptance criteria
Design

[x] Design includes latest content updates
[x] Capture questions that come up during the content and design work in the Website redesign research plan

Review

[x] Reviewed with design and/or design lead
[x] Reviewed with content strategist for final approval
[x] Reviewed with product and engineering

Follow up ticket needed: To update content

accidental duplicate


feat: lazygit integration

https://github.com/NotAShelf/neovim-flake/issues/46
a simple lazygit thing using toggleterm, derived from their help file

LGTM! I'll update the docs and should be ready to merge afterwards.

Sorry it took me too long, had to go outside and touch grass for a bit. The alejandra formatting should fix the CI fail.
P.S. The styling CI looks like it's failing due to some Github networking error, Alejandra returns a positive exit code and I think we are good for a merge.
Thanks for your contribution.


ci: test on alternative operating systems; ignore doc updates

Stops CI from running of documentation is updated (waste of a run)
Tests on macOS and Windows alongside Ubuntu, to ensure fluent-json-schema runs on all operating systems (Windows is the big one to test, just due to how it handles forward/back slashes differently, and single quotation marks from unix)

Checklist

[x] commit message and code follows the Developer's Certification of Origin
and the Code of conduct

Pull Request Test Coverage Report for Build 839090865

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 99.554%

Totals

Change from base Build 834338069:
0.0%

Covered Lines:
408

Relevant Lines:
408

💛  - Coveralls

LGTM


subprocess error when fine tunning llama2

Trying to finetunning llama2 model
here is the exception info:
Traceback (most recent call last):
File "/data1/xbz/LLM-Pretrain-SFT-master/llm_pretrain/pretrain3.py", line 76, in
trainer_stats = trainer.train()
File "", line 121, in train
File "", line 359, in _fast_inner_training_loop
File "/home/lantu_mp/.local/lib/python3.9/site-packages/transformers/trainer.py", line 2902, in training_step
loss = self.compute_loss(model, inputs)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/transformers/trainer.py", line 2925, in compute_loss
outputs = model(**inputs)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/accelerate/utils/operations.py", line 822, in forward
return model_forward(*args, **kwargs)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/accelerate/utils/operations.py", line 810, in call
return convert_to_fp32(self.model_forward(*args, **kwargs))
File "/home/lantu_mp/.local/lib/python3.9/site-packages/torch/amp/autocast_mode.py", line 16, in decorate_autocast
return func(*args, **kwargs)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/unsloth/models/llama.py", line 883, in PeftModelForCausalLM_fast_forward
return self.base_model(
File "/home/lantu_mp/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/peft/tuners/tuners_utils.py", line 179, in forward
return self.model.forward(*args, **kwargs)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/accelerate/hooks.py", line 166, in new_forward
output = module._old_forward(*args, **kwargs)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/unsloth/models/llama.py", line 814, in _CausalLM_fast_forward
outputs = self.model(
File "/home/lantu_mp/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/accelerate/hooks.py", line 166, in new_forward
output = module._old_forward(*args, **kwargs)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/unsloth/models/llama.py", line 651, in LlamaModel_fast_forward
hidden_states = Unsloth_Offloaded_Gradient_Checkpointer.apply(
File "/home/lantu_mp/.local/lib/python3.9/site-packages/torch/autograd/function.py", line 539, in apply
return super().apply(*args, **kwargs)  # type: ignore[misc]
File "/home/lantu_mp/.local/lib/python3.9/site-packages/torch/cuda/amp/autocast_mode.py", line 113, in decorate_fwd
return fwd(*args, **kwargs)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/unsloth/models/_utils.py", line 385, in forward
output = forward_function(hidden_states, *args)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/accelerate/hooks.py", line 166, in new_forward
output = module._old_forward(*args, **kwargs)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/unsloth/models/llama.py", line 433, in LlamaDecoderLayer_fast_forward
hidden_states = fast_rms_layernorm(self.input_layernorm, hidden_states)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/unsloth/kernels/rms_layernorm.py", line 190, in fast_rms_layernorm
out = Fast_RMS_Layernorm.apply(X, W, eps, gemma)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/torch/autograd/function.py", line 539, in apply
return super().apply(*args, **kwargs)  # type: ignore[misc]
File "/home/lantu_mp/.local/lib/python3.9/site-packages/unsloth/kernels/rms_layernorm.py", line 144, in forward
fx[(n_rows,)](
File "", line 63, in _rms_layernorm_forward
File "/home/lantu_mp/.local/lib/python3.9/site-packages/triton/compiler/compiler.py", line 425, in compile
so_path = make_stub(name, signature, constants)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/triton/compiler/make_launcher.py", line 39, in make_stub
so = _build(name, src_path, tmpdir)
File "/home/lantu_mp/.local/lib/python3.9/site-packages/triton/common/build.py", line 90, in _build
ret = subprocess.check_call(cc_cmd)
File "/usr/lib64/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/local/bin/gcc', '/tmp/tmpka0ojl2z/main.c', '-O3', '-I/home/lantu_mp/.local/lib/python3.9/site-packages/triton/common/../third_party/cuda/include', '-I/usr/include/python3.9', '-I/tmp/tmpka0ojl2z', '-shared', '-fPIC', '-lcuda', '-o', '/tmp/tmpka0ojl2z/_rms_layernorm_forward.cpython-39-x86_64-linux-gnu.so', '-L/usr/lib64', '-L/lib', '-L/usr/lib64', '-L/lib']' returned non-zero exit status 1.
here is my code:
max_seq_length = 4096  # Choose any! We auto support RoPE Scaling internally!
dtype = None  # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
load_in_4bit = True  # Use 4bit quantization to reduce memory usage. Can be False.

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="/data1/xbz/llama2_expand/",
    max_seq_length=max_seq_length,
    dtype=dtype,
    load_in_4bit=load_in_4bit,
    # token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
)

model = FastLanguageModel.get_peft_model(
    model,
    r=32,  # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj", ],
    lora_alpha=16,
    lora_dropout=0,  # Supports any, but = 0 is optimized
    bias="none",  # Supports any, but = "none" is optimized
    # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
    use_gradient_checkpointing="unsloth",  # True or "unsloth" for very long context
    random_state=3407,
    use_rslora=False,  # We support rank stabilized LoRA
    loftq_config=None,  # And LoftQ
)

dataset_dict = load_dataset("/data1/xbz/llama3_ddl/")
train_test_split_dataset = dataset_dict['test'].train_test_split(test_size=0.01)
train_test_split_dataset2 = train_test_split_dataset['test'].train_test_split(test_size=0.2)
train_test_split_dataset3 = train_test_split_dataset2['test'].train_test_split(test_size=0.2)
train_test_split_dataset4 = train_test_split_dataset3['test'].train_test_split(test_size=0.2)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=train_test_split_dataset4['train'],
    eval_dataset=train_test_split_dataset4['test'],
    dataset_text_field="content",
    max_seq_length=max_seq_length,
    dataset_num_proc=1,
    packing=True,  # Can make training 5x faster for short sequences.
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        learning_rate=2e-4,
        fp16=not is_bfloat16_supported(),
        bf16=is_bfloat16_supported(),
        logging_steps=10000,
        optim="adamw_8bit",
        weight_decay=0.01,
        lr_scheduler_type="cosine",
        seed=3407,
        output_dir="/data1/xbz/llama2_new/",
    ),
)

trainer_stats = trainer.train()

model.save_pretrained("/data1/xbz/lora_model") # Local saving
tokenizer.save_pretrained("/data1/xbz/lora_model")

can you check ur GCC, Triton, Pytorch and CUDA versions.

can you check ur GCC, Triton, Pytorch and CUDA versions.
gcc version 7.5.0 (GCC)
CUDA Version: 11.8
torch.version：'2.1.0+cu118'

I have a similar issue, when running phi3 mini everything works, when I change the model to phi3 medium instruct/llama 3 8b instruct or mistral v3 instruct it fails

@Atomheart-Father Did you use the exact install instructions for torch 2.1 & CUDA 11.8 on our Github repo?
@antolo-arch Oh you might have to update Unsloth maybe? (Unless you already did)
pip uninstall unsloth -y
pip install --upgrade --force-reinstall --no-cache-dir git+https://github.com/unslothai/unsloth.git

If you did, that's a super weird bug

@danielhanchen I did upgrade the package, I went to google collab with an example for Llama 3 8b, when I run the code after a long track of errors I get this one
  raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/bin/gcc', '/scratch-local/scur2122.6532017/tmp67tc2ubd/main.c', '-O3', '-I/home/scur2122/.conda/envs/unsloth_env/lib/python3.10/site-packages/triton/common/../third_party/cuda/include', '-I/home/scur2122/.conda/envs/unsloth_env/include/python3.10', '-I/scratch-local/scur2122.6532017/tmp67tc2ubd', '-shared', '-fPIC', '-lcuda', '-o', '/scratch-local/scur2122.6532017/tmp67tc2ubd/_rope_embedding.cpython-310-x86_64-linux-gnu.so', '-L/lib64', '-L/lib64']' returned non-zero exit status 1.

and this one
/home/scur2122/.conda/envs/unsloth_env/include/python3.10/Python.h:116:10: fatal error: namespaceobject.h: No such file or directory
 #include "namespaceobject.h"
          ^~~~~~~~~~~~~~~~~~~
compilation terminated.

as soon as i change the model to phi3 mini everything works fine 😬

@Atomheart-Father Did you use the exact install instructions for torch 2.1 & CUDA 11.8 on our Github repo?
@antolo-arch Oh you might have to update Unsloth maybe? (Unless you already did)
pip uninstall unsloth -y
pip install --upgrade --force-reinstall --no-cache-dir git+https://github.com/unslothai/unsloth.git

If you did, that's a super weird bug

My working environment is unable to connect to the internet, so I can not use pip or connect to github directly.
I can only download the zip file from the github page and transfer it to the environment.
how can I download the torch 2.1 & CUDA 11.8 version of zip file?I will try to reinstall it again.

@antolo-arch Great donwgrading and creating a new env works!
@antolo-arch Wait is there no way to connect to the internet? It's very hard getting torch 2.1 and CUDA as a zip

@antolo-arch Great donwgrading and creating a new env works! @antolo-arch Wait is there no way to connect to the internet? It's very hard getting torch 2.1 and CUDA as a zip

Yeah,due to some "security policy", github was blocked. I will close this issue.
Will you consider providing different versions of the zip package in the future, perhaps by adding different branches?

Oh a security policy? Interesting hm ill have to check that


SSH No Ports GUI unable to use Control C

Describe the bug
When using a Linux terminal session Control C and Control V are  useful for cut and pasting and they work in the GUI. But Control C is also used to interupt programs and it is not sent to the terminal session at present.
Steps to reproduce

First I connect to a machine using the GUI
Then I run top
And then I hit Control C  (nothing happens)

Expected behavior
When something is highlight then Control C copies what is highlighted to the cut/past buffer.
When something is not highlighted then Control C is sent on to the terminal session
Were you using an atApplication when the bug was found?

Windows SSH No Ports GUI

Additional context
What might also be useful is after highlighting right clicking the mouse unhighlights and puts content into cut/paste buffer

@cconstab Do you want control + C to exit from top when nothing is highlighted?

@cconstab Do you want control + C to exit from top when nothing is highlighted?

Yes, he does. if you run top in a normal terminal you can use q or <C-c> to exit top.
In our app you can only use q.
While this isn't a problem with top, it is a big problem for something like:
while true; do sleep 1; echo "asdf"; done
In fact, in an ideal world, we pass all control sequences to the terminal
For example:
<C-p> - reloads the previous command
<C-l> - clears the screen

When things are highlight cntl c should copy (windows norm)
When nothing is highlighted then the ctnl should be sent to session

On windows and linux we can also:

make  (Ctrl + shift + c) copy
make  (Ctrl + c) send the control signal to the terminal

This is default behaviour of vscode, and I think windows terminal

Closing this ticket, These features available from build 1.0.0 (4)


save user language

when a user signs up, we should set the language in
dosomething_user_user_insert()
we can call to convert the country code to the associated language

actually, it does look like we are doing this
it just isn't working

this happens when you visit thor.dosomething.org/mx (or br) explicitly.

also, see dosomething_user_set_fields

this seems to be working now. (tested locally and on thor with torguard.)

Duplicate of #5553


New version: WebIO v0.8.13

Registering package: WebIO
Repository: https://github.com/JuliaGizmos/WebIO.jl
Created by: @travigd
Version: v0.8.13
Commit: e160f9551b36b85da89bc0d18f30fdc4bf7b2468
Reviewed by: @travigd
Reference: https://github.com/JuliaGizmos/WebIO.jl/commit/e160f9551b36b85da89bc0d18f30fdc4bf7b2468#commitcomment-36538385

[noblock]
Skipping v0.8.12 because it shouldn't be released but NPM packages were already uploaded and can't be changed (#7031).


Limit amount of files in test dump

We previously have looked into reducing the size of config dumps from test failures. However, the number of files is also an issue as the prow code iterates them all. For example, https://prow.istio.io/view/gs/istio-prow/pr-logs/pull/istio_istio/34274/integ-ipv6-k8s-tests_istio/1423786671932968960 will not load.
We should see if we can reduce this at all to avoid these issues

not stale


Time is not imported in Cluster.py

File "/gfs/devel/ddanko/CGATPipelines/CGATPipelines/Pipeline/Execution.py", line 529, in run ignore_errors=ignore_errors)
File "/gfs/devel/ddanko/CGATPipelines/CGATPipelines/Pipeline/Cluster.py", line 260, in collectSingleJobFromCluster stdout, stderr = getStdoutStderr(stdout_path, stderr_path)
File "/gfs/devel/ddanko/CGATPipelines/CGATPipelines/Pipeline/Cluster.py", line 292, in getStdoutStderr time.sleep(1)
NameError: global name 'time' is not defined
Seems like a minor issue; I've only had it come up a few times.

Thanks for your feedback. This was fixed:
https://github.com/CGATOxford/CGATPipelines/blob/master/CGATPipelines/Pipeline/Cluster.py#L38


i have a table inside my table with the automation id "sgData" and I want to access the data inside it by clicking it. Does anyone know how to do that?

@hassanuz @alakshmi1030 @JunlinZhu  How can I click inside the table I have.
@varun210619 Did you ever find the solution for this? Also when you said it worked usinf action class for you. can you please share the code?

Are elements visible in the visual tree (e.g. Inspect.exe)? Hard to say if the controls were exposed to UI Automation.

@pm90pl No. They are not visible. It just shows pane for the whole table.

@NikitaTurakhia Can you please show(screenshot if possible) what you get in inspect.exe.
On which technology your application is based?
Some workarounds -

If you cannot get any row, can you get any option in application to export data and then work?
You can also try keyboard actions on application - pressing tabs and then clicks on particular rows.


Test that libbpf-rs compiles without error in CI

Right now, there are no tests beyond ensuring that libbpf_set_print can be called. This ensures that something hasn't gone horribly wrong, but it doesn't do much to prevent API breakage.
In libbpf/libbpf-rs#303, a minor release of libbpf-sys broke libbpf-rs because bindgen's default behavior changed. This would have been prevented if we somehow tested compiling libbpf-rs in CI, but I don't know enough about GitHub CI to implement that.

I was going to suggest doing just that, but then it struck me as too libbpf-rs centric for me to propose here :-) Glad you suggested this step.


Update language-dtml.cson

Removed duplicate entries that causes an error message at the startup of atom

Please provide a better description.

Done


Move drag&drop to nehuba container

Why is this change required?

Why is this change required?

Because it is not possible to make drag&drop feature for plugins and everywhere outside Nehuba viewer.

Can you elaborate on drag and drop off plugin?

To import user landmarks, one of the ways is drag&drop. Currently, when the user drags the JSON file on the plugin, in parallel IAV expects to drop the file in it.


Add support for ScriptHookVDotNetCore

Nothing need to be changed since it shares identical scripting API with SHVDN.

Please merge master onto the PR so it can actually try and build with .NET 7 on CI

Turns out, I forgot to change the build .NET version. Whops.
Can you merge master again? Thanks!

No problem

Looks like it copies the SHVDNCore DLL to the output directory. Could you please change that?
You could take a look at the SHVDN3 csproj if you are not sure how to do it :)

Also, I don't think that the deps.json is required. That can be disabled with:
<GenerateDependencyFile>false</GenerateDependencyFile>

Done. Do i need to squash this to one commit?

Sure

Just letting you know: There is nothing wrong with this PR, I just want to do some stuff before I merge and release :)

Sure, no porblem 🙆♂

Closing this to avoid confusion as SHVDNC might get merged into SHVDN itself in the future

I have been following the issues on the SHVDN repo.
Can you keep this around while .NET Core support is fully implemented? So i can do some testing locally and prepare for the Core version.

So, it was decided in scripthookvdotnet/scripthookvdotnet#1036 that Core/5+ support will not be added to the existing version of SHVDN.
@Sardelka9515 do you plan to keep working on the core version in your repo?

The core version can be maintained with minimum efforts (I can easily merge the the SHVDN upstream code at anytime) it currently have some issues with exception handling when running alongside SHVDN. it was resolved in scripthookvdotnet#1167 but the new version is not released yet. Maybe kagikn will make another version for .NET Core in the future but SHVDNC should work fine for now


Clip offline tiles to downloaded bounds

Includes some small refactors of tile overlay and MOG impl.
Fixes #1850
Fixes #1756
Screen_recording_20230910_174722.webm
@shobhitagarwal1612 PTAL?

@shobhitagarwal1612 Thanks for reviewing! RFAL


i18n(fr): update guides/authentication.mdx

Description (required)
Removes Lucia from library examples in the French translation of guides/authentication.mdx (see #10064)

Related issues & labels (optional)

Suggested label: i18n

Lunaria Status Overview
🌕 This pull request will trigger status changes.

Learn more
By default, every PR changing files present in the Lunaria configuration's files property will be considered and trigger status changes accordingly.
You can change this by adding one of the keywords present in the ignoreKeywords property in your Lunaria configuration file in the PR's title (ignoring all files) or by including a tracker directive in the merged commit's description.

Tracked Files

File
Note

fr/guides/authentication.mdx
Localization changed, will be marked as complete.

Warnings reference

Icon
Description

🔄
The source for this localization has been updated since the creation of this pull request, make sure all changes in the source have been applied.


AkashJS Get-Deployments Error

@akash-mm commented on Wed Jun 15 2022
When using this AkashJS example:
https://github.com/ovrclk/akashjs/blob/main/examples/get_deployments.ts
Via:
        const request2 = QueryDeploymentsRequest.fromJSON({
          filters: {
            owner: 'akash1hsjdca4tfcth9mlleq2g05wr9f0jfw3ewgglg3',
          },
        })

        const client2 = new QueryClientImpl(getRpc(RPC))
        const response2 = await client2.Deployments(request2)
        const data2 = QueryDeploymentsResponse.toJSON(response2)
        console.log(data2)

Receive this error in browser developer tools > network tab:
{
  "jsonrpc": "2.0",
  "id": 0,
  "error": {
    "code": -32602,
    "message": "Invalid params",
    "data": "error converting json params to arguments: encoding/hex: invalid byte: U+002C ','"
  }
}

If no filters are based in the query is successful.

This should be fixed in<EMAIL_ADDRESS>


Refrences problem

I cant find the supported CortrolzEx dll for it do you guys have  pre compiled build

Fixed, references should be now on GitHub.


Accidently restored after dualboot

I did not know the remove dualboot command existed so i just restored my iphone with finder when something went wrong now im getting this error when trying to dualboot again

the only solution that i know for now is that using linux you can jump that error because linux uses another method


prediction in untrimmed audio

Hi, is the model capable to predict the age gender in the continous audio?
Thanks

The audio should be fed first into a voice activity detector (VAD), n order to run the model only on the speech parts.
The model will crash if the signal is not at least 0.025 s long. But we also highly discourage to use the model on segments shorter than 3 s as the performance degrades drastically for very short segments.


How can set null or '' value to my type via Expression?

Description
I have customized a type using Formly, similar to a text input field. Now, I want to clear my custom type when a different value is selected in a dropdown Select box. I've written this logic using the Expression property. However, I've noticed that when I return '', null, or undefined in the expression, the system encounters errors, and sometimes it doesn't update as instructed to be empty or null.
Minimal Reproduction
Your Environment

Angular version: 16
Formly version: 6

Additional context
My TS Code as below：
expressions: {
'model.C_TENOR_TYPE': (field: FormlyFieldConfig) => {
if (field.model.C_TENOR_TYPE=== 'A' || field.model.C_TENOR_TYPE=== 'B' || field.model.C_TENOR_TYPE=== 'C') {
return '';
}
return field.model.C_TENOR_TYPE;
},

see https://formly.dev/docs/guide/expression-properties#3-get-notified-about-an-expression-changes


error  css_zero [not found]

Hello - on a new Rails 8 app, I add the css-zero gem to gem file and bundle,
next, I run the generator:
% bin/rails generate css_zero:install
create  app/assets/stylesheets/base.css
force  app/views/layouts/application.html.erb
create  app/assets/stylesheets/layouts.css
create  app/assets/images/menu.svg
create  app/assets/stylesheets/input.css
create  app/assets/images/select-arrow.svg
create  app/assets/stylesheets/switch.css
create  app/assets/stylesheets/button.css
create  app/assets/images/loader-circle.svg
create  app/assets/stylesheets/flash.css
create  app/javascript/controllers/element_removal_controller.js
create  app/assets/stylesheets/alert.css
However, when I create a basic controller, I get an error that css_zero is not found.
% rails generate controller css_sample home
create  app/controllers/css_sample_controller.rb
route  get "css_sample/home"
error  css_zero [not found]
invoke  test_unit
create    test/controllers/css_sample_controller_test.rb
invoke  helper
create    app/helpers/css_sample_helper.rb
invoke    test_unit
The CSS files created by the generator are in the app/assets/stylesheets directory.
Help is greatly appreciated and I think this looks like an awesome project and can"t wait to start using it!

Hey, thanks for reporting that; it's fixed on 0.0.64.
https://github.com/lazaronixon/css-zero/releases/tag/v0.0.64

Thank you, and I hope you have a great day!!


Include collection_id and dataset_id as env vars for all jobs to help with troubleshooting

Further: make it more clear when the dataset version id refers to the id of the previous version vs the next (migrated) version.

This should be tackled as part of the Improving Efficiency of Schema Migrations work

This should be tackled as part of the Improving Efficiency of Schema Migrations work

Removing from #6790


Missing keystore file for SCIM

Current oxAuth/oxTrust images don't have scim-rp.jks or any SCIM-related files inside the container.

Resolved in https://github.com/GluuFederation/docker-oxauth/commit/fa42a0a1fb173c48caec47dcc7d84a686a362999 (oxAuth) and https://github.com/GluuFederation/docker-oxtrust/commit/da60016a014766af980c8de8a8d8e83b84cbc23b (oxTrust)


gh-action: use yq as a builtin

Describe the changes
See if yq is actually just a builtin

This is what I need help with

Link to Phabricator

Prior discussion

FWIW we confirmed this by looking in https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md


Init view's cache

Fix for issue #519

Thank you for the PR. I'll merge it tomorrow together with others


Better determine when an image has been cropped vs not cropped.

From Trello card description:
"Currently the logic found within https://github.com/parameter1/base-cms/blob/master/packages/image/src/crop-rectangle.js results in if a crop rectangle has coordinates of x = 0 and y = 0 it determines that an image has not been cropped, by default x and y are set to x1 and y1 meaning that if x1 and y1 are both 0 it determines the image has not been cropped regardless of it cropDimensions are set on the image or not.
x1 and y1 of 0 and 0 respectively are perfectly valid so this needs to be corrected to perform some other sort of logic in order to determine whether or not an image has been cropped."
Corrected Item:

GraphQL:

MongoDB Asset Collection:

Logging GraphQL Server Response:

Unaffected Item:

GraphQL:

MongoDB Asset Collection:

 Correct logic for determining whether or not an image has been cropped


Fix GH212, PP308

Fix a few warnings by repo review: GH212, PP308

[x] xref #8239

Great, thanks @tqa236
Setting to auto-merge. There's unrelated test failure from a dependency which should clear in a few hours (but will require re-running the test). Feel free to ping if it doesn't merge!


[SNOWFLAKE] Can't open SQL Stored Procedures parameters or code on schemas with underscore _

System information:

Operating system (distribution) and version: Windows 10 Enterprise 21H2
DBeaver version: 22.1.2
Java version: 1.8.0_331-b04
Additional extensions

Connection specification:

Database name and version: Snowflake 6.23.0
Driver name: JDBC Snowflake 3.13.6
Do you use tunnels or proxies (SSH, SOCKS, etc)? No

Describe the problem you're observing:
When trying to visualize the parameters and source code for stored procedures on schemas with underscore, I get the following error:
/* ERROR WHILE READING SOURCE:
SQL Error [1003] [42000]: SQL compilation error:
parse error line 1 at position 35 near '95'.
parse error line 1 at position 39 near '95'.
*/
On snowflake, it seems that DBeaver is trying to escape the underscore on the schema name on the desc procedure command:

Same doesn't happen on desc function, there no escape of the underscore is done and everything works fine.

Steps to reproduce, if exist:
Create a procedure on a schema that has underline on the schema name. DBeaver won't be able to show the parameters/source code of the procedure.
Create any other object (View, function) inside the same schema name. DBeaver is able to read and get the source code without any issue.
Include any warning/errors/backtraces from the logs

procedure_error.log

Hello @danryushin
As I can see, your issue is solved in the last Snowflake driver version.
You can change it in your connection settings.

Please try it and give us feedback - is driver upgrading helped you?

Hello,
Thanks, with the new driver the procedures are now working fine for visualization


Ensure that the Sidetree node's IPFS component is setup to only replicate/store Sidetree data

Most IPFS installs allow the IPFS instance to replicate and store random data from around the network. This is something we need to block in our use. The Sidetree node should only be coded to replicate and store Sidetree-specific data.

Opened this issue in sidetree-ipfs: https://github.com/decentralized-identity/sidetree-ipfs/issues/30


Change default top_k to 1

If the current default top_k values we sometimes get really bad answers, which may make LitGPT look really bad. Just an example:
from litgpt import LLM

llm = LLM.load("mistralai/Mathstral-7B-v0.1")
print(llm.generate("What is 1+2?"))

 In your job, what is the average time people spend using your app? These can have wildly different answers depending upon the dataset, the timeframe for the dataset, and how the data is being calculated or stated, disaggregated and re

With
print(llm.generate("What is 1+2?", top_k=1))

 1+2 equals 3.

Overall, I think it's better to err on the side of generating the same answer, and if people want more variation, they can change the top_k setting. I think this is better than generating garbage by default and people trying to figure out how to make it work properly.

While I agree with the reasoning, I'm concerned that this happens in the first place.
Let's merge the PR and later come back to it.
Current default values are quite safe and that shouldn't happen, in theory.

@rasbt How do you arrive at the conclusion that top_k needs to be set to 1 in this case? What do the other outputs look like under top_k > 1 and does the original model also exhibit this behavior? What about the other models (e.g., see other references)?

It's basically something observed at the workshop last week too, it takes several takes sometimes to get reasonable results, which looks kind of bad. Here for phi-2:

For top_k=1 you will always get

' Llamas are herbivores and primarily eat grass, hay, and other plant material. They have a unique digestive system that allows them to efficiently extract nutrients from fibrous plant material.\n'

Other values like top_k=5 or 10 also works ok. But isn't it better to make it deterministic by default to ensure somewhat  good results? Either way people may want to change the setting, but I think it's better if it looks good by default then producing weird results and people thinking it's an error, issue or bug.

Looks like the mileage varies depending on what model we look at, but I think that 50 is definitely too high. Would something like 5 be a good compromise where we have it not strictly deterministic but also not quite as random? What do you think @awaelchli @Andrei-Aksionov ?

Could you then also create an issue with these findings?
Definitely something is off.

Yes, let's stick to 50.
Create an issue.
I'll take a look at it when I have time.

Ok sounds good. In the meantime, in this PR I changed everything to 50 then. It was 200 in some places.

What I don't understand is how the conclusion from "model does not produce good outputs" can be to reduce top_k. If top_k is really so impactful, shouldn't this be a problem in other implementations as well?
For example:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.set_default_device("cuda")

model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype="auto", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2", trust_remote_code=True)

inputs = tokenizer("What do Llamas eat?", return_tensors="pt", return_attention_mask=False)

outputs = model.generate(**inputs, max_length=200, top_k=200, do_sample=True)
text = tokenizer.batch_decode(outputs)[0]
print(text)

produces various examples of similar quality as what you have shared as acceptable.
Maybe I'm doing something wrong here on the HF transformers side to make this comparison. Please correct me if I'm wrong.

@awaelchli  I think I got bad results before but never that bad. I think it's possibly also something related to the kv-cache but not sure. I opened an issue here #1593

This hints at an issue, either in my transformers comparison code, or in LitGPT's generation code. If the latter is true, setting topk=1 should be considered a temporary hack in my opinion.

What makes me suspicious now is the the generation between litgpt chat and the Python API look different. The one from the Python API looks much worse. It can't be a total bug because otherwise lower top_k values wouldn't work. I need to investigate more. Thanks for calling this out though, it didn't occur to me that it could be anything other then top_k setting but now I realize that it might be something different.


We should add a medicine delete date

This helps us to know which medicine patients actually took at a specific point in time

information will probably be very unreliable anyhow.
Nice to know data, but actually in the real world people probably won't update dose/medication
Limited number of medicine available. Maybe even yes/no medication is sufficient information.
Also dose will most likely correlate to the stage of the disease


Date is not JSON serializable

Bug report
Bug summary
If a node_attribute, e.g. a Date object, is not JSON serializable, I get the following error:
      1 widget1 = ipycytoscape.CytoscapeWidget()
----> 2 widget1.graph.add_graph_from_neo4j(subgraph1)

~/miniconda3/envs/dkg/lib/python3.7/site-packages/ipycytoscape/cytoscape.py in add_graph_from_neo4j(self, g)
    739             # create node
    740             node_instance = Node()
--> 741             _set_attributes(node_instance, node_attributes)
    742             node_list.append(node_instance)
    743

~/miniconda3/envs/dkg/lib/python3.7/site-packages/ipycytoscape/cytoscape.py in _set_attributes(instance, data)
    250             setattr(instance, k, v)
    251         else:
--> 252             instance.data[k] = v

...

~/miniconda3/envs/dkg/lib/python3.7/site-packages/jupyter_client/jsonutil.py in json_default(obj)
    124         return float(obj)
    125
--> 126     raise TypeError("%r is not JSON serializable" % obj)
    127
    128

TypeError: Date(2020, 11, 6) is not JSON serializable

Code for reproduction
# Paste your code here
#
#

Actual outcome

Expected outcome

Version Info

ipycytoscape version (import ipycytoscape; print(ipycytoscape.__version__)) :
Python version:
Jupyter(Lab) version:

I'm thinking if all data shouldn't be serialized?
What if you try to convert it with dumps?
Idk if this make sense, sorry if it doesn't, but I'm struggling to find a good generic solution for this.
You think we should convert it always before calling this method add_graph_from_neo4j, for example? Or do you have any input on what should we do here?
Thanks for opening the issue! Never thought about this before.

Ah, this issue was fixed by your PR, correct?
Sorry for the confusion :)

Yes, the pr fixes the issue in a general way.
On Tue, Jan 18, 2022 at 11:28 AM Mariana Meireles @.***>
wrote:

Ah, this issue was fixed by your PR, correct?
Sorry for the confusion :)
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you authored the thread.Message ID:
@.***>


Support "versionAsFor" for streaming from delta table

It seems currently there is no good way how to start streaming from table from particular batch or timestamp. It is possible to apply filtering after reading, like:
spark.readStream
      .format("delta")
      .load(tablePath)
      .where(col("dateColumn") >= today)
      .writeStream...

but it would really useful to be able to set starting batchId or timestamp streaming should be started from. Sort of:
spark.readStream
      .format("delta")
      .option("versionAsOf", "5")

This is a great feature request. Would be great if someone in the community adds this feature.

This looks useful. Since this is not time traveling, I'd suggest to use startingVersion instead.

#474 has been released


Better Tekton API package

I wanted to create this ticket to kick off the discussion around how we can make Tekton API (pkg/apis package) easier to use. Currently, due to the heavy coupling issues in Tekton API, the vendors building on the top of Tekton Pipelines have to import some unnecessary dependencies on the rest of tree of Tekton Pipelines, such as Triggers which only needs to import Tekton API, and Tekton Chains, Results and CLI who need both API and Client.
Therefore, segmenting Tekton API pkg/apis and client pkg/client into their own modules might help those vendors reduce the number of dependencies and make it easier to use for all future vendors who only need API and/or client.
The current coupling issues in pkg/apis package:

Internal coupling issue: some _type.go files also define validation functions which call other validation code defined in _validation.go files. Should we add a new design principle that _types.go files should only contain struct definitions, not any other function definitions? That means we might need to move some validation functions to _validation.go files and perhaps move some getters/setters/MarkStatus to somewhere else as well.

External coupling issue: To segment pkg/apis to its own module, we need to make sure it doesn’t import other packages in Tekton Pipelines. But now it depends on a number of packages:

pkg/reconciler/testing: example
pkg/spire/config: example
pkg/substitution: example
pkg/resolution/common: example
pkg/reconciler/pipeline/dag: example

Since Client only depends on API package in Tekton Pipelines, if we clean up the APIs, client should be slimmer too.

cc @wlynch @dibyom

I'm generally in favor of this! (though be mindful of breaking changes)

Internal coupling issue: some _type.go files also define validation functions which call other validation code defined in _validation.go files. Should we add a new design principle that _types.go files should only contain struct definitions, not any other function definitions?

These are defined because of the apis.Validatable interface which knative uses to configure the admission webhooks along with Defaultable. These likely need to stay in place attached to the types. There may be some opportunities to move behavior out of the admission webhooks into the reconciler, but we'd have to look at those on a case-by-case basis.

Also, I am wondering if mirroring api and client into their own repos will help us in the long term esp as tekton grows.
Kubernetes has the staging folder in the home repo where all the development happens and periodically syncs individual packages into own repos.

but I think we should at least move validation out of _types.go file. For example, _validation.go might be a good place for it.

I think that's what we generally aim for (though I wouldn't be surprised if we're not strictly following this everywhere).
This should all be the same package though, so it shouldn't make a functional difference which file it lives in.

Also, I am wondering if mirroring api and client into their own repos will help us in the long term esp as tekton grows.
Kubernetes has the staging folder in the home repo where all the development happens and periodically syncs individual packages into own repos.

My preference would be to use Workspaces with a separate module in the same repo to avoid having to maintain extra automation.
I can't think of any benefits for separate repos that a separate module wouldn't also give us.

I think moving api and client to its own module makes sense. It would be tough to move the validation/defaulting/conversion functions since the knative/pkg conventions depend on our types having those functions. Refactoring the package to not depend on other packages in tekton sounds like a good idea though.
I'm not sure about a separate repo, not sure how much benefit that will bring over the cost of maintaining another repo.

I think that's what we generally aim for (though I wouldn't be surprised if we're not strictly following this everywhere).
This should all be the same package though, so it shouldn't make a functional difference which file it lives in.

I agree that files doesn't make a functional differences given that they are in the same package anyways, but I think clearly separating _type.go and _validation.go files will help the vendors who is able to cherrypick which files to import from tekton pipelines. For instance, when they only need _types.go in their third_party, this will help avoid importing unnecessary dependencies.

My preference would be to use Workspaces with a separate module in the same repo to avoid having to maintain extra automation.

👍

separating _type.go and _validation.go files will help the vendors who is able to cherrypick which files to import from tekton pipelines.

I would not rely on file names as a stable reference. You're welcome to selectively parse/transform files in compliance with the LICENSE, but I doubt we will guarantee compatibility. Package boundaries are probably safest.

It would be tough to move the validation/defaulting/conversion functions since the knative/pkg conventions depend on our types having those functions.

Thanks for the input. @dibyom!

I would not rely on file names as a stable reference. Package boundaries are probably safest.

That's a good point.
I will think about this a bit more.
From today (2023-04-03) API WG discussion, we got community agreement, and we will start from step 1 and see how it goes.

I think moving api and client to its own module makes sense. It would be tough to move the validation/defaulting/conversion functions since the knative/pkg conventions depend on our types having those functions. Refactoring the package to not depend on other packages in tekton sounds like a good idea though.
I'm not sure about a separate repo, not sure how much benefit that will bring over the cost of maintaining another repo.

Suggestion: let's complete the currently-defined steps

Step 1: move all the validation funcs from _type.go files to _validation.go files
Step 2 (not too sure): move getter/setter/markstatus funcs from _types.go files to somewhere else??
Step 3: makes sure pkg/apis package doesn’t depend on other packages in Tekton Pipelines
Step 4: segment api and client into their own go modules.

and then proceed to evaluate the trade-offs in moving all the API definitions into a separate repo.
On my mind: providers, vendors, and users who have a dependency on Tekton really should have no requirement to clone the entire pipelines repo as part of their third-party vendoring process. Perhaps Go modules are the right approach, or perhaps a more portable approach is to move all our public APIs that users code against into Protocol Buffer definitions in a stand-alone repo -- this enables API dependencies with no code / implementation dependencies whatsoever?
Having tossed that idea into this issue, I don't think it's worth debating this until we have a stand-alone Go module, IMO that's a clear first step toward this path anyway, and it may be all that is needed today.


Spawning test in separate process

We have both require (old variant) and spawn working: require for ios (for now) and spawn for desktop and android.
Closes #1228.

This change is

PR is added to the queue for testing as 1. task. (368e99e)

Test (Success) 88523129 build is completed (368e99e)
See https://github.com/ThaliTester/TestResults/tree/88523129368e99e_Spawning_test_in_separate_process_andrew-aladev/ for the logs

Test<PHONE_NUMBER>8e99e(368e99e) has failed
See https://github.com/ThaliTester/TestResults/tree/88523129368e99e_Spawning_test_in_separate_process_andrew-aladev/ for the fail logs

Reviewed 12 of 12 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


api can be intercepted by Burp after implemented it if android API level < 24

after implemented it in android, the api still can be intercepted by Burp, you can see the request/response and modify them, if api level <24, it is not supposed to happen.

Have you used this in your config: <trustkit-config enforcePinning="true"> ?

@nabla-c0d3  yes, I did, but that does not help.

Hey @wxhawk are you still experiencing this issue ? Because I was not able to reproduce the problem.

@jobot0 good news!!!
How did you reproduce it? did you use Burp as proxy to intercept all the network traffic from mobile app?
Can you share the details what you did in your app so that I can try on my side?

@wxhawk sorry for the delay of my answer. I just did the minimal Burp test (meaning just installed Burp on a device (23), trying to intercept the request) and the TrustKit seems to work. I need to setup a Reactive Native environment. I have not done myself the support for React Native but I'll definitely have a look to understand why you still have the issue.

I am having the same issue with a native app.

Hey @wxhawk quick update I'm still investigating the issue 😅  Sorry for the delay I had a lot of on my plate recently.
@ppamorim are you talking about a Reactive Native app or native app written only with the Android framework ?

Facing similar issue. my targetSdkVersion is 26 And enforcePinning is true.

TrustKit does not officially support react-native (we've never tested it) so I'll close this issue. If the problem is still happening with a non-react app, please open a new ticket with a copy of your network policy configuration.
Thanks!


Support floats in JSON

Support float types in JSON

Thanks @igor-plotnikov  for opening a PR.
There seems to be an issue with the float precision in the tests. I will have a look at it too.

@igor-plotnikov  thanks for your contribution, please review #48.


Sign release build with distribution profile

Otherwise it breaks CI or machines where only distribution profiles are installed, but not development ones.

Too old and not sure to get why


How to get image_list.txt?

Hi, I download the dataset from here then ran python dataset/synth-text/make_list.py and I couldn't find image_list.txt.
Is it necessary to download the dataset from this link?

Sorry, it is my fault, make_list.py is only for making data/synth-text/gt/*.txt and without image_list.txt, for getting image_list.txt you can simply run
ls gt/ > image_list.txt

in synth-text root. Or you can download here: image_list.txt

It works. Thanks. :)

I get an error at random times:
loss_pos = F.cross_entropy(predict[pos], target[pos], reduction='sum')   File "/home/sherif/Envs/TextSnake_venv/lib/python3.6/site-packages/torch/nn/functional.py", line 1550, in cross_entropy     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)   File "/home/sherif/Envs/TextSnake_venv/lib/python3.6/site-packages/torch/nn/functional.py", line 975, in log_softmax     return input.log_softmax(dim) RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

I seems that in some situations, the n_pos might be 0(due to blank annotation or too heavy augmentation) so it causes this error. This is almost impossible  if your batch size is big enough, maybe your batch size is set too small, try to use bigger batch size.
Another solution is change function ohem as following:
def ohem(self, predict, target, train_mask, negative_ratio=3.):
    pos = (target * train_mask).byte()
    neg = ((1 - target) * train_mask).byte()

    n_pos = pos.float().sum()

    if n_pos.item() > 0:
        loss_pos = F.cross_entropy(predict[pos], target[pos], reduction='sum')
        loss_neg = F.cross_entropy(predict[neg], target[neg], reduction='none')
        n_neg = min(int(neg.float().sum().item()), int(negative_ratio * n_pos.float()))
    else:
        loss_pos = 0.
        loss_neg = F.cross_entropy(predict[neg], target[neg], reduction='none')
        n_neg = 100
    loss_neg, _ = torch.topk(loss_neg, n_neg)

    return (loss_pos + loss_neg.sum()) / (n_pos + n_neg).float()

I can't use a batch size larger than 2.
Thanks. :)

I still get the error.

The cause is sometimes tcl_pred[train_mask * tr_mask] is tensor([], grad_fn=).

Well, ohem has no problem now. it can be resolved by a few code modifying, add a condition to set to 0.0 if there are not any positive samples.


Two test failures in french phonemizer sqlite3 connection

Not entirely sure what db_path it's trying to access here, and these are the only two tests failing for me.
======================================================================
ERROR: test_last_token (tests.test_fr_phonemizer.FrenchPhonemizerTestCase)
Ensure liason does not leave last token
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/source/tests/test_fr_phonemizer.py", line 75, in test_last_token
    phonemes = text_to_phonemes("Est-ce-que", lang="fr")
  File "/build/source/gruut/__init__.py", line 224, in text_to_phonemes
    return_tuples.extend(
  File "/build/source/gruut/__init__.py", line 224, in <genexpr>
    return_tuples.extend(
  File "/build/source/gruut/lang.py", line 858, in phonemize
    for (token1, token1_pron), (token2, token2_pron) in pairwise(
  File "/build/source/gruut/utils.py", line 125, in pairwise
    next(b, None)
  File "/build/source/gruut/phonemize.py", line 271, in phonemize
    token_pron = self.get_pronunciation(token)
  File "/build/source/gruut/phonemize.py", line 328, in get_pronunciation
    word_prons = list(pron for _word, pron in self.select_prons(word))
  File "/build/source/gruut/phonemize.py", line 328, in <genexpr>
    word_prons = list(pron for _word, pron in self.select_prons(word))
  File "/build/source/gruut/phonemize.py", line 607, in select_prons
    self._connect()
  File "/build/source/gruut/phonemize.py", line 481, in _connect
    self.db_conn = sqlite3.connect(self.db_path)
sqlite3.OperationalError: unable to open database file

======================================================================
ERROR: test_liason (tests.test_fr_phonemizer.FrenchPhonemizerTestCase)
Test addition of liason
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/source/tests/test_fr_phonemizer.py", line 51, in test_liason
    sentence = self._without_and_with_liason(
  File "/build/source/tests/test_fr_phonemizer.py", line 93, in _without_and_with_liason
    phonemes_no_liason = list(phonemizer_no_liason.phonemize(sentence.tokens))
  File "/build/source/gruut/lang.py", line 849, in phonemize
    yield from token_phonemes
  File "/build/source/gruut/phonemize.py", line 271, in phonemize
    token_pron = self.get_pronunciation(token)
  File "/build/source/gruut/phonemize.py", line 328, in get_pronunciation
    word_prons = list(pron for _word, pron in self.select_prons(word))
  File "/build/source/gruut/phonemize.py", line 328, in <genexpr>
    word_prons = list(pron for _word, pron in self.select_prons(word))
  File "/build/source/gruut/phonemize.py", line 607, in select_prons
    self._connect()
  File "/build/source/gruut/phonemize.py", line 481, in _connect
    self.db_conn = sqlite3.connect(self.db_path)
sqlite3.OperationalError: unable to open database file

----------------------------------------------------------------------

I tried to debug this by printing out the db_path and it is only opening :memory: and gruut/data/en-us/lexicon.db.
diff --git a/gruut/phonemize.py b/gruut/phonemize.py
index 4650fd3..934af86 100644
--- a/gruut/phonemize.py
+++ b/gruut/phonemize.py
@@ -477,8 +477,13 @@ class SqlitePhonemizer(Phonemizer):
         if self.db_conn is None:
             assert self.db_path is not None, "No sqlite3 database path"

-            _LOGGER.debug("Connecting to %s", self.db_path)
+            _LOGGER.error("Connecting to %s", self.db_path)
             self.db_conn = sqlite3.connect(self.db_path)
+            _LOGGER.error("OK")
+
+            import sys
+            sys.stdout.flush()
+            sys.stderr.flush()

             if not self.feature_to_id:
                 # Try to load feature names from the database

Oddly, there are more failing tests with regard to gruut when building the tests for tts 0.1.2.

DEBUG    gruut.phonemize:phonemize.py:480 Connecting to /nix/store/lgiasx76vy73z6r2hq6rbmmwp3620qvi-python3.8-gruut-1.2.0/lib/python3.8/site-packages/gruut/data/en-us/espeak/lexicon.db

But the espeak directory does not exist. There is a lexicon.db in the directory above.
❯ find /nix/store/lgiasx76vy73z6r2hq6rbmmwp3620qvi-python3.8-gruut-1.2.0/lib/python3.8/site-packages/gruut/data/
/nix/store/lgiasx76vy73z6r2hq6rbmmwp3620qvi-python3.8-gruut-1.2.0/lib/python3.8/site-packages/gruut/data/
/nix/store/lgiasx76vy73z6r2hq6rbmmwp3620qvi-python3.8-gruut-1.2.0/lib/python3.8/site-packages/gruut/data/en-us
/nix/store/lgiasx76vy73z6r2hq6rbmmwp3620qvi-python3.8-gruut-1.2.0/lib/python3.8/site-packages/gruut/data/en-us/g2p
/nix/store/lgiasx76vy73z6r2hq6rbmmwp3620qvi-python3.8-gruut-1.2.0/lib/python3.8/site-packages/gruut/data/en-us/g2p/model.crf
/nix/store/lgiasx76vy73z6r2hq6rbmmwp3620qvi-python3.8-gruut-1.2.0/lib/python3.8/site-packages/gruut/data/en-us/pos
/nix/store/lgiasx76vy73z6r2hq6rbmmwp3620qvi-python3.8-gruut-1.2.0/lib/python3.8/site-packages/gruut/data/en-us/pos/model.crf
/nix/store/lgiasx76vy73z6r2hq6rbmmwp3620qvi-python3.8-gruut-1.2.0/lib/python3.8/site-packages/gruut/data/en-us/lexicon.db

I forgot to include the espeak lexicon.db file for English in gruut 1.2.0. Please upgrade to 1.2.2 and try again.
Regarding the French test failures, you will need to pip install gruut-lang-fr first. Sorry, this was not mentioned anywhere in the docs.

I forgot to include the espeak lexicon.db file for English in gruut 1.2.0. Please upgrade to 1.2.2 and try again.

Indeed, that fixes the TTS test failures.
There does not seem to be a tag for versions 1.2.1 and 1.2.2, those versions are only up on PyPi.

Regarding the French test failures, you will need to pip install gruut-lang-fr first. Sorry, this was not mentioned anywhere in the docs.

The way we package them up means that gruut has all its language-specific packages accessible via its PYTHONPATH, but other than that they're all in their individual site-packages directory and are only linked together via that environment variable.
I probably have to read up on how it finds paths to the language packs.

Ah, it seems my language files packaging is at fault.
/nix/store/fih6hsnikzc4pmb6ri6nri58jghacz7z-python3.8-gruut-lang-fr-1.2.2/
/nix/store/fih6hsnikzc4pmb6ri6nri58jghacz7z-python3.8-gruut-lang-fr-1.2.2/lib
/nix/store/fih6hsnikzc4pmb6ri6nri58jghacz7z-python3.8-gruut-lang-fr-1.2.2/lib/python3.8
/nix/store/fih6hsnikzc4pmb6ri6nri58jghacz7z-python3.8-gruut-lang-fr-1.2.2/lib/python3.8/site-packages
/nix/store/fih6hsnikzc4pmb6ri6nri58jghacz7z-python3.8-gruut-lang-fr-1.2.2/lib/python3.8/site-packages/gruut_lang_fr
/nix/store/fih6hsnikzc4pmb6ri6nri58jghacz7z-python3.8-gruut-lang-fr-1.2.2/lib/python3.8/site-packages/gruut_lang_fr/__pycache__
/nix/store/fih6hsnikzc4pmb6ri6nri58jghacz7z-python3.8-gruut-lang-fr-1.2.2/lib/python3.8/site-packages/gruut_lang_fr/__pycache__/__init__.cpython-38.pyc
/nix/store/fih6hsnikzc4pmb6ri6nri58jghacz7z-python3.8-gruut-lang-fr-1.2.2/lib/python3.8/site-packages/gruut_lang_fr/VERSION
/nix/store/fih6hsnikzc4pmb6ri6nri58jghacz7z-python3.8-gruut-lang-fr-1.2.2/lib/python3.8/site-packages/gruut_lang_fr/__init__.py
/nix/store/fih6hsnikzc4pmb6ri6nri58jghacz7z-python3.8-gruut-lang-fr-1.2.2/lib/python3.8/site-packages/gruut_lang_fr-1.2.dist-info
/nix/store/fih6hsnikzc4pmb6ri6nri58jghacz7z-python3.8-gruut-lang-fr-1.2.2/lib/python3.8/site-packages/gruut_lang_fr-1.2.dist-info/INSTALLER
/nix/store/fih6hsnikzc4pmb6ri6nri58jghacz7z-python3.8-gruut-lang-fr-1.2.2/lib/python3.8/site-packages/gruut_lang_fr-1.2.dist-info/METADATA
/nix/store/fih6hsnikzc4pmb6ri6nri58jghacz7z-python3.8-gruut-lang-fr-1.2.2/lib/python3.8/site-packages/gruut_lang_fr-1.2.dist-info/RECORD
/nix/store/fih6hsnikzc4pmb6ri6nri58jghacz7z-python3.8-gruut-lang-fr-1.2.2/lib/python3.8/site-packages/gruut_lang_fr-1.2.dist-info/REQUESTED
/nix/store/fih6hsnikzc4pmb6ri6nri58jghacz7z-python3.8-gruut-lang-fr-1.2.2/lib/python3.8/site-packages/gruut_lang_fr-1.2.dist-info/WHEEL
/nix/store/fih6hsnikzc4pmb6ri6nri58jghacz7z-python3.8-gruut-lang-fr-1.2.2/lib/python3.8/site-packages/gruut_lang_fr-1.2.dist-info/direct_url.json
/nix/store/fih6hsnikzc4pmb6ri6nri58jghacz7z-python3.8-gruut-lang-fr-1.2.2/lib/python3.8/site-packages/gruut_lang_fr-1.2.dist-info/top_level.txt

The code for locating language files is here if you're interested: https://github.com/rhasspy/gruut/blob/6590a671fa4fea914f3bdd0a9e17e57e68309298/gruut/utils.py#L17

Locating the packages works, just the data files given in the setup.py are nowhere to be found.
https://github.com/rhasspy/gruut/blob/6590a671fa4fea914f3bdd0a9e17e57e68309298/gruut-lang-fr/setup.py#L26-L35
❯ cd gruut-lang-fr
❯ fd
LANGUAGE
gruut_lang_fr
gruut_lang_fr/VERSION
gruut_lang_fr/__init__.py
gruut_lang_fr/pos
gruut_lang_fr/pos/results.txt
setup.py

Sorry, I don't want to bother you too much.
The package_data referenced in the language packages are not part of the repository, but part of the pypi sdist. Can you add them to the repository?

Please keep bothering me :slightly_smiling_face:
The problem here is that the lexicon.db files for some languages are over the size limit that GitHub has for files stored in a repo. So I either need to use GitLFS or assume that the files can be pulled from a release.
Any suggestions?

With my package maintainer hat on: We would be fine with fetching via git-lfs, and it is probably easier to work with, than having to track/manage separate release artifacts.

I've pushed the files up to a side branch: https://github.com/rhasspy/gruut/tree/v1.2
Turns the biggest file was still under the limit!

Awesome, finally a non-packaging issue! :grinning:
======================================================================
FAIL: test_liason (tests.test_fr_phonemizer.FrenchPhonemizerTestCase)
Test addition of liason
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/build/source/tests/test_fr_phonemizer.py", line 51, in test_liason
    sentence = self._without_and_with_liason(
  File "/build/source/tests/test_fr_phonemizer.py", line 104, in _without_and_with_liason
    self.assertEqual(word_phonemes, with_phonemes)
AssertionError: Lists differ: ['l', 'e'] != ['l', 'e', 'z']

Second list contains 1 additional elements.
First extra element 2:
'z'

- ['l', 'e']
+ ['l', 'e', 'z']
?          +++++

----------------------------------------------------------------------
Ran 44 tests in 1.516s

FAILED (failures=1)

The PYTHONPATH needs to be augmented if you don't install the gruut_lang_* modules. See this example in run-tests.sh. The liason test is failing because it can't locate the French part of speech model.
I'm not a French speaker, but as I understand it, liason in French is the addition of "blending sounds" between certain words. The rules can get quite complex, so I've only implemented a few of them. But they depend on part of speech tagging, which is why they need the POS model.

The PYTHONPATH needs to be augmented if you don't install the gruut_lang_* modules. See this example in run-tests.sh. The liason test is failing because it can't locate the French part of speech model.

The PYTHONPATH looks pretty good, I introduced newlines after each : to improve readability.
/nix/store/8n4mdbzwklnmvf9z6ydr30jrnwc6mpcz-python3.8-gruut-1.2.2/lib/python3.8/site-packages:
/nix/store/6cfajs6lsy9b4wxp3jvyyl1g5x2pjmpr-python3-3.8.9/lib/python3.8/site-packages:
/nix/store/5yfmn9py4bjn5hqzrz6qpbbc3l07d697-python3.8-setuptools-54.2.0/lib/python3.8/site-packages:
/nix/store/vq4cs34z21hbri21pq5l5avc7qdw54rx-python3.8-wheel-0.36.2/lib/python3.8/site-packages:
/nix/store/xzgkgip067x7zxlrkfaj6nbasl7k4l46-python3.8-pip-21.0.1/lib/python3.8/site-packages:
/nix/store/m1xzfbib6f5ci9yrjny3p7xfdw21gzlp-python3.8-Babel-2.9.1/lib/python3.8/site-packages:
/nix/store/8q8qxp9gcmrdby0hnkybkbc6yyhxpiwg-python3.8-pytz-2021.1/lib/python3.8/site-packages:
/nix/store/fkjwqvsghaj2as0n46v773y3gj0nknxx-python3.8-gruut-ipa-0.9.2/lib/python3.8/site-packages:
/nix/store/p0jn1g0gkz8n5vd2a1vd3lia0xd3vclf-python3.8-jsonlines-1.2.0/lib/python3.8/site-packages:
/nix/store/hllx5h4hsrk6v0lcgnvibg7vfi5xkxmb-python3.8-six-1.16.0/lib/python3.8/site-packages:
/nix/store/azk4ycpllqz6yzwpwazlzldkdvh5bmij-python3.8-num2words-0.5.10/lib/python3.8/site-packages:
/nix/store/09nswns2axpcaqq8gm4slk0dv76sj21n-python3.8-docopt-0.6.2/lib/python3.8/site-packages:
/nix/store/lbzvq16abvw2sa0iyld9lxm79psmmrsp-python3.8-python-crfsuite-0.9.7/lib/python3.8/site-packages:
/nix/store/wi47akarfaq9isjsqnzk7j7h0r01ajhj-python3.8-gruut-lang-cs-1.2.2/lib/python3.8/site-packages:
/nix/store/9lb84rdf2dfdz42vll1a0vy9k8i18q3w-python3.8-gruut-lang-de-1.2.2/lib/python3.8/site-packages:
/nix/store/p0s2abvlrp6nrj1zwn6nrpkwx2a9b3g0-python3.8-gruut-lang-es-1.2.2/lib/python3.8/site-packages:
/nix/store/hkzjvzjngjn81qxjczlcf9rq2na7bp30-python3.8-gruut-lang-fr-1.2.2/lib/python3.8/site-packages:
/nix/store/q5dvx0jrvy4xkyzpp1azc4rb8za5yyzd-python3.8-gruut-lang-it-1.2.2/lib/python3.8/site-packages:
/nix/store/5458ncn0c9r9if0s7zxpm6b89znpk0sz-python3.8-gruut-lang-nl-1.2.2/lib/python3.8/site-packages:
/nix/store/bc66a637skc1j07wv3qbirhhc1xj0bw4-python3.8-gruut-lang-pt-1.2.2/lib/python3.8/site-packages:
/nix/store/5qikp9816449i7a7m1zhcxv8zcs5lcni-python3.8-gruut-lang-ru-1.2.2/lib/python3.8/site-packages:
/nix/store/9bzx7algn8scp45nflqbqa1xxwphc1ck-python3.8-gruut-lang-sv-1.2.2/lib/python3.8/site-packages:
/nix/store/i9514df0cphjbkrb86xmb2ps3a93mcm8-python3.8-gruut-lang-sw-1.2.2/lib/python3.8/site-packages

I'm not a French speaker, but as I understand it, liason in French is the addition of "blending sounds" between certain words. The rules can get quite complex, so I've only implemented a few of them. But they depend on part of speech tagging, which is why they need the POS model.

Pretty sure you are talking about a liaison, like how the article and the word following it connect which each other. See https://github.com/juliacarbajal/french_phonologizer/blob/master/phonologize.py#L87.

Lol, I missed the extra "i" in liaison this whole time :man_facepalming:

The POS model files are not part of the package_data! :smile:
https://github.com/rhasspy/gruut/blob/6e261552db2f18f7cdac9391618d3b0b29e19b99/gruut-lang-fr/setup.py#L28-L35

I might wear out the facepalm emoji here :man_facepalming:
OK, got the POS model added to the French model. I can at least confirm that tests pass for me with a clean pull of v1.2 after running the create-venv.sh script.
Thanks for your patience :+1:

Exciting. Things are falling into place. All that is left now is the kind ask for a git tag to build from, as opposed to a random commit. Would that be possible?
Happy to be your rubber duck! :duck:

Sure! I've moved the v1.2.0 tag to this most recent commit. Will that work?

Yep, that works! Thank you.


Images Shortcodes do not seem to influence image size examples.

On this page: https://themes.gohugo.io/theme/docdock/shortcodes/image/
This code is not making the image show as 80px in height.
![Hackathon](https://github.com/vjeantet/vjeantet.fr/raw/master/static/images/sgthon/C.jpg?height=80px)

Good catch, I'll fix that

I really like the system/theme. Thank you for putting it together.
On Jun 14, 2017 14:17, "Valere JEANTET"<EMAIL_ADDRESS>wrote:

Good catch, I'll fix that
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/vjeantet/hugo-theme-docdock/issues/17#issuecomment-308560941,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AF3qD8mZ0zQ3Xr0hyLgYX8iJ6nYxA5jeks5sEE3vgaJpZM4N6cxG
.

fixed


bring back caching

!backup caching was disabled due to the repo refacto, now that we're past that, it needs to be brought back.

investigating a promising way to resolve the multi-writer locking issue behind our cache.

This will be tackled later, there's a fair amount of caching already


微调后的模型推理问题

您好！我想使用自定义数据进行微调VisRAG-Ret 我使用的命令为 CUDA_VISIBLE_DEVICES=7 bash scripts/train_retriever/train.sh 2048 16 1 0.02 1 true false config/deepspeed.json 1e-5 false wmean causal 1 true 2 false /data/code/RAG/VisRAG-Ret  /data/code/RAG/parquet 其中parquet文件夹是我按照训练集格式收集的自定义数据 但是训练完后我发现推理时缺失tokenizer.py、modeling_minicpm.py类似的这种文件 我从VisRAG-Ret文件夹中复制了同名文件到训练权重中，这样可以推理但是推理的结果和原始模型完全一致 我感觉模型并没有更新 请问这是什么原因

@tuncha 您好，如果您想要使用自己训练的权重进行推理的话，只需要把VisRAG-Ret仓库中的权重替换为您自己训练得到的即可，请问您的步骤是这样的吗，如果是这样的话，照理来说并不会出现以上问题。

请问有没有遇到
RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the `forward` function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations.
[rank1]: Parameter at index 677 with name lm_q.vpm.blocks.25.mlp.fc2.bias has been marked as ready twice. This means that multiple autograd engine  hooks have fired for this particular parameter during this iteration.

这个报错，如何解决？谢谢

您好！我想使用自定义数据进行微调VisRAG-Ret 我使用的命令为 CUDA_VISIBLE_DEVICES=7 bash scripts/train_retriever/train.sh 2048 16 1 0.02 1 true false config/deepspeed.json 1e-5 false wmean causal 1 true 2 false /data/code/RAG/VisRAG-Ret /data/code/RAG/parquet 其中parquet文件夹是我按照训练集格式收集的自定义数据 但是训练完后我发现推理时缺失tokenizer.py、modeling_minicpm.py类似的这种文件 我从VisRAG-Ret文件夹中复制了同名文件到训练权重中，这样可以推理但是推理的结果和原始模型完全一致 我感觉模型并没有更新 请问这是什么原因

请问您有没有出现以下报错，
RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the forward function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple checkpoint functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations.
[rank1]: Parameter at index 677 with name lm_q.vpm.blocks.25.mlp.fc2.bias has been marked as ready twice. This means that multiple autograd engine  hooks have fired for this particular parameter during this iteration.
想问一下，如何解决？谢谢