[BUG] Policies without assignments, cause exit code 1

Describe the bug
Device Configuration Policies without assignments, cause the Pipeline "Find change author & commit the backup" to fail.
To Reproduce
Create a device configuration policy with a few settings and don't assign it.
Expected behavior
Backup to finish

Mode: 1
Client Pipeline
Version 2.0.7

To be honest, I'm not sure anymore if it only impacts Config Profiles without assignments, now also receiving it for another iOS profile that has an assignment.
One step prior it has been backed up

Hi - what is your pipeline looking like?

I.e. are you doing anything outside of just a normal backup?

Hi,
I used the pipeline from here: https://github.com/ztrhgf/DevOps_Pipelines/blob/main/intune-backup-pipeline.yml and pretty much followed a tutorial from this guy:
https://doitpsway.com/how-to-easily-backup-your-intune-environment-using-intunecd-and-azure-devops-pipeline
I renamed the Policy and removed the brackets []. It seems to go further now, but now I end up here:

Sorry, probably some newbie mistakes I'm doing. Is there any "official" pipeline which I can use and amend?

I think that is something for Ondrej who created the powershell script to look at in that case 😄  to run IntuneCD without any 3rd party script only run the backup/document steps. I believe @aaronparker has some examples

In its simplest form you could just do something like this: https://github.com/almenscorner/IntuneCD/wiki/Backup#dev-env-variables


Evalulate enchants

Hello, if Trade macro you can select the values you like to evaluate.
I tried to evaluate some boots that had a LAB enchant but  I didn't get the enchantmanet evaluted.

0.6.4 (2020-03-15)

fix enchant stats not working (#320)

I think this is still occurring in 0.6.4

0.6.5 (2020-03-16)

fix singular/ plural stats (#320)


Save the image’s alt-text to its EXIF data

As the title says, if the images in a tweet contain alt-text, this change will try to record it as part of the image’s EXIF data (using https://github.com/hMatoba/piexifjs ). It only works for JPG images, but support for writing PNG comments or other formats can be added fairly easily.
Additionally, this makes a slight improvement to Tweetdeck support. Sometimes, when Tweetdeck displays images of a tweet (e.g. in Notifications), the images in the DOM don’t have alt-text, but clicking on the tweet to expand it re-renders it with alt-text. This change calls update_stored_tweet_info as part of the click handler, to make sure the DOM gets rescanned with the latest info. It would be better to change this part to fetch the tweet info via the API, like the main_react script does, but that’s a larger change…

This changes image's hash and may break lots of people's overflow (including mine). I'd say at best it should be optional and opt-in.


Theme submission - anatole

Theme submission
Please make sure that you have read the theme submission guidelines before submitting a theme. The guidelines provide all relevant information and requirements that have to be fulfilled before the submission. Please tick the relevant boxes for your theme in the checklist below:
https://github.com/lxndrblz/anatole
I made sure that...

[x] the repository contains a good README.md describing my theme
[x] an open source license has been added to LICENSE.md
[x] all metadata have been added to theme.toml
[x] screenshots have been added in the images/ folder with the required dimensions
[x] in case I'm using a customized demo via the exampleSite folder that

[x] https://example.com is set as base url in exampleSite/config.{toml, yaml, json} to avoid the abuse of unused domains
[x] I tested that my theme's demo works with the content directory of gohugoio/HugoBasicExample

[x] I tested my theme against the gohugoio/HugoBasicExample

[x] I've checked the developer tools' console in my browser for error messages

[x] in case my theme is using Hugo Pipes features like toCSS and PostCSS that I have committed the /resources directory with all generated assets, for my theme to work in the basic version of Hugo

N.B. By submitting a theme to the Hugo Themes Showcase you understand that you need to maintain your theme. You will need to keep an eye on whether your theme's demo generates on the Hugo website, whenever there is a new release. We are a very small team and we just cannot keep up opening issues in the various theme repositories to notify authors. Whenever a theme demo breaks and remains broken then at some point it will be removed from the list without prior warning. If you no longer wish to maintain a theme please let us know.
New themes will usually be promoted on Hugo's official Twitter account. If you would like to be mentioned in the tweet please add your Twitter username to this submission.
Link to my Twitter account (optional): REPLACE-WITH-URL
Feel free to ask questions. We're glad to help.

Hello @lxndrblz
Thank you for the submission. I have added the theme to the repo in e38c688

@digitalcraftsman
This is a responsive blog theme that features two columns at bigger displays.

Thank you for your theme submission, @lxndrblz. Your theme is already live on Hugo's theme site and it will be featured soon on Hugo's official Twitter account.


error at def expBaseAE(adata, exp_metrics):

Hi dear team, thanks for the package. I get an error could you mind give some tips.
def expBaseAE(adata, exp_metrics):
    n_cells, n_genes = adata.X.shape
    in_dim = n_genes
    z_dim = args.z_dim
    h_dim = args.h_dim

    model = get_baseline_AE(in_dim, z_dim, h_dim).to(device)
    model = main_AE(args, model, save_name=f"baseAE_{args.model_name}")
    model.eval()
    with torch.no_grad():
        x = model.encoder(tensor_x)
        s = model.encoder(tensor_s)
        u = model.encoder(tensor_u)

        v = estimate_ld_velocity(s, u, device=device).cpu().numpy()
        x = x.cpu().numpy()
        s = s.cpu().numpy()
        u = u.cpu().numpy()

    adata = new_adata(adata, x, s, u, v, g_basis=args.nb_g_src)
    scv.tl.velocity_graph(adata, vkey='new_velocity')

    scv.pl.velocity_embedding_stream(adata, vkey="new_velocity", basis='X_umap', color='leiden',
                                    title="Baseline AutoEncoder",
                                    )
    scv.tl.velocity_confidence(adata, vkey='new_velocity')
    exp_metrics['Baseline AutoEncoder'] = evaluate(adata, cluster_edges, 'leiden', "new_velocity")

expBaseAE(adata, exp_metrics)

Train Epoch: 100/20000 	Loss: 58.533680
Train Epoch: 200/20000 	Loss: 58.349663
Train Epoch: 300/20000 	Loss: 58.188026
Train Epoch: 400/20000 	Loss: 58.048077
Train Epoch: 500/20000 	Loss: 57.929665
.......
Train Epoch: 11400/20000 	Loss: 20.946295
Train Epoch: 11500/20000 	Loss: 20.501766
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Input In [10], in <module>
     26     scv.tl.velocity_confidence(adata, vkey='new_velocity')
     27     exp_metrics['Baseline AutoEncoder'] = evaluate(adata, cluster_edges, 'leiden', "new_velocity")
---> 29 expBaseAE(adata, exp_metrics)

Input In [10], in expBaseAE(adata, exp_metrics)
      5 h_dim = args.h_dim
      7 model = get_baseline_AE(in_dim, z_dim, h_dim).to(device)
----> 8 model = main_AE(args, model, save_name=f"baseAE_{args.model_name}")
      9 model.eval()
     10 with torch.no_grad():

Input In [6], in main_AE(args, model, lr, weight_decay, save_name)
      9 while i < args.n_epochs:
     10     i += 1
---> 11     loss = train_step_AE([tensor_s, tensor_u], model, optimizer, xyids=[0, 1], device=device)
     12     losses.append(loss)
     13     if i % args.log_interval == 0:

File ~/miniconda3/envs/velo/lib/python3.8/site-packages/veloproj/util.py:370, in train_step_AE(Xs, model, optimizer, xyids, device, aux_weight, rt_all_loss, perc, norm_lr)
    367     lr_loss = vloss.item()
    368     loss += vloss
--> 370 loss.backward()
    371 optimizer.step()
    372 if rt_all_loss:

File ~/miniconda3/envs/velo/lib/python3.8/site-packages/torch/_tensor.py:307, in Tensor.backward(self, gradient, retain_graph, create_graph, inputs)
    298 if has_torch_function_unary(self):
    299     return handle_torch_function(
    300         Tensor.backward,
    301         (self,),
   (...)
    305         create_graph=create_graph,
    306         inputs=inputs)
--> 307 torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)

File ~/miniconda3/envs/velo/lib/python3.8/site-packages/torch/autograd/__init__.py:154, in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    151 if retain_graph is None:
    152     retain_graph = create_graph
--> 154 Variable._execution_engine.run_backward(
    155     tensors, grad_tensors_, retain_graph, create_graph, inputs,
    156     allow_unreachable=True, accumulate_grad=True)

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [100]], which is output 0 of IndexPutBackward0, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Thank you!
It seems the vanila AE is projecting the input matrices into spurious representations that resulted in "NaN" values when fitting the low-dimensional linear regression.
The code in current line 427 of the model.py filters the nan values by assigning 0 values:
offset[torch.isnan(offset)], gamma[torch.isnan(gamma)] = 0, 0
I guess the error is raised by this line of code.
The good news is that a safer nan filtering function (i.e., torch.nan_to_num) has been provided by pytorch since version >= 1.8, updated in veloAE with the hope to tackle the issue.
Hope it helps!


Update pip window when largest video changes

After PiP window has been created, monitor every second if largest playing video changed so that video conferencing web apps always show the most appropriate video.
@mounirlamouri PTAL

Could we use the ResizeObserver API when available in order to avoid polling?

This API is only useful for existing videos. We would still need to poll for new videos sadly

The way I see it, there is only one largest video, if we assume that the main video has to be resized in order for it to not be the main video, wouldn't that resize be caught by the ResizeObserver and at that point, we could run the algorithm to check which video is now the main video?

Your observation is correct. https://github.com/GoogleChromeLabs/picture-in-picture-chrome-extension/pull/42/commits/d250e99a3693f360642e93d2edcecd706570c5b0 addresses it. Let me know what you think.


runtime: excessive memory use between 1.21.0 -> 1.21.1

What version of Go are you using (go version)?

$ go version
go version go1.21.4 linux/amd64

Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (go env)?
go env Output
$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/builder/.cache/go-build'
GOENV='/home/builder/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/builder/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/builder/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.4'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='0'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3335037910=/tmp/go-build -gno-record-gcc-switches'

What did you do?
Our production service was recently shut down by the OOM, which led us to inspect its memory usage in details.
We discovered that, when running, our process had a memory consumption (RSS) that grew over time and never shrank.
sample runtime.MemStats report :
	"Alloc": 830731560,
	"TotalAlloc":<PHONE_NUMBER>56,
	"Sys":<PHONE_NUMBER>,
	"Lookups": 0,
	"Mallocs":<PHONE_NUMBER>,
	"Frees":<PHONE_NUMBER>,
	"HeapAlloc": 830731560,
	"HeapSys":<PHONE_NUMBER>,
	"HeapIdle":<PHONE_NUMBER>,
	"HeapInuse": 920240128,
	"HeapReleased":<PHONE_NUMBER>,
	"HeapObjects": 4088086,
	"StackInuse": 15204352,
	"StackSys": 15204352,
	"MSpanInuse": 8563968,
	"MSpanSys": 17338944,
	"MCacheInuse": 4800,
	"MCacheSys": 15600,
	"BuckHashSys": 5794138,
	"GCSys": 146092920,
	"OtherSys": 3020358,
	"NextGC":<PHONE_NUMBER>,
	"LastGC":<PHONE_NUMBER>506142728,
	"PauseTotalNs": 108783964,

at that time, the reported RSS for our process was 3,19Gb.
We looked at our history in more details, and observed that we had a big change in production behavior when we upgraded our go version from 1.19.5 to 1.20.0 - we unfortunately didn't notice the issue at that time, because we upgrade (and restart) our service on a regular basis.
To confirm this theory, we have downgraded our go version back to 1.19.13, and our memory consumption is now small and stable again.
Here is a graph of the RSS of our service over the last 48h, the drop corresponds to our new deployment with go 1.19.13  :

It should be noted that our production kernel is a hardened kernel based on grsecurity 5.15.28, which may be related to this issue (randomized heap addresses ?)
What did you expect to see?
A constant and stable memory usage.
What did you see instead?
The go runtime seems to not release memory back to the system.

Unfortunately, we have only been able to observe this issue on our production system, in production conditions.
We were not yet able to reproduce the issue on other systems, or by running isolated features in test programs deployed on our production infrastructure.

perhaps try looking at profiling

@seankhliao : we did, the OOM killer incident was a combination of high memory usage and a bug on our side, which we fixed.
After that fix, memory profiles would report only small amounts for 'inuse_size', in the 300-400 Mb range when our process's RSS would be several GB, and more importantly: the RSS of our process would grow over time, but the profiles would report no extra used memory.
We could definitely optimize allocations at large, but the fact is: compiling the exact same code in 1.19 produces a binary which doesn't leak.

@seankhliao  (sidenote: I work with @LeGEC):  we did extensive memory profiling of our processes during several weeks (and can provide full data privately if necessary) but from the point of view of the memory profiler nothing was leaking and nothing would explain the constant grow in memory usage, mostly unreleased to the OS.
This is definitely a runtime issue as running the exact same code with go 1.19.13 leads to a drastically better behavior... (see graph in first post)
We are also willing to test any intermediate compiler version or run any profiling command that would be useful to track down this issue.

I remember some things changed regarding huge pages on linux and were backported to 1.21, that may be related.
https://github.com/golang/go/issues/61718
Worth it to give at a read at the wiki for THP and GOGC https://go.dev/doc/gc-guide#Linux_transparent_huge_pages

cc @mknyszek

@mauri870  on our production system:
$ cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never

If huge page are indeed involved: is there a way to disable transparent_hugepage for a single process / without root access ? (Alternatively, is there a way to patch the go compiler to prevent transparent_hugepage usage ?)

Memstats of our process in go1.19:
	"Alloc": 710580968,
	"TotalAlloc":<PHONE_NUMBER>88,
	"Sys":<PHONE_NUMBER>,
	"Lookups": 0,
	"Mallocs":<PHONE_NUMBER>,
	"Frees":<PHONE_NUMBER>,
	"HeapAlloc": 710580968,
	"HeapSys":<PHONE_NUMBER>,
	"HeapIdle": 517931008,
	"HeapInuse": 842268672,
	"HeapReleased": 372064256,
	"HeapObjects": 3670670,
	"StackInuse": 15532032,
	"StackSys": 15532032,
	"MSpanInuse": 7786368,
	"MSpanSys": 11976192,
	"MCacheInuse": 4800,
	"MCacheSys": 15600,
	"BuckHashSys": 4959765,
	"GCSys": 64211392,
	"OtherSys": 2561923,
	"NextGC": 992932464,
	"LastGC":<PHONE_NUMBER>049685014,
	"PauseTotalNs": 101655886,

@LeGEC RSS mentioned initially (at the top)  is not correlating to the mem stat report.
Sys: 8GB
But the RSS mentioned was just 3GB

The huge page related changes can't be the culprit, because huge pages were only forced by the runtime between Go 1.21.0 and Go 1.21.3 (inclusive). Go 1.21.4 no longer adjusts huge page settings at all. I don't see any of those versions in the conversation above.

How are you measuring RSS?
Also, as a side note, "Sys" is virtual memory footprint. It's not going to match up with RSS. "Sys - HeapReleased" will be much closer.
Also, is it possible this is related to a regression in a specific standard library package for instance? IIRC we didn't see anything like this back when Go 1.20 was released, so it's a bit surprising. It's still possible it's related to the compiler/runtime, so I don't mean to rule it out.
Is it possible to reproduce this on a smaller scale? The difference seems fairly big and "obvious" so if you can bisect down to the commit between Go 1.19 and Go 1.20 that causes it, that would immediately tell us what happened.

@mknyszek yes we also thought possible that it could be related to changes in some standard library packages.
However unless those have specific hooks into the runtime or make use of unsafe, those allocations should normally have been visible in the memory profiling, no ?
I can share - privately - a heap dump obtained using /pprof/heap endpoint when the relevant service was on go 1.21.4 if it can help.
Bisecting beetween 1.19 and 1.20 is a possibility, we can do that, but we can only try 1 version / day in order to reproduce since the issue looks dependent on user / service activity... Can you suggest a list of commits to try ?

@vanackere There were a lot of commits that went into Go 1.20, it would take quite a while to enumerate them and try to guess what went wrong. 😅
The commit range is e99f53fed98b0378c147588789b8c56b0305469b..9088c691dac424540f562d6271c5ee479e9f9d80 which contains 2018 commits.
$ git log --oneline e99f53fed98b0378c147588789b8c56b0305469b..9088c691dac424540f562d6271c5ee479e9f9d80 | wc -l
2018

log2(2018) ~= 11 so it would take around 11 attempts to identify the culprit. I realize that's a lot of days to by trying this out in production. Is there really no other way to create a smaller reproducer?
Although... it occurs to me that Go 1.20 had some significant crypto regressions that could make TLS (and/or other similar operations like validating JWT tokens) slower. I could imagine that perhaps that could cause requests to pile up in your service? I would've expected them to have been mitigated somewhat in Go 1.21, but perhaps that's worth looking into?
See #64332 and #59442 maybe?

@mknyszek if u lay out the steps i can pitch in. Not new to git bisect!!

one update: we compiled and deployed our server using go 1.20.0, and the leaky behavior isn't observed.
We're pondering between "testing" the 1.20.11 (the latest 1.20)  and 1.21.0 .
Would you have some insight about these two versions ?
for example: were there some fixes or updates in memory handling in one of the 1.20 minor versions ?

update: we tried go 1.20.11 today, and still have a reasonable (for us ...) RSS value
behavior change happens between 1.20.11 and 1.21.4. updating the title accordingly

@LeGEC can u also try gotip. Try goup or if it's a docker image : u can try tag tip or gotip.. i believe

@Nasfame Unfortunately I don't think either of us can bisect because we can't run the reproducer.
@LeGEC Thanks for the updates. I think the only thing I'll say is it seems like the crypto-related regressions are unrelated. At this point bisection and/or a reproducer seems like it would be the most fruitful path forward. As before, it may be worthwhile to look at a CPU profile before and after, which might reveal other unexpected changes in application behavior that lead to the culprit.

Unfortunately neither of us can bisect because we can't run the reproducer.

@mknyszek : yes, we're trying to run the parts of our services in isolation, unfortunately none of the would be culprits seem to be guilty alone (or, more probably, we haven't identified the right culprit).
Today we rebuilt our server with gotip (golang master version 0c7e5d3) and deployed, and the same buggy memory behavior clearly shows up.

@mknyszek :
I will repeat one point: we suspect that the behavior we observe on our production server may be due to an interaction between the golang runtime and the kernel on our production server. The system on our servers is managed by our host provider, and they run a kernel based on grsecurity 5.15.28.
There definitely are some features activated around memory safety, we are trying to get more details on which ones are.
One extra information, if it is of any help:
when trying to run a binary compiled with the race detector, it panicked with an error (unfold details for complete stack trace) :

fatal error: too many address space collisions for -race mode

runtime.(*mheap).alloc.func1()
    /usr/local/go/src/runtime/mheap.go:968 +0x5c fp=0x73783ad4f7f8 sp=0x73783ad4f7b0 pc=0x4637fc
runtime.(*mheap).alloc(0x2780160?, 0x274e720?, 0x19?)
    /usr/local/go/src/runtime/mheap.go:962 +0x5b fp=0x73783ad4f840 sp=0x73783ad4f7f8 pc=0x46375b
runtime.(*mcentral).grow(0x73783ad4f8c0?)
    /usr/local/go/src/runtime/mcentral.go:246 +0x52 fp=0x73783ad4f880 sp=0x73783ad4f840 pc=0x451e52
runtime.(*mcentral).cacheSpan(0x277aea8)
    /usr/local/go/src/runtime/mcentral.go:166 +0x306 fp=0x73783ad4f8d8 sp=0x73783ad4f880 pc=0x451cc6
runtime.(*mcache).refill(0x689f9e4c0108, 0x0?)
    /usr/local/go/src/runtime/mcache.go:182 +0x153 fp=0x73783ad4f918 sp=0x73783ad4f8d8 pc=0x451413
runtime.(*mcache).nextFree(0x689f9e4c0108, 0x1c)
    /usr/local/go/src/runtime/malloc.go:925 +0x85 fp=0x73783ad4f960 sp=0x73783ad4f918 pc=0x447745
runtime.mallocgc(0xc0, 0x1656b60, 0x1)
    /usr/local/go/src/runtime/malloc.go:1112 +0x448 fp=0x73783ad4f9e0 sp=0x73783ad4f960 pc=0x447d08
runtime.newobject(0x73783ad4f9f8?)
    /usr/local/go/src/runtime/malloc.go:1324 +0x25 fp=0x73783ad4fa08 sp=0x73783ad4f9e0 pc=0x448305
internal/cpu.doinit()
    /usr/local/go/src/internal/cpu/cpu_x86.go:51 +0x1e fp=0x73783ad4fa68 sp=0x73783ad4fa08 pc=0x43b1be
internal/cpu.Initialize({0x0, 0x0})
    /usr/local/go/src/internal/cpu/cpu.go:125 +0x1d fp=0x73783ad4fa88 sp=0x73783ad4fa68 pc=0x43ac1d
runtime.cpuinit({0x0?, 0x800000000?})
    /usr/local/go/src/runtime/proc.go:639 +0x1f fp=0x73783ad4faa8 sp=0x73783ad4fa88 pc=0x476b1f
runtime.schedinit()
    /usr/local/go/src/runtime/proc.go:729 +0xbd fp=0x73783ad4faf0 sp=0x73783ad4faa8 pc=0x476d7d
runtime.rt0_go()
    /usr/local/go/src/runtime/asm_amd64.s:349 +0x11c fp=0x73783ad4faf8 sp=0x73783ad4faf0 pc=0x4a8dfc

Unfortunately neither of us can bisect because we can't run the reproducer.

yes, we're trying to run the parts of our services in isolation, unfortunately none of the would be culprits seem to be guilty alone (or, more probably, we haven't identified the right culprit).

Just to be clear I didn't mean that in a bad way -- I definitely understand the difficulty of creating reproducers. 😅 And unless the reproducing code is open source it's difficult and/or impossible to share too many details. I was just clarifying for @Nasfame. Thanks for your continued communication here!

One extra information, if it is of any help:
when trying to run a binary compiled with the race detector, it panicked with an error (unfold details for complete stack trace) :

This is something I would expect with a kernel forcing address space randomization. The race detector (TSAN) requires all heap memory to exist in a very specific range of addresses. This is unfortunately not easy to change. IIUC it's fairly fundamental to the technique.

Therefore, the issue has to lie in something very specific to @LeGEC's application, either in how their application uses Go or in how Go is interacting with their environment (even if the issue is ultimately in the Go runtime).

True, but my intention is to see if we they can reproduce the issue without enabling these features.

blind shot: did something land on go1.21 which changed the way a go process returns memory to the OS ?

@LeGEC Yes; part of the huge page work was making the runtime a little less eager to return memory to the OS, but it hasn't meaningfully affected actual memory use so far. This is by design; the same memory that would get released is only released one GC cycle later than it otherwise would. (Even more details: the heap is divided into hugepage-ish-sized chunks and the runtime avoids releasing small amounts of memory in these chunks if they're going to get dense in the next GC cycle anyway. Otherwise it's a total waste of a syscall.)
I think it's unlikely that this is the source of your issues, but perhaps one detail is relevant. The metadata the runtime uses for managing these hugepage-ish-sized chunks has increased 64x. This is up from 1 bit per 4 MiB of heap to 8 bytes per 4 MiB of heap, or 1 MiB for a 2 TiB heap. But this calculation is assuming the heap is contiguous. The metadata is sized proportionally to [max heap address - min heap address]. With ASLR turned on, that range can clearly get much wider.
This sounds pretty damning, but I didn't mention it before because that's just mapped memory. In theory, even if your heap is 2 TiB in size, the OS will only actually back the parts of this data structure that are accessed.
... Unfortunately I am only now realizing that the runtime does iterate over this whole data structure in the background (yielding its time if it takes a while, so it won't noticeably impact throughput or latency). 🤦  That will effectively force the OS to back the entire memory region.
Would it be difficult to graph runtime.MemStats.GCSys before and after? If that's the source of the difference, I think that fully explains the problem. I did have a hunch this could be a problem earlier, but your MemStats dumps didn't implicate GCSys at all, not on the order of GiB anyway. But maybe that was just taken at an inopportune time?

One good-ish news with the experiments we ran today, we narrowed the scope for our issue to go1.21.0..go.1.21.1
It turns out the memory issue does not show up with go1.21.0, but does appear with go1.21.1.

@mknyszek : thank you very much for this insight.

That is good news! It narrows it down to just 36 commits https://github.com/golang/go/compare/go1.21.0...golang:go:go1.21.1?expand=1

One good-ish news with the experiments we ran today, we narrowed the scope for our issue to go1.21.0..go.1.21.1

Huh. That does seem to run counter to my theory in https://github.com/golang/go/issues/64332#issuecomment-1830174854.
Also, in a 48-bit address space on linux/amd64 the worst case size of that metadata is only 128 MiB. That can't ever really explain several GiB of memory use, unless you're using some of the fancy 52-bit addressable hardware and have it enabled in your kernel.
I see you're editing your previous comment, so I'll hold additional commentary until you're done. :)

When doing git diff, these CLs could be the ones
https://go-review.googlesource.com/c/go/+/522317/3/src/runtime/map.go
HugePage related CLs (discussed already): Final CL: https://go-review.googlesource.com/c/go/+/523655

In the commit range go1.21.0..go1.21.1,  we went for the obvious suspect 06df3292a8 (the one which brings the most changes to memory management to src/runtime/):
[release-branch.go1.21] runtime: avoid MADV_HUGEPAGE for heap memory

and we can confirm that, in our setup, this commit alone triggers our memory issue.
(I think Nasfame mentioned it as the second CL ?)

@mknyszek : is there a way to entirely disable the hugepage specificities in the runtime on linux ?
I'm trying to read the code in src/runtime/ : at the moment (on master), hugepage management gets enabled when the garbage collector computes that its next heap collection should be triggered with a size exceeding 1Gb ?
am I understanding correctly the code in mgc.go ?

also : I have very little knowledge on how linux allocates memory
couldn't this set of lines :
https://github.com/golang/go/blob/master/src/runtime/malloc.go#L758-L766
			l2 = (*[1 << arenaL2Bits]*heapArena)(sysAllocOS(unsafe.Sizeof(*l2)))
			if l2 == nil {
				throw("out of memory allocating heap arena map")
			}
			if h.arenasHugePages {
				sysHugePage(unsafe.Pointer(l2), unsafe.Sizeof(*l2))
			} else {
				sysNoHugePage(unsafe.Pointer(l2), unsafe.Sizeof(*l2))
			}

trigger weird things when address randomization is enabled ?

What I'm confused by in all of this is that nearly all calls to sysHugePage and sysNoHugePage have been removed in Go 1.21.4 (and on tip-of-tree). All calls to madvise(v, n, _MADV_COLLAPSE), which was added in 06df3292a8, were also removed in Go 1.21.4. Is this all still reproducible in Go 1.21.4?

couldn't this set of lines : ... trigger weird things when address randomization is enabled ?

Possibly, but those lines are also there in Go 1.21.0. So that doesn't explain the difference. That's also another one of those cases where the worst case can't possibly account for GiB of memory use. That map is worst-case 2^48 B / 64 MiB * 8 bytes = 32 MiB on linux/amd64.
I suppose it's possible these lines are involved as well but this is another case where I fail to see how that could account for all the memory use. (Also this memory is also all attributed to GCSys. In the worst case huge pages are doubling this memory overhead, but in practice I doubt that's true. And even then I still don't think it's enough to account for everything?)

is there a way to entirely disable the hugepage specificities in the runtime on linux ?

Yeah, if you want to rule out (or confirm) anything hugepage related, you can do a few things. On the machine you can run echo never >/sys/kernel/mm/transparent_hugepage/enabled.
Another option is disabling hugepages at the process level with something like prctl(PR_SET_THP_DISABLE, ...). (You can use the golang.org/x/sys/unix package for this. There's a Prctl function and PR_SET_THP_DISABLE is also available as a constant.)

@mknyszek : yes you are right, the lines I mentioned do not explain the difference in behavior.
looking at the differences introduced in 06df329a8 which are still present in go1.21.4 or master, there are modifications in mgcscavenge.go L774-L784 :
@@ -771,7 +771,7 @@ func (p *pageAlloc) scavengeOne(ci chunkIdx, searchIdx uint, max uintptr) uintpt
-			shouldNoHugePage := p.scav.index.setNoHugePage(ci)
+			p.scav.index.setNoHugePage(ci)

			// With that done, it's safe to unlock.
			unlock(p.mheapLock)
			if !p.test {
				pageTraceScav(getg().m.p.ptr(), 0, addr, uintptr(npages))

				// Only perform sys* operations if we're not in a test.
				// It's dangerous to do so otherwise.
-				if shouldNoHugePage {
-					sysNoHugePage(unsafe.Pointer(chunkBase(ci)), pallocChunkBytes)
-				}

on master, the call to p.scav.index.setNoHugePage(ci) got removed altogether.
If I understand correctly: in go1.21.0 and before, there was an opportunity for a memory range that was flagged as "hugePage" when allocated to be unmarked when freed, and this opportunity is not present anymore in go1.21.1 and later.
Did I get that part right ?
This may be combined with an issue in the way our customized version of the kernel reclaims (or not) memory from a process.

we are currently testing a program compiled with the partial revert : https://github.com/trustelem/go/commit/840fc3c97ac3b7d086d941f1a59e95dedc92949d
we will keep you posted with our observations

we can confidently say that, on our production setup, the above patch is enough to fix our issue

we are still missing a targeted reproducer, though
we are guessing the bug can be triggered by following a sequence of allocations
@mknyszek : given the patch above, would you have an idea of a kind of sequence that would be affected by that patch ?

Oof.
So, that partial revert is going to, in the fullness of time, mark quite a lot of memory as MADV_NOHUGEPAGE. Removing the call to sysNoHugePage is very intentional. I don't think the Go runtime should be in the business of trying to mark memory as MADV_HUGEPAGE or MADV_NOHUGEPAGE. Part of the problem is the Linux API. There's no way to unmark MADV_NOHUGEPAGE, only to mark as MADV_HUGEPAGE. And the latter is problematic. (That is, there are 3 states memory could be in. "None", "NOHUGEPAGE", and "HUGEPAGE". System calls exist to go from "None" to "NOHUGEPAGE" and "HUGEPAGE", and between "NOHUGEPAGE" and "HUGEPAGE". But not back to "None".)
Versions of Go prior to Go 1.21.0 would somewhat unpredictably mark memory as NOHUGEPAGE and then back to HUGEPAGE. I would also rather not return to this state. See #64342.
My recommendation is to try and adjust your system's configuration according to https://go.dev/doc/gc-guide#Linux_transparent_huge_pages, or to just disable huge pages altogether. Specifically, I would suggest trying to set /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none to 0. The fact that MADV_NOHUGEPAGE matters so much for your application suggests to me that khugepaged is being really overzealous with backing memory with huge pages. Versions of Go before Go 1.21.4 tried to work around this, but I do not think this is worth doing (again, because of the issues with MADV_HUGEPAGE and the inability to properly undo MADV_NOHUGEPAGE).
I understand this does not sound ideal, but I'm reluctant to do anything else. I went down a long and painful road to try and figure this out and make it all work, but it just doesn't. The whole design of huge pages on Linux is focused around application owners controlling their environment, not around language runtimes making decisions for them. (There are some claims that MADV_COLLAPSE for instance is meant for memory allocators, which is why I tried that in the culprit you found, but I've found in practice it does not work well for that use-case in its current iteration.)

we are still missing a targeted reproducer, though

Given what I think I'm seeing here (an overzealous khugepaged), it's going to be really hard to generate a reproducer. I've tried in the past to do so and failed many times, even though the behavior was reproducible in real applications. There's clearly something I was missing, and I would commend you for trying again, but I wouldn't sink too much time into it.

@mknyszek : this is really unfortunate since we don't have any easy way to disable huge pages on our production systems (and it would also impact other services, not only our go programs)...
I understand NOHUGEPAGE on linux is a can of worms... So is possible couldn't we consider adding a setting to the go runtime - eg controlled by GODEBUG or any other specific environment variable - to explicitely disable huge page usage by the go runtime altogether ? This would at least give an easy workaround for people like us, and might also be useful in the future to diagnose similar issues.

I think a compatibility GODEBUG is reasonable (and I'd go back and update the release notes to mention it as well).
Just in the interest of exploring all options, is the following a reasonable workaround for you to disable huge pages?
import "golang.org/x/sys/unix"

...
unix.Prctl(unix.PR_SET_THP_DISABLE, ...)
...

If not, why not? (Perhaps I'm missing something about Linux capabilities -- I'm definitely not an expert in this realm.)

@mknyszek : we have to actually try and confirm that the workaround actually works (we'll try tomorrow)
However, from my point of view, even if this works it would far from ideal since it would be much more complex to manage on the long term :

we would have to ensure all processes compiled for this particular system have this workaround or implement a generic compatibility layer wired in all our processes that would basically emulate what the GODEBUG env var would go (but more error prone)
moreover this would also not work for other precompiled go programs not built by us that might exihibit the bad behaviour whereas a global env var would neatly solve this...

@LeGEC @vanackere Just also wanted to say thanks for the thorough investigation and for bearing with me on this.

@mknyszek:  thank you for the patient support :smile:

Just another update: our production has been running flawlessly for 3 days with huge page disabled. Memory usage is so much better and predictable that I even think hugepage allocations by the go runtime should be made opt-in instead of opt-out so that only people interested in trading (much) higher memory usage for better throughput actually enable this.
Of course I realize everybody has different goal, and this is just my opinion, but I bet most people would prefer a slightly slower server over one easily killed because of OOM. (By the way since the go GC is not a moving one, how are huge pages behaving in case of memory fragmentation ? In other terms: can a set of huge pages stay pinned indefinitely because of a small set of live object ?)
Another suggestion / idea : perhaps we could make minHeapForMetadataHugePages settable by an environment variable (instead of the GODEBUG env var to disable hugepages), and perhaps also with an initial value meaning that the feature is disabled ?
In any case, as long as we have an option, I think we're unlikely to want enabling hugepage again on our production systems.

Memory usage is so much better and predictable that I even think hugepage allocations by the go runtime should be made opt-in instead of opt-out so that only people interested in trading (much) higher memory usage for better throughput actually enable this.

To be absolutely clear, huge pages are no longer controlled by the Go runtime at all. There's no distinction of "opt-in" vs. "opt-out" to be made here. Except for two very specific niche cases, the Go runtime, as of Go 1.21.4, does not touch any memory's huge page state.
Changing the transparent huge pages opt-in vs. opt-out is a request that should go to Linux or the Linux distro you're using. We filed such a bug to discuss changing a default a long time ago, and there has been no movement over the years. Some Linux distros took it into their own hands and disabled huge pages by default unless the program explicitly requests it. (I believe some popular Debian images disable it.) Go no longer requests it.
At a higher level, I do not feel comfortable making this decision for application owners. I also do not feel particularly good about a Go runtime option about disabling huge pages at all; after all, there are already Linux configuration options for that, and ways to disable it per-process.
However, the Go runtime used to touch huge page state, so I recognize the need for compatibility across Go versions and that is why I'm on-board with the GODEBUG setting. For better or for worse, I took Go out of this entire space; it is far too messy and confusing to have the OS and the runtime make two different choices. As of Go 1.21.4, the choice lies entirely with Linux. The runtime now just tries its best to be friendly to huge pages in its policies (while avoiding detriment to the non-huge-page cases), but that's the extent of it.

(By the way since the go GC is not a moving one, how are huge pages behaving in case of memory fragmentation ? In other terms: can a set of huge pages stay pinned indefinitely because of a small set of live object ?)

Yes, fragmentation can definitely lead to poor utilization of huge pages. That being said, this is not usually a problem for most Go programs. The GC frees memory in batches, so most Go heaps tend to get very dense by the end of the GC cycle and all page-level fragmentation gets filled.
If your application makes a lot of large irregularly-sized memory allocations (on the order of MiB) and not many small-sized allocations, then that increases the chance that there are persistent holes in the heap. There exists a background system goroutine that finds these holes and returns them to the OS, but the OS can sometimes make a poor decision and undo that work. The default Linux configuration is such that it can always make this poor decision; that's the default value of /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none I mentioned earlier. However, note that if you find yourself in such a situation, pooling those large allocations is likely a good idea anyway and may mitigate some of the issue.

Another suggestion / idea : perhaps we could make minHeapForMetadataHugePages settable by an environment variable (instead of the GODEBUG env var to disable hugepages), and perhaps also with an initial value meaning that the feature is disabled ?

To be clear, that variable only gets used in some very niche circumstances. It does not apply to heap memory. I don't expect changing that to meaningfully affect anybody's memory usage in practice.

@mknyszek : thanks for the clarifications !

If your application makes a lot of large irregularly-sized memory allocations (on the order of MiB) and not many small-sized allocations, then that increases the chance that there are persistent holes in the heap. There exists a background system goroutine that finds these holes and returns them to the OS, but the OS can sometimes make a poor decision and undo that work. The default Linux configuration is such that it can always make this poor decision; that's the default value of /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none I mentioned earlier. However, note that if you find yourself in such a situation, pooling those large allocations is likely a good idea anyway and may mitigate some of the issue.

Our application indeed keeps, for performance reasons, several in-memory caches of database views (where entries are invalidated by triggers/notifications). This means that in practice objects are usually allocated "in batch" and some of those may stay alive for a very long time while others are unallocated / invalidated and replaced by new values regularly, perhaps this pattern could explain the poor behavior we were seeing ?
Anyway we now have at least a stable workaround, so thanks again for helping us on this :smile:

@mknyszek This problem becomes more pronounced in conjunction with GOMEMLIMIT, in our production systems we have default config (GOGC:100 and go1.21+), which led to low heaps but high GC CPU, but there was nothing that we were able to do as such because otherwise it can lead to OOM. We tried GOMEMLIMIT and it worked for us, so we disabled GOGC (off) in some of the non impacting services, but again due to wierdly abandoning THP configurations by go1.21+ we saw random huge spikes in memory and OOM behaviour in those services after disabling GOGC, we downgraded the go version to go1.20.4 and it worked like a charm.
I understand go runtime do not want to handle this "OS stuff" and leave it to the devs to handle (as suggested by the linux doc according to you), but it becomes difficult for application devs when suddenly after one release they see a diff runtime behaviour (OOM, etc) and they have to take a decision which they were not taking otherwise
A debug.DisableTHP() would be great :)
Thanks

@Nasfame No, this is for a diff Org

@sakib-malik-7417 Thanks for reporting.

I understand go runtime do not want to handle this "OS stuff" and leave it to the devs to handle (as suggested by the linux doc according to you), but it becomes difficult for application devs when suddenly after one release they see a diff runtime behaviour (OOM, etc) and they have to take an OS decision which they were not taking otherwise

I was fully prepared for the Go runtime to handle this "OS stuff," but it's just not really reasonable to handle on our end with the current Linux APIs, and I had to learn the hard way. We may return to this in the future; I don't know.

A debug.DisableTHP() would be great :)

To be clear, I think that all we're going to do is GODEBUG=<something>, not a new API. If you want to set this at run-time, there's already a way:
import "golang.org/x/sys/unix"

...
unix.Prctl(unix.PR_SET_THP_DISABLE, 1, 0, 0, 0)
...

Our application indeed keeps, for performance reasons, several in-memory caches of database views (where entries are invalidated by triggers/notifications). This means that in practice objects are usually allocated "in batch" and some of those may stay alive for a very long time while others are unallocated / invalidated and replaced by new values regularly, perhaps this pattern could explain the poor behavior we were seeing ?

Just the fact that you have long-lived allocations isn't enough on its own. If the program makes enough small allocations to fill the holes in between the long-lived (possibly fragmented) allocations by the end of a GC cycle, it shouldn't matter. But it sounds like maybe that's not always true in your application, so yeah, maybe that's it.

@gopherbot Please open a backport issue for Go 1.21.
This issue is difficult to work around as part of upgrading Go since it involves adjusting the execution environment, and the proposed fix (a compatibility GODEBUG) is a comparatively simple, safe, and reasonable workaround.

Regarding the allocated pages for mheap.arenas and pageAlloc.chunks (@mknyszek : thanks for your patience, I finally wrapped my head around the fact that only these pages were explicitly marked with madvise(..., MADV_{NO,}HUGEPAGE) :sweat_smile:) :
what is the expected gain from backing these pages with hugepages (or not) ?

what is the expected gain from backing these pages with hugepages (or not) ?

It's less about backing with huge pages and more about disabling huge pages for small heaps. These mappings tend to be quite large in comparison to small heaps which might lead to the kernel backing them with a huge page, leading to large proportional overheads.
The main reason to keep the mitigation in this case is that the runtime doesn't ever return any of this memory to the OS, so even if the Linux configuration doesn't have a high max_ptes_none we might still end up with an unnecessary huge page on this metadata with nothing else we can do about it. With the general heap we regularly return memory to the OS, so if max_ptes_none is zero, that will stick.
If it were up to me we wouldn't call MADV_HUGEPAGE at all, but these mappings grow very slowly compared to the heap. So although the initial mapping size is big, there won't actually be very many calls to and/or initial accesses to newly MADV_HUGEPAGE'd memory, mitigating most of the stall issues that make MADV_HUGEPAGE problematic. Also, the math works out that the worst-case additional overhead for asking the OS to provide a huge page is proportionally very small, mitigating that issue as well.
Overall, I think the cost/benefit just works out in favor of keeping this particular mitigation. It helps small programs stay small without really hurting anyone else in practice.

Our production applications also experienced similar behavior and we used the GODEBUG=disablethp=1 environment as suggested in this issue. We still couldn't figure out how to validate this fix in the production system as the issue is reproduced only in long-living nodes [>15hours]. We don't want to wait for a long time to validate this fix and then explore alternative fixes. Is there any information that we can read from system files to validate that the THP is disabled for a particular go process?

One thing you can do is dump /proc/<pid>/smaps for the process and observe that memory regions have the nh ("no hugepage") attribute.

Yes. I can see nh attribute in a few memory regions. We explicitly disabled the flag in the go code as well. To rule out any memory leaks, we deployed one node with binary built with go1.18 and 2 nodes with go1.21 toolchain. The issue persists only the binaries built with go1.21 and the other node's memory stays flat. Check the attached screenshot for reference.
S.txt

unix.Prctl(unix.PR_SET_THP_DISABLE, 1, 0, 0, 0)
On a side, Is this fixed in Go 1.22 by any chance? We are anyways experimenting with the new version. I will update after observing the metrics for a day.

@LeGEC ,we use go1.22.5 and met the same question. Have you resolved this issue? Can you provide us with some suggestions?

@lojies : in our specific setting, yes, setting GODEBUG=disablethp=1 is enough to fix this very specific issue.
Note however that we first ran several memory profiles on our app to rule out any "regular" memory leaks in our code -- and first found and fixed several candidates for memory consumption by doing so.

@lojies : in our setting, yes, setting GODEBUG=disablethp=1 is enough to fix this very specific issue.
Note however that we first ran several memory profiles on our app to rule out any "regular" memory leaks in our code -- and we found and fixed several candidates for memory consumption by doing so.

Thank you for your suggestion. We tried, but it had no effect.


[backport] Use cudnn.DropoutStates instead of calling API directly

Backport of #3717

Sorry DropoutStates was not bakported to cupy.

OK


v4: useQuery and potentially others are broken with latest @vue/composition-api (0.6.1)

Describe the bug
useQuery  seems to be watching for optionRef  and documentRef  but watch  is not executed immediately unless {immediate: true} specified, so you end up with undefined values causing execution to fail with quite weird errors.
To Reproduce
Create a vue component.
in setup() call useQuery()
Observation:
the query will never be executed. if you force start you will get error due to undefined currentOption.
Expected behavior
A clear and concise description of what you expected to happen.
Versions
"vue": "^2.6.11",
"@vue/composition-api": "^0.6.1",
"@vue/apollo-composable": "^4.0.0-alpha.8",
"apollo-boost": "^0.4.9",
Additional context
https://github.com/vuejs/vue-apollo/blob/v4/packages/vue-apollo-composable/src/useQuery.ts#L318-L352
  let currentDocument: DocumentNode // will be undefined
  watch(documentRef, value => {
    currentDocument = value
    restart()
  })

  // Applying variables
  let currentVariables: TVariables  // will be undefined
  let currentVariablesSerialized: string  // will be undefined
  watch(variablesRef, (value, oldValue) => {
    const serialized = JSON.stringify(value)
    if (serialized !== currentVariablesSerialized) {
      currentVariables = value
      restart()
    }
    currentVariablesSerialized = serialized
  }, {
    deep: true,
  })

  // Applying options
  const currentOptions = ref<UseQueryOptions<TResult, TVariables>>()  // will be undefined
  watch(() => isRef(optionsRef) ? optionsRef.value : optionsRef, value => {
    if (currentOptions.value && (
      currentOptions.value.throttle !== value.throttle ||
      currentOptions.value.debounce !== value.debounce
    )) {
      updateRestartFn()
    }
    currentOptions.value = value
    restart()
  }, {
    deep: true,
  })

Confirm this issue

temporally works downgrading to v.0.5.0

One more confirmation.

I guess this can be closed since https://github.com/vuejs/vue-apollo/pull/1000 got merged and released with v4.0.0-alpha.10 ? Or is it still happening?


[3.x] Fix ViewportTexture error when viewport is used from a child scene

This should hopefully fix some of the viewport errors mentioned in https://github.com/godotengine/godot/issues/27790

Is this also relevant for 4.0?

Is this also relevant for 4.0?

That's a good question.  I think when I initially made the change, I was having trouble getting 4.0 to run.  I seem to remember something about the viewport stuff changing, but looking at master, that function appears to be the same.  Perhaps I should do the proper fix in master then pull it over to 3.x.

I pushed an update to improve the code changes, as the check was already being done just a couple lines below.

Thanks for taking care of that.  Was off on vacation + busy with other stuff, so I didn't get around to cleaning this up.

Thanks!


How to deal with extracted URLs?

Currently extracted URLs are handled exactly like keywords (from UI perspective). Next to breaking the layout (https://github.com/redlink-gmbh/smarti/issues/128) questions about:

how to handle URLs in general,
what is their conversational value,
should we promote them via preview
what do URL tell us about the content of the conversation
...

came up.
This issue has being opened in order to start a discussion about URLs from the end user perspective. How should URLs represented inside the SmartiWidget and can we use them to get a better search result ranking?

As a first step I can map them into an own field in the LATCH template (used by all Information Retrieval QueryBuilder). This will allow to ignore them for now and implement a special handling later on

Is this issue also existing when using the conversation-search query builder?


Implements symbologyIdentifier for Code 39 (#214)

Part-implements #214
Implements symbologyIdentifier for Code 39.
Adds unit test.

This PR is ready for review.

Force-push ecf12a1 added uniit test "ODCode39ReaderTest.cpp" to "test/unit/CMakeLists.txt"; also made symbologyModifiers array in Code39Reader::decodePattern() const.
This PR is now ready for review.

Just to have said it: it would be totally fine to have more than one commit (for more than one symbology) in one single PR :).

Ah cool, was going to do one PR for a number of the 1-d barcodes which have trivial Symbology Identifiers, but not for others like ITF where it's slightly more involved. I do find though having separate commits in one PR error-prone on my part, especially with the way I'm doing it (porting from my local "diagnostics2" branch). Will give it a lash with the next PR...


Identify critical business signals

As a project lead, I want to identify the resources and signals most important to the business and application, so that the team can begin collecting and analyzing the requisite data

[ ] Design doc created that outlines telemetry signals to prioritize for the application
[ ] Document references resource-specific and application-specific telemetry signals[ ] Monitor reference docs linked for each infrastructure resource (e.g. https://learn.microsoft.com/en-us/azure/service-bus-messaging/monitor-service-bus-reference)

Mentioned in #332


operators [R] etcd (0.6.1 0.9.0 0.9.2 0.9.2-clusterwide 0.9.4 0.9.4-clusterwide)

Thanks submitting your Operator. Please check below list before you create your Pull Request.
Fix #193  remove the olm.maxOpenShiftVersion=4.8 since this etcd opeataor's CRDs already use the v1 version.
New Submissions

[ ] Are you familiar with our contribution guidelines?
[ ] Have you packaged and deployed your Operator for Operator Framework?

[ ] Have you tested your Operator in all supported installation modes?
[ ] Have you considered whether you want use semantic versioning order?
[ ] Is your submission signed?
[ ] Is operator icon set?

Updates to existing Operators

[ ] Did you create a ci.yaml file according to the update instructions?
[ ] Is your new CSV pointing to the previous version with the replaces property if you chose replaces-mode via the updateGraph property in ci.yaml?
[ ] Is your new CSV referenced in the appropriate channel defined in the package.yaml or annotations.yaml ?
[ ] Have you tested an update to your Operator when deployed via OLM?
[ ] Is your submission signed?

Your submission should not

[ ] Modify more than one operator
[ ] Modify an Operator you don't own
[ ] Rename an operator - please remove and add with a different name instead
[ ] Modify any files outside the above mentioned folders
[ ] Contain more than one commit. Please squash your commits.

Operator Description must contain (in order)

[ ] Description about the managed Application and where to find more information
[ ] Features and capabilities of your Operator and how to use it
[ ] Any manual steps about potential pre-requisites for using your Operator

Operator Metadata should contain

[ ] Human readable name and 1-liner description about your Operator
[ ] Valid category name1
[ ] One of the pre-defined capability levels2
[ ] Links to the maintainer, source code and documentation
[ ] Example templates for all Custom Resource Definitions intended to be used
[ ] A quadratic logo

Remember that you can preview your CSV here.
--
1 If you feel your Operator does not fit any of the pre-defined categories, file an issue against this repo and explain your need
2 For more information see here

/merge possible

/authorized-changes

Hi @camilamacedo86 Could you help add the authorized-changes label? Thanks!

/merge possible


test: add e2e test and upgrade dependencies

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@e3bc76d). Click here to learn what that means.
Patch has no changes to coverable lines.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #96   +/-   ##
=======================================
  Coverage        ?   75.65%
=======================================
  Files           ?       18
  Lines           ?      645
  Branches        ?       73
=======================================
  Hits            ?      488
  Misses          ?      156
  Partials        ?        1

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


AppWrappers with same name with generic items that also have the same name but are deployed in different namespaces fail

This is a very obscure bug, and potentially not worth fixing. The issue can be replicated with the following code. The only difference between the two AppWrappers is the fact that one is deployed to namespace-1 and the other to namespace-2, everything else is the same, including the names for the AppWrappers and the generic items. This deployment fails for one of the AppWrappers as MCAD thinks there is already a dispatched AppWrapper but its generic items are non existent in its corresponding namespace. Changing the names of either the AppWrapper or the generic items fixes the issue. It is very unlikely anyone would have the exact same names across two namespaces, so no need to worry about this for now.
apiVersion: mcad.ibm.com/v1beta1
kind: AppWrapper
metadata:
    namespace: namespace-1
    name: my-aw
spec:
    resources:
        GenericItems:
            - replicas: 1
              custompodresources:
              - replicas: 1
                requests:
                    cpu: 500m
                    nvidia.com/gpu: 0
                    memory: 500Mi
                limits:
                    cpu: 500m
                    nvidia.com/gpu: 0
                    memory: 500Mi
              generictemplate:
                  apiVersion: v1
                  kind: Pod
                  metadata:
                      namespace: namespace-1
                      name: my-pod
                  spec:
                      containers:
                          - name: my-pod
                            image: ubuntu:latest
                            imagePullPolicy: IfNotPresent
                            command:
                                - sh
                                - -c
                                - |
                                  sleep infinity
                            resources:
                                requests:
                                    cpu: 500m
                                    nvidia.com/gpu: 0
                                    memory: 500Mi
                                limits:
                                    cpu: 500m
                                    nvidia.com/gpu: 0
                                    memory: 500Mi
---
apiVersion: mcad.ibm.com/v1beta1
kind: AppWrapper
metadata:
    namespace: namespace-2
    name: my-aw
spec:
    resources:
        GenericItems:
            - replicas: 1
              custompodresources:
              - replicas: 1
                requests:
                    cpu: 500m
                    nvidia.com/gpu: 0
                    memory: 500Mi
                limits:
                    cpu: 500m
                    nvidia.com/gpu: 0
                    memory: 500Mi
              generictemplate:
                  apiVersion: v1
                  kind: Pod
                  metadata:
                      namespace: namespace-2
                      name: my-pod
                  spec:
                      containers:
                          - name: my-pod
                            image: ubuntu:latest
                            imagePullPolicy: IfNotPresent
                            command:
                                - sh
                                - -c
                                - |
                                  sleep infinity
                            resources:
                                requests:
                                    cpu: 500m
                                    nvidia.com/gpu: 0
                                    memory: 500Mi
                                limits:
                                    cpu: 500m
                                    nvidia.com/gpu: 0
                                    memory: 500Mi

May be this is an anti-pattern, users in one namespace submit AW to another namespace

Well, I'm thinking the case where a user has access to two namespaces and decides to submit the exact same job to both of them, for some reason. Then it would fail. But that may be a very very unlikely scenario.


No Support for Foreign Characters

...at least in the loading_phrase text. For example, in this image the missing character between "D" and "d" is supposed to be ø, the Nordic o with a slash. I would like to see support added for foreign characters in Moonscraper.

This is an issue with Clone Hero. Re-open if you can get these characters to appear in Clone Hero without the use of Moonscraper.


handle required mod downloadlinks

broadcasts links to the masterserver and makes them available for the client
depends on R2Northstar/NorthstarMasterServer#58 and R2Northstar/NorthstarMods#309

This would be perfect! I hate having to manually install everything and then manually uninstall/disable when I play on another server.

@iCrazyBlaze This PR doesn't implement automatic mod downloads. Only a menu is shown that links to the download pages missing mods have specified. If you're looking for automatic mod enabling/disabling, I made another PR for that here: R2Northstar/NorthstarMods#298

succeeded by automatic mod downloads that will come at some point in this century


ContactPairView::collider2 can panic when using in_fixed_schedule

Hi I recently switched my app to using in_fixed_schedule and sometimes (generally when entities are being deleted)  ContactPairView::collider2 panics on unwrapping a None value.
For context, I'm trying to get the collider of a contact from RapierContext::contacts_with and this is happening in a system which is not in the fixed schedule.
I had never seen this behaviour before switching to in_fixed_schedule and it's happening all the time now so I'm very confident the change has something to do with that.
If you like I can try to produce a minimal example of this.

A minimal example would be helpful! My gues right now is that rapier's cache of collider <-> entity maps is getting cleared too early here.


A few additional details on the early history of RAP

After chatting with Matt U about the early work on eesectors, Chris Fairbanks worked with me to build a RAP as an exemplar project on the MoJ's new analytical platform.  (He led this work, I just helped out with a bit of code and aspects of good practice/git, and ensuring reproducibility on our specific infrastructure)
He eventually won a young statisticians award for this work

Thanks Robin! We're just incorporating these into our "hidden" dev repo and we will get it published ASAP!

Great, thanks so much!

This has been resolved in a different pull request :) #44


Environment selector server

Implemented Day/night selector for servers.
TODO: Look at feasibility of implementing other weather conditions selector, and how to implement that for consoles

tested from YSCE server and YS2015 client - works great!

Approved and merged!


chore(starknet_api): add sierra version utils

#2811
#2809
#2752
#2828  👈 (View in Graphite)
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Merge activity

Dec 19, 11:06 AM EST: A user merged this pull request with Graphite.


gpu is always active when Hyper is active

[v] I am on the latest Hyper.app version
[v] I have searched the issues of this repo and believe that this is not a duplicate

Windows 10

OS Name	Microsoft Windows 10 Home
Version	10.0.16299 Build 16299

1.4.8 stable

Gist of .hyper.js

N/A

Is Vanilla

Issue

at first I was curious about why my laptop fans are on when I have Hyper.js open.
so after a little bit of monitoring with task manager I can see that the GPU is always on. and by always i literally mean always, not the integrated with cpu gpu but the other gpu, my GTX 1050.
I'm not sure if this is by design or just something that slipped through, but all I can tell is that Hyper is using atleast 300MB of memory and a <1% spike in gpu rendering every minute.

@crostine You can try in the canary version, maybe the consumption of the cpu is lower.

I found it, when I tried to force Hyper to use the integrated card though NVidia control panel, it was already set to prefer the dedicated gpu.
I changed this to automatic/global setting and the dedicated gpu is now turned off.

Great to hear this got resolved. Thanks for updating us and closing 👍

@crostine Same problem here. Can you elaborate on what you did to fix?

Open up NVIDIA Control Panel
Select "Manage 3D settings" in the left menu
Select the "Program Settings" tab
Use the dropdown to find Hyper (if its not listed there are no different settings from global settings)
Here I noticed that the selected preferred graphics processor was on "High-performance NVIDIA processor"
Change this option to "Use global setting" or "Integrated graphics"
Click "Apply" in the lower right corner

I hope this helps


My 2nd custom Filter : BloomFilter  & There is a problem :(

I create a  bloom filter .
Demo :
http://fattyboy.cn/pt/?t=123
Code :
https://github.com/finscn/pixi.js/blob/master/src/extension/BloomFilter.js
https://github.com/finscn/pixi.js/blob/master/src/extension/BrightnessFilter.js
References :
http://learnopengl.com/#!Advanced-Lighting/Bloom
It looks correct.
But when I add the filter to  a sprite with PerspectiveRenderer ( more detail at  https://github.com/pixijs/pixi.js/issues/3227 ) , the sprite can't display and no any error in console.
Would you like to suggest some possible reasons why this is ?
Thanks.
cc @ivanpopelyshev  @englercj

User the WebGL inspector and check what your buffers look like. I don't have time atm to debug a custom renderer :(

I found the reason :  it's   PerspectiveRenderer's fault.
BloomFilter is OK.
I hope PIXI.js could include a filter like  bloom, you could write a better one, I think.
Thanks

@finscn check out https://github.com/pixijs/pixi-filters there's a Bloom filter there.

I've transport the  BloomFilter in https://github.com/pixijs/pixi-filters  to PIXI v4.
I found it's algorithm is different from  http://learnopengl.com/#!Advanced-Lighting/Bloom
And it can't custom light of bloom .


Review sniff handling of PHP 7.4 null coalesce equal operator

Since PHP 7.4 null coalesce equal operator was introduced (??=).
Now instead of using null coalesce operator when assigning to a variable like:
$_GET['variable'] = $_GET['variable'] ?? 'default';

You can write
$_GET['variable'] ??= ''default';

We need to see what sniffs cover usage of this operator upstream and in the utils and if we can just use them in the WPCS ruleset.
Refs:

PHP RFC: https://wiki.php.net/rfc/null_coalesce_equal_operator

Related to #764

All sniffs have been reviewed and fixed to support modern PHP in as far as my imagination reached.
If any more situations would be discovered which need to be taken into account, a new issue should be opened with code samples etc.
Closing this now as fixed.


Announcement 2.5.0 Release

bugs

(#1325) Missing @RequestParam on a boolean parameter causes Swagger page to not render that controller, and all controllers alphabetically after has-workaround @Thunderforge
(#1321)	Cannot fully change swagger and swagger ui path has-workaround @nikit-cpp
(#1133) Respect ApiModelProperty(hidden=true) on @ModelAttribute annotated models has-workaround @cm325
(#1317)	@ApiModelProperty(value="something") on bean annotated with @ModelAttribute changes dataType to "ref" @mpostelnicu
(#1310)	'enum' query parameter type(annotated with @ModelAttribute) is 'ref'.  @namkee
(#1306)	Maps as parameters were not rendered correctly @aqlu
(#1285)	dataType="file" is not working in springfox 2.4.0 @HDBandit
(#1282)	Missing model definitions in swagger json document when return type is array syntax (CustomModel[]) @namkee
(#1280)	@ApiModelProperty.position not respected @marceloverdijk
(#1268)	Compatible issue encountered with Spring Boot 1.4.0.M2 and Spring 4.3.RC @hantsy
(#1260)	Not working proper with query params not-reproducable @Gaurav-Deshkar
(#1258)	Fields not visible from children when implementing interfaces in parent request objects @stashthecode    2 of 2
(#1249)	Bug: Templated url are submited with the template part @anthofo
(#1241)	Overloaded method resolution was incorrect causing ArrayIndexOutOfBoundsException @tdeverdiere
(#1238)	Swagger UI showing incorrect URLs @martin3361
(#1211)	ApiParam allowableValues string with spaces @pvpkiran
(#1209)	Query parameters - complex data types are coerced to 'string' type, especially collections @jkasmann
(#1207)	CachingModelPropertiesProvider - NullPointerException @bharatkaushik
(#1203)	swagger-ui location /configuration/ui configurable?  @flexguse
(#1196)	Springfox not emitting attributes for definitions with some kinds of circular references @benfowler

features

(#1296) Support for JSR-303: @Pattern annotation @ashutosh-shirole
(#1291) Make 'supportedSubmitMethods' configurable in springfox-swagger-ui @thomseno
(#1287) Feature: Headers in @RequestMapping are not documented @ry4n-sc0tt
(#1244) @ApiParam - Allowable values not displayed in Swagger API docs @jfiala

maintenance

(#1332) Support for jdk 8 build @dilipkrish
(#1326) Simplify the gradle build @dilipkrish
(#1322) Copyright check for license should allow for newer years @dilipkrish
(#1276) character encoding error in springfox-staticdocs @catinred2
(#1275) Added message about starring the repository  PR @dilipkrish
(#1253) BeanPropertyDefinitions not compatible with Jackson 2.7.x @Stephan202
(#1162) java.sql.Date creates an useless "Date" definition @cbornet

PRs

(#1331) Fix for #1282: Array dependent types not part of models  @namkee
(#1319) Fix for #1318: fixing favicon images urls  @rubencepeda
(#1316) Fix for #1296: Adding Pattern annotation support  @ashutosh-shirole
(#1314) Fix for #1296: Support for JSR-303: @Pattern annotation  @igor-sokolov
(#1233) Fix for #1207: Skip events from child application context @praveen12bnitt
(#1295) Make 'supportedSubmitMethods' configurable in springfox-swagger-ui / Enable and disable the "Try it out!" functionality  @thomseno
(#1292) Added support for documenting headers in @RequestMapping  @ry4n-sc0tt
(#1284) Fix for broken contributing link in README  @ry4n-sc0tt
(#1277) Set encoding to utf-8 for multilanguage support.  @catinred2
(#1275) Added message about starring the repository  maintenance @dilipkrish
(#1265) Don't include a license object when both license and licenseUrl are e…  @wgreven-ibr
(#1239) Fix for the context refresh ordering issue. Support for spring-cloud brixton  @dilipkrish
(#1218) Only encode the API key when passed in URL query  @franklloydteh
(#1217) adds basePath from x-forwarded-prefix for reverse proxy scenarios  @matonthecat
(#1213) Fixed some wrong asciidoc typos  @RobWin

Not to mention all the questions and suggestions by the community!! :metal:

This release has been published to maven central and jcenter


Create claps or applause or like feature

Liked an article?
Let's Create it!
Can be similar to medium, facebook or any site!
Prerequisite

You have to be absolute beginner
You should have a motivation for building this feature

Reference
https://github.com/ColinEberhardt/applause-button

@piyush97  can i work on this?

@abhaybhatt Go Ahead!

@piyush97 i am facing problem in setting up the project in my machine?

Can you explain what issue you are facing @abhaybhatt ?

Hey! Great issue. Just clarifying, the requirement is to build a component that can be integrated into the end of the blog posts. Anything that I'm missing ?

Correct @sindhujaponnuru

@piyush97 I'm unable to run the application :

@sindhujaponnuru
npm install

npm run dev

@piyush97 I know React & JS pretty well, gatsby would be new for me but yeah I would love to explore and try to make the best out of it. I would add a clap animation or a favorite icon animation below the blog component. Can I PR?

@piyush97 I know React & JS pretty well, gatsby would be new for me but yeah I would love to explore and try to make the best out of it. I would add a clap animation or a favorite icon animation below the blog component. Can I PR?

Go for it!

If facing issue

How to store likes count?

https://www.gatsbyjs.com/plugins/gatsby-source-firestore/
or infact any of the recommended Db
https://www.gatsbyjs.com/docs/sourcing-from-databases/
You can implement it and store the number of likes in the Db
:)

How do I store the likes count

@bufgix

is this issue currently assigned to anyone

No @ogheneovo12

I'm a beginner in JS , Can I work on this ??

Please go ahead @khush31

Please go ahead @khush31

But I don't know where to contribute. Could you please Guide me

can i also work on it

Hey i made a PR, please review and let me know if it is fine!
#101


Adding Kaeforest for the webring

I'm about to fill the pages with the content I desire, and will probs do so in a few days, also to include research into my given areas of synthesis.
Though hopefully this will be good for the time being :)
https://kae.si/home

Nice wiki :) welcome to the webring!


Global defines and file specific defines

Problem shown by trying to port the GCC Asm test project to use IAR Toolchain.
The GCC Asm Target.cproj included in the test-suite lists that same defines both on a global level and for specific asm-files. Running the cmake generation script will with the current design add both global defines and file-local defines to the command line resulting in compilation errors with the IAR Assembler as it does not allow listing the same define twice.
How should one go about this? As I see it there are three ways:

Modify the IAR toolchain file to filter out any defines that is being re-added.
Enforce the writer of the cproj fileto not pass defines that are added on a global level on file-specific levels as well.
Have the define-system as overrides, meaning that a file with local overrides does not use the global defines.

Personally, I leaning towards number 2. @brondani @JonatanAntoni opinions on this?

This is a know problem (see #607).  We will first solve it by applying define: and add-path: only to C and C++ source files.

The problem here is the hierarchical processing even if settings are only applied to a single language type. A define that is specified on a global level maybe duplicated by a define on the file level. The tool may not allow to specify the same define twice on the command line.
@brondani does csolution have the ability to
a) identify duplicate defines
b) remove duplicates when generating cprj

CMSIS-Toolbox 1.5.0


Fault node status doesn't go to RUNNING state during experiment execution

What happened:
Fault node status doesn't go to RUNNING state during experiment execution, it is stuck at INITIALIZED state.
![image (15)](https://github.com/litmuschaos/litmus/assets/28081206/67a7dfa5-d642-4f91-ae36-99aedd497f7b
What you expected to happen:
Fault node status should get updated to RUNNING state when experiment starts running.

Where can this issue be corrected? (optional)

How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:

Hello @Saranya-jena  I am interested to work on this issue. Please assign this to me.

Hello @Saranya-jena  could you please review my pull request #4733
If any changes are needed, please let me know and I will try to make them
Thank you


Lecterns don't show tabs

Lecterns are recognized as having an inventory, so show up in the tabs.

Yeah, lecterns only show up as an inventory tab when they have a book in them. The book GUI is kinda funky, I haven't yet mixed into it to get the tabs to render there


Possibility to add custom message to initial post

Would it be possible to provide a custom/additional message to the initial post created by the bot?
<script src="https://utteranc.es/client.js"
        repo="[ENTER REPO HERE]"
        issue-term="[ENTER TERM HERE]"
        issue-body="[CUSTOM MESSAGE HERE]"
        async>
</script>

I think that would be an easy change.  Here's the current issue body template for reference:
https://github.com/utterance/utterances/blob/c31b9579a2bf0da3b9b7fd2a092993d9a4203ba7/src/github.ts#L194
Before I make any modifications, could you provide an example of what your custom template would look like? I want to make sure what I'm thinking about doing would meet the need.

Oh perfect. I would suggest to make it look like this
 body: `# ${title}\n\n${description}\n\n ${customMessage}\n\n[${documentUrl}](${documentUrl})`


circup: 1.2.1 -> 1.2.3

Description of changes

Things done

Built on platform(s)

[X] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[X] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[X] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[X] Fits CONTRIBUTING.md.

close #243313

Please use nix-update or add the link to the changelog manually as stated in CONTRIBUTING.md.

I used nix-update, what makes you think I didn't?

Result of nixpkgs-review pr 243314 run on x86_64-linux 1

2 packages built:

circup
circup.dist

I used nix-update, what makes you think I didn't?

Because the changelog is not part of the commit message. nix-update adds the changelog. Not sure when the feature was added to nix-update but it's already present for a while.

I have nix-update 0.18.0 and used nix-update circup --commit which didn't add a change log to the commit message.

Hmmm, seems to work on NixOS unstable.
$ nix-env --query --available --attr-path nix-update
nixos.nix-update  nix-update-0.18.0

$ nix-update circup --build --commit
[...]
} --extra-experimental-features flakes nix-command
$ git -C /home/fab/Documents/repos/nixpkgs commit --verbose --message circup: 1.2.1 -> 1.2.3

Diff: https://github.com/adafruit/circup/compare/refs/tags/1.2.1...1.2.3

Changelog: https://github.com/adafruit/circup/releases/tag/1.2.3 /home/fab/Documents/repos/nixpkgs/pkgs/development/tools/circup/default.nix
[circup-bump 914fefa3c20] circup: 1.2.1 -> 1.2.3
 1 file changed, 2 insertions(+), 2 deletions(-)


UI for Forgot password page Fixed

Related Issue
Fixes:  #1342
Description
Before :

After :

Type of PR

[x] Bug fix
[x] Feature enhancement
[ ] Documentation update
[ ] Other (specify): _______________

Screenshots / videos (if applicable)
Checklist:

[x] I have made this change from my own.
[x] I have taken help from some online resources.
[x] My code follows the style guidelines of this project.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] My changes generate no new warnings.
[x] I have tested the changes thoroughly before submitting this pull request.
[x] I have provided relevant issue numbers and screenshots after making the changes.

@huamanraj issue no #1342 resloved


No model files

Hi, when I run it, I got this error: No such file or directory: '/.../.../reactiondataextractor2/reactiondataextractor/configs/../models/cnn_weights/torch_arrow_detector_classifier.pt'.
As I checked this the folder 'configs', I found that there is no folder named 'models'. Could you please help me to locate the folder and model files? Thank a lot.

I didn't check README carefully, my bad.


Update README.md

Added detail of my upcoming talk at .Net South West (UK) with Carl Sargunar

I've put me in as a separate entry and removed my name from Carl's line.
Carl mentioned that it's good for these to go in as multiple events/PRs.
Happy to change as needed though.
Thanks!

Hello @AdamPrendergast ,
Yes! It's definitely OK to create a separate entry and get PR credit for yourself 👍
And don't hesitate to do the same with the event on October 31st  😁🎃
Cheers!


About train GlaS Dataset

I had a problem learning XNet, your great research.
The GlaS dataset, a 2D image, was preprocessed using wavelet2D.py to obtain LF and HF images of [1,128,128] testers, respectively.
After passing through the XNet model, are the shapes of outputs_train1 and outputs_train2 [2,128,128] respectively?
And when we learn GlaS dataset on branch1 and branch2 on the XNet model, will 3 and 1 channels be right?
# branch 1
self.b1_1_1 = nn.Sequential(
conv3x3(in_channels, l1c), #in_channels = 3
conv3x3(l1c, l1c),
BasicBlock(l1c, l1c)

# branch 2
self.b2_1_1 = nn.Sequential(
conv3x3(1, l1c),
conv3x3(l1c, l1c),
BasicBlock(l1c, l1c)

Thank you for your answers!!

The XNet architecture was written after other baseline models, so the in_channels parameter is habitually retained and the default value is set to 3. In actual use, you can adjust it according to your own dataset.
For the GlaS dataset, the input is [Batch size, 1, 128, 128]，the output is [Batch size, 2, 128, 128].
By the way, the wavelet transform for RGB three-channel images is very flexible. There are two strategies: (1) You can first convert the three-channel image into a grayscale image, and then perform wavelet transform to generate L and H, so that L and H are the single channel images. (2) You can also perform wavelet transformation on the three channels R, G, and B respectively, and concatenate the results, so that L and H are three-channel images.

Thank you for your kind answer.
Also, if you import and use a mask from GlaS Dataset, it comes out as a [128,128] tensor, so I can't calculate Dice loss by output_train of [Batch size, 2, 128, 128].
Does GlaS dataset require additional preprocessing?

Can you give me the error print?

This is my tensor shape for training.
inputs_train_shape1 =  torch.Size([2, 1, 128, 128])
inputs_train_shape2 =  torch.Size([2, 1, 128, 128])
outputs_train_shape1 =  torch.Size([2, 2, 128, 128])
outputs_train_shape2 =  torch.Size([2, 2, 128, 128])
mask_train =  torch.Size([128, 128])
And this is my error print.
Traceback (most recent call last):
File "train_sup_XNet.py", line 235, in
File "train_sup_XNet.py", line 235, in
loss_train_sup1 = criterion(outputs_train1, mask_train)
File "/home/gpuadmin/anaconda3/envs/xnet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/data/LAB/cha/XNet-main/loss/loss_function.py", line 140, in forward
loss_train_sup1 = criterion(outputs_train1, mask_train)
File "/home/gpuadmin/anaconda3/envs/xnet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
return self._base_forward(output, target_one_hot, valid_mask)
File "/data/LAB/cha/XNet-main/loss/loss_function.py", line 107, in _base_forward
result = self.forward(*input, **kwargs)
File "/data/LAB/cha/XNet-main/loss/loss_function.py", line 140, in forward
return self._base_forward(output, target_one_hot, valid_mask)
File "/data/LAB/cha/XNet-main/loss/loss_function.py", line 107, in _base_forward
dice_loss = dice(predict[:, i], target[..., i], valid_mask)
dice_loss = dice(predict[:, i], target[..., i], valid_mask)
IndexError: index 2 is out of bounds for dimension 1 with size 2
IndexError: index 2 is out of bounds for dimension 1 with size 2

mask_train = torch.Size([128, 128])？
This seems to be because the shape of the mask is inconsistent with the shape of the image. mask is missing the dimension of batch size

I check it, then i have same error
input_train_shape1 =  torch.Size([2, 1, 128, 128])
input_train_shape2 =  torch.Size([2, 1, 128, 128])
mask_train =  torch.Size([2, 128, 128])
outputs_train_shape1 =  torch.Size([2, 2, 128, 128])
outputs_train_shape2 =  torch.Size([2, 2, 128, 128])
Traceback (most recent call last):
File "train_sup_XNet.py", line 235, in
File "train_sup_XNet.py", line 235, in
loss_train_sup1 = criterion(outputs_train1, mask_train)
File "/home/gpuadmin/anaconda3/envs/xnet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/data/LAB/cha/XNet-main/loss/loss_function.py", line 140, in forward
return self._base_forward(output, target_one_hot, valid_mask)
File "/data/LAB/cha/XNet-main/loss/loss_function.py", line 107, in _base_forward
dice_loss = dice(predict[:, i], target[..., i], valid_mask)
IndexError: index 2 is out of bounds for dimension 1 with size 2
loss_train_sup1 = criterion(outputs_train1, mask_train)
File "/home/gpuadmin/anaconda3/envs/xnet/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/data/LAB/cha/XNet-main/loss/loss_function.py", line 140, in forward
return self._base_forward(output, target_one_hot, valid_mask)
File "/data/LAB/cha/XNet-main/loss/loss_function.py", line 107, in _base_forward
dice_loss = dice(predict[:, i], target[..., i], valid_mask)
IndexError: index 2 is out of bounds for dimension 1 with size 2

I have one more question from the author. I wonder why the output channel comes out as 2 according to NUM_CLASSES in For the GlaS data set, the input is [Batch size, 1, 128, 128] and the output is [Batch size, 2, 128, 128] that you mentioned last time. And in each image, the mask is 1 channel, and I wonder how it is used to calculate the 2 channel image and loss.

Before calculating loss, the code will perform one hot for the mask (loss/loss_function.py line 119 ), which means that the mask will be converted from [Batch size,1,128,128] to [Batch size,NUM_CLASSES,128,128].

I checked the raw dataset again and I think I know where your error is. You did not preprocess the mask of the raw dataset. Please note that the raw mask is instance segmentation annotation.

Preprocessing is as simple as you said, foreground = 1, background = 0.

Then, isn't it the shape of the tensor of [2,128,128] corresponding to NUM_CLASSES??
If you need to make it in that shape, please tell me how to pre-process it. thank you.

Author, the problem I asked earlier has been solved.
I have a new question. In wavelet2D, is it possible for us to change the threshold other than specifying --wavelet_type?

I had a problem learning XNet, your great research. The GlaS dataset, a 2D image, was preprocessed using wavelet2D.py to obtain LF and HF images of [1,128,128] testers, respectively. After passing through the XNet model, are the shapes of outputs_train1 and outputs_train2 [2,128,128] respectively?
And when we learn GlaS dataset on branch1 and branch2 on the XNet model, will 3 and 1 channels be right?
# branch 1
self.b1_1_1 = nn.Sequential(
conv3x3(in_channels, l1c), #in_channels = 3
conv3x3(l1c, l1c),
BasicBlock(l1c, l1c)

# branch 2
self.b2_1_1 = nn.Sequential(
conv3x3(1, l1c),
conv3x3(l1c, l1c),
BasicBlock(l1c, l1c)

Thank you for your answers!!

Hello, I would like to ask if you sliced the dataset using a python file. The dataset downloaded from the official website is in HDF format. Is it necessary to convert it into a image using a python file before processing it? Also, what is the step size you set when processing the dataset for slicing

I had a problem learning XNet, your great research. The GlaS dataset, a 2D image, was preprocessed using wavelet2D.py to obtain LF and HF images of [1,128,128] testers, respectively. After passing through the XNet model, are the shapes of outputs_train1 and outputs_train2 [2,128,128] respectively?
And when we learn GlaS dataset on branch1 and branch2 on the XNet model, will 3 and 1 channels be right?
# branch 1
self.b1_1_1 = nn.Sequential(
conv3x3(in_channels, l1c), #in_channels = 3
conv3x3(l1c, l1c),
BasicBlock(l1c, l1c)

# branch 2
self.b2_1_1 = nn.Sequential(
conv3x3(1, l1c),
conv3x3(l1c, l1c),
BasicBlock(l1c, l1c)

Thank you for your answers!!

Hello, I would like to ask if you sliced the dataset using a python file. The dataset downloaded from the official website is in HDF format. Is it necessary to convert it into a image using a python file before processing it? Also, what is the step size you set when processing the dataset for slicing

The GlaS dataset does not seem to require the kind of preprocessing you mentioned. The images I downloaded are in bmp format.

I had a problem learning XNet, your great research. The GlaS dataset, a 2D image, was preprocessed using wavelet2D.py to obtain LF and HF images of [1,128,128] testers, respectively. After passing through the XNet model, are the shapes of outputs_train1 and outputs_train2 [2,128,128] respectively?
And when we learn GlaS dataset on branch1 and branch2 on the XNet model, will 3 and 1 channels be right?
# branch 1
self.b1_1_1 = nn.Sequential(
conv3x3(in_channels, l1c), #in_channels = 3
conv3x3(l1c, l1c),
BasicBlock(l1c, l1c)

# branch 2
self.b2_1_1 = nn.Sequential(
conv3x3(1, l1c),
conv3x3(l1c, l1c),
BasicBlock(l1c, l1c)

Thank you for your answers!!

Hello, I would like to ask if you sliced the dataset using a python file. The dataset downloaded from the official website is in HDF format. Is it necessary to convert it into a image using a python file before processing it? Also, what is the step size you set when processing the dataset for slicing

The GlaS dataset does not seem to require the kind of preprocessing you mentioned. The images I downloaded are in bmp format.

作者大大您好 非常感谢您这么及时回复 我能问一下在处理CREMI数据集的时候用滑动窗口裁剪数据集 这个有步长的设置吗 还是说就用256x256规格对原数据集进行裁剪

I had a problem learning XNet, your great research. The GlaS dataset, a 2D image, was preprocessed using wavelet2D.py to obtain LF and HF images of [1,128,128] testers, respectively. After passing through the XNet model, are the shapes of outputs_train1 and outputs_train2 [2,128,128] respectively?
And when we learn GlaS dataset on branch1 and branch2 on the XNet model, will 3 and 1 channels be right?
# branch 1
self.b1_1_1 = nn.Sequential(
conv3x3(in_channels, l1c), #in_channels = 3
conv3x3(l1c, l1c),
BasicBlock(l1c, l1c)

# branch 2
self.b2_1_1 = nn.Sequential(
conv3x3(1, l1c),
conv3x3(l1c, l1c),
BasicBlock(l1c, l1c)

Thank you for your answers!!

Hello, I would like to ask if you sliced the dataset using a python file. The dataset downloaded from the official website is in HDF format. Is it necessary to convert it into a image using a python file before processing it? Also, what is the step size you set when processing the dataset for slicing

The GlaS dataset does not seem to require the kind of preprocessing you mentioned. The images I downloaded are in bmp format.

作者大大您好 非常感谢您这么及时回复 我能问一下在处理CREMI数据集的时候用滑动窗口裁剪数据集 这个有步长的设置吗 还是说就用256x256规格对原数据集进行裁剪

For train set, a sliding window with an overlap rate of 0.25 is used for sampling and pure negative examples are removed. For test set, a sliding window without overlap is used for sampling.

I had a problem learning XNet, your great research. The GlaS dataset, a 2D image, was preprocessed using wavelet2D.py to obtain LF and HF images of [1,128,128] testers, respectively. After passing through the XNet model, are the shapes of outputs_train1 and outputs_train2 [2,128,128] respectively?
And when we learn GlaS dataset on branch1 and branch2 on the XNet model, will 3 and 1 channels be right?
# branch 1
self.b1_1_1 = nn.Sequential(
conv3x3(in_channels, l1c), #in_channels = 3
conv3x3(l1c, l1c),
BasicBlock(l1c, l1c)

# branch 2
self.b2_1_1 = nn.Sequential(
conv3x3(1, l1c),
conv3x3(l1c, l1c),
BasicBlock(l1c, l1c)

Thank you for your answers!!

Hello, I would like to ask if you sliced the dataset using a python file. The dataset downloaded from the official website is in HDF format. Is it necessary to convert it into a image using a python file before processing it? Also, what is the step size you set when processing the dataset for slicing

The GlaS dataset does not seem to require the kind of preprocessing you mentioned. The images I downloaded are in bmp format.

作者大大您好 非常感谢您这么及时回复 我能问一下在处理CREMI数据集的时候用滑动窗口裁剪数据集 这个有步长的设置吗 还是说就用256x256规格对原数据集进行裁剪

For train set, a sliding window with an overlap rate of 0.25 is used for sampling and pure negative examples are removed. For test set, a sliding window without overlap is used for sampling.

Glas数据集官方的链接似乎已经失效了 作者这边能提供一份网盘的链接吗

作者，我之前问的问题已经解决了。我有一个新问题。在 wavelet2D 中，除了指定 --wavelet_type 之外，我们是否可以更改阈值？
Hello! I have also encountered this type of problem during my training. Can you please offer your help?

Before calculating loss, the code will perform one hot for the mask (loss/loss_function.py line 119 ), which means that the mask will be converted from [Batch size,1,128,128] to [Batch size,NUM_CLASSES,128,128].

Hello! I have also encountered this type of problem during my training. Can you please offer your help?


Create Api.dropbox

Browser

@edimaliusman Can you clarify what the purpose of this PR is?

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Create test data for load_signal

This is now being generated by an octave script.  Eventually it would be nice to do this in Python.


Serialiization error when Slack returns an error response from Chat.Post

I'm getting some sort of error back from Slack, but when the library tries to deserialize the error response from Slack for a call to Chat.Post:
Exception_Type=JsonSerializationException
Exception_Message="Error converting value ''failed to match all allowed schemas [json-pointer:/blocks/0/text]'' to type 'Slack.NetStandard.WebApi.WebApiError'. Path 'errors[0]', line 1, position 117."
InnermostException_Type=ArgumentException
InnermostException_Message="Could not cast or convert from System.String to Slack.NetStandard.WebApi.WebApiError."

I'm not sure if there is a way to get the JSON of the response. Is there a setting somewhere that would allow me to see it?

Hi @mapitman - I’ll take a look at fixing the serialisation over the weekend, but the error itself is the ‘failed to match all allowed schemas’ text in the message you’re getting.
It appears that either one of the values in the text block at the start is invalid or possibly some of the text in that block is too long.
As I say, I’ll leave this open and hopefully get a fix released soon.

Oh, thanks for that bit of info. What's weird is the code works on one Slack workspace and fails on another. I made a change to an existing block. It is a Button and I added Style = ButtonStyle.Primary. That was the only change. I rolled my change back and it works fine now. 🤷♂

OK, now I see my issue. I have a TextBlock  where the text is too long. Nothing to do with the button style. I was testing with some different data and that was the cause.

Glad you were able to track the issue down. I was able to reproduce the serialization problem you had and correct it in 3.8.0-beta3, which has just been published.


Questioning accuracy

All the user-story charts (for my different projects), have the same unlikely trend of very high cycle time in the past and very low in the present. The past doesn't show any of the short-cycle user stories that I know were accepted back then. A couple of months ago I tried the app and had the same result.
I can't be the only community user with this problem. What exactly is the rule by which stories are chosen, and then how is the cycle time calculated?
This app is far superior to the library app but I can't use it because its results are so patently wrong in my case.

Problem appears to goes away when I leave Accepted out of the State list, but to be honest without knowing how things are calculated it's hard for me to stand by the chart in front of management and ask for resources. It might be nice to have a very simple document describing how the points are identified and how their cycle time is counted, based on the states and Completed state choices..


refactor: encapsulate some globals 3

Continuing https://github.com/D-Programming-Language/dmd/pull/5066

Auto-merge toggled on


Enable private repos

Would be good.

Works for personal private repos, but the token doesn't seem to apply to private repositories for organizations you're involved in.

Dependent on this. https://github.com/kevva/github-repositories/pull/10

Closed in v2, with #10.