Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "VectorCalculationsThread 10"

Issue Description
This was the stdout before the issued happened:
2017-04-21 16:51:55,178  INFO [main] (Nd4jBackend.java:194) - Loaded [CpuBackend] backend
2017-04-21 16:51:55,271  INFO [main] (NativeOpsHolder.java:49) - Number of threads used for NativeOps: 6
2017-04-21 16:51:55,307 DEBUG [main] (Reflections.java:186) - going to scan these urls:
jar:file:/musixmatch/jar/MXMDeepLearningDL4J.jar!/
2017-04-21 16:51:55,507  INFO [main] (Reflections.java:229) - Reflections took 197 ms to scan 1 urls, producing 29 keys and 189 values
2017-04-21 16:51:55,698  INFO [main] (Nd4jBlas.java:36) - Number of threads used for BLAS: 6
2017-04-21 16:51:55,705  INFO [main] (DefaultOpExecutioner.java:544) - Backend used: [CPU]; OS: [Linux]
2017-04-21 16:51:55,706  INFO [main] (DefaultOpExecutioner.java:545) - Cores: [12]; Memory: [8.7GB];
2017-04-21 16:51:55,706  INFO [main] (DefaultOpExecutioner.java:546) - Blas vendor: [OPENBLAS]
2017-04-21 16:52:00,456 DEBUG [Thread-10] (WordVectorSerializer.java:2336) - Trying full model restoration...
2017-04-21 16:52:06,403  INFO [Thread-10] (WordVectorSerializer.java:2821) - Projected memory use for model: [18.04 MB]
2017-04-21 16:52:20,539  INFO [Thread-10] (SequenceVectors.java:127) - Starting vocabulary building...
2017-04-21 16:52:20,539 DEBUG [Thread-10] (VocabConstructor.java:180) - Target vocab size before building: [0]
2017-04-21 16:52:20,543 DEBUG [Thread-10] (VocabConstructor.java:197) - Trying source iterator: [0]
2017-04-21 16:52:20,543 DEBUG [Thread-10] (VocabConstructor.java:198) - Target vocab size before building: [0]
2017-04-21 16:52:43,803 DEBUG [Thread-10] (VocabConstructor.java:286) - Wating till all processes stop...
2017-04-21 16:52:43,810 DEBUG [Thread-10] (VocabConstructor.java:296) - Vocab size before truncation: [70211],  NumWords: [5654882], sequences parsed: [150], counter: [5654739]
2017-04-21 16:52:43,874 DEBUG [Thread-10] (VocabConstructor.java:389) - Scavenger: Words before: 70211; Words after: 23784;
2017-04-21 16:52:43,874 DEBUG [Thread-10] (VocabConstructor.java:303) - Vocab size after truncation: [23784],  NumWords: [5573770], sequences parsed: [150], counter: [5654739]
2017-04-21 16:52:52,814  INFO [Thread-10] (VocabConstructor.java:372) - Sequences checked: [150], Current vocabulary size: [23784]; Sequences/sec: [4.65];
2017-04-21 16:52:52,815  INFO [Thread-10] (WordVectorSerializer.java:2821) - Projected memory use for model: [54.44 MB]
2017-04-21 16:52:52,884  INFO [Thread-10] (InMemoryLookupTable.java:141) - Initializing syn1...
2017-04-21 16:52:52,915  INFO [Thread-10] (SequenceVectors.java:160) - Building learning algorithms:
2017-04-21 16:52:52,915  INFO [Thread-10] (SequenceVectors.java:168) -           building SequenceLearningAlgorithm: [PV-DBOW]
2017-04-21 16:52:52,917  INFO [Thread-10] (SequenceVectors.java:176) -           building ElementsLearningAlgorithm: [SkipGram]
2017-04-21 16:52:52,917  INFO [Thread-10] (SequenceVectors.java:273) - Starting learning process...

The full stacktrace is here
Exception in thread "VectorCalculationsThread 0" Exception in thread "VectorCalculationsThread 10" Exception in thread "VectorCalculationsThread 1" Exception in thread "VectorCalculationsThread 4" java.lang.OutOfMemoryError: Java heap space
        at org.bytedeco.javacpp.Pointer.<init>(Pointer.java:83)
        at org.bytedeco.javacpp.PointerPointer.<init>(PointerPointer.java:134)

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "VectorCalculationsThread 10"
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:3181)
        at java.util.ArrayList.grow(ArrayList.java:261)
        at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:235)
        at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:227)
        at java.util.ArrayList.add(ArrayList.java:458)
        at org.nd4j.linalg.api.ops.aggregates.impl.AggregateSkipGram.<init>(AggregateSkipGram.java:49)
        at org.nd4j.linalg.api.ops.aggregates.impl.AggregateSkipGram.<init>(AggregateSkipGram.java:21)
        at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.iterateSample(SkipGram.java:273)
        at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.skipGram(SkipGram.java:215)
        at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.learnSequence(SkipGram.java:171)
        at org.deeplearning4j.models.sequencevectors.SequenceVectors.trainSequence(SequenceVectors.java:349)
        at org.deeplearning4j.models.sequencevectors.SequenceVectors$VectorCalculationsThread.run(SequenceVectors.java:1165)
java.lang.OutOfMemoryError: Java heap space
        at org.nd4j.linalg.api.ops.aggregates.BaseAggregate.<init>(BaseAggregate.java:15)
        at org.nd4j.linalg.api.ops.aggregates.impl.AggregateSkipGram.<init>(AggregateSkipGram.java:31)
        at org.nd4j.linalg.api.ops.aggregates.impl.AggregateSkipGram.<init>(AggregateSkipGram.java:21)
        at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.iterateSample(SkipGram.java:273)
        at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.skipGram(SkipGram.java:215)
        at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.learnSequence(SkipGram.java:171)
        at org.deeplearning4j.models.sequencevectors.SequenceVectors.trainSequence(SequenceVectors.java:349)
        at org.deeplearning4j.models.sequencevectors.SequenceVectors$VectorCalculationsThread.run(SequenceVectors.java:1165)
java.lang.OutOfMemoryError: Java heap space
        at java.lang.Double.valueOf(Double.java:519)
        at org.nd4j.linalg.api.ops.aggregates.BaseAggregate.<init>(BaseAggregate.java:20)
        at org.nd4j.linalg.api.ops.aggregates.impl.AggregateSkipGram.<init>(AggregateSkipGram.java:31)
        at org.nd4j.linalg.api.ops.aggregates.impl.AggregateSkipGram.<init>(AggregateSkipGram.java:21)
        at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.iterateSample(SkipGram.java:273)
        at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.skipGram(SkipGram.java:215)
        at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.learnSequence(SkipGram.java:171)
        at org.deeplearning4j.models.sequencevectors.SequenceVectors.trainSequence(SequenceVectors.java:349)
        at org.deeplearning4j.models.sequencevectors.SequenceVectors$VectorCalculationsThread.run(SequenceVectors.java:1165)
Exception in thread "VectorCalculationsThread 8" java.lang.OutOfMemoryError: Java heap space

We are using at launch the env
ENV MALLOC_ARENA_MAX 4
and
-Xmx10000m -XX:+UseParallelGC -XX:-UseGCOverheadLimit
Version Information
Please indicate relevant versions, including, if relevant:

Deeplearning4j version
0.8.0

platform information (OS, etc)
Docker + Ubuntu 16.0.4 LTS

CUDA version, if used
none

NVIDIA driver version, if in use
none

We are aware of an issue of a possible memory leak of the JVM/JRE inside Docker, so this could be related to https://github.com/moby/moby/issues/15020

Can you run your code outside of docker?

@raver119 We are investigating now this, since we have other java libs (CoreNLP) running smoothly within Docker, we are now enabling dump and adding further logging here, please wait.

also, please get rid of 2 this params: -XX:+UseParallelGC -XX:-UseGCOverheadLimit

@raver119 do you mean to avoid to use them?

Yes.

Thanks we are running directly in Ubuntu 16.0.4 LTS, let's see what happens.

any news guys?

This seems to work https://twitter.com/loretoparisi/status/855489627871997952
@loretoparisi Would you mind documenting precisely your memory setup so that we can find this one if it occurs again ? We'd need physical memory available, JVM parameters at launch, and ideally the source Docker image you were using, and if other containers were running on the machine (also,  any unusual cgroup parameters if any).

	So, the physical available memory is 16GB. The cpu is a Intel Xeon with 8 cores. The GPU (not uses in this case) are 2 Nvidia GeForce 1080.

We are testing the run outside Docker right now, please wait we will back with results.
All investigation let to think that this is related to a Docker issue with Java settings, but we are not 100% sure, before getting results from the run in Ubuntu.
	Get Outlook for iOS
_____________________________

From: François Garillot<EMAIL_ADDRESS>Sent: sabato, aprile 22, 2017 4:22 AM
Subject: Re: [deeplearning4j/nd4j] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "VectorCalculationsThread 10" (#1785)
To: deeplearning4j/nd4j<EMAIL_ADDRESS>Cc: Mention<EMAIL_ADDRESS>Loreto Parisi<EMAIL_ADDRESS>This seems to work https://twitter.com/loretoparisi/status/855489627871997952
@loretoparisi Would you mind documenting precisely your memory setup so that we can find this one if it occurs again ? We'd need physical memory available, JVM parameters at launch, and ideally the source Docker image you were using, and if other containers were running on the machine (also,  any unusual cgroup parameters if any).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@huitseeker thanks for your help, we have reported this issue to Docker/moby, since it seems to be related to a combination of factors between Docker, Linux and Java, as several evidences show, and other similar issues referring OOM, memory leaks, GC issues, etc.
I have collected all the info we have at this point, and hopefully get a solution here!

Hi guys,
I run the same code on my computer (MacOS with IntellyJIDEA) without docker and I have the same issues.
12:41:17.153 [main] DEBUG o.d.m.w.wordstore.VocabConstructor - Target vocab size before building: [0]
12:41:17.153 [main] DEBUG o.d.m.w.wordstore.VocabConstructor - Trying source iterator: [0]
12:41:17.153 [main] DEBUG o.d.m.w.wordstore.VocabConstructor - Target vocab size before building: [0]
12:41:43.625 [main] DEBUG o.d.m.w.wordstore.VocabConstructor - Wating till all processes stop...
12:41:43.638 [main] DEBUG o.d.m.w.wordstore.VocabConstructor - Vocab size before truncation: [69546],  NumWords: [5658961], sequences parsed: [150], counter: [5658812]
12:41:43.736 [main] DEBUG o.d.m.w.wordstore.VocabConstructor - Scavenger: Words before: 69546; Words after: 23731;
12:41:43.736 [main] DEBUG o.d.m.w.wordstore.VocabConstructor - Vocab size after truncation: [23731],  NumWords: [5578790], sequences parsed: [150], counter: [5658812]
12:41:55.677 [main] INFO  o.d.m.w.wordstore.VocabConstructor - Sequences checked: [150], Current vocabulary size: [23731]; Sequences/sec: [3.89];
12:41:55.677 [main] INFO  o.d.m.e.loader.WordVectorSerializer - Projected memory use for model: [54.32 MB]
12:41:55.803 [main] INFO  o.d.m.e.inmemory.InMemoryLookupTable - Initializing syn1...
12:41:55.871 [main] INFO  o.d.m.s.SequenceVectors - Building learning algorithms:
12:41:55.871 [main] INFO  o.d.m.s.SequenceVectors -           building SequenceLearningAlgorithm: [PV-DBOW]
12:41:55.873 [main] INFO  o.d.m.s.SequenceVectors -           building ElementsLearningAlgorithm: [SkipGram]
12:41:55.873 [main] INFO  o.d.m.s.SequenceVectors - Starting learning process...
Exception in thread "VectorCalculationsThread 1" java.lang.OutOfMemoryError: GC overhead limit exceeded
	at org.bytedeco.javacpp.Pointer.<init>(Pointer.java:83)
	at org.bytedeco.javacpp.FloatPointer.<init>(FloatPointer.java:84)
	at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(NativeOpExecutioner.java:853)
	at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(NativeOpExecutioner.java:916)
	at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.learnSequence(SkipGram.java:176)
	at org.deeplearning4j.models.sequencevectors.SequenceVectors.trainSequence(SequenceVectors.java:349)
	at org.deeplearning4j.models.sequencevectors.SequenceVectors$VectorCalculationsThread.run(SequenceVectors.java:1165)
Exception in thread "VectorCalculationsThread 2" Exception in thread "VectorCalculationsThread 0" java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.Arrays.copyOf(Arrays.java:3181)
	at java.util.ArrayList.grow(ArrayList.java:261)
	at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:235)
	at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:227)
	at java.util.ArrayList.add(ArrayList.java:458)
	at org.nd4j.linalg.api.ops.aggregates.impl.AggregateSkipGram.<init>(AggregateSkipGram.java:49)
	at org.nd4j.linalg.api.ops.aggregates.impl.AggregateSkipGram.<init>(AggregateSkipGram.java:21)
	at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.iterateSample(SkipGram.java:273)
	at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.skipGram(SkipGram.java:215)
	at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.learnSequence(SkipGram.java:171)
	at org.deeplearning4j.models.sequencevectors.SequenceVectors.trainSequence(SequenceVectors.java:349)
	at org.deeplearning4j.models.sequencevectors.SequenceVectors$VectorCalculationsThread.run(SequenceVectors.java:1165)
Exception in thread "VectorCalculationsThread 3" java.lang.OutOfMemoryError: GC overhead limit exceeded
	at org.nd4j.linalg.api.ops.aggregates.BaseAggregate.<init>(BaseAggregate.java:17)
	at org.nd4j.linalg.api.ops.aggregates.impl.AggregateSkipGram.<init>(AggregateSkipGram.java:31)
	at org.nd4j.linalg.api.ops.aggregates.impl.AggregateSkipGram.<init>(AggregateSkipGram.java:21)
	at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.iterateSample(SkipGram.java:273)
	at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.skipGram(SkipGram.java:215)
	at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.learnSequence(SkipGram.java:171)
	at org.deeplearning4j.models.sequencevectors.SequenceVectors.trainSequence(SequenceVectors.java:349)
	at org.deeplearning4j.models.sequencevectors.SequenceVectors$VectorCalculationsThread.run(SequenceVectors.java:1165)
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(NativeOpExecutioner.java:867)
	at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.exec(NativeOpExecutioner.java:916)
	at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.learnSequence(SkipGram.java:176)
	at org.deeplearning4j.models.sequencevectors.SequenceVectors.trainSequence(SequenceVectors.java:349)
	at org.deeplearning4j.models.sequencevectors.SequenceVectors$VectorCalculationsThread.run(SequenceVectors.java:1165)
13:19:22.022 [main] INFO  o.d.m.s.SequenceVectors - Epoch: [1]; Words vectorized so far: [330431];  Lines vectorized so far: [8]; learningRate: [1.0E-4]
13:19:22.027 [main] INFO  o.d.m.s.SequenceVectors - Time spent on training: 2246154 ms

the process ends, but during training (towards the end) it generates the error.
Ps:

java options: -Xmx6096m
DL4J 0.8.0

So, to be clear i have full picture:
You have 150 documents, that are pretty big, total number of words in them 5658961, that is over 30k words per document.
Is that right?

Yes,
this is a list of bytes for documents:
421444
249191
219447
194504
189205
256930
277615
9213
457026
206989
164136
274170
248763
397965
387589
338711
138704
186301
176580
341769
292862
104474
720548
483972
85291
189924
107998
317284
53175
357864
265972
126917
261711
166198
107776
148389
76166
438136
199964
247937
212330
297942
281270
50708
319681
207567
197229
88466
426991
127050
133561
221187
205460
226542
46298
229513
272800
211847
156433
497760
203695
194232
180754
250869
48900
11909
304941
192749
274922
542491
164059
284845
281407
172305
163607
161062
104781
406640
182510
227050

Adding that if this analysis is correct i.e. 150 documents for a total of 5658961 words
and the exception takes place in the ArrayList while adding:
at java.util.Arrays.copyOf(Arrays.java:3181)
   at java.util.ArrayList.grow(ArrayList.java:261)
   at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:235)
   at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:227)
   at java.util.ArrayList.add(ArrayList.java:458)

this means that something occurs while calling
org.nd4j.linalg.api.ops.aggregates.impl.AggregateSkipGram.<init>(AggregateSkipGram.java:49)

like it cannot hold a document of that size?

Other info:
BEFORE PRE-PROCESS
WORDS NUMBER: 6167000
AVG WORDS NUMBER FOR DOC: 41113

AFTER PRE-PROCESS
TOKENS NUMBER: 5658812
AVG TOKENS NUMBER FOR DOC: 37725

PRE-PROCESS = Tokenizations

Right, logic flaw. Fix incoming, but will be merged slightly later

@raver119 that sounds great! to be clear, was it an issue in the iterateSample ?

Fix applied. Will be merged later.
TL/DR: purge of batches happens at document's end. But if you have huge docuemts - that will lead to huge batches. So, internally batches should be dispatched in such cases.

Thanks for highlighting this issue.

@raver119 ok thanks a lot, waiting the fix then to procede with the complete training! Very glad to help you and come to a solution so quickly!

@raver119 sorry do you have a eta for this fix te be on maven? Thanks a lot for your help guys!

@loretoparisi no - just use snapshots for now. We don't guarantee a release cycle for the open source.

@agibsonccc I perfectly understand, so we can go simply to the master?

@agibsonccc I should use 0.8.1-SNAPSHOT?

Yes, with the snapshot repo: https://github.com/deeplearning4j/dl4j-examples/blob/master/dl4j-examples/pom.xml#L16

Ok, now everything works well :)

Great, thanks for highlighting this issue

It may be the result of glibc...
http://codearcana.com/posts/2016/07/11/arena-leak-in-glibc.html
https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en
https://github.com/cloudfoundry/java-buildpack/issues/320


error: pathspec 'gh-pages' did not match any file(s) known to git

When making a release on a fork this error shows up towards the end just before cleanup. What are the outcomes of this error?

Also, what does -dirty mean?

Reopen, if necessary


Add button to import data library datasets

Instead of just importing the tables when selecting them.

When do you mean? In which moment?

In the dashboard, if you click inside the "box" of the table, (the dark blue area that appears when a dataset is hovered, picture below) the table gets imported.
What about just importing if you click a "import button", or explicitly on the table name at least?  If not, I think that people will start downloading a bunch of tables without noticing, or at least I do it when I test the new dashboard and I navigate through Data Library.

Uhm... that layout is broken! :)
It should get imported when clikcing on the name, not in the box. Anyway, I've an option with a button ready at create-modal-1.2.sketch

Yeah, I just logged into a random staging account with Data Library to get the "blue box" image. :-) If you've changed the import to need a click in the name, that's perfect then. I'll test all this in a couple of hours.

Button or name?

@saleiva, where?

The behaviour has changed, so closing this ticket ;)


Update json-eval expression to use entire json body.

Purpose

Describe the problems, issues, or needs driving this feature/fix and include links to related issues in the following format: Resolves issue1, issue2, etc.

Goals

Describe the solutions that this feature/fix will introduce to resolve the problems described above

Approach

Describe how you are implementing the solutions. Include an animated GIF or screenshot if the change affects the UI (email<EMAIL_ADDRESS>to review all UI text). Include a link to a Markdown file or Google doc if the feature write-up is too long to paste here.

User stories

Summary of user stories addressed by this change>

Release note

Brief description of the new feature or bug fix as it will appear in the release notes

Documentation

Link(s) to product documentation that addresses the changes of this PR. If no doc impact, enter “N/A” plus brief explanation of why there’s no doc impact

Training

Link to the PR for changes to the training content in https://github.com/wso2/WSO2-Training, if applicable

Certification

Type “Sent” when you have provided new/updated certification questions, plus four answers for each question (correct answer highlighted in bold), based on this change. Certification questions/answers should be sent to<EMAIL_ADDRESS>and NOT pasted in this PR. If there is no impact on certification exams, type “N/A” and explain why.

Marketing

Link to drafts of marketing content that will describe and promote this feature, including product page changes, technical articles, blog posts, videos, etc., if applicable

Automation tests

Unit tests

Code coverage information

Integration tests

Details about the test cases and coverage

Security checks

Followed secure coding standards in http://wso2.com/technical-reports/wso2-secure-engineering-guidelines? yes/no
Ran FindSecurityBugs plugin and verified report? yes/no
Confirmed that this PR doesn't commit any keys, passwords, tokens, usernames, or other secrets? yes/no

Samples

Provide high-level details about the samples related to this feature

Related PRs

List any other related PRs

Migrations (if applicable)

Describe migration steps and platforms on which migration has been tested

Test environment

List all JDK versions, operating systems, databases, and browser/versions on which this feature/fix was tested

Learning

Describe the research phase and any blog posts, patterns, libraries, or add-ons you used to solve the problem.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@arunans23 , please confirm this fix.

Contributor License Agreement is signed.

It seems Integration Studio is not supporting json-eval($.) for some reason. Rather than using $.*, I prefer just using $ which will work as well. @oprabin WDYT?

I will test the above and confirm this asap.

Hi @Mariangela,
My bad. If that works, feel good to merge it. I won't be able to test thing for a while.
Thanks.


Added support from thunder.token() from WebPlatformEmbedded (a97e0ef6…

…fcf634ac7da74dcac3bc2732b7e7c069)

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

https://jenkins.code.rdkcentral.com/job/github-triggered/350/ started


Add Google as a social provider

Is your feature request related to a problem? Please describe.
As a developer, I want my users to be able to log in to my system with their google account.
Describe the solution you'd like
We need to follow the RESTful guidelines here: https://developers.google.com/youtube/v3/guides/auth/server-side-web-apps#httprest_1
Describe alternatives you've considered
n/a
Additional context
n/a

using Google as a provider could look like this.
final var googleProviderTemplate = new ExternalIdentityProviderTemplateEntity();
    googleProviderTemplate.setAccessTokenRequestBaseUrl("https://oauth2.googleapis.com/token");
    googleProviderTemplate.setProfileRequestBaseUrl("");
    googleProviderTemplate.setName("google");
    googleProviderTemplate.setIconPath("");
    googleProviderTemplate.setBaseAuthorizationUrl("https://accounts.google.com/o/oauth2/v2/auth");
    entityManager.persist(googleProviderTemplate);


Issue 1542: DirectoryListingService instances to be managed by Spring

Refactored the initialization to WebConfig configuration class.

Pull Request Description
I have found the following references of the DirectoryListingService getting initialized directly in the code.

org.carlspring.strongbox.controllers.BrowseController
org.carlspring.strongbox.controllers.LoggingManagementController

I have refactored these initialization steps to WebConfig and provided these two beans through Spring configuration by referring to them using their names.
This pull request closes #1542
Acceptance Test

[x] Building the code with mvn clean install -Dintegration.tests still works.
[x] Running mvn spring-boot:run in the strongbox-web-core still starts up the application correctly.
[x] Building the code and running the strongbox-distribution from a zip or tar.gz still works.
[x] The tests in the strongbox-web-integration-tests still run properly.

Questions

Does this pull request break backward compatibility?

[ ] Yes
[x] No

Does this pull request require other pull requests to be merged first?

[ ] Yes, please see #...
[x] No

Does this require an update of the documentation?

[ ] Yes, please see strongbox/strongbox-docs#{PR_NUMBER}
[x] No

Hi!
Thank you for your contribution!
Would you mind signing the ICLA, as described in the Contributing page?
Also, please, feel free to join our chat channel, if you'd like to learn more about the project and/or like to find out what else you could help with.
Kind regards,
Martin

Hi!
Thank you for your contribution!
Would you mind signing the ICLA, as described in the Contributing page?
Also, please, feel free to join our chat channel, if you'd like to learn more about the project and/or like to find out what else you could help with.
Kind regards,
Martin

FYI, I have shared the final ICLA document with @carlspring in email.

Thanks for signing the ICLA, @saratbeesa ! :)

Status so far :
Pending tests from strongbox-web-integration-tests. I have tried to run them locally and the test seems to fail during initialization due to environment issues. I am investigating them now.

@carlspring ,
I have run the web-integration-tests locally and they're successful. However, the Jenkins job still fails with some error. I have checked the job status and found that it is the gradle web-integration-tests that are failing.
Can you confirm if these failures are out of scope of this issue ?
With the test run complete, this issue can be merged.

Guys, could you please review this? The tests are now passing.

Guys, could you please review this? The tests are now passing.

@carlspring @saratbeesa unresolved comments left

@sbespalov , could you please confirm that all your comments and change requests have been addressed? Thanks!

Hi guys,
I have done a mistake by pushing latest merge commit to the feature branch instead of my fork/master branch. But as per the coding guidelines this action is not recommended. Hence I created a fresh branch for this bug fix and cherry-picked all the correct commits to the branch.
I created a fresh PR for the same https://github.com/strongbox/strongbox/pull/1556
I request you guys to finalize the review in that PR instead. This PR can be marked obsolete. Sorry for the inconvenience.
Thanks,
Sarat kumar Beesa.
CC: @carlspring , @sbespalov , @fuss86 , @ptirador

Closing in favour of #1556 .


[Graph] Move to the official react-flow-renderer 10.0.0 release

Motivation
We are currently fixed the version of react-flow-rendered to "react-flow-renderer": "10.0.0-next.30",  as the newer version is breaking our graph due to d3select package update, which leads to have two different, conflicting d3select package version to be installed with our app.
We use this library to render our graph.
However it is not an official version and force sticking to it - doesn't allow us to remove old packages and fix some vulnerabilities. Further more it stops us from consuming new features that plan to come into this packge
Goal
Fix our code so updating to latest won't break graph

Included with v0.55.0


Swadge Zine

Summary
Wouldn't it be neat to have a little comic zine on each Swadge, just like Tea Room has?

Sounds like a good use case for resurrecting the markdown renderer! And I think the markdown parsing could actually be moved to be part of asset processing instead of doing it at runtime, so we'd just load up a static markdown tree and the accompanying text blob

Oh, I figured this would be more of an image sideshow

Ahh, I totally missed the "comic" part, woops. One of these days I'll find a problem that can be solved with markdown!!

There was some chatter about a dating sim or text adventure or something, which would be ripe for MD!

Artist wrangling is hard, so I'm gonna shelve this for now.


SAK-30989 Duplicate close button deleted from page

Deleted the second close button from the page

Right now the second button has sense because if you have lot of information the scroll moves the top button up, so if you remove the bottom button and go to the end of the list you have to scroll up again to close the window, so the bottom button is needed too.
I think the right fix for this one is remove both buttons and add some standard "X" button in the top bar on the right corner, and make that bar fixed so the scroll doesn't move it.

Note that your pull request comes from your master branch. You should use a new branch for every feature/bug fix. Please see here:
https://help.github.com/articles/using-pull-requests/


flaky test: TestFileCache_{compiler, wazevo}

We recently are having the flaky seg fault as in

https://github.com/tetratelabs/wazero/actions/runs/7853356149/job/21432882550?pr=2036
https://github.com/tetratelabs/wazero/actions/runs/7856559936/job/21439463102

since the failure is happening for both wazevo and old compiler, I am pretty confident that
this is something to do with life cycle issue around mmap/finalizer in file caches.

relevant: https://github.com/tetratelabs/wazero/pull/1815

Unrelated, but I am noticing that if the function returns early (line 36) then the finalizer is not set.
https://github.com/tetratelabs/wazero/blob/99c057bcb5f3f4311815c884ba7113c8d52b3316/internal/engine/compiler/engine_cache.go#L35-L54
As for the actual issue. I still don't know, but one thing that I notice is that before this commit the original code was a little more defensive see here and especially here
e.g. originally:
	e := cm.executable
	if e == nil {
		return // already released
	}

	// Setting this to nil allows tests to know the correct finalizer function was called.
	cm.executable = nil
	if err := platform.MunmapCodeSegment(e); err != nil {
		// munmap failure cannot recover, and happen asynchronously on the finalizer thread. While finalizer
		// functions can return errors, they are ignored.
		panic(fmt.Errorf("compiler: failed to munmap code segment: %w", err))
	}

and I wonder if the defensive check against nil was actually needed. WDYT @achille-roussel could this be related at all?

My understanding of this code is that we don't need to set the finalizer on values we got from the in-memory table of compiled modules because they have already been attached a finalizer if needed (e.g., if they were obtained from the cache).
I think the current code we have still has the same validation, it's just been moved into the asm.(*CodeSegment).Unmap function https://github.com/tetratelabs/wazero/commit/9780f0f4a019ebf8c657fd74f65ce25e2046e507#diff-7257d290f32f0a9c96c45985d8131d1a37533bfb68bb6cff425cf11d15f25757R66-R75
func (seg *CodeSegment) Unmap() error {
	if seg.code != nil {
		if err := platform.MunmapCodeSegment(seg.code[:cap(seg.code)]); err != nil {
			return err
		}
		seg.code = nil
		seg.size = 0
	}
	return nil
}

I can reproduce the same error (although I don't know if am reproducing the exact same conditions) if I try to releaseCompiledModule(cm) while that module is being executed. Essentially:
		go modInst.ExportedFunction("1").Call(testCtx)
		releaseCompiledModule(cm)

So my guess is the finalizer might have to be guarded to check whether we are unmapping a region that is still in use because of some race condition:

we scheduled a region to be released
the module is no longer being executed/deleted, so eventually the GC will want to collect
the module is retrieved again from the file cache because we reloaded it concurrently
the finalizer is still scheduled to unmap, so it kills the mmap'd region while we are inside it

more observations that might be useful.

The first failure is on macOS amd64 during a normal run of the test suite.
The second failure is on linux amd64 but it is a scratch container. So the test is first compiled and then the container is created.

Now, because tests are compiled by package:
# use CGO_ENABLED=0 to ensure the build does not link against glibc
CGO_ENABLED=0 go test ./internal/integration_test/filecache/... -c -o filecache.test

it means that the executable has run for a relatively short time, thus it can't have accumulated "a lot" of garbage, and that garbage is only related to the tests in that sub-package. So this rules out any interference from "other" tests.
This is how I am running filecache.test similarly to how it runs on CI (case 2)
#Dockerfile
FROM scratch
CMD ["/test"]

For completeness, I am running my tests on Fedora using buildah/podman:
buildah build -t wazero:test  .
podman run --cpus=0.5 --cpuset-cpus=0,1 --memory=300m  -v $PWD/filecache.test:/test --tmpfs /tmp -it localhost/wazero:test /test

(--cpuset-cpus= might need to be configured)
and then I am creating a stresstest on the same cpus
taskset -c 0,1 stress --cpu 4 --vm 4 --vm-hang 1

https://github.com/tetratelabs/wazero/pull/2091 follows up to #2088 and hardens the cache

https://github.com/tetratelabs/wazero/actions/runs/8077516060/job/22067963967?pr=2099 this hasn't been resolved yet I think

it's interesting that in both cases when the failure was in the old compiler it was amd64+macos-12 🧐

I think I got it in https://github.com/tetratelabs/wazero/pull/2102
tl;dr: there's a call_indirect in a spec test that points to a function that does not exist because its module has been collected earlier


msg time iterating over file is seconds, not ticks

1.2.10 docs say:
The time attribute is used in several different ways:
inside a track, it is delta time in ticks. This must be an integer.
in messages yielded from play(), it is delta time in seconds (time elapsed since the last yielded message)

import mido
f = mido.MidiFile('demo/cwm_rhondda.mid')
for m in f.play():
    print(m.time)

shows time in seconds as expected
...
0
0.4739583333333333

f.tracks[0][13]
shows the time in ticks as expected:
Message('note_on', channel=0, note=55, velocity=0, time=455)
for m in f:
    print(m.time) shows the time in seconds, not ticks.

There doesn't appear to be a way to get all messages across all tracks in the proper time sequence as with play while getting the time in ticks. I believe this behavior is broken.

Yes, I believe that the time attribute should be revamped to not mix units, allow conversion to and from each and support both relative and absolute time.
This is something I have had in the back of my mind for quite a while but I haven't looked at it seriously.
Thanks for the reminder and, to anyone, feel free to discuss and suggest ways to improve this area.

I haven't dug into the code but it seems to me that iterating over the file should return the actual messages, and play should preserve its current behavior.  An alternative might be to mirror the time field with a ticks field but my head isn't clear on this right now. It might have the advantage of not breaking anything written against current behavior though.
I don't know how people are using this. I myself am using  for msg in midifile.play(): when I actually play the notes, and trying to do operations building the midifile using for msg in midifile:
(I've got a web app MIDI metronome.  I recently switched from just sending midi messages directly to building a file so I don't have to rebuild my table whenever tempo  or subdivision note or volume changes. Only if  the time signature changes or the swing setting changes do I rebuild the midifile. But since I'm using a midifile now I added the ability to load midi files and play them at tempo. I found that measure/beats were fine but once I added eighth/sixteenths/clock messages things no longer matched the loaded file. I'm not yet sure if this is related to this bug/my handling of the files yet, or something else.)


RX65N RTC Driver added

RTC Driver files and related modifications added to the rx65n.

When you change the headers on all of the new files to the new Apache 2.0 license I will be happy to merge the code.  Sorry to trouble you with this, but these are the rules for new files since NuttX in an Apache project (We will eventually get all of the older, existing file headers changed too).
In general the code looks good.  Thanks!  It is, however, difficult to review large chanages online.  Did you run the code through tools/nxstyle to assure that it meets the coding standard?

Hello @patacongo,
Apache License Header Added to all the new files and the modified files.
Please check and merge if OK.

When you change the headers on all of the new files to the new Apache 2.0 license I will be happy to merge the code. Sorry to trouble you with this, but these are the rules for new files since NuttX in an Apache project (We will eventually get all of the older, existing file headers changed too).
In general the code looks good. Thanks! It is, however, difficult to review large chanages online. Did you run the code through tools/nxstyle to assure that it meets the coding standard?

Yes, we have checked for the coding style.


sound problem on some devices

hello, I have problems with sounds on some devices
sound is loaded like this:
game.load.audio('explodeSoundBomb', ['assets/audio/bomba.wav', 'assets/audio/bomba.mp3']);
this.soundExplosionBomb = game.add.audio('explodeSoundBomb', 1, false);
this sound plays when user hit bomb in game and bomb explodes
and played like this:
this.soundExplosionBomb.play();
I have problem on two devices:
HTC Desire Z running Android 2.3.3 (tested on Dolphin and standard browser)

when I enter new level on my game, background music is playing, and it is playing all the time
bomb explosion sound plays normally at begginig, but after I hit bomb several times (this sound plays several times), sound stops playing at all

Samsung Galaxy S running CyanogenMod Android 4.1.1 (tested on standard browser)

when I enter new level on my game, background music is playing
then I hit bomb and sound of bomb explosion mentioned earlier plays, but it stops background music; since then I hear only bomb explosion sound

I dont know where could be the problem, whether I am doing something wrong, or there is bug in framework or there is problem in operating system. Do you have any ideas ?
Thank you for your help

I also have the problem on Xbox360 (sounds stop after few times) and on Wii U (no sound).

Does it throw any kind of error, or just not play? (should be easy enough to debug Android, less so I expect for xbox/wii u)

Are you using Audio Sprites for this? If not, you probably ought to be - I'm highly doubtful Android 2.x can play multi-channel audio at all, so you're probably running out of sound channels as the previous effects lock onto the old ones. For Android 2.x (and even 4.x in stock browser) I would only ever use 1 single audio file personally.

I've just pushed up some changes to the Sound class which may help in these cases (if the browser can use Web Audio). For legacy audio, see my comment above. I would need actual errors / test cases if the recent changes didn't help.


Make LM preparation working with custom profiles

The LM preparation will get an extreme slow down for custom profiles that exclude e.g. motorways (occurs only for world wide instances). Related to #1835.
Usually we just need to pick one node of the returned subnetwork in LM preparation and use that to calculate the coordinates of the landmarks. But likely because of some old code that tried to make this more robust, we loop over all nodes of this subnetwork, to try again with nodes that couldn't be explored.
Interestingly even with this loop it is fast as we exit the loop as soon as a largish subnetwork was found. But now with the CustomWeighting the Tarjan algorithm incorrectly returns a large subnetwork that is splitted by a oneway motorway (Tarjan only sees access not the weight) and then the initial search takes 30 sec (in the large subnetwork) and then due to the next landmarks searches it gets stuck in the small subnetwork (due to the oneway motorway and the filter just uses access flags) and so it returns too few nodes for the last explorer (explorer.getFromCount) so that the next node of the large subnetwork is tried where the same happens again and again likely until all nodes of the large subnetwork were tried. Somewhere here.
This PR should fix this problem (not yet tried for world wide instances).
Performance seems to be fine.

Very nice you figured this out! Thanks :)

It still does not work in rare cases. The reason is that the custom profile can create islands. E.g. "exclude motorway" can disconnect otherwise accessible roads from the main network. To make Tarjan algo aware of this and handle islands that are partially connect we need to replace FiniteWeighting with a directed EdgeFilter:
EdgeFilter accessFilter = edgeState -> edgeState.get(encoder.getAccessEnc()) && Double.isFinite(weighting.calcEdgeWeight(edgeState, false));

Otherwise the algo won't split e.g. the graph 1-2-3->4-5 into two networks 1-2-3 and 4-5. Which can lead to a problem in the next step where LandmarkExplorer selects landmarks and picks a random start node from a network like 1-2-3 but ends with e.g. 4 as next node and because 4-5 is a network from which 1-2-3 is not reachable and which is also too small it will be skipped and a new start node e.g. 2 is tried and and could fail again and again. Which is the problem here as for the planet-wide import the main network "1-2-3" consists of millions of nodes and so exploration takes half a minute for every node leading to a too long waiting time.
Unfortunately this is not enough to fix the slow down for world wide LM preparation.
The second problem is that LandmarkExplorer is not aware of the created islands (from the customization) and a graph exploration there can cross "island boundaries" leading to the same problem. To fix that we mark the main road network and force the exploration (in LandmarkExplorer) to stay there.
Unfortunately this causes the routing requests to return incorrect approximations. I think there is a bug which was mostly covered until now. Will need to investigate this further.
Likely these problems won't occur when #2273 is done.

To make Tarjan algo aware of this and handle islands that are partially connect we need to replace FiniteWeighting with a directed EdgeFilter:

Yes, currently in master we use : DefaultEdgeFilter.outEdges(accessEnc). Using FiniteWeightFilter would indeed me wrong because it accepts an edge even when only the reverse direction has finite weight.

This finally goes through also for a planet-wide import and I was able to remove a bit more code :)
The minor question is now if we should require the subnetwork EncodedValue and adapt the tests. Or if we should keep it like it currently is (only include subnetwork EV if it exists).
For that let me explain why the subnetwork EncodedValue is used. Because it is not required as we do a Tarjan in LandmarkStorage again. And we could do this TarjanSCC also edge-based and then use the created subnetworks instead of relying on the work of PrepareRoutingSubnetwork (which could even use a different minimum network size).
I tried this but as everything regarding landmarks is still node-based this opened up a rabbit hole (required to change the stored LM weights to edge-based and maybe even the LM query to be clean). IMO this is a separate issue and it is not clear if this would improve something as memory usage could increase (higher edge key than nodes?). So I decided to keep using the node-based Tarjan but use the subnetwork EncodedValue from PrepareRoutingSubnetwork that is also correct regarding turn restrictions. So basically what we did before but now cleaner and based on the Profile not FlagEncoder only.


500 error on initial login?

Hi,
I just did a fresh install of Gitlab on RHEL 6.5 / Puppet 3.8.1 using the module, and I'm getting a 500 error when trying to access /users/sign_in (see below).  I found a lot of references to this type of error for upgrades / database migrations, but less for new installations.  I'm assuming that the "projects" table would be a default (since I don't have any data that I've added to gitlab yet)?
I'll keep digging, but I thought it was worth asking in case you've seen this error before after an install.
The production.log file says:
Started GET "/users/sign_in" for <IP_ADDRESS> at 2015-10-27 17:20:15 -0400
Processing by SessionsController#new as HTML
PG::UndefinedTable: ERROR:  relation "projects" does not exist
LINE 5:                WHERE a.attrelid = '"projects"'::regclass
^
:               SELECT a.attname, format_type(a.atttypid, a.atttypmod),
pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod
FROM pg_attribute a LEFT JOIN pg_attrdef d
ON a.attrelid = d.adrelid AND a.attnum = d.adnum
WHERE a.attrelid = '"projects"'::regclass
AND a.attnum > 0 AND NOT a.attisdropped
ORDER BY a.attnum
Completed 500 Internal Server Error in 15ms (ActiveRecord: 9.7ms)

When I tried to troubleshoot by running gitlab-ci-rake commands, it complained about not being able to load files under /opt/gitlab/etc/gitlab-ci - that directory does not appear to exist.
[root@linvm2346 gitlab-rails]# gitlab-ci-rake db:migrate:status
/usr/bin/gitlab-ci-rake error: could not load /opt/gitlab/etc/gitlab-ci/gitlab-ci-rc
Either you are not allowed to read the file, or it does not exist yet.
You can generate it with:   sudo gitlab-ctl reconfigure
[root@linvm2346 gitlab-rails]# cd /opt/gitlab/etc
[root@linvm2346 etc]# ls -la
total 48
drwxrwxr-x 3 root root  4096 Oct 27 17:01 .
drwxrwxr-x 9 root root  4096 Oct 27 17:01 ..
drwx------ 3 git  root  4096 Oct 27 17:01 gitlab-rails
-rw-rw-r-- 1 root root 35191 Oct 27 08:43 gitlab.rb.template
I tried running gitlab-ctl reconfigure by hand to look for errors, etc., but there weren't any obvious issues (and it still didn't create the gitlab-ci directory).

Just realized why gitlab-ci-rake of course wouldn't work - here's the output of the correct command:
[root@linvm2346 etc]# gitlab-rake db:migrate:status
Schema migrations table does not exist yet.

I figured out that I needed to have the following code run:
if is_hash($postgresql) {
unless $postgresql[enable] {
exec { 'gitlab_setup':
command     => '/bin/echo yes | /usr/bin/gitlab-rake gitlab:setup',
refreshonly => true,
timeout     => 1800,
require     => Exec['gitlab_reconfigure'],
unless      => "/bin/grep complete ${git_data_dir}/postgresql.setup"
}
->
file { "${git_data_dir}/postgresql.setup":
content => 'complete'
}
I don't see anything in the module documentation as to what should be placed into the postgresql hash - I would have tried just using enable => true, but it looks like that would also keep the setup code from firing, which is a bit confusing - am I reading this incorrectly?  Is there a basic postgresql hash that can be used for an install that wants to use the defaults but configure the provided postgres db?
Also, it looks like $git_data_dir should be set to /var/opt/gitlab/postgresql/data (when using the default install location from the omnibus / module)?  It looks like it defaults to undef, which means that the postgresql.setup file won't be created anywhere useful.  The init file doesn't provide much clarification.
I'm happy to make a doc PR after I have this all figured out - I just want to make sure whatever I submit is correct.  Thanks!

Maybe we can ping @robbruce / @sd-robbruce, he did the implementation of this part in the module in https://github.com/vshn/puppet-gitlab/pull/15

As I didn't hear back any news on this issue, I'll close it right now. Feel free to open a new one.

I re-opened this, as I ran into this block of code while working on #159.
I'm going to remove this block entirely, as it's likely to cause problems, (specifically this will cause #195).
Running this command will rebuild authorized keys, and will migrate the db, and neither will be wanted by the admin.
With gitlab 10.4 onward, gitlab has built in support for HA, which has many scenarios where postgres in disabled on a node, but the postgresql hash still has configuration in it allowing the application_role, consul_role, and pgbouncer_role nodes to connect to the postgres database.


Incorrect result after an INSERT violates a UNIQUE constraint

Consider the following statements:
CREATE TABLE t0(c0 INT);
CREATE UNIQUE INDEX i0 ON t0(c0);
INSERT INTO t0(c0) VALUES (1);
INSERT INTO t0(c0) VALUES (1); -- Failed to commit: Constraint: PRIMARY KEY or UNIQUE constraint violated: duplicated key
SELECT * FROM t0 WHERE t0.c0 = 1; -- expected: {1}, actual: {}

Unexpectedly, the SELECT does not fetch a row. There is one row c0=1 present in the table. It seems that the second, failing INSERT triggers this bug; if this statement is removed, the query works as expected.
Detailed output:
SQLite version DuckDB c52fc9ba
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> CREATE TABLE t0(c0 INT);
sqlite> CREATE UNIQUE INDEX i0 ON t0(c0);
sqlite> INSERT INTO t0(c0) VALUES (1);
sqlite> INSERT INTO t0(c0) VALUES (1);
Error: TransactionContext: Failed to commit: Constraint: PRIMARY KEY or UNIQUE constraint violated: duplicated key
sqlite> SELECT * FROM t0 WHERE t0.c0 = 1; -- expected: {1}, actual: {}
sqlite>

I can reproduce this on the latest master (commit c52fc9bafd22e41a5554d3305530ebc1bf8364f9) and the current fix branch (commit 5c4cde52e3c3ddc4ad4d78a7adf0101f7e61861f).

This is likely related to an incorrect cleanup of the index structure after the transaction abort; looks to me like the row is removed from the index where it shouldn't be.

Fixed in 0d29535fe6aefdf2fbe8a7de0775c253f8038a1a, the problem was that when the transaction failed to commit it would try to remove entries from the index again. This behavior is there so the index is properly maintained if a conflict occurs later on, e.g. if we have INSERT (1), (2), (3), with a conflict on (3), we want to remove (1) and (2) again from the index.
However, before it performed this cleanup only by value, and did not correctly consider the row id. This caused the first row to be incorrectly removed from the index in the cleanup of the second insertion.


Develop to next post 1.5.18

Post-release housekeeping.

WARNING  -  A relative path to 'models/dcim/devicerole.md' is included in the
            'nav' configuration, which is not found in the documentation files
WARNING  -  A relative path to 'models/dcim/rackrole.md' is included in the
            'nav' configuration, which is not found in the documentation files
WARNING  -  A relative path to 'models/ipam/aggregate.md' is included in the
            'nav' configuration, which is not found in the documentation files
WARNING  -  A relative path to 'models/ipam/role.md' is included in the 'nav'
            configuration, which is not found in the documentation files

Looks like we have some docs to rip out for no-longer-relevant models.


base_url() Mac OS X

CodeIgniter 3.0.3 - Website release
If base_url is not defined in config it is populated as http://::1 on Mac OS X. Removing that entry from my local host file results in http://<IP_ADDRESS>
Tested version 3.0.2 correctly display base_url as http://localhost:8888

Read the upgrade instructions.

My apologies. Thank you very much, I understand now.


colored text glued to the screen on scrolling

├──<EMAIL_ADDRESS>$ atom -v
0.209.0

use pigments
https://github.com/abe33/atom-pigments

Cool, thanks!


LibRetro.RetroArch.Nightly.64-bit

[x] Have you signed the Contributor License Agreement?
[x] Have you checked that there aren't other open pull requests for the same manifest update/change?

[x] Does your manifest conform to the 1.1 schema?

Note: <path> is the name of the directory containing the manifest you're submitting.

I have this PR open but I decided to go with separate architecture PR for future proof. While that PR open but fails installer HASH validation, this PR uses same links and everything but doesn't fail installer hash on my local environment. So I'm fine with either one of these PR get merged so I can ninja release on our Stable so I can make PR for stable version.
Microsoft Reviewers: Open in CodeFlow

Hi @ItzLevvie this could be just me but I wasn't expected this is going to be that much exhausting. Could you review this one? First PR contains both 32bit and 64bit builds but fails hash validation. This PR is using same build but only for 64bit, however fails at installation. Anychance manual approve can be considered at this point? Thanks in advance for your review.

It is still failing the SHA256 validation, but just in a different place
2022-05-01 01:41:48.007 [CLI ] WinGet invoked with arguments: 'install' '--manifest' 'D:\TRANSFER\ab04b65d-2ac9-ec11-950a-000d3a9ed19d\Manifest.yaml' '--accept-package-agreements'
2022-05-01 01:41:48.008 [CLI ] Found subcommand: install
2022-05-01 01:41:48.008 [CLI ] Leaf command to execute: root:install
2022-05-01 01:41:48.021 [CLI ] Executing command: install
2022-05-01 01:41:48.024 [CLI ] Manifest fields: Name [RetroArch Nightly], Version [<IP_ADDRESS>]
2022-05-01 01:41:48.028 [CLI ] Starting installer selection.
2022-05-01 01:41:48.028 [CLI ] Completed installer selection.
2022-05-01 01:41:48.067 [CLI ] Generated temp download path: C:\Users\VALIDA~1\AppData\Local\Temp\WinGet\Libretro.RetroArch.Nightly.64-bit.<IP_ADDRESS>\aab8a8bd4a382011c9417dbc363db98cf2be49ab8d0eff103b93978fe805c64a
2022-05-01 01:41:48.068 [CORE] Downloading to path: C:\Users\VALIDA~1\AppData\Local\Temp\WinGet\Libretro.RetroArch.Nightly.64-bit.<IP_ADDRESS>\aab8a8bd4a382011c9417dbc363db98cf2be49ab8d0eff103b93978fe805c64a
2022-05-01 01:41:48.068 [CORE] DeliveryOptimization downloading from url: https://buildbot.libretro.com/nightly/windows/x86_64/RetroArch-Win64-setup.exe
2022-05-01 01:41:50.695 [CORE] Download completed.
2022-05-01 01:42:04.504 [CORE] Started applying motw to C:\Users\VALIDA~1\AppData\Local\Temp\WinGet\Libretro.RetroArch.Nightly.64-bit.<IP_ADDRESS>\aab8a8bd4a382011c9417dbc363db98cf2be49ab8d0eff103b93978fe805c64a with zone: 3
2022-05-01 01:42:04.515 [CORE] Finished applying motw
2022-05-01 01:42:04.515 [CLI ] Package hash verification failed. SHA256 in manifest [aab8a8bd4a382011c9417dbc363db98cf2be49ab8d0eff103b93978fe805c64a] does not match download [be29572a70d4f90af16c8990d6235fa7c375fa8de1aaa21ecfd3961acb5e2d88]

It is still failing the SHA256 validation, but just in a different place
2022-05-01 01:41:48.007 [CLI ] WinGet invoked with arguments: 'install' '--manifest' 'D:\TRANSFER\ab04b65d-2ac9-ec11-950a-000d3a9ed19d\Manifest.yaml' '--accept-package-agreements'
2022-05-01 01:41:48.008 [CLI ] Found subcommand: install
2022-05-01 01:41:48.008 [CLI ] Leaf command to execute: root:install
2022-05-01 01:41:48.021 [CLI ] Executing command: install
2022-05-01 01:41:48.024 [CLI ] Manifest fields: Name [RetroArch Nightly], Version [<IP_ADDRESS>]
2022-05-01 01:41:48.028 [CLI ] Starting installer selection.
2022-05-01 01:41:48.028 [CLI ] Completed installer selection.
2022-05-01 01:41:48.067 [CLI ] Generated temp download path: C:\Users\VALIDA~1\AppData\Local\Temp\WinGet\Libretro.RetroArch.Nightly.64-bit.<IP_ADDRESS>\aab8a8bd4a382011c9417dbc363db98cf2be49ab8d0eff103b93978fe805c64a
2022-05-01 01:41:48.068 [CORE] Downloading to path: C:\Users\VALIDA~1\AppData\Local\Temp\WinGet\Libretro.RetroArch.Nightly.64-bit.<IP_ADDRESS>\aab8a8bd4a382011c9417dbc363db98cf2be49ab8d0eff103b93978fe805c64a
2022-05-01 01:41:48.068 [CORE] DeliveryOptimization downloading from url: https://buildbot.libretro.com/nightly/windows/x86_64/RetroArch-Win64-setup.exe
2022-05-01 01:41:50.695 [CORE] Download completed.
2022-05-01 01:42:04.504 [CORE] Started applying motw to C:\Users\VALIDA~1\AppData\Local\Temp\WinGet\Libretro.RetroArch.Nightly.64-bit.<IP_ADDRESS>\aab8a8bd4a382011c9417dbc363db98cf2be49ab8d0eff103b93978fe805c64a with zone: 3
2022-05-01 01:42:04.515 [CORE] Finished applying motw
2022-05-01 01:42:04.515 [CLI ] Package hash verification failed. SHA256 in manifest [aab8a8bd4a382011c9417dbc363db98cf2be49ab8d0eff103b93978fe805c64a] does not match download [be29572a70d4f90af16c8990d6235fa7c375fa8de1aaa21ecfd3961acb5e2d88]

How can I fix this? Should I just replace current SHA with be29572a70d4f90af16c8990d6235fa7c375fa8de1aaa21ecfd3961acb5e2d88

I’m not sure, because when I download the file I get a hash of 494bbf97e55299f79f89e61a350cf31a76f07c22cb71b4c2cc913e5a57ae2321
Is there a dynamic CDN which changes the file that gets downloaded? Could it be a CDN caching issue?

I’m not sure, because when I download the file I get a hash of 494bbf97e55299f79f89e61a350cf31a76f07c22cb71b4c2cc913e5a57ae2321
Is there a dynamic CDN which changes the file that gets downloaded? Could it be a CDN caching issue?

I also get 494BBF97E55299F79F89E61A350CF31A76F07C22CB71B4C2CC913E5A57AE2321. I'll check CDN later today. I believe there is nothing else we can do at this moment(?)

Yes

With the SHA256 sum, you need to change it to the sum that winget got. What CDN are you using also?

/AzurePipelines run

@wingetbot run

@msftbot I have reviewed the changes. @fpscan needs to fix these for the pull request (PR) to succeed.

The problem is, you cannot use Nightly builds, you would have to change the hash every day...

The problem is, you cannot use Nightly builds, you would have to change the hash every day...

Thank you for your effort. Ah, in this case we are going to ninja release on stable build so we can use stable version in here instead nightly. Is there any roadmap that changes how hash handles for everyday change in future?

@wingetbot run


Core dump SIGILL on Ubuntu arm64/aarch64 on Rpi4.

The current version v2.1.3 rev: 884 runs fine on a Rpi4 on 32-bit Raspberry Pi OS.
Migrating the same install (with same config file) over to another Rpi4 running 64-bit Ubuntu - it core dumps on start:
# uname -a
Linux pi4.my.lan 5.4.0-1047-raspi #52-Ubuntu SMP PREEMPT Wed Nov 24 08:16:38 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

# dpkg -l unifi-poller
||/ Name           Version      Architecture Description
+++-==============-============-============-====================================================================
ii  unifi-poller   2.1.3-884    arm64        Polls a UniFi controller, exports metrics to InfluxDB and Prometheus

# strace /usr/bin/unifi-poller
execve("/usr/bin/unifi-poller", ["/usr/bin/unifi-poller"], 0xffffd72a92b0 /* 38 vars */) = 0
mmap(0xb10000, 3774884, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb10000
--- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPC, si_addr=0xea8214} ---
+++ killed by SIGILL (core dumped) +++
Illegal instruction (core dumped)

Can you elaborate on migrating? Does it work if you install it from a package?

oh it's SIGILL. This is a weird issue that comes up on some arm64 systems. Running the 32 bit version (armhf) will work.

Yeah, I was already running the packaged version.
Tried the 32bit one as you suggested and it worked.  Thanks for that.
For those with a similar issue before you can install it you need to run:
dpkg --add-architecture armhf
Also, I think there's a bug - if you explicitely state "debug = false" in the config file, it stil outputs lots of info to syslog.
If you comment out all 'debug' statements, then the default works and all is nice and quiet.
So, I think if "debug" is found in the config file at all, it defaults to debug=true.

If debug is set true in the config or environment it will enable the extra output. Thanks for sharing your fix!

The bug is:

if debug is set at all (to either 'true' or 'false') in the config file, it defaults to 'on'.
You have to comment it out to set it to 'off'.

I'm not able to reproduce that behavior.

Yeah, after a reboot I'm not able to reproduce it anymore either.  Odd...

I think I found another fix/workaround for the SIGILL on arm64 platforms.
sudo apt install upx || sudo yum install upx
sudo upx -d /usr/bin/unifi-poller

Nice one, David - it works!!
I guess I'll have to remember to uncompress the binary again after each upgrade.. ;-)
FYI: on an Ubuntu system the package to install is "upx-ucl"
Thanks again!


[TPC-H] Dask query 19

Part of https://github.com/coiled/dask-engineering/issues/234

Failures unrelated

FAILED tests/tpch/test_correctness.py::test_dask_results[9] - distributed.scheduler.KilledWorker: Attempted to run task ('readparquet-fused-chunk-03c3a346f7973a36c8db77225bf8e915', 1) on 4 different workers, but all those workers died while running it. The last worker that attempt to run the task was tcp://<IP_ADDRESS>:33209. Inspecting worker logs is often a good next step to diagnose what went wrong. For more information see https://distributed.dask.org/en/stable/killed.html.
FAILED tests/tpch/test_dask.py::test_query_9 - distributed.scheduler.KilledWorker: Attempted to run task ('readparquet-bec6f578be4fe0e15801f53f9dbeb108', 5) on 4 different workers, but all those workers died while running it. The last worker that attempt to run the task was tls://<IP_ADDRESS>:45841. Inspecting worker logs is often a good next step to diagnose what went wrong. For more information see https://distributed.dask.org/en/stable/killed.html.
FAILED tests/tpch/test_dask.py::test_query_10 - RuntimeError: shuffle_barrier failed during shuffle b7a495806f2cd27b1a3444c0db304e59


intero fails to start

Created new project with
stack new intero-demo

Opened intero-demo/app/Main.hs
Emacs buffer spit out
Usage: stack ghci [TARGET] [--ghc-options OPTION] [--with-ghc GHC] [--no-load]
                  [--package ARG] [--main-is TARGET]
  Run ghci in the context of project(s) (experimental)

---

This is the buffer where Emacs talks to intero. It's normally hidden,
but a problem occcured.

TROUBLESHOOTING

It may be obvious if there is some text above this message
indicating a problem.

The process ended. Here is the reason that Emacs gives us:

  exited abnormally with code 1

For troubleshooting purposes, here are the arguments used to launch intero:

  stack ghci --with-ghc intero "--docker-run-args=--interactive=true --tty=false" --no-build --no-load --ghci-options -odir=/var/folders/9w/zyp9c7rx1hz5bz6pfjx6z2rc0000gn/T/intero67322_Ek --ghci-options -hidir=/var/folders/9w/zyp9c7rx1hz5bz6pfjx6z2rc0000gn/T/intero67322_Ek intero-demo

WHAT TO DO NEXT

If you fixed the problem, just kill this buffer, Intero will make
a fresh one and attempt to start the process automatically as
soon as you start editing code again.

If you are unable to fix the problem, just leave this buffer
around in Emacs and Intero will not attempt to start the process
anymore.

You can always run M-x intero-restart to make it try again.

Trying to run that same command from command line
[intero-demo]$ stack ghci --with-ghc intero "--docker-run-args=--interactive=true --tty=false" --no-build --no-load --ghci-options -odir=/var/folders/9w/zyp9c7rx1hz5bz6pfjx6z2rc0000gn/T/intero67322_Ek --ghci-options -hidir=/var/folders/9w/zyp9c7rx1hz5bz6pfjx6z2rc0000gn/T/intero67322_Ek intero-demo
Usage: stack ghci [TARGET] [--ghc-options OPTION] [--with-ghc GHC] [--no-load]
                  [--package ARG] [--main-is TARGET]
  Run ghci in the context of project(s) (experimental)

Was able to load ghci in the shell after stripping away all command arguments
[intero-demo]$ stack ghci --with-ghc intero intero-demo
* * * * * * * *
The main module to load is ambiguous. Candidates are:
Package `intero-demo' component exe:intero-demo-exe with main-is file: /Users/adams/workspace_opensource/haskell-trash/intero-demo/app/Main.hs
Package `intero-demo' component test:intero-demo-test with main-is file: /Users/adams/workspace_opensource/haskell-trash/intero-demo/test/Spec.hs
None will be loaded. You can specify which one to pick by:
 1) Specifying targets to stack ghci e.g. stack ghci intero-demo:exe:intero-demo-exe
 2) Specifying what the main is e.g. stack ghci --main-is intero-demo:exe:intero-demo-exe
* * * * * * * *
Configuring GHCi with the following packages: intero-demo
Intero 0.1.15 (GHC 7.10.3)
Type :intro and press enter for an introduction of the standard commands.

[1 of 1] Compiling Lib              ( /Users/adams/workspace_opensource/haskell-trash/intero-demo/src/Lib.hs, interpreted )
Ok, modules loaded: Lib.
Collecting type info for 1 module(s) ...
*Lib>

stack.yaml:
resolver: lts-6.1
packages:
- '.'
extra-deps: []
flags: {}

User fail, I had a really old version of stack in my path.

Another case where https://github.com/commercialhaskell/intero/issues/217 would have been helpful!


Python: improve determinism

Issue description

Python 3 interpreters are not build deterministic (http://hydra.nixos.org/build/44710243#tabs-buildsteps)
Python packages are not build determinic (should be addressed by 2bcd41b9ffc629c7c9df402b7a39700340fdf52e)

Related issue: https://github.com/NixOS/nixpkgs/pull/2281

We might do some shopping at Debian

2.7: https://sources.debian.net/src/python2.7/2.7.12-7/debian/patches/
3.4: https://sources.debian.net/src/python3.4/3.4.2-1/debian/patches/
3.5: https://sources.debian.net/src/python3.5/3.5.2-8/debian/patches/
3.6: https://sources.debian.net/src/python3.6/3.6.0~rc1-1/debian/patches/

Hasn't this been fixed?

yep, it has! At least, for most interpreters.

There's a PR to incorporate in CPython some of the modifications I made to our CPython.
https://github.com/python/cpython/pull/296


Bumped to 1.3.5

As simple as that, our jenkins pipeline is currently blocked as we added 2.5 ruby support and we don't want to go back.
FYI @ohadlevy @witlessbird the commit we need is: https://github.com/svenfuchs/safemode/commit/65fcbbf933821ce1f5b5a0117b81e1122eb8d937
Thanks!

Merged, thanks @lzap

https://rubygems.org/gems/safemode shows 1.3.5 as a release from January, was there some problem back then?


Please help with the version number of python you are using

Please help with the version number of python you are using.      My installation with python=3.10.6 always fails

I use 3.8.5.

can python3.11 run the program?i failed when trying it

Always have a error when i try to install the list

Not sure about this. Downgrade python to 3.10 or 3.8 if convenient.

Google Colab seems impossible for this then, been at ut for some hours now. Its either problems here:
/content/Inpaint-Anything Traceback (most recent call last):   File "/content/Inpaint-Anything/remove_anything.py", line 9, in <module>     from lama_inpaint import inpaint_img_with_lama   File "/content/Inpaint-Anything/lama_inpaint.py", line 19, in <module>     from saicinpainting.evaluation.utils import move_to_device   File "/content/Inpaint-Anything/lama/saicinpainting/evaluation/__init__.py", line 6, in <module>     from saicinpainting.evaluation.losses.base_loss import SSIMScore, LPIPSScore, FIDScore   File "/content/Inpaint-Anything/lama/saicinpainting/evaluation/losses/base_loss.py", line 15, in <module>     from .lpips import PerceptualLoss   File "/content/Inpaint-Anything/lama/saicinpainting/evaluation/losses/lpips.py", line 15, in <module>     from saicinpainting.utils import get_shape   File "/content/Inpaint-Anything/lama/saicinpainting/utils.py", line 12, in <module>     from pytorch_lightning import seed_everything   File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/__init__.py", line 28, in <module>     from pytorch_lightning import metrics  # noqa: E402   File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/metrics/__init__.py", line 14, in <module>     from pytorch_lightning.metrics.classification import (  # noqa: F401   File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/metrics/classification/__init__.py", line 14, in <module>     from pytorch_lightning.metrics.classification.accuracy import Accuracy  # noqa: F401   File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/metrics/classification/accuracy.py", line 18, in <module>     from pytorch_lightning.metrics.functional.accuracy import _accuracy_compute, _accuracy_update   File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/metrics/functional/__init__.py", line 14, in <module>     from pytorch_lightning.metrics.functional.accuracy import accuracy  # noqa: F401   File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/metrics/functional/accuracy.py", line 18, in <module>     from pytorch_lightning.metrics.classification.helpers import _input_format_classification, DataType   File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/metrics/classification/helpers.py", line 19, in <module>     from pytorch_lightning.metrics.utils import select_topk, to_onehot   File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/metrics/utils.py", line 18, in <module>     from pytorch_lightning.utilities import rank_zero_warn   File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/utilities/__init__.py", line 18, in <module>     from pytorch_lightning.utilities.apply_func import move_data_to_device  # noqa: F401   File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/utilities/apply_func.py", line 29, in <module>     from torchtext.legacy.data import Batch ModuleNotFoundError: No module named 'torchtext.legacy'
But it's guranteed fail in the lama :
`Run inpainting
Traceback (most recent call last):
File "/content/lama/bin/predict.py", line 32, in
from saicinpainting.training.trainers import load_checkpoint
File "/content/lama/saicinpainting/training/trainers/init.py", line 3, in
from saicinpainting.training.trainers.default import DefaultInpaintingTrainingModule
File "/content/lama/saicinpainting/training/trainers/default.py", line 10, in
from saicinpainting.training.modules.fake_fakes import FakeFakesGenerator
File "/content/lama/saicinpainting/training/modules/fake_fakes.py", line 2, in
from kornia import SamplePadding
ImportError: cannot import name 'SamplePadding' from 'kornia' (/usr/local/lib/python3.10/dist-packages/kornia/init.py)
IndexError                                Traceback (most recent call last)
 in <cell line: 12>()
10
11 plt.rcParams['figure.dpi'] = 200
---> 12 plt.imshow(plt.imread(f"/content/output/{fname.split('.')[1].split('/')[2]}_mask.png"))
13 _=plt.axis('off')
14 _=plt.title('inpainting result')
IndexError: list index out of range`
You need getting past the installing problems though. For me the requrements for lama looks solved like this:
first failing:
ERROR: Failed building wheel for scikit-image ERROR: No matching distribution found for torch==1.8.0+cu111 ERROR: No matching distribution found for opencv-python-headless==<IP_ADDRESS>
Then theese corrects:
!pip install --upgrade pip !pip uninstall --yes --quiet osqp !pip install -U scikit-survival !pip install opencv-contrib-python !pip install opencv-python !pip install opencv-python-headless !pip install pytorch-lightning !pip uninstall kornia !pip install kornia --no-dependencies !pip install hydra-core --upgrade !pip install webdataset
Im thinking downgrading to Python 3.8 won't cut it, but maby it does.


Web Domain to masked DDNS - CSS fails to load

Im not very sure if this is an issue but when using domain that masks a dynamic DNS, the CSS on YOURLS fails to load.
Here is an example:
"dynamicdns.dns.com" points to the YOURLS install folder.
"www.mydomain.com" is forwarded to the dynamic dns address "dynamicdns.dns.com" with masking.
When accessing the YOURLS admin page using "www.mydomain.com", no CSS & Images load.
When accessing the YOURLS admin page using "dynamicdns.dns.com", everything works fine.
This seems like rather weird behaviour so i'm just putting this here.
[EDIT] Install failed because of MySQL keyword limit, so fixed and removed from post.

CSS (and JS and images) are loaded using the domain specified in the config. HTML doesn't understand domain forwarding (ie if you say <img src="domain.com/blah.gif"> the image has to be on domain.com, your browser won't follow redirection to load the image.


Corrected VC compile warning C4090.

looks okay (or harmless and not needed) to me; @dot-asm ?

Looks like VC trying to be a C++ compiler instead of a C compiler ;)

VC tries to be a both compiler...

@rrip, do you consider this a trivial change (I do).  If not, we need you to sign our CLA.

I don't think this is good idea.

Looks like VC trying to be a C++ compiler instead of a C compiler

Well, I'd instead ask what are the exact circumstances when the warning is issued. If C compiler actually requires the kind of cast you think is problematic here, then it doesn't comply with standard. And when it doesn't comply it's called a compiler bug. And then question is if we should work around compiler bugs. Well, we did it earlier, but we never aimed to work around every compiler bug. In other words they have to be evaluated somehow. What we are looking at here is C4090 warning that actually has a record of being buggy, it's even suppressed when OpenSSL itself is built. So what's going on here? It ought to be that warning is issued when user compiles own application code. But what is this warning? It's revolving about const qualifier. So here we have two options. Either user passes const object, or warning is actually unjustified, i.e. it's bug in warning subsystem. But neither seems to justify the modification. Because if warning is actually justified, then user should comply with declaration. And if it's bug, then note that it would effectively mask justified warning.

user should comply with declaration

One can wonder if *_insert should actually accept const void *. But regardless outcome of this query a) suggested way is not the way to solve it; b) it can't be done in 1.1.x(*).
(*) We touched this before, we either make no changes to public headers, or explicitly formulate rigid rules what can be done in letter releases. Well, since no changes would be permitted to public headers till discussion has taken place, so for now there is no or alternative.

We didn't say no changes to public headers, we said no incompatible changes.  But since you feel strongly that this is wrong, and you know more than I do, I remove my approval.

We didn't say no changes to public headers, we said no incompatible changes.

Right, my formulation was a bit sloppy. But so is the quoted one :-) What was said that till we formulate rigid rules for modifications of prototypes in public headers, we tolerate exclusively additions. Well, naturally even additions has to be compatible, e.g. we shouldn't tolerate masking #if or #undefs...
However! One can argue that suggested modification is compatible. After all is doesn't modify any public prototype, only inline implementation. But one again, what is the actual problem? If it's compiler bug, then why are we risking masking legitimate warnings? If it's legitimate warning, then (given the fact that it's C4090 warning) it ought to be application problem...

The problem is whenever lhash.h is included into an application, the warning shows up, regardless if you actually use lh_##type##_insert() function or not. When the macro DEFINE_LHASH_OF is expanded because of the line DEFINE_LHASH_OF(OPENSSL_CSTRING); the warning pops up all over an application that would normally compile warning-free.  The warning occurs because OPENSSL_LH_insert requires a non-const parameter "data", and a const parameter is being passed in because OPENSSL_CSTRING is a const.
The underlying issue is whether the "void *data" value in structure lhash_node_st should be changed to "const void *data"? Then the OPENSSL_LH_insert function could be changed to const void *data. But that would generate 5 valid compiler warnings and would require OPENSSL_LH_DOALL_FUNC and OPENSSL_LH_DOALL_FUNCARG definitions to be changed to include const.

The warning occurs because OPENSSL_LH_insert requires a non-const parameter "data", and a const parameter is being passed in because OPENSSL_CSTRING is a const.

In other words it's legitimate warning(*). But in either case suggested solution is not right. I mean casting away const is not right solution, never is, it's inappropriate to break that kind of contract. Simplest example. Compiler is free to place constant object to write-protected region, and it would be dead-wrong to pass it to a function that doesn't promise to not modify it.

The underlying issue is whether the "void *data" value in structure lhash_node_st should be changed to "const void *data"? Then the OPENSSL_LH_insert function could be changed to const void *data.

That would be right solution, but a) it's not given that it's possible, i.e. OPENSSL_LH_insert might modify data under some circumstances (I haven't looked); b) but even if it is possible, it's not something one can formally do in letter or even minor release (at least not till team discusses and agrees what is acceptable to do in letter or minor release). Alternative can be to introduce additional API, OPENSSL_LH_insert_const perhaps, that would accept const void *data and act accordingly. This can be done in minor release. But once again, casting away const is not right solution. It's very unfortunate, but we seem to be stuck with the warning for now... At least I'd argue that it's better to leave warning as remainder than to mask it, because if masked, real problem won't be fixed.
(*) I'm surprised that other compilers don't catch this. For reference lhash was not the reason for suppressing the warning in question in VC builds. The reason was actual bug in warning subsystem, I don't recall where exactly right now... At any case, idea was to rely on other compilers to catch real problems, and the seem to fail us...

I have submitted another commit with improvements that should satisfy all of the concerns.
The object within struct lhash_node_st has been changed to "const void *data". That is a promise that lhash functions will not modify the data (as they shouldn't because the data can be any type of object).  However the data is returned back as "void *" by lhash functions (without the const). This is similar to the way C library function strchr() function handles "const".
The code compiles without any additional C4090 warnings.

you have to update your cla to include your GH address or change the author on these commits to be<EMAIL_ADDRESS>Sorry to be a nuisance.

Considering all that @dot-asm has said, I'm not sure I see the new solution as distinctly better.  The troube is that there's no direct correlation between the (possibly really const) stuff that's being inserted into the lhash and the stuff (possibly const, but just as possibly non-const) that's retrieved from it.
Refering to strchr() doesn't change this, because with strchr(), there's a direct correlation between what goes in and what comes out.  There's no in between storing of values, and that makes it very different.
I'm leaning toward the same conclusion as @dot-asm, that this warning is best left there as a reminder, rather than being hidden away with ugly and potentially dangerous casts.

We have also stumbled across this warning in our project. We traced the problem back to a MSVC compiler problem. Here you can see the related Microsoft Connect issue.
I believe that the original code should compile without warnings on a well-behaving compiler, and I am not confident in that the semantics should be changed only for the sake of getting rid of these warnings.
Wouldn't it be more correct to just disable (pragma) this warning specifically on the affected compiler versions?

Yes, we just disabled the pragma in #3420.  Closing this.


Return nicer error message when all ranked abundance ranking values are 0

From microbiomeComputations created by asizemore: VEuPathDB/microbiomeComputations#22
Found when qa-ing the beta site. HMP WGS, ranked abundance box, ranking method median, data Normalized number of ...

The abundances are so low in this collection that i think all of the medians = 0. We need to return a nicer error message to pass to the client, and include a suggestion of what to do (rank by max, for example)
Edit
This will also require

[ ] mockups
[ ] web-eda work to display the message wherever we decide to put it

Update. The error is different now. The user sees that the compute fails, and the rserve log reveals that it's the validObject call that errs because invalid class “VariableMetadata” object: Members must be non-empty for collection variables..
We could add a check on if all median=0 somewhere above, even before the ranking occurs, and then return the message and display that for the user.

So each compute can return a computeMessage. This could display the crucial points like "kicked out 80 samples because we didn't like them" or for the situation here "all medians are 0, nothing to rank".
The question becomes, where does this information go in the viz?? We'd ideally keep it brief, but it's possible it could get longer....
Maybe an element only appears when there is a message? Some little warning thing like "hey the compute sent you a message, please read!"

I feel like if this is a possibility for a particular compute, then that compute plugin should produce a JSON stats file that contains something like:
{
     "status": "no-data",
     "message": "The computation did not produce any usable data.  Please try with different parameters."
}

The visualizations that depend on that compute can hit the stats endpoint up front, check the status, and return whatever response is appropriate to the client (400? 422?) along with the compute message or a custom message.

Bc the request didn't succeed?


Add Default implementation

This PR implements Default for VecCell by simply forwarding to Vec::new(). This allows a VecCell to be used in other default-impl'd types, e.g.
#[derive(Default)]
struct Container {
    vec: VecCell<bool>
}

let container = Container::default();

Oh, I must've forgotten to run the code in clippy


[1.20.1] Crash when viewing a camera from the mod SecurityCraft

Describe the bug
When Very Many Players is installed alongside SecurityCraft, trying to view a camera from SecurityCraft that is far away (and hence needs new chunks to load) crashes the server after a short while
To Reproduce

Install SecurityCraft v1.9.11 and Very Many Players 0.2.0+beta.7.101 on your client
Start the game and join a singleplayer world
Teleport to a far away place (I used 1000 100 1000), place a SecurityCraft camera there and link a SecurityCraft camera monitor to it
Teleport back to spawn
Rightclick the camera monitor and select the linked camera
Observe that the game first hangs and then crashes after a short while

Expected behavior
The game would not crash, and the player's camera would move to the position of the SecurityCraft camera so the player can view the surroundings of that camera
Runtime info (please complete the following information):

OS: Windows 11
Minecraft version: 1.20.1, Forge version 47.2.0
Mod version: 0.2.0+beta.7.101

Crash reports / logs
Crash report: https://pastebin.com/FUmBgExD
Other mods
Should be visible in the crash report; to my knowledge, only SecurityCraft and VMP modify the server in a way that is meaningful to this issue
Checklist

[x] I am using the official version of the mod.
[ ] I tried the latest development version but the issue persists.
[x] I searched for similar open issues and could not find an existing bug report on this.

Try reproduce with this mod installed: https://modrinth.com/mod/uwrad/version/1.1.0+forge

I installed the mod, the crash occurred in the same manner as before.
Log file (crash report should already be included in there): https://pastebin.com/n4jHLJFi

The only way this can happen is due to the use of unsafe threading in other mods (most likely SecurityCraft itself) that have corrupted the state vmp uses.
Please consider reporting this to SecurityCraft as well.

I am one of the developers of SecurityCraft, and after reporting this internally we concluded that, to quote bl4ckscor3, "We're not doing anything off-thread". My personal guess is that this issue could arise from SecurityCraft trying to load and send chunks far away from the player in a faulty/unexpected way, since this crash does not occur when trying to view a SecurityCraft camera that is directly next to the player.

Cannot reproduce the crash.

Alright I apologize for the confusion, the crash I experienced was likely in part caused by me having WorldEdit installed as well (though now I can also no longer reproduce it, my game only starts to hang indefinitely until it runs out of memory with SC, VMP and WorldEdit).
I've also now investigated why mounting the camera with SecurityCraft and VMP, while not crashing, also does not work as expected (the player stays where it was, instead of mounting the camera), and the most likely cause of that is VMP queueing the sending of newly added server entities to the client (see MixinThreadedAnvilChunkStorage#redirectUpdateOnAddEntity in VMP), so when the client wants to mount SecurityCraft's camera entity, it has not yet been sent to the client, and thus this process fails. This is something that can be fixed by SecurityCraft though, by just having the client wait until the camera entity arrives.
So since the actual crash seems to be a weird 3-mod-incompatibility and not directly related to the SC+VMP interaction, feel free to close this issue if you want.


Debugger displays contents of _Uint8List incorrectly (lower nibbles only)

This code works correctly
 Uint8List testList = Uint8List.fromList([0x00, 0x01, 0x0f, 0x10, 0x11, 0x1f, 0xf0, 0xf1, 0xff]);
 if (kDebugMode) {
   print(testList);
 }

Output:
flutter: [0, 1, 15, 16, 17, 31, 240, 241, 255]

but in the iOS debugger, the contents of the list is displayed incorrectly, only the lower nibbles are printed:

% flutter upgrade
Flutter is already up to date on channel stable
Flutter 3.7.0 • channel stable • https://github.com/flutter/flutter.git
Framework • revision b06b8b2710 (8 days ago) • 2023-01-23 16:55:55 -0800
Engine • revision b24591ed32
Tools • Dart 2.19.0 • DevTools 2.20.1

Hi @mh-, I think this might be an issue with the flutter plugin for intellij as I cannot reproduce this with vscode. Do you still face the issue in vscode? If not you might have to file an issue over on the intellij plugin repo

Sorry, I don't have a VSCode setup to try this, only Android Studio - I will file the issue in the other project.


EventHubs components to use Azure SDK for Go ("Track 2 SDK")

The EventHubs components should be refactored to use the Azure SDK for Go (Track 2 SDK)
https://github.com/Azure/azure-sdk-for-go/tree/sdk/messaging/azeventhubs/v0.2.0
(Check for a newer version when working on this issue)
As part of this issue support for AAD auth should be added - see #1103.

Consider adding https://github.com/dapr/components-contrib/issues/2176 (Websocket connection support) with the new SDK.

Ping @ItalyPaleAle  :)

It's on my TODO for next week!


US 01.02.01 - Unpublish Experiment

As an owner, I want to unpublish an experiment.

Done. Maybe remove the Unpublish option in the floating hamburger. But not a big deal if it doesn't get removed since it will only re-overwrite the data.
Also could have an alert dialog for a successful or an error/failure to unpublish.


range ivar_points

If a variable has a lot of unique values and empirical = TRUE, then it can happen, that pd is only calculated for a snippet of the range.

Messed up the closing thing.
See commit 0d6a for the fix


Addition: New Use Case Compute Time

Addition: New Use Case Compute Time, computes the time between each attraction, factoring in start time, end time, and travel time.

Hi guys, please check my PR for any issues. Thanks


save user data faill

Description
  customConfig: {
    userDataDir: path.resolve(process.cwd(),'user_data')
  }

Unable to start after configuration
Full steps to reproduce the issue
    const { browser, page } = await connect({       headless: false,       args: [],       customConfig: {         userDataDir: path.resolve(process.cwd(),'user_data')       },       turnstile: true,       connectOption: {},       disableXvfb: false,       ignoreAllFlags: false     })
Issue Type
Others
Operating System
Mac OS
Do you use Docker?
None

what's the error?

what's the error? i can run the browser smoothly with that configuration

The log did not report any errors, but after logging into a certain website, when I closed the project and restarted it, I still did not log in and did not save my information. You should try to find a website that can log in to, whichever one works

it save my information in my case. I try to login to app.daily.dev, close the browser and rerun the code, open the page again, and i already logged in.

it save my information in my case. I try to login to app.daily.dev, close the browser and rerun the code, open the page again, and i already logged in.

emmm，I try it again


Append tag name to the default image

This PR fixes an issue where default image did not have a tag attached and OpenShift automatically appended latest tag, which is not correct.
It also moves the default annotation to the tag level to make sure we can make sure a specific tag is used as default

@vpavlin Is there a difference between recommended and default on tags?

That's an interesting point...the main differenc in my mind is that default is set only once and it is the image used when a user does not select anything.
Recommended should be set in one tag for each image to make sure UI can select the right tag when user only picks an image.
Do you want me to add comments for this?

Do you want me to add comments for this?

Up to you, I just need to know which tag the UI should select by default (I assume recommended if there is one, otherwise default, otherwise latest version).


SauceLabs setup

TODOs in subsequent pull requests:

[ ] run tests on IE6,7,8 (currently unsupported by Karma)
[ ] clean up the travis file (develop a karma plugin to limit how many browsers they start at a single time, they have no interest in limiting based on IRC chats)

This is a continuation and resolves #1835
Do not merge until travis passes (I'll work on this again tongiht)

IT WORKS! https://travis-ci.org/jashkenas/underscore/jobs/51519224

:balloon: This is sorely needed, especially after yesterday. :+1:

Sweet its working after #2069. Everyone happy with this?

Bravo!


Fix integration test "FreeText Editor FreeText rotation must check that the dimensions of a rotated annotations are correct after a font size change"

Introduction date: July 4, 2023
Introduction PR: https://github.com/mozilla/pdf.js/commit/1ce6668a7071178bd08e5b817473eea780cd4d07 (/cc @calixteman)
Type: intermittent (sometimes it works, indicating a possible timing/waiting issue)
OS: Windows
Logs: http://<IP_ADDRESS>:8877/d67e98b920493b4/output.txt
Traceback:
Failures:
1) FreeText Editor FreeText rotation must check that the dimensions of a rotated annotations are correct after a font size change
  Message:
    In firefox: Expected false to equal true.
  Stack:
        at <Jasmine>
        at C:\pdfjs\botio-files-pdfjs\private\d67e98b920493b4\test\integration\freetext_editor_spec.js:1341:14
        at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
        at async Promise.all (index 0)
        at async UserContext.<anonymous> (C:\pdfjs\botio-files-pdfjs\private\d67e98b920493b4\test\integration\freetext_editor_spec.js:1273:7)

@calixteman Was this completely fixed by PR #17036, or is there more work to do here?

Normally the mentioned PR fixed it.


testing installation: error

running through installation as described in Readme on 2018-04-16 on my Mac:
pip3 install gnpy --> works
Traceback (most recent call last):
ggrammel-mbp:gnpy ggrammel$ python examples/transmission_main_example.py
File "examples/transmission_main_example.py", line 106, in
exit(main(args))
File "examples/transmission_main_example.py", line 41, in main
network = network_from_json(json_data)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gnpy/core/network.py", line 29, in network_from_json
g.add_node(getattr(elements, el_config['type'])(el_config))
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gnpy/core/elements.py", line 111, in init
self.dispersion = self.params.dispersion  #s/m/m
AttributeError: 'ConfigStruct' object has no attribute 'dispersion'

Checking the timestamps of files and folders and conclude those were not in sync with the timestamps on current master. So this appears to be an error caused by an installation or configuration issue. probably something in the Radme is missing.

Procedure doesn't seem to work if an older GNPY repository is lingering on the same computer. README may be misleading.

does pip install gnpy install also /gnpy/examples?
if so, where?
if not, README is misleading as it suggests examples could be run by simply installing gnpy


Gutter width seems to always be 30px

Hey,
I've been checking out this library for a new project I'm working on. I've got the following:
// other imports
import { Theme, ThemeProvider } from 'react-rasta';

const theme: Theme = {
  gutterWidth: 20,
}

export const App = () => (
  <Provider>
    <ThemeProvider theme={theme}>
      <Router>
        <Routes />
      </Router>
    </ThemeProvider>
  </Provider>
)

However, the gutter width seems to stay 30px in my app:

Am I doing something wrong?

Thank you for reporting. Fixed in version 1.0.2.


Is this production ready?

Hi @daviddias ,
Is the code base production ready? Also i could not find a way to support media calls (audio, video). Is repo not meant for media calls?
Thanks in advance

Hi @anandarivu! What are you trying to build? This code was built for Research for my M.Sc thesis and it met its purpose (transferring data and computation over an overlay network), i twas not design for calls. Read more about it at:

http://daviddias.me/blog/webrtc-explorer-2-0-0-alpha-release/
http://daviddias.me/blog/enter-webrtc-explorer/
http://daviddias.me/blog/webrtc-ring/

Closing this issue for now. Let me know if any other questions arise :)


Issues of Extract operation in preview

There are some issues with the extract operation in the current preview:

the POI to be placed does not always follow the mouse pointer directly, but often with a random offset.
the POI to be placed is invisible while the mouse is over the toolbar. The user might assume the operation is already complete.
when pressing ESC before placing the POI this results in 2 changes to be saved but no undo available.
the favorites should not be shown before the operation is complete.
the safe button should better be replaced with cancel (like the drawing mode) until the POI is placed.

I've noticed some of these issues and I think my preferred solution would be to instantly place the point in the middle of the feature (or as close as possible in-frame). The user can then move the point as needed. This seems simpler than building out a workflow, plus it's easier on touchscreens.

my preferred solution would be to instantly place the point in the middle of the feature

I made it work like this. It's definitely simpler, consistent with the related Combine and Downgrade operations, and touchscreen friendly. Plus it automatically puts points in the center of areas, so mappers won't have to worry about placing points perfectly and can oftentimes skip moving them altogether.
I also realized that the "Extract this point" validation fix for point-as-vertex warnings already works likes this, so there's precedent within iD. It's good making them consistent.


clSparse build fails on K40 system or any NVidia systems

The build fails when trying to look for OpenCL version 1.2 not found,

The driver supports 1.2, but the problem is with the header files.
@kvaragan You can download OpenCL 1.2 headers from Khronos to fix the builds.

@kvaragan I think you can close this issue; it's resolved right?

I have to check this issue, then I will close it, till now I have not used clSparse on NV, I have been only using cuSPARSE.


Can't add an account with an existing password via Manage Windows Credentials

This is what happens when I try:

If I try again after this, I get VS crashing.

Ah, the ':' in the project name is causing trouble. This will affect users using the organization. Will fix right away.

There's a fix for this in review.


NameService: minor improvements, part 1

Refactor code a bit and make it closer to https://datatracker.ietf.org/doc/html/rfc1034.

Just to give some background, we have an NNS contract in (https://github.com/nspcc-dev/neofs-contract/), it was originally based on C# one, but it diverged over time as we solved some problems related to NeoFS CDN and dApp needs. This contract is used in NeoFS and it also powers dApp examples shown on Aug 30, so it's tested and known to work. We think changes made there are very much relevant for real-life applications and standard public NNS contract will benefit from them.

@shargon, @superboyiii, @erikzhang? This part is rather trivial, but we have more invasive things in the queue, so maybe we can deal with this one first and then discuss other things.


Include the possibility to define uiManagerFactoryFunction

Description

Native SDKs support defining uiManagerFactoryFunction via:

Android: https://cdn.bitmovin.com/player/android/3/docs/player-ui-web/com.bitmovin.player.api.ui/-ui-config/-web-ui/-variant/index.html
iOS: https://cdn.bitmovin.com/player/ios/3/docs/Classes/BitmovinUserInterfaceConfig.html#/:~:text=uiManagerFactoryFunction

On Android this is an important use case, since the default small screen UI doesn't support TV spatial navigation out of the box.
For the public API in this PR I tried to keep it close to what we have on Android.
Right now the sample is integrated in the basic playback sample, meaning that other samples won't have the TV spatial navigation enabled by default. I'm a bit unsure if we should just add it to all the samples, as I would assume this should be enough.
Changes

Update Android Bitmovin player to 3.90.0
Sample: https://github.com/bitmovin/bitmovin-player-react-native/pull/546/commits/9a526c8949672d2cbcd2fa75def2fca450c25379

Checklist

[x] 🗒 CHANGELOG entry

In an ideal world we would have a tv sample, maybe we can add one in a follow up ticked if we have time.