testing: document cap the number of concurrent goroutines with -race

What version of Go are you using (go version)?
1.9.2
GOOS=linux
GOARCH=amd64
What did you do?
package main

import (
	"math/rand"
	"strconv"
	"testing"
	"time"
)

func TestGoRoutines(t *testing.T) {
	for i := 0; i < 1<<15; i++ {
		t.Run(strconv.Itoa(i), func(t *testing.T) {
			t.Parallel()
			time.Sleep(time.Duration(rand.Intn(500000000)))
		})
	}
}

saved as x_test.go and run with go test -race
What did you expect to see?
The tests to pass. Instead,
race: limit on 8192 simultaneously alive goroutines is exceeded, dying

As I understand it, this is likely a technical limitation on Go's race detector. However, it would be nice for the testing package to be aware of this and, if raceenabled, cap the number of parallel tests at something below the 8192 limit (say, 7000?). Perhaps a better way would be a warning, but I know Go doesn't really like warnings.

I don't think is possible.  There is already a limit on the number of tests that are run in parallel, controlled by the -test.parallel option, whose default is runtime.GOMAXPROCS(0).  It doesn't help, though, because by the time we know that the test is going to run in parallel we have already started it.  We can't not start a test, because then we would never finish running all the non-parallel tests.
I'm going to close this as impossible but I'd be willing to hear a proposal for how to implement  it.

@ianlancetaylor fair enough. Could this be a doc fix, though? It was confusing the first time I stumbled upon it. I mean, the docs give an example (which I quoted above) that could easily be used to hit the 8,192 limit.

I'm OK with a doc fix but I'm not sure where it should go.  Maybe in https://blog.golang.org/race-detector ?

Throwing a couple ideas out there:

A comment under theTestGroupedParallel example that warns against using too many parallel tests if race is enabled. Perhaps something like, "Note: because it can be easy to run many parallel tests, care should be taken when the -race flag is toggled, as Go's race detector will kill the program if more than 8,192 goroutines are run simultaneously."

A // BUG(...) comment which warns against it, either in testing or race. (But, it's not really a bug per se...)

An addendum to the blog post is a good idea, but I've a feeling most people won't see it. (Could be mitigated if the text race: limit on 8192 simultaneously alive goroutines is exceeded, dying is added verbatim, since people will likely copy and paste that into Google.)

I'm fine with a comment in TestGroupedParallel, want to send a pull request?

@ianlancetaylor yessir. I'll get to it tonight. Do you want it inside TestGroupedParallel or just above/below it?

I would guess below it, actually.

@ianlancetaylor great that's where I added it. And darn, I should've tried making it with the new GitHub deal instead of git-codereview ;)


feat: deploy to NuGet Repository

Proposed Changes
Deploy to Nuget repository.
Checklist

[ ] Rebased/mergeable
[ ] Tests pass
[ ] Commit messages are conventional
[ ] Sign CLA (if not already signed)

Codecov Report

Merging #20 (847e226) into main (f96b77a) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head 847e226 differs from pull request most recent head 5b22db6. Consider uploading reports for the commit 5b22db6 to get more accurate results

:exclamation: Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.
@@            Coverage Diff            @@
##              main       #20   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           10        10
  Lines          736       736
=========================================
  Hits           736       736


Will support hardhat?

Will support hardhat?
Error:
Truffle Flattener must be run inside a Truffle or Buidler project:
truffle.js, truffle-config.js, buidler.config.js, nor buidler.config.ts found

Sure. Would you send a PR? It should look really similar to this: https://github.com/nomiclabs/truffle-flattener/commit/e1f253871a456283bc9f57c70dde69b91f86a8ac

@alcuadrado  Had send a PR. please check.

@alcuadrado Had send a PR. please check.

https://github.com/nomiclabs/truffle-flattener/pull/66 - this one.

@alcuadrado Had send a PR. please check.

https://github.com/nomiclabs/truffle-flattener/pull/66 - this one.

Hi, has this been fixed? I am still getting this error. Please help. Thank you!

Hi, has this been fixed? I am still getting this error. Please help. Thank you!

Hardhat has a built-in, much more advanced, flattening system.


Case insensitive find packages for NugetV2

I think I figured how to do pull requests :P

Ah yes. That makes sense. Thx


Update logo_outlined.svg

IMPROVED

That R looks really rough. we cant merge that. Also commit messages should be more than "Update X"

sure, there is maybe a better font to use instead of impact

Personally I like Inter Alia

Personally I like InterAlia

This is cool

We have a new logo now. We need to ask the designer to see if we can get an svg and high rez png with transparency. Then you can use that to create everything else.


(Mainnet Verification) Node uptime remains at 0% despite being initiated a few days ago - Hristo Damianov<EMAIL_ADDRESS>

Uptime stays on 0% since they were started a few days ago
Node:6539 ( Created on : 3/15/24, 11:12 PM)
Node:6540 ( Created on : 3/16/24, 12:43 AM)
Node:6539 Last uptime recorded: Wed Mar 20 2024 09:30:36 GMT+0000
Node:6540 Last uptime recorded: Wed Mar 20 2024 09:31:30 GMT+0000

Update:
Investigation and Solution:
the issue is with getNodeMintingFixupReceipts, the request fails and this is not handled
need to move this function into  inner try block

       GET https://alpha.minting.tfchain.grid.tf/api/v1/node/6539 404 (Not Found)

verified and working on mainnet
V 2.4.2
minting requests response are working now for the nodes giving the same starting time
https://alpha.minting.tfchain.grid.tf/api/v1/node/6539
https://alpha.minting.tfchain.grid.tf/api/v1/node/6540

@Cameron1986labeyrie reply and reopen if the issue is still there but for now the minting for the nodes are working fine

Thank you for the update @khaledyoussef24


Error when field has no values

When an optional field has 0 values, the generated file seems unreadable by parquet-tools, the error I get is can not read class org.apache.parquet.format.PageHeader: Required field 'num_nulls' was not found in serialized data! Struct: DataPageHeaderV2(num_values:10, num_nulls:0, num_rows:10, encoding:PLAIN, definition_levels_byte_length:20, repetition_levels_byte_length:0).
When I forced the num_nulls value to be no less than 1, it started working. I'm not sure if this is an issue with the module or with the way I'm trying to use it, so I'm letting you know.

uh oh. yes, this is a problem with the thriftjs library that we are already working around in a number of places (the problem is that if the value is a literal >0< thriftjs will not acutally encode the field but leave the tag out instead).
will add a regression case and fix for this.

github automatically closed this issue because I pushed 86682e6e66222b73de9467362f993c89b58ece0c. the fix is in 0.8.0


add PropTypes for AutofitImage component

I add some PropTypes for AutofitImage component. Hope it helps :)

Close due to unacitvity


Voice recognition and speech

It would be great if we could use the story-writer using speech. In an extreme scenario, there is no keyboard attached to the computer.
CC @autmoate

This could be done using blablado


I want to use easy ocr on different GPU

I want to use easy ocr on different GPU, is it possible?

Hello,
I have not tried to use it on GPU but the Reader is instantiated with mostly the default arguments.
From their code, it seems that it is supposed to support usage with GPU https://github.com/xavctn/img2table/blob/main/src/img2table/ocr/easyocr.py#L28


Naming of config variables

Hey folks, I just added Flask-S3 to a project, and I think the naming of the config variables ought to be changed to be specific to this module; perhaps with a FLASKS3_ prefix (like how the Flask-Security module does it).
I already have a config var called S3_BUCKET in my project, and now I have S3_BUCKET_NAME.. Which doesn't provide the clearest distinction in the code base.
Just a thought. Interested to see what you think.

:+1:

Unfortunately, this would break backwards compatibility, and as most users probably don't pin their dependency versions, it would break their things.

@SunDwarf how about throwing deprecation warnings, and falling back to old naming structure if new structure is not used? I'd be happy to spend some time over the weekend and send a pull request.
(I also have some variables in my config starting with S3_ but thankfully no conflicts!)

I was thinking of @bool-dev's approach as well (which is how Django handle's this kind of thing). Support both variable names, and show deprecation warnings until the next major release.

I think that's a better idea.

@SunDwarf We definitely can't just change the names of the config variables on everyone! ;)


It occur error when I run 'make test'

[ERROR*][54027] *** stderr:
[ERROR*][54027] File "_build/.jbuilds/test/jbuild", line 28, characters 14-18:
[ERROR*][54027] Error: Library "utop" not found.
[ERROR*][54027] Hint: try: jbuilder external-lib-deps --missing _build/test/infer.bc _build/test/scripts/checkCopyright.bc _build/test/inferunit.bc _build/test/infertop.bc
[ERROR*][54027] make[1]: *** [test] Error 1

You need to install utop (opam install utop). This and more is done by make devsetup as listed in CONTRIBUTING.md.


samples/mgmt/smp_svr: Increase main stack size

Some build variants will fail runtime for certain boards (e.g. nRF52832)
due to stack overflow. Avoid this by increasing the stack size.
Signed-off-by: Håkon Øye Amundsen<EMAIL_ADDRESS>

@hakonfam Should this be included into LTS?

@hakonfam Should this be included into LTS? Also, can you provide PR that does the same here: https://github.com/apache/mynewt-mcumgr

The sample will work on more platforms if it is included into LTS, so if it is low cost to get it in then I would say yes.
I will open a PR there as well.


MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size when inserting 10m records into LMDB

Current Behavior
This may be a 5.0 bug b/c 4.3.10 seemed to work.  Generated 10m records using SP2B data generator tool (https://dbis.informatik.uni-freiburg.de/index.php%3Fproject=SP2B%252Fdownload.php.html)
tried to insert it via REST interface:
curl -X POST -H "Content-type: text/turtle" --data-binary @10m.ttl http://localhost/rdf4j-server/repositories/reponame/statements
Received:
org.eclipse.rdf4j.http.server.ServerHTTPException: org.eclipse.rdf4j.repository.RepositoryException: org.eclipse.rdf4j.sail.SailException: java.io.IOException: MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size
        at org.eclipse.rdf4j.http.server.repository.statements.StatementsController.getAddDataResult(StatementsController.java:420)
        at org.eclipse.rdf4j.http.server.repository.statements.StatementsController.handleRequestInternal(StatementsController.java:121)
        at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:177)
        at org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle(SimpleControllerHandlerAdapter.java:51)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:681)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at com.github.ziplet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:263)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.valves.HealthCheckValve.invoke(HealthCheckValve.java:102)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1787)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.eclipse.rdf4j.repository.RepositoryException: org.eclipse.rdf4j.sail.SailException: java.io.IOException: MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size
        at org.eclipse.rdf4j.repository.sail.SailRepositoryConnection.commit(SailRepositoryConnection.java:228)
        at org.eclipse.rdf4j.http.server.repository.statements.StatementsController.getAddDataResult(StatementsController.java:402)
        ... 36 common frames omitted
Caused by: org.eclipse.rdf4j.sail.SailException: java.io.IOException: MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size
        at org.eclipse.rdf4j.sail.lmdb.LmdbSailStore$LmdbSailSink.approveAll(LmdbSailStore.java:609)
        at org.eclipse.rdf4j.sail.base.Changeset.sinkApproved(Changeset.java:857)
        at org.eclipse.rdf4j.sail.base.SailSourceBranch.flush(SailSourceBranch.java:519)
        at org.eclipse.rdf4j.sail.base.SailSourceBranch.flush(SailSourceBranch.java:484)
        at org.eclipse.rdf4j.sail.base.SailSourceBranch.flush(SailSourceBranch.java:299)
        at org.eclipse.rdf4j.sail.base.SailSourceBranch.autoFlush(SailSourceBranch.java:392)
        at org.eclipse.rdf4j.sail.base.SailSourceBranch$1.close(SailSourceBranch.java:197)
        at org.eclipse.rdf4j.sail.base.SailSourceBranch.flush(SailSourceBranch.java:303)
        at org.eclipse.rdf4j.sail.base.UnionSailSource.flush(UnionSailSource.java:74)
        at org.eclipse.rdf4j.sail.base.SailSourceConnection.commitInternal(SailSourceConnection.java:506)
        at org.eclipse.rdf4j.sail.lmdb.LmdbStoreConnection.commitInternal(LmdbStoreConnection.java:91)
        at org.eclipse.rdf4j.sail.helpers.AbstractSailConnection.commit(AbstractSailConnection.java:524)
        at org.eclipse.rdf4j.repository.sail.SailRepositoryConnection.commit(SailRepositoryConnection.java:226)
        ... 37 common frames omitted
Caused by: java.io.IOException: MDB_BAD_VALSIZE: Unsupported size of key/DB name/data, or wrong DUPFIXED size
        at org.eclipse.rdf4j.sail.lmdb.LmdbUtil.E(LmdbUtil.java:64)
        at org.eclipse.rdf4j.sail.lmdb.ValueStore.lambda$findId$10(ValueStore.java:758)
        at org.eclipse.rdf4j.sail.lmdb.LmdbUtil.readTransaction(LmdbUtil.java:86)
        at org.eclipse.rdf4j.sail.lmdb.ValueStore.readTransaction(ValueStore.java:814)
        at org.eclipse.rdf4j.sail.lmdb.ValueStore.findId(ValueStore.java:662)
        at org.eclipse.rdf4j.sail.lmdb.ValueStore.getId(ValueStore.java:900)
        at org.eclipse.rdf4j.sail.lmdb.ValueStore.storeValue(ValueStore.java:1221)
        at org.eclipse.rdf4j.sail.lmdb.LmdbSailStore$LmdbSailSink.approveAll(LmdbSailStore.java:589)

Expected Behavior
for it to work
Steps To Reproduce

redhat linux
running 5.0 sdsk wars against tomcat 9.0.64
created repo with configs:

#!/bin/bash

REPO_NAME="reponame"
REPO_DESC="reponame with lmdb rdf4j 5.0"
REPO_TRIPLE_SIZE_IN_GB=100
REPO_VALUE_SIZE_IN_MB=100
REPO_TRIPLE_INDEX="spoc,posc,ospc"

REPO_CONFIG=$(cat <<-END

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>.
@prefix rep: <http://www.openrdf.org/config/repository#>.
@prefix sr: <http://www.openrdf.org/config/repository/sail#>.
@prefix sail: <http://www.openrdf.org/config/sail#>.
@prefix ns: <http://www.openrdf.org/config/sail/native#>.
@prefix sb: <http://www.openrdf.org/config/sail/base#>.
@prefix lmdb: <http://rdf4j.org/config/sail/lmdb#>.

[] a rep:Repository ;
   rep:repositoryID "$REPO_NAME" ;
   rdfs:label "$REPO_DESC" ;
   rep:repositoryImpl [
       rep:repositoryType "openrdf:SailRepository" ;
       sr:sailImpl [
           sail:sailType "rdf4j:LmdbStore" ;
           sb:defaultQueryEvaluationMode "STANDARD";
           lmdb:tripleIndexes "$REPO_TRIPLE_INDEX";
           lmdb:tripleDBSize $(($REPO_TRIPLE_SIZE_IN_GB*1024*1024*1024));
           lmdb:valueDBSize $(($REPO_VALUE_SIZE_IN_MB*1024*1024));
           lmdb:autoGrow "true";
       ]
   ].
END

)

echo "$REPO_CONFIG" > data.fil
CMD='curl -d<EMAIL_ADDRESS>-H "Content-Type: text/turtle" -X PUT '"localhost:19440/rdf4j-server/repositories/$REPO_NAME"
eval "$CMD"
rm data.fil

generated 10m triples with sp2bench (see above)
used curl to upload to system

note:
received multiple (4) "Resizing map from xxx to 2xxx" before it failed.
I bumped up valueDBSize to 100GB and 10m worked but 100m failed with same MBD error as above (but I did not see a Resizing message)
Version
5.0
Are you interested in contributing a solution yourself?
None
Anything else?
No response

Might be related: https://github.com/meilisearch/meilisearch/issues/2743

The key size is checked at this stage. What is new is the error check that was not present before.

The only thing that could happen is that the key size is zero:
https://github.com/LMDB/lmdb/blob/ddd0a773e2f44d38e4e31ec9ed81af81f4e4ccbb/libraries/liblmdb/mdb.c#L7129
That supports my reasoning above. A quick fix would be to either removed the additional E(...) check around the get invocation and or to check that hashLength is smaller than hashIdBb size.
@hmottestad I don't have access to a Computer in the next days. Could you support this?

The only thing that could happen is that the key size is zero:
https://github.com/LMDB/lmdb/blob/ddd0a773e2f44d38e4e31ec9ed81af81f4e4ccbb/libraries/liblmdb/mdb.c#L7129
That supports my reasoning above. A quick fix would be to either removed the additional E(...) check around the get invocation and or to check that hashLength is smaller than hashIdBb size.
@hmottestad I don't have access to a Computer in the next days. Could you support this?

I think it's probably best if you do. I don't exactly understand the reason for why the key would be zero length to begin with.
I'm going on holiday tomorrow for about a month.  We have a script and also some documentation (which might be a bit out of date).
Script: https://github.com/eclipse-rdf4j/rdf4j/blob/main/scripts/release.sh
Documentation: https://rdf4j.org/documentation/developer/releases/
The script automates pretty much everything in the documentation. Feel free to email me at the RDF4J developer mailing list if you have any issues.

If you can quickly remove the E(...) check today and make a new release then I'll investigate this further in one or two weeks..If this is not possible then @nguyenm100 has to create his own local fix for now.

I can try locally, where is the E()  and hashlength check?

Just remove E(...) on this line:
https://github.com/eclipse-rdf4j/rdf4j/blob/8e9043b4d5a5f87cf61d4bed0cb70c4e5c0fde6b/core/sail/lmdb/src/main/java/org/eclipse/rdf4j/sail/lmdb/ValueStore.java#L758

@hmottestad The length can be zero for the first found hash entry. This only happens if multiple values map to the same hash (collision).

Just to be absolutely sure, we're removing the E(..) wrapping call and leaving the mdb _get() and its return conditional check.

@kenwenzel I could have sworn that I had removed the E(...) calls where the return value was already being checked.

I pulled down tag 5.0.0 and recompiled locally and deployed just to baseline my setup.  Seeing smth different now while trying to load 10m with same config.
No MDB error but logs look funny.  It's resizing as expected but looks like it restarts the resize midway through.  will share some logs below:
Deployed compiled wars and reran load.  What's a little odd was that after this run there was no data in the repo (perhaps it was still flushing??) as count report 0.. I should have waited but I reran the load a second time.
2024-06-28 09:02:35,610 INFO  o.e.r.h.s.r.s.StatementsController.handleRequestInternal(120) [repositories/matt] -  POST data to repository
2024-06-28 09:04:55,069 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 104857600 to 209715200
2024-06-28 09:04:59,698 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 209715200 to 419430400
2024-06-28 09:05:10,426 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 419430400 to 838860800
2024-06-28 09:05:38,545 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 838860800 to<PHONE_NUMBER>
2024-06-28 09:07:46,755 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 104857600 to 209715200
2024-06-28 09:07:49,736 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 209715200 to 419430400
2024-06-28 09:08:16,304 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 419430400 to 838860800
2024-06-28 09:09:02,014 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 838860800 to<PHONE_NUMBER>
Second run same repo (note resizing patterns are different from first).  This time all 10M made it in and i was able to get a count of 10m.
2024-06-28 09:12:09,959 INFO  o.e.r.h.s.r.s.StatementsController.handleRequestInternal(120) [repositories/matt] -  POST data to repository
2024-06-28 09:14:32,116 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 104857600 to 209715200
2024-06-28 09:14:36,652 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 209715200 to 419430400
2024-06-28 09:14:49,309 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 419430400 to 838860800
2024-06-28 09:16:59,401 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 104857600 to 209715200
2024-06-28 09:17:01,588 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 209715200 to 419430400
2024-06-28 09:17:16,172 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 419430400 to 838860800
2024-06-28 09:17:36,665 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 838860800 to<PHONE_NUMBER>
3rd run I deleted the repo and recreated it and reran load.  This time got some strange vocab discrepancy errors which I've not seen before.  All 10M went it w/o incident.
2024-06-28 09:20:39,949 INFO  o.e.r.h.s.r.h.DefaultRepositoryRequestHandler.handleDeleteRepositoryRequest(49) [repositories/matt] -  DELETE request invoked for repository 'matt'
2024-06-28 09:20:40,360 INFO  o.e.r.h.s.r.h.DefaultRepositoryRequestHandler.handleDeleteRepositoryRequest(60) [repositories/matt] -  DELETE request successfully completed
2024-06-28 09:20:40,406 INFO  o.e.r.h.s.r.TupleQueryResultView.logEndOfRequest(109) [http-nio-19440-exec-6] -  Request for query 3392903 is finished
2024-06-28 09:21:03,123 INFO  o.e.r.h.s.r.h.DefaultRepositoryRequestHandler.handleCreateOrUpdateRepositoryRequest(77) [repositories/matt] -  PUT request invoked for repository 'matt'
2024-06-28 09:21:03,125 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:21:03,125 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:21:03,125 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:21:03,125 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:21:03,125 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:21:03,125 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:21:03,126 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:21:04,477 INFO  o.e.r.h.s.r.TupleQueryResultView.logEndOfRequest(109) [http-nio-19440-exec-4] -  Request for query 3392903 is finished
2024-06-28 09:21:10,504 INFO  o.e.r.h.s.r.s.StatementsController.handleRequestInternal(120) [repositories/matt] -  POST data to repository
2024-06-28 09:21:27,023 INFO  o.e.r.h.s.r.TupleQueryResultView.logEndOfRequest(109) [http-nio-19440-exec-6] -  Request for query 3392903 is finished
2024-06-28 09:21:52,559 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 104857600 to 209715200
2024-06-28 09:22:50,902 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 209715200 to 419430400
2024-06-28 09:25:17,749 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 419430400 to 838860800
2024-06-28 09:29:43,045 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 104857600 to 209715200
2024-06-28 09:29:44,236 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 209715200 to 419430400
2024-06-28 09:29:55,104 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 419430400 to 838860800
2024-06-28 09:30:13,736 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 838860800 to<PHONE_NUMBER>
20
fourth run, i deleted repo and restarted service and got different resize results from run 3.
2024-06-28 09:31:15,008 INFO  o.e.r.h.s.r.h.DefaultRepositoryRequestHandler.handleDeleteRepositoryRequest(49) [repositories/matt] -  DELETE request invoked for repository 'matt'
2024-06-28 09:31:15,292 INFO  o.e.r.h.s.r.h.DefaultRepositoryRequestHandler.handleDeleteRepositoryRequest(60) [repositories/matt] -  DELETE request successfully completed
2024-06-28 09:31:15,347 INFO  o.e.r.h.s.r.TupleQueryResultView.logEndOfRequest(109) [http-nio-19440-exec-3] -  Request for query 3392903 is finished
2024-06-28 09:31:58,335 INFO  o.e.r.c.a.l.l.LogbackConfiguration.init(52) [main] -  Logback logging API implementation is configured.
2024-06-28 09:31:59,010 INFO  o.s.w.s.DispatcherServlet.initServletBean(547) [main] -  Completed initialization in 1013 ms
2024-06-28 09:32:00,945 INFO  o.e.r.h.s.r.TupleQueryResultView.logEndOfRequest(109) [http-nio-19440-exec-3] -  Request for query 3392903 is finished
2024-06-28 09:32:01,166 INFO  o.e.r.h.s.r.TupleQueryResultView.logEndOfRequest(109) [http-nio-19440-exec-4] -  Request for query 3392903 is finished
2024-06-28 09:32:03,916 INFO  o.e.r.h.s.r.TupleQueryResultView.logEndOfRequest(109) [http-nio-19440-exec-7] -  Request for query 3392903 is finished
2024-06-28 09:33:56,820 INFO  o.e.r.h.s.r.h.DefaultRepositoryRequestHandler.handleCreateOrUpdateRepositoryRequest(77) [repositories/matt] -  PUT request invoked for repository 'matt'
2024-06-28 09:33:56,877 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:33:56,879 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:33:56,880 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:33:56,889 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:33:56,891 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:33:56,904 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:33:56,906 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:34:00,903 INFO  o.e.r.h.s.r.TupleQueryResultView.logEndOfRequest(109) [http-nio-19440-exec-1] -  Request for query 3392903 is finished
2024-06-28 09:34:02,611 INFO  o.e.r.h.s.r.TupleQueryResultView.logEndOfRequest(109) [http-nio-19440-exec-8] -  Request for query 3392903 is finished
2024-06-28 09:34:02,618 INFO  o.e.r.h.s.r.TupleQueryResultView.logEndOfRequest(109) [http-nio-19440-exec-9] -  Request for query 3392903 is finished
2024-06-28 09:34:02,795 INFO  o.e.r.h.s.r.TupleQueryResultView.logEndOfRequest(109) [repositories/matt] -  Request for query 3392903 is finished
2024-06-28 09:34:02,857 INFO  o.e.r.h.s.r.TupleQueryResultView.logEndOfRequest(109) [http-nio-19440-exec-4] -  Request for query 3392903 is finished
2024-06-28 09:34:24,479 INFO  o.e.r.h.s.r.s.StatementsController.handleRequestInternal(120) [repositories/matt] -  POST data to repository
2024-06-28 09:36:50,215 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 104857600 to 209715200
2024-06-28 09:36:54,246 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 209715200 to 419430400
2024-06-28 09:37:03,712 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 419430400 to 838860800
2024-06-28 09:37:28,338 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 838860800 to<PHONE_NUMBER>
2024-06-28 09:40:14,844 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 104857600 to 209715200
2024-06-28 09:40:16,028 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 209715200 to 419430400
2024-06-28 09:40:26,670 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 419430400 to 838860800
2024-06-28 09:40:44,893 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt] -  Resizing map from 838860800 to<PHONE_NUMBER>
5th run, I created a new repo (matt1) and reran load and got similar results as run 3.
2024-06-28 09:41:46,917 INFO  o.e.r.h.s.r.h.DefaultRepositoryRequestHandler.handleCreateOrUpdateRepositoryRequest(77) [repositories/matt1] -  PUT request invoked for repository 'matt1'
2024-06-28 09:41:46,919 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt1] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:41:46,919 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt1] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:41:46,919 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt1] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:41:46,919 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt1] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:41:46,919 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt1] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:41:46,920 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt1] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:41:46,920 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt1] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 09:41:57,474 INFO  o.e.r.h.s.r.TupleQueryResultView.logEndOfRequest(109) [http-nio-19440-exec-4] -  Request for query 3392903 is finished
2024-06-28 09:41:57,525 INFO  o.e.r.h.s.r.TupleQueryResultView.logEndOfRequest(109) [http-nio-19440-exec-6] -  Request for query 3392903 is finished
2024-06-28 09:42:01,359 INFO  o.e.r.h.s.r.s.StatementsController.handleRequestInternal(120) [repositories/matt1] -  POST data to repository
2024-06-28 09:44:23,543 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from 104857600 to 209715200
2024-06-28 09:44:28,454 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from 209715200 to 419430400
2024-06-28 09:44:42,081 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from 419430400 to 838860800
2024-06-28 09:47:50,892 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from 104857600 to 209715200
2024-06-28 09:47:52,305 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from 209715200 to 419430400
2024-06-28 09:48:03,275 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from 419430400 to 838860800
2024-06-28 09:48:21,710 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from 838860800 to<PHONE_NUMBER>

i will try 100m next to see if i can get the mdb errors with my locally built wars.  probably still makes sense to remote the E() check but odd why no mdb errors didn't happen today for me.

Just to be absolutely sure, we're removing the E(..) wrapping call and leaving the mdb _get() and its return conditional check?

Yes. Only the wrapping E(...) call.

deleted and recreated the repo.  ran through 100m with no errors, but once load was complete, repo size is 0 (verified by file sizes in the repo).  this happened the very first time i tried to load 10m but could not repro with subsequent runs.  this is the first time i've tried to load 100m on this machine.  could it be an async write thing with lmdb?
2024-06-28 10:25:13,553 INFO  o.e.r.h.s.r.h.DefaultRepositoryRequestHandler.handleCreateOrUpdateRepositoryRequest(77) [repositories/matt1] -  PUT request invoked for repository 'matt1'
2024-06-28 10:25:13,554 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt1] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 10:25:13,554 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt1] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 10:25:13,554 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt1] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 10:25:13,554 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt1] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 10:25:13,555 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt1] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 10:25:13,555 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt1] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 10:25:13,555 WARN  o.e.r.m.util.Configurations.logDiscrepancyWarning(238) [repositories/matt1] -  Discrepancy between use of the old and new config vocabulary.
2024-06-28 10:25:25,223 INFO  o.e.r.h.s.r.TupleQueryResultView.logEndOfRequest(109) [http-nio-19440-exec-1] -  Request for query 3392903 is finished
2024-06-28 10:26:56,944 INFO  o.e.r.h.s.r.s.StatementsController.handleRequestInternal(120) [repositories/matt1] -  POST data to repository
2024-06-28 10:29:26,712 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from 104857600 to 209715200
2024-06-28 10:29:31,333 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from 209715200 to 419430400
2024-06-28 10:29:42,136 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from 419430400 to 838860800
2024-06-28 10:30:08,220 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from 838860800 to<PHONE_NUMBER>
2024-06-28 10:37:50,402 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from<PHONE_NUMBER> to<PHONE_NUMBER>
2024-06-28 10:51:43,772 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from<PHONE_NUMBER> to<PHONE_NUMBER>
2024-06-28 11:19:34,307 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from<PHONE_NUMBER> to<PHONE_NUMBER>0
2024-06-28 11:48:52,264 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from 104857600 to 209715200
2024-06-28 11:49:03,677 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from 209715200 to 419430400
2024-06-28 11:49:35,017 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from 419430400 to 838860800
2024-06-28 11:50:46,206 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from 838860800 to<PHONE_NUMBER>
2024-06-28 11:52:09,216 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from<PHONE_NUMBER> to<PHONE_NUMBER>
2024-06-28 11:54:11,548 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from<PHONE_NUMBER> to<PHONE_NUMBER>
2024-06-28 12:10:08,243 INFO  o.e.rdf4j.sail.lmdb.ValueStore.resizeMap(588) [repositories/matt1] -  Resizing map from<PHONE_NUMBER> to<PHONE_NUMBER>0

It seems that it writes the data two times as it resizes the map also a second time. Maybe this is für to the memory overflow. Which isolation level are you using?

I'm not specifying in the curl call (above) so whatever is the default for rdf4j 5.0?

It seems that it writes the data two times as it resizes the map also a second time. Maybe this is due to the memory overflow. Which isolation level are you using?
Please try with isolation level NONE and transactions of size around 100,000 - 500,000 triples each.
However, it should not happen that the repo is empty after inserting the data.

it definitely work when we split the file to 100k each.

@nguyenm100 You could try to increase the size of the chunks to 500,000 or more. By passing transaction-setting__org.eclipse.rdf4j.common.transaction.IsolationLevel=NONE as HTTP parameter the insertion performance should also be improved.

We're in an enterprise env so our standard vms are 32g and goes up from there.
Would be interested in understanding optimized configs for lmdb if you have them. 1b load in an hour is impressive. How much mem did you have? What are your setting for your jvm mx as a percentage of your total mem usually?

@nguyenm100 Are you using SSDs directly attached to the server?
@kenwenzel Could it be that since we don't call the MDB sync method that there is some chance that data isn't written to disk?

The vms are all accessed through ceph. We will look into bare metal at a later date.

@nguyenm100 Are you using SSDs directly attached to the server?
@kenwenzel Could it be that since we don't call the MDB sync method that there is some chance that data isn't written to disk?

@hmottestad It might be the case but 30k TPS seem slow.
@nguyenm100 Does the data set have larger literals?

I meant about the issue when loading all the data in the first transaction where the DB is still empty afterwards.

I meant about the issue when loading all the data in the first transaction where the DB is still empty afterwards.

@hmottestad Ah OK. But then I wonder that reading from the memory overflow model seems to work?!

There are no large literals. All the records can be generated by the sp2b benchmarking framework as stated above

@nguyenm100 Could you share your particular repository configuration and the used parameters for SP2bench?

configuration is provided in the bug (how to reproduce).
download sp2bench from https://dbis.informatik.uni-freiburg.de/index.php%3Fproject=SP2B%252Fdownload.php.html
expand and run ./sp2b_gen -t


Faster build when using -DskipTests

Faster build when using -DskipTests
BEFORE: 1:35 min
AFTER: 0:42 min
Please check the relevant options

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Dependency update
[x] Refactoring
[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] This change requires a documentation update
[x] This change requires execution against OCP (use run tests phrase in comment)

Checklist:

[ ] Example scenarios has been updated / added
[x] Methods and classes used in PR scenarios are meaningful
[x] Commits are well encapsulated and follow the best practices

Remote dev tests are failing

run tests

run tests

Rebased after https://github.com/quarkus-qe/quarkus-test-framework/pull/735 merge

OCP failed in JVM mode on ISPN test which was green in the run before the rebase to pull in update of Apicurio to get rid of the marshaling issues with the latest Quarkus main  (unrelated to the ISPN test).
Triggered OCP JVM run again, but imho good to merge @mjurc

Rebased because of the conflict in pom.xml after latest merge


bump: :editor evil

emacs-evil/evil@b00018bf550f -> emacs-evil/evil@a5fd96dadc44
Bump evil to include replace state map changes.

What changes?

What changes?

Replace state map changes are b00018bf550fbbe1b55165579e6ede973d70f53b..b5c038729958856b0b194a45daf988c42e97bc58, this update also include other commits.

The module was overdue a more comprehensive bump, which I've done in 8603413. Thanks for the reminder!


vym: 2.6.11. -> 2.7.0 and fix qt5

Motivation for this change
New major version of vym was released and the application was affected by the qt5 xcb plugin error.
Things done
Version update and addition of wrapQtAppsHook.

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Notify maintainers
cc @AndersonTorres

Closing in favor of existing PR with these changes (and 2.7.1).  Thanks for submitting, though! Does that PR address the issue you mention?
https://github.com/NixOS/nixpkgs/pull/73483

The mentioned pull Request does not incorporate the wrapQtAppsHook, so the executable probably won't run properly. Also I would like to mention, that version 2.7.1 is a development snapshot of vym.


Remove parameters from TableAttribute constructor and make properties internal

We do not want to support, for v1.0.0, the ability for tables to modify the method names the SDK calls.
We also should investigate if we want the InternalTableAttribute

For documentation sake, we decided to still let users change these method names via extending the TableAttribute class


CUD Import gets slower and slower

We are ingesting messages with the CUD strategy. The first ~100k nodes are created quite fast but then the ingestion gets slower and slower and nearly comes to a stop.
We store the database on a persistent volume even after a restart of the pod the ingestion is still very slow.
I have no idea how to debug the problem. The container has enough CPU and memory (MAX_HEAP set to 40 in the conf but somehow still is set to 12,5g) available.
The debug.log is also not helpful, at least not for me:
2020-11-15 19:50:42.024+0000 WARN [s.k.KafkaEventRouter] [neo4j] The BROKER config `auto.create.topics.enable` is false, the following topics need to be created into the Kafka cluster otherwise the messages will be discarded: [neo4j, neo4j]
2020-11-15 19:50:42.025+0000 INFO [s.StreamsEventRouterAvailabilityListener] [neo4j] Streams Source transaction handler initialised
2020-11-15 19:50:42.296+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=139, gcTime=146, gcCount=1}
2020-11-15 19:50:47.721+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=109, gcTime=121, gcCount=1}
2020-11-15 20:08:08.260+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] [neo4j] Checkpoint triggered by "Scheduled checkpoint for every 15 minutes threshold" @ txId: 181593 checkpoint started...
2020-11-15 20:08:08.347+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] [neo4j] Checkpoint triggered by "Scheduled checkpoint for every 15 minutes threshold" @ txId: 181593 checkpoint completed in 83ms
2020-11-15 20:08:08.358+0000 INFO [o.n.k.i.t.l.p.LogPruningImpl] [neo4j] No log version pruned. The strategy used was '104857600 size'. Last checkpoint was made in log version 0.

We are using the latest version of streams: 4.0.6 and Neo4j Community 4.1.1.

Did you check your indexes/constraints so that they are used in your insert operations?

We don't have any indexes or constraints defined.
As far as I understand these concepts they could only slow things down, don't they?

Not really.
If nodes are to be inserted uniquely with Merge or relationships created by looking up nodes by id the the constraint prevents duplicates and prevents full database scans and does a very fast lookup instead.
The native indexes/constraints for ids affect write performance only minimally.
@conker84 do we auto create those for CUD? Or at least warn if missing on the first occurrence?

I've created an INDEX and the situation seems to be a bit better. At the moment Neo4j is ingesting 30 nodes/sec.
Unfortunatly, this is an order of magnitude to slow for our usecase. We need at least 2000 nodes/sec.
After reading the Architectural guidancedocumentation I suppose that the Pattern/Relationship-Strategy should be the most performant strategy, because Neo4j streams can ingest multiple topics concurrently. Am I right here?
What ingestion strategy is recommended if we have full control over the structure of the events and need very high troughput?

@FrankR85 we need:

the structure of your topic message
the neo4j.conf parameters of neo4j streams

Not really.
If nodes are to be inserted uniquely with Merge or relationships created by looking up nodes by id the the constraint prevents duplicates and prevents full database scans and does a very fast lookup instead.
The native indexes/constraints for ids affect write performance only minimally.
@conker84 do we auto create those for CUD? Or at least warn if missing on the first occurrence?

No we don't the CUD generates a cypher query at runtime so we expect that the indexes/constraints are online before the ingestion process starts

Turned out I forget to create an index for one type of our nodes. Now the ingestion is really fast, thank you both for the pointers! 👍
This issue can be closed. Although the question about the ingestion strategy with the highest throuput remains. 🙂


Allow helm to be used in a genrule

This allows the helm toolchain to be used to produce arbitrary build outputs that the ruleset may not support yet, e.g., generating templates.
This follows the same patterns that bazel-lib uses for jq, yq, etc.: https://github.com/aspect-build/bazel-lib/blob/1697a3275becc29d0c002f6ef18bd63885b4d758/lib/private/jq_toolchain.bzl#L109

By the way @abrisco are you in the Bazel slack group? I may want to make some further contributions to rules_helm but it would be nice to run them past you first.


FLEX-3543 Push to devices now gets properly saved in the database

SonarQube analysis reported 7 issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 FirmwareManagementService.java#L437: The Cyclomatic Complexity of this method "addFirmware" is 13 which is greater than 10 authorized.
 FirmwareManagementService.java#L437: Method has 8 parameters, which is greater than 7 authorized.
 FirmwareManagementService.java#L439: Define and throw a dedicated exception instead of using a generic one.
 FirmwareManagementService.java#L515: Define and throw a dedicated exception instead of using a generic one.
 FirmwareManagementService.java#L530: Method has 8 parameters, which is greater than 7 authorized.
 FirmwareManagementService.java#L577: Define and throw a dedicated exception instead of using a generic one.
 FirmwareManagementService.java#L668: Define and throw a dedicated exception instead of using a generic one.

All is well.

SonarQube analysis reported 7 issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 FirmwareManagementService.java#L437: The Cyclomatic Complexity of this method "addFirmware" is 11 which is greater than 10 authorized.
 FirmwareManagementService.java#L437: Method has 8 parameters, which is greater than 7 authorized.
 FirmwareManagementService.java#L439: Define and throw a dedicated exception instead of using a generic one.
 FirmwareManagementService.java#L510: Define and throw a dedicated exception instead of using a generic one.
 FirmwareManagementService.java#L525: Method has 8 parameters, which is greater than 7 authorized.
 FirmwareManagementService.java#L580: Define and throw a dedicated exception instead of using a generic one.
 FirmwareManagementService.java#L671: Define and throw a dedicated exception instead of using a generic one.

All is well.

SonarQube analysis reported 8 issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 FirmwareManagementService.java#L437: Method has 8 parameters, which is greater than 7 authorized.
 FirmwareManagementService.java#L437: The Cyclomatic Complexity of this method "addFirmware" is 13 which is greater than 10 authorized.
 FirmwareManagementService.java#L439: Define and throw a dedicated exception instead of using a generic one.
 FirmwareManagementService.java#L515: Define and throw a dedicated exception instead of using a generic one.
 FirmwareManagementService.java#L530: Method has 8 parameters, which is greater than 7 authorized.
 FirmwareManagementService.java#L530: The Cyclomatic Complexity of this method "changeFirmware" is 11 which is greater than 10 authorized.
 FirmwareManagementService.java#L587: Define and throw a dedicated exception instead of using a generic one.
 FirmwareManagementService.java#L678: Define and throw a dedicated exception instead of using a generic one.

All is well.

SonarQube analysis reported 7 issues
Note: The following issues were found on lines that were not modified in the pull request. Because these issues can't be reported as line comments, they are summarized here:

 FirmwareManagementService.java#L437: Method has 8 parameters, which is greater than 7 authorized.
 FirmwareManagementService.java#L437: The Cyclomatic Complexity of this method "addFirmware" is 11 which is greater than 10 authorized.
 FirmwareManagementService.java#L439: Define and throw a dedicated exception instead of using a generic one.
 FirmwareManagementService.java#L510: Define and throw a dedicated exception instead of using a generic one.
 FirmwareManagementService.java#L525: Method has 8 parameters, which is greater than 7 authorized.
 FirmwareManagementService.java#L582: Define and throw a dedicated exception instead of using a generic one.
 FirmwareManagementService.java#L673: Define and throw a dedicated exception instead of using a generic one.

All is well.


6LoWPAN: Introduce test vectors and improvements

This PR introduces a new 6LoWPAN test vectors (~60) and fixes some bugs.
Modification in the LoWPAN:

Fixed bug of performing stateful compression with context that has compression flag disable.
Fixed bug of decompression unspecified address "::"
Improve compression of Traffic Class and Flow Label fields (when DSCP is 0, and ECN is not)
Throw parse error when DAC=1 and DAM=00.
Add possibility to compress Unicast-Prefix-based IPv6 Multicast Addresses (however i've added support only if the address match context 0 - for performance reason - but we can change it in the future)

An example of the unit test log:
=== Test name: Extension Header - Hop-by-Hop with PadN2 ===

Expected error -------------- no
UDP present ----------------- no
Extension Headers present --- yes
IP-in-IP present ------------ no
IPHC length ----------------- 13
IPv6 uncompressed offset ---- 48

Expected IPv6 uncompressed packet:
60 00 00 00 00 10 00 40  fd 00 ca fe fa ce 12 34
00 00 00 ff fe 00 00 00  ff 03 00 00 00 00 00 00
00 00 00 00 00 00 00 01  3a 00 6d 02 60 11 01 00
80 00 01 02 03 04 05 06

Expected IPHC compressed frame:
7e 7a 03 00 00 01 e0 3a  04 6d 02 60 11 80 00 01
02 03 04 05 06

Resulted IPHC compressed frame:
7e 7a 03 00 00 01 e0 3a  04 6d 02 60 11 80 00 01
02 03 04 05 06

Resulted IPv6 uncompressed packet:
60 00 00 00 00 10 00 40  fd 00 ca fe fa ce 12 34
00 00 00 ff fe 00 00 00  ff 03 00 00 00 00 00 00
00 00 00 00 00 00 00 01  3a 00 6d 02 60 11 01 00
80 00 01 02 03 04 05 06

PASS

Current coverage is 72.62% (diff: 96.66%)

Merging #947 into master will decrease coverage by 0.68%

@@             master       #947   diff @@
==========================================
  Files           118        117     -1
  Lines         17554      17564    +10
  Methods        2450       2449     -1
  Messages          0          0
  Branches       2120       2121     +1
==========================================
- Hits          12869      12756   -113
- Misses         3988       4115   +127
+ Partials        697        693     -4

Powered by Codecov. Last update 266232e...757fe91

I've added comments as @turon requested and changed IPHC to Iphc and NHC to Nhc in function names.
Additional one more test have been added to fix one corner situation in TC and FL.


API testing  - guest acknowledgement text

Should not be string. Should be Ticket ID. I.e. for my "Azadi TestGuest1" guest booking today the Ticket ID below should be in the Acknowledgement text field. (If you can include the text "Ticket ID: " then great. If not, no worries. )
Ticket ID: zys9i03-2

But there's no one-to-one link between a ticket and a transaction.
Eg. there will be one transaction if you buy 15 tickets.
I will set AcknowledgementText to the ticket id of the first ticket. Hope that's ok?

That works great. thanks
direct: +44 20 3286 6595 | mobile: +44 7837 566 595 | mail: azadi<EMAIL_ADDRESS>Productle Ltd, 3rd Floor, Buckingham House, Buckingham Street Aylesbury, HP20 2LA
From: Samuel Colvin<EMAIL_ADDRESS>Sent: 16 October 2018 11:34
To: samuelcolvin/nosht<EMAIL_ADDRESS>Cc: Azadi Sheridan<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [samuelcolvin/nosht] API testing - guest acknowledgement text (#159)
But there's no one-to-one link between a ticket and a transaction.
Eg. there will be one transaction if you buy 15 tickets.
I will set AcknowledgementText to the ticket id of the first ticket. Hope that's ok?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/samuelcolvin/nosht/issues/159#issuecomment-430187671, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AncozwLTLK92ZLngdbIZCkXp1CB4iAGGks5ulbYegaJpZM4Xc-fN.


CNV performallelefractionsegmentation: tag doc

No example command since there is no WDL for me to reference. FYI @LeeTL1220 @achevali.
tag doc and mention that the tumorHets file is from GetHetCoverage.

Copying comments from closed branch:

No example command since there is no WDL for me to reference. FYI @LeeTL1220 @achevali.

There is no WDL because this tool is experimental.  In general, you should be able to put together example command lines from the integration tests for a tool.

tag doc and mention that the tumorHets file is from GetHetCoverage.

We should probably say that the tumorHets file is from GetBayesianHetCoverage, preferably, but can also come from GetHetCoverage.

@samuelklee I looked into the IntegrationTest and did not see an obvious command structure or references to files in our resource suite except for the file that I give an example of.

The following block from PerformAlleleFractionSegmentationIntegrationTest can be translated into a command line (with a little bit of work):
final String[] arguments = {
            "-" + ExomeStandardArgumentDefinitions.TUMOR_ALLELIC_COUNTS_FILE_SHORT_NAME, snpFile.getAbsolutePath(),
            "-" + PerformAlleleFractionSegmentation.INITIAL_NUM_STATES_SHORT_NAME, Integer.toString(initialNumStates),
            "-" + ExomeStandardArgumentDefinitions.SEGMENT_FILE_SHORT_NAME, outputSegmentFile.getAbsolutePath()
    };


about datasets

Dear Teacher Author,
Hello, I would like to ask about the overall structure of the dataset you used for training or inference. During the process of reproduction, I encountered the following two minor issues, and I suspect that there might be a problem with the structure of my dataset, or there might be an error in my reproduction process. Could you please help me clarify this?

Thank you for paying attention to our work. I can't see the screenshot you provided. Please show your question again.

Thank you for paying attention to our work. I can't see the screenshot you provided. Please show your question again.

Thank you teacher for your reply. I would like to ask what is the organizational structure of the data set. Can you post a picture if it is convenient?

ValueError: too many values to unpack (expected 2)
And what is the reason for this error?

@henrryzh1
May I ask if this code is executable? I'm encountering the same issue, as I also get an error when running ucf_main.py
for example, the dataloader return data,label,name but you only expect data,label
https://github.com/henrryzh1/UR-DMU/blob/master/ucf_test.py#L30

Thank you for paying attention to our work. I can't see the screenshot you provided. Please show your question again.
requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine('SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.10\r\n'))
2%|███▏                                                                                | 51/3000 [07:05<18:03:03, 22.04s/it]
Is this running away?


[Proto-goal]: Define standard hub offerings and a list of extras

Context
The topic of defining hub types, or hub offerings has been touched in various contexts so far, https://github.com/2i2c-org/infrastructure/issues/1798, https://github.com/2i2c-org/infrastructure/issues/1917 and most recently during the team retreat.
I believe this goal qualifies as a cross-team (engineering + partnerships) goal that could and should be tackled across both of these teams for better impact.
Historically, the basehub and daskhub helm charts tried to define the two main hub types that we can support. A basehub is mostly used for educational hubs that we deploy and the daskhub for some of the research ones.
But there are lots of ways we can configure these two charts or add on top of them in order to create hub deployments that serve communities' needs with no way of knowing what's "standard".
I believe we need to intentionally use the deployments we manage so far to identify a couple of types of hubs that qualify as "standard offerings" and what are the extras we are supporting right now.
Outputs
### Done means:
- [ ] We have a list of "standard hubs" and a list of extras that communities can choose from
- [ ] We can deploy the standard hubs fast and in an automated fashion using the deployer
- [ ] We update our _New Hub_ issue template to match the new structure

Outcomes
### Achieving this goal will
- [ ] Improve understanding (cross-team) of what is or what should be straightforward and efficient to build and what requires additional work and time and adds maintainability toil
- [ ] We create a hash map of `(community request, infrastructure hub feature)` pairs that will make partnership's job of translating community needs into engineering requirements easier
- [ ] Templating and automating the standard and most common deployments will, free up engineering time which will eventually translate into smaller base offering costs. And this will make us more accessible for communities will less access to funding.
- [ ] Improve the billing story, by understanding and separating "standard" from "extra". This way we could have more predictable cost offerings that stay real to the actual work that we are doing

Proposed plan
### Cross-team research needed
- [ ] Starting from the _New Hub_ issue template and using other assets leveraged by partnerships when a new managed hub contract is being discussed, identify what qualifies as _"hub feature"_
- [ ] In the list of the current hubs we manage, identify the most common features that the hub we deploy have and can be considered "basic" for a functional area
- [ ] Categorize the most common features into two/three types of _"standard offerings"_
- [ ]  Create a list of remaining "extras" that we currently provide that are less used and weight more in the cost we're paying to maintain it in the infrastructure

### Engineering work needed
- [ ] Create template value files for these couple of standard hubs that we ndentify
- [ ] Use the `deployer` to generate them in the appropriate location for any new such hub (similar with what's happening for new GCP cluster config files; https://github.com/2i2c-org/infrastructure/issues/2187 is also relevant and should be tackled as it's useful for this goal)

### Partnerships work needed
- [ ] Update the assets that are used and shared with the communities when a new hub contract is decided.
- [ ] Update the _New Hub_ issue template to match the new offerings

Related PR:

https://github.com/2i2c-org/docs/pull/187

Note that I've updated the top comment to match current proto-goal proposed structure.
I know we're not supposed to be proposing plans in this stage, but in the References and links to issues I tracked down existing issues related to the goal and tried to order them a bit.


reuse packing algorithm for 32 and 64 bits

64 bits packing produces the same exact values bit by bit for values 32 bits or less.

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Send app location in WalletLink request

Summary
This PR adds the app's window.location to the WalletLink event. We already do this for the origin, but there's cases where Wallet needs the full path.
How did you test your changes?
Tested manually and verified that adding the field doesn't cause issues with older versions of Wallet.


Initial cut at rly tx pkt

This now works in general.  Have a look at rly tx send-packet --help for details.
soil:relayer michael$ rly tx send-packet -d ibc0 ibc1 'hello from ibc0!'
I[2020-04-20|23:17:11.747] ✔ [ibc0]@{6197} - msg(0:sendpacket) hash(B8461AF5F4B8AE55E558391CF0192956ED9AFA5D180B2D83EA9DE4719D3A9FD3)
soil:relayer michael$ rly start ibczeroibcone
I[2020-04-20|23:18:12.659] - listening to tx events from ibc0...
I[2020-04-20|23:18:12.659] - listening to block events from ibc0...
I[2020-04-20|23:18:12.660] - listening to tx events from ibc1...
I[2020-04-20|23:18:12.660] - listening to block events from ibc1...
I[2020-04-20|23:18:13.686] ✔ [ibc1]@{6255} - msg(0:update_client,1:ics04/opaque) hash(5BD3DB812B0C2A8E06257C0E98E62A2FAAE82908DE79F26751E4B283D6234B81)
I[2020-04-20|23:18:13.686] ★ Relayed 1 packets: [ibc0]port{echo}->[ibc1]port{echo}
I[2020-04-20|23:18:13.687] • [ibc1]@{6255} - actions(0:update_client,1:ics04/opaque) hash(5BD3DB812B0C2A8E06257C0E98E62A2FAAE82908DE79F26751E4B283D6234B81)
^CSignal Recieved: interrupt
I[2020-04-20|23:18:16.862] - [ibc0]:{echo} <-> [ibc1]:{echo} relayer shutting down
soil:relayer michael$

Hi @jackzampolin, this is ready for you to have a look.  It would be great to land, but please let me know if you have any changes (such as removing the swingset dependencies).

Will take a peek tomorrow morning, but don't mind the swingset dep

👍 on the approach. Only question is if that packet type should live in a chain (i.e. swingset or cosmos-sdk).

Can you explain what you have in mind?  I didn't want to introduce a dependency from the relayer on Agoric, since we haven't yet published our Go modules.


filterModel returns CSV list without escaping commas

The ag-grid default 'set' filter implementation returns a CSV list of all the values being checked in the column filter checkbox list.
However, if any of these values contains a comma by itself, then this item is not getting escaped by surrounding it with quotes. So any parser is going to misinterpret the CSV list.
This looks like a bug to me.
When returning a CSV list, ag-grid should comply to RFC 4180.

According to filtering example - http://www.ag-grid.com/angular-grid-filtering/index.php
It looks like you pass array of string for values instead of comma separated string.
I tried to change

var irishAthletes = ['Michael Phelps', .... to
var irishAthletes = ['Michael, Phelps', ...  and changed underlying example data to Michael, Phelps, It seems to be working here.

Absolutely. Sorry, I didn't JSON.Stringify() again here. My mistake.


Add new selenium test case to test clear output button

##What is this PR for?
Add a new test case for testing the clear output button.
Clear output button clears the output on the paragraph
##What type of PR is it?
Test
##Is there a relevant Jira issue?
NA
##How should this be tested?
On OSX, you'll need firefox 42.0 installed, then you can run with
PATH=~/Applications/Firefox.app/Contents/MacOS/:$PATH CI=""
mvn -Dtest=org.apache.zeppelin.ParagraphActionsIT -Denforcer.skip=true
test -pl zeppelin-server
##Questions:
Does the licenses files need update?NO
Is there breaking changes for older versions?NO
Does this needs documentation?NO

Tested, CI green, LGTM.

CI is now failing on:
Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 25.538 sec <<< FAILURE! - in org.apache.zeppelin.integration.ParagraphActionsIT
testClearOutputButton(org.apache.zeppelin.integration.ParagraphActionsIT)  Time elapsed: 5.039 sec  <<< ERROR!
org.openqa.selenium.NoSuchElementException: Unable to locate element: {"method":"xpath","selector":"//div[@ng-controller=\"ParagraphCtrl\"][1]//div[contains(@class,'text ng-scope')]"}

This should catch all Exception not just ElementNotVisibleException.

@prabhjyotsingh  @corneadoug
Changes has been done, Please have a look.

@prabhjyotsingh used handleException, please have a look

Question, why is that test deleting: testCreateNewButton? Is that a result of the rebase?

@corneadoug Corrected bad rebase.

LGTM


Add top and bottom margin around images

Currently, the images appear very close to the text.
E.g. https://wagtail--6930.org.readthedocs.build/en/6930/editor_manual/finding_your_way_around/the_explorer_page.html#

@tbrlpld This link takes me to missing page, so I can't exactly find what the issue is. Please provide me with some more information and relevant link.

It appears this has been fixed.


Redesign edit Topic interface

As part of the overall dashboard redesign #75 the edit topic UI for creating and editing topics needs to be rethought.

somwhat complete, closing in light of new effort underway to holistically re-imagine the UX


Problems with the Fetch API

Type: Bug
I am not sure if this is really a problem with VS Code or not. I use the Fetch API in JavaScript. When I run the Fetch directly in JavaScript with a Chrome browser, I have no problems. The Fetch API executes fine. A very important point is that I don't see an OPTIONS request being sent. However, when I run the same Fetch using exactly the same JavaScript (using VS Code) I do see an OPTIONS being sent. Even worse, the Fetch (under JavaScript, under Chrome, under VS Code) does not wait for the OPTIONS to complete. The Fetch just fails with a 'failed to fetch' error. I don't understand this. Why should VS Code make any difference? What am I doing wrong? What additional information should I provide? How should this problem be debugged?
VS Code version: Code 1.81.0 (6445d93c81ebe42c4cbd7a60712e0b17d9463e97, 2023-08-02T12:37:13.485Z)
OS version: Windows_NT x64 10.0.19045
Modes:

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-4702HQ CPU @ 2.20GHz (8 x 2195)

GPU Status
2d_canvas: enabledcanvas_oop_rasterization: disabled_offdirect_rendering_display_compositor: disabled_off_okgpu_compositing: enabledmultiple_raster_threads: enabled_onopengl: enabled_onrasterization: enabledraw_draw: disabled_off_okvideo_decode: enabledvideo_encode: unavailable_offvulkan: disabled_offwebgl: enabledwebgl2: enabledwebgpu: enabled

Load (avg)
undefined

Memory (System)
15.91GB (5.79GB free)

Process Argv
--folder-uri file:///c%3A/Users/Peter/Documents/Visual%20Studio%20Code/Projects/WebApplication5/WebApplication5 --crash-reporter-id 7c17602d-d116-41bf-9eeb-6f288773aaef

Screen Reader
no

VM
0%

Extensions (10)

Extension
Author (truncated)
Version

copilot
Git
1.100.306

python
ms-
2023.14.0

vscode-pylance
ms-
2023.8.10

jupyter
ms-
2023.7.1002162226

jupyter-keymap
ms-
1.1.2

jupyter-renderers
ms-
1.0.17

vscode-jupyter-cell-tags
ms-
0.1.8

vscode-jupyter-slideshow
ms-
0.1.5

live-server
ms-
0.4.9

LiveServer
rit
5.7.9

A/B Experiments
vsliv368cf:30146710
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vstes627:30244334
vslsvsres303:30308271
vserr242:30382549
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscoreces:30445986
vscod805cf:30301675
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593cf:30376535
pythonvs932:30410667
vsclangdf:30486550
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30695312
azure-dev_surveyone:30548225
vsccc:30803844
282f8724:30602487
f6dab269:30613381
showlangstatbar:30737416
vsctsb:30748421
03d35959:30757346
pythonfmttext:30731395
pythoncmvfstrcf:30756944
fixshowwlkth:30771522
showindicator:30805244
pythongtdpath:30769146
i26e3531:30792625
gsofb:30804716
pythonnosmt12:30797651
pythonidxpt:30805730
pythonnoceb:30805159
e537b577:30795824
dsvsc013:30795093
dsvsc014:30804076

/question

We closed this issue because it is a question about using VS Code rather than an issue or feature request. Please search for help on StackOverflow, where the community has already answered thousands of similar questions. You may find their guide on asking a new question helpful if your question has not already been asked. See also our issue reporting guidelines.
Happy Coding!

Hi,
This is not a question. This is (in my opinion) a bug/issue/problem/etc. Code (in JavaScript) that works properly in native mode fails under VSC. The problem is actually worse than it sounds. I was able to get rid of the OPTIONS HTTP request, by removing my headers from the request. However, I still get a ‘Failed to fetch’ error.
Thank you
Peter Schaeffer
From: John Murray @.>
Sent: Sunday, August 6, 2023 2:17 PM
To: microsoft/vscode @.>
Cc: Peter Schaeffer @.>; Author @.>
Subject: Re: [microsoft/vscode] Problems with the Fetch API (Issue #189761)
/question
—
Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode/issues/189761#issuecomment-1666947939 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4LSSTWXD26EZFI6CGIPETXT7URZANCNFSM6AAAAAA3F7EYHI .
You are receiving this because you authored the thread.  https://github.com/notifications/beacon/AA4LSSSNKHDASBENTYQTT3DXT7URZA5CNFSM6AAAAAA3F7EYHKWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTTDLONWG.gif Message ID: @.*** @.***> >

See #189774

Hi,
Warning. This may be a duplicate.
I added what follows to my launch.json file and the problem went away.
"runtimeArgs": ["--disable-web-security","--user-date-dir=c:\chrome-browser"]
Putting all these things together, it looks like when I ran without VSC, CORS is/was somehow (automatically? partially?) disabled. When I run inside VSC, CORS is not disabled. As a consequence, CORS must be manually disabled. At least this explanation fits the facts. What is really going on, I don’t know.
Thank you
Peter Schaeffer
P.S. I would suggest adding (in some form) this explanation to issue 189774, so other folks can search for it
From: John Murray @.>
Sent: Sunday, August 6, 2023 4:53 PM
To: microsoft/vscode @.>
Cc: Peter Schaeffer @.>; Author @.>
Subject: Re: [microsoft/vscode] Problems with the Fetch API (Issue #189761)
See #189774 https://github.com/microsoft/vscode/issues/189774
—
Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode/issues/189761#issuecomment-1666975850 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4LSSVZYFKOWHRSLKYK72DXUAGZ5ANCNFSM6AAAAAA3F7EYHI .
You are receiving this because you authored the thread.  https://github.com/notifications/beacon/AA4LSSVQY6LSSYWI55VH62DXUAGZ5A5CNFSM6AAAAAA3F7EYHKWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTTDLQEGU.gif Message ID: @.*** @.***> >


pull request for e2e test at 12/15/2016 12:56:17

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Add support for smart PVC cloning

This PR adds the support for rbd to create smart cloning from a PVC, to create a new image from an existing image we are following below steps
Create a PVC from PVC

Create a temporary snapshot from the parent volume
Clone a new image from a temporary snapshot with options  --rbd-default-clone-format 2 --image-feature layering,deep-flatten
Delete temporary snapshot created
Create a snapshot with requested Name from cloned volume
Create a clone from a snapshot  with user-provided image features

Delete a PVC

Move the image to the trash
Add a task to remove the image from the trash

example commands:-
1) rbd snap create <RBD image for src k8s volume>@<random snap name>
2) rbd clone --rbd-default-clone-format 2 --image-feature
layering,deep-flatten <RBD image for src  k8s volume>@<random snap <RBD image for temporary snap image>
3) rbd snap rm <RBD image for src k8s volume>@<random snap name>
4) rbd snap create <RBD image for temporary snap image>@<random snap name> <k8s snap name>
5) rbd clone --rbd-default-clone-format 2 --image-feature <k8s dst vol config>
<RBD image for temporary snap image>@<random snap name> <RBD image for k8s dst vol>
6) rbd snap rm <RBD image for temporary snap image>@<random snap name>

Fixes: #675
Note: This PR is built on top of https://github.com/ceph/ceph-csi/pull/693, will squash/remove commits once #693 is merged

@Madhu-1 can you revisit this PR ?

this will be based on https://github.com/ceph/ceph-csi/pull/693, once it is merged we will revisit this one

[status update]
Last week we had a meeting around snapshot/clone design/implementation we introduced recently with this PR, to support further use cases like Bakcup/DR , Data replication..etc. The main reason behind this meeting was to avoid a revamp again with snapshot/clone implementation while we start supporting these use cases too in near future. Atm, we are listing down the requirements to support it along with possible changes/enhancements we need with current implementation/design. Further technical notes will be followed/pointed here to those discussion. Considering we are at code freeze of v2.1.0 and the state we are with this features, the decision is to defer it to next month release (v3.0.0 #865 ) and release v2.1.0 as planned with other bug fixes and enhancements.

[status update]
Last week we had a meeting around snapshot/clone design/implementation we introduced recently with this PR, to support further use cases like Bakcup/DR , Data replication..etc. The main reason behind this meeting was to avoid a revamp again with snapshot/clone implementation while we start supporting these use cases too in near future. Atm, we are listing down the requirements to support it along with possible changes/enhancements we need with current implementation/design. Further technical notes will be followed/pointed here to those discussion. Considering we are at code freeze of v2.1.0 and the state we are with this features, the decision is to defer it to next month release (v3.0.0 #865 ) and release v2.1.0 as planned with other bug fixes and enhancements.

Details of other use-cases is present here.
The decision is to continue with the current design and implementation as it does not impede the future uses as understood at present.


Class name with a middle blank space

Selector doesn't work for this case. Is there any way to  handle it.

Finally, I found the solution:
div.cell.item
from here

Thanks. I need to write a FAQ and include this, as it does cause a lot of confusion.


Hackpi doesn't work

Hi, I plugged the key to my laptop before looking at the github and the key was launching a program. Now I have followed the setup steps, my key doesn't work. Can you help me ?

For me using "firmware.uf2" causes this problem and returns "No module named future" when trying to run code in Thonny. After using the official CircuitPython "adafruit-circuitpython-raspberry_pi_pico-en_US-8.0.5.uf2" (link at the end of step 2) everything is back to normal.

Oh bro, THANK YOU, it works !
Have a good day.


[google_sign_in] iOS PlatformException (PlatformException(missing-config, GoogleService-Info.plist file not found, null, null))

[VERBOSE-2:dart_vm_initializer.cc(41)] Unhandled Exception: PlatformException(missing-config, GoogleService-Info.plist file not found and clientId was not provided programmatically., null, null)
#0      StandardMethodCodec.decodeEnvelope
package:flutter/…/services/message_codecs.dart:653
#1      MethodChannel._invokeMethod
package:flutter/…/services/platform_channel.dart:296

#2      GoogleSignIn._callMethod
package:google_sign_in/google_sign_in.dart:271

#3      GoogleSignIn.signIn.isCanceled
package:google_sign_in/google_sign_in.dart:407

Hi @abdelrahman-abied, what version of google_sign_in are you using?
Please provide the output of flutter doctor -v.
Also, did you follow the setup instructions in the plugin readme or did you do the initialization through flutterfire configure?


[Kubernetes] Knative Serving

Implement knative serving so we can scale down to zero the backend when running on a kubernetes cluster.
Relates to KaotoIO/kaoto-backend#15

not needed anymore as there is no backend in kaoto-next


Field validate function should receive field props

Are you submitting a bug report or a feature request?

Feature Request
What is the current behavior?

Signature is:
function validate(props, allValues, formProps)

What is the expected behavior?
function validate(props, allValues, formProps, fieldProps)

Other information

I have a setup where the validate definition is decoupled from the props. I want to be able to change the validation at runtime based on config variables in the props. This needs to be done on a per-field basis.

Just wondering what the latest on this proposal is. It seems to me that a more robust pattern (at least what I personally have implemented) is to have the fields responsible for their own validation rather than the form. That being the case, having access to their own state will provide greater granular control

@erikras There seems to be some growing interest in this, do you have any thoughts around it?

@bradywatkinson how are you picturing implementing validation in the WrappedField?

This is definitely desirable, PR's welcome.

I'm shocked that this isn't already baked in. I just combed through all the issues, but didn't even see a roundabout hack - is there one? I just want to know if a particular field is dirty as part of validation (for a use case where it has an initialValue set, but I want to ensure the final submitted value was explicitly set by the user, regardless of it is the same as the initialValue or not)

As #4016 points out, one precedence this might create is that validation will run on any props change, so we need to tread carefully in that regard.

@dfagan2 because you can't currently put props based validation logic 'underneath' the Field - in it's validation function, it presently must go 'above'. This is done by dynamically changing a validation function (or one in an array) passed to the validate prop during a render which will re-run (fixed in 7.4.2) the validation with that new function immediately. 7.4.2 will fail fast with an infinite loop if you don't manage this carefully (i.e. don't naively create a new validate prop on every render). I'm trying (being the keyword) to round the 'above' the Field approaches / patterns in #4017 in lieux of / as well as this being implemented.

@danielrob - Ok. Is there a way to extract field level meta props like dirty and pristine "above" the Field? I didn't see a Redux-Form method or selector at that level that could achieve that...

To be honest since working with redux-form a little longer I'v managed to do everything I needed to do without this feature.  It was strange at first because I expected the props to be there, but since diving into the api and using the action creators, and my own stateful component where my form lives I'v managed to do everything I wanted

@conor909 - care to share your wisdom? I'm still not sure how to accomplish field level dirty/pristine detection during validation (without this proposal)

@dfagan2  I was trying to achieve something different, but I just thought I'd share my experience.

@dfagan2 it's been a while since I've done a personal round up of the latest in this area, but:

start with getFormMeta, it's had more and more added to it over time.
Move on to this (in this context apologetically chatty) discussion about this question over time at #3289
If all else fails, somewhere floating around there's @erikras's comment pointing out there's nothing stopping you using Field as a meta selector. e.g. have a Field with a component that renders a Field. It does feel ugly, but it is just calling connect() at the end of the day which is what the selectors do anyway.


Re-enable parallel sphinx builds by default

Summary
With the recent merge of #9346 that stopped using the problematic jupyter-sphinx plugin we're no long blocked from running sphinx-build in parallel anymore. We had originally enabled this in #4477 to get a speed boost during docs builds. However, we were forced to revert that in #6539 because jupyter-sphinx was unreliable and had a race condition when 2 jupyter-execute blocks were run at the same time (see #5904 for more details). Since we've removed the jupyter-sphinx usage that is no longer a problem and we can reliably run parallel sphinx builds again. This commit makes that change and changes the default sphinx-build behavior when run via tox (as is done via ci) to use all available system cores to perform a sphinx build. Annecdoteally this provides a nice speed up, on my local workstation a full docs build went from taking 8min 42 seconds for a serial build to 2min 35sec with a parallel build.
Details and comments

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 84.579%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
64200

Relevant Lines:
75905

💛  - Coveralls


Could Unalign::update cause issues for types that are aware of their memory location? Is this a soundness hole?

Rust doesn't allow the following code:
#[derive(Copy, Clone)]
#[repr(C, packed)]
struct Unalign<T>(T);

It won't emit a Clone impl - instead, you have to write it yourself, using a T: Copy bound. It occurred to me that they could use the same trick we use in Unalign::update, using ptr::read to move to the stack, calling clone on the now-aligned value, and then mem::forgeting the copy. That would not play nicely with interior mutability (not a problem for us - Unalign::update takes a &mut self), but I realized that it also might cause problems if you have a type whose semantics include its memory location. This could be for soundness reasons (e.g., pinned types) or for safety reasons (user-defined invariants).
Need to think through this more to figure out whether Unalign::update is actually unsound.
cc @djkoloski since you reviewed that PR and might have thoughts

You don't have to worry. For types that may be self-referential or otherwise immovable, there is a burden on the type itself to never allow a T to be produced (or &mut T to be produced in a situation where the user is allowed to move the value). You can confidently say that immovable values must always be located behind a Pin, and values that are not behind a Pin are not immovable. Unless you have a special way to call methods on Pin<&mut Unalign> to get an &mut or something like that, you probably don't have to worry about immovable types.

Sounds good, thanks for the insight!


Please update doc to include - promote staging slot to proction before migrating to ARM

[Enter feedback here]
Cloud Service | Cloud Service with a deployment in a single slot only.
Cloud Services containing a prod slot deployment can be migrated. It is not reccomended to migrate staging slot as this can result in issues with retaining service FQDN
Can this be updated to include promote staging to production and migrate proeduction slot to ARM. Also it would be helpful if  explained further with details like FQDN becomes [deploymentid].cloudapp.net when upgraded and the existing DNS name gets suspended and hence not recommended.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: cbd3beac-4b11-5d20-8633-9bd9a633e575
Version Independent ID: cb7fe932-92f0-ec71-d6e1-71be702c9e2b
Content: Migrate Azure Cloud Services (classic) to Azure Cloud Services (extended support)
Content Source: articles/cloud-services-extended-support/in-place-migration-overview.md
Service: cloud-services-extended-support
Sub-service: classic-to-arm-migration
GitHub Login: @hirenshah1
Microsoft Alias: hirshah

@SauravB-MS
Thanks for the feedback! I have assigned the issue to content author to check and update the document as appropriate.

@hirenshah1
Can you please check and add your comments on this doc update request as applicable.

Updated content - https://github.com/MicrosoftDocs/azure-docs-pr/pull/234798


Option (or bug), should not crawl escaped HTML

Let's say I'm crawling a site of docs. In the HTML for the docs I have

As it is simplecrawler will try to crawl that (and of course fail).
Is the the intended behavior?

Kind of, yes. There are a lot of scenarios to account for when looking for resources in a page, and simplecrawler has chosen to use a rather dumb catch-everything approach for its default discovery mechanism.
The discovery mechanism is almost intended to be augmented, and it is common to use something like cheerio to parse pages into a DOM-like structure where you can be sure to get eg. just the actual href attributes of  elements.
The discovery mechanism is the most common topic that people ask about, and I intend to add some details about the philosophy around it to the README ASAP.

What @fredrikekelund said. :)
But it might be worth translating HTML entities into their URI-safe versions in future. And sorry about the ambiguity — I really should've added better docs in the first place. Maybe adding an FAQ.md, or TROUBLESHOOTING.md. :)

When you say "use something like cheerio to parse pages into a DOM like structure" are you saying that I should first parse a page using cheerio, then get cheerio's output and pass that to simplecrawler?
I guess what I'm asking is how to stop the false positives? I'm trying to scan my docs for broken links but it's reporting links from the sample code which aren't actually links.
So what's the recommended way of either rejecting false positives or preventing them in the first place?

It's hard to optimise the discovery mechanism for every use case — a broken link checker wants as few false positives as possible, but a site archiver wants to get as much as possible, even extracting content from binary files, and the cost of false positives is low.
So we try to get everything, and offer some key places to tweak the algorithm. The inbuilt discovery function simply uses a list of naive regex. You could first limit the patterns only to those you know will work for your use case, by customising it like below:
// Tweak as necessary
crawler.discoverRegex = [
    /\s(?:href|src)\s?=\s?(["']).*?\1/ig,
    /\s(?:href|src)\s?=\s?[^"'\s][^\s>]+/ig
];

Alternatively, replacing the discovery function with something less naive is an easy operation.
var cheerio = require("cheerio")

crawler.discoverResources = function(resourceData, queueItem) {
    var $ = cheerio.load(resourceData.toString("utf8"));
    return [].slice.call($("[href],[src]"), 0).forEach(function(item) { return item.attribs.href || item.attribs.src; });
};


Resolved issue #329

The Pull Request resolves Issue #[329]
Description:
I added all three of the words(Befuddle, Bemused, Baffle)

Hey @Chillers1101
Issue #329  is resolved already. Please contribute to other open issues. I'm closing this PR
Thanks for contributing


Login / Micro CMS Functionality

Features:

Uses Oauth / Passport etc for Login functionality
User / Author schema that integrates into the blog feature

Passport login is about 50% done. Its working with mongodb but /admin needs to be locked down so it is only accessible when logged.


remove seemingly unnecessary step

This will bring a stream into memory, which can take a lot of memory unnecessarily
Admittedly there is a lot of intermediate code that I can't attach here, but replacing the finch adapter with the httpc adapter looks like this with a 10G multipart request body:
Finch adapter:

httpc adapter:

(Notice that the units are different between the two screenshots, one is in MB and the other in GB)
I can try to characterize the memory issue I am seeing with a test if you would like to see, but uploading a big file to see a memory spike in a test seems like a test you wouldn't want in the long term.

🚀 💜


Improve build and execution

Build process improvement:

Run the build process inside a container

Dockerized solution improvement :

Make image configurable using env vars
Shrink Image size from 251.2 MB  to 12.17 MB

Metrics:

Add response time metric

Release:

Added bin directory and SHA512 sum for the binary

Documentation

Detailled metrics explanation
How to configure the image
How to build the images

Thanks for your contribution! I'll review this pr ASAP


How do I set style for Select Component

💬 Question
I am unable to set style prop for Select Component. I tried setting style with Style property but it doesn't work (attached screenshot). Am I doing right ?
Here is my code for the Select Component
 <View                 style={{                   backgroundColor: colors.primary,                   borderRadius: 20,                   elevation: 1,                   justifyContent: 'center',                 }}               >                 <Select                   selectedIndex={categoryIndex}                   children={Categories.map((item) => {                     return (                       <SelectItem                         title={item.value}                         style={{                           borderTopColor: Colors.secondary,                           borderWidth: 1,                           borderRadius: 5,                           fontFamily: 'SFUIText-Regular',                         }}                       />                     );                   })}                   onSelect={(index) => selectCategory(index)}                   value={category}                   style={{                     flex: 1,                     backgroundColor: colors.background,                     borderRadius: 10,                     fontFamily: 'SFUIText-Regular',                   }}                 />               </View>

UI Kitten and Eva version

Package
Version

@eva-design/eva
2.0.0

@ui-kitten/components
5.0.0

Hi @thegreatdanton, I followed the Customize Component Mapping tutorial  tutorial to style my Select component, look into the mapping.json file to find the options.
{   "components": {     "Select": {       "meta": {         "parameters": {           "backgroundColor": "white"         }       },       "appearances": {         "default": {           "mapping": {             "labelFontWeight": "400"           },           "variantGroups": {             "status": {               "basic": {                 "backgroundColor": "white",                 "borderColor": "white"               }             },             "size": {               "small": {                 "paddingVertical": 3,                 "textFontWeight": "400"               },               "medium": {                 "paddingVertical": 7,                 "textFontWeight": "400"               },               "large": {                 "paddingVertical": 11,                 "textFontWeight": "400"               }             }           }         }       }     },     "SelectOption": {       "meta": {         "parameters": {           "backgroundColor": "white"         }       },       "appearances": {         "default": {           "mapping": {             "textFontWeight": "400"           }         }       }     }   } }

@thegreatdanton sorry for the late reply. Please follow the answer above to make style changes with that component.

@rjsajnani Thank you for the help. I ll try this.
I am learning React Native and UI Kitten helped me a lot but isn't tiresome to create custom styling with lot of code rather than doing by using component API ?

I also agree with you, also if you see the API for select component it states we can use touchable opacity props, so by right we must be able to use style props.
I'm also new to UI kitten, maybe someone from the team or anyone else can answer your question

Apologies to keep thread going on, I would like to know how to use TouchableOpacity props from API of select component
@artyorsh any help here ?

Hi @thegreatdanton, I followed the Customize Component Mapping tutorial to style my Select component, look into the mapping.json file to find the options.
{ "components": { "Select": { "meta": { "parameters": { "backgroundColor": "white" } }, "appearances": { "default": { "mapping": { "labelFontWeight": "400" }, "variantGroups": { "status": { "basic": { "backgroundColor": "white", "borderColor": "white" } }, "size": { "small": { "paddingVertical": 3, "textFontWeight": "400" }, "medium": { "paddingVertical": 7, "textFontWeight": "400" }, "large": { "paddingVertical": 11, "textFontWeight": "400" } } } } } }, "SelectOption": { "meta": { "parameters": { "backgroundColor": "white" } }, "appearances": { "default": { "mapping": { "textFontWeight": "400" } } } } } }

Is possible change the border radius?? I really try but I not able to change, I just need to change the border radius could you have a little example


[DC-727] Develop a script to determine the number of EHR and RDR data rows in each dataset for a set of participants

Add participant count script
Add check for ehr and rdr
Use client library instead of bq_utils

Thanks @nishanthpp93 ! I really like the comments/docstrings and also the incremental tests. Could we use "row_count" instead of "prevalence"? The latter may sound like we're talking about disease frequency (although retracted pids can make me feel unwell).
In addition to feedback here I have some questions which I think are better to discuss over voice. I'll set up time with you offline.

Will this script be used during retraction? If so, maybe we can include it somehow so it happens automatically.

We said it would be run before retraction. In the future we might consider sandboxing records to be deleted.

There is probably a good reason, but not sure why there is a separation between the person table and all other CDM tables.

This was done to facilitate distinction between EHR and RDR records. We will refactor to use the mapping tables which will obviate the need for this and also allow scaling for other sources of records.

I find the test data scenarios hard to intuit; any ideas on how to make it easier to comprehend? Could we initialize dataframes with list[dict] or some other representation?

I've made all the changes @mark-velez, please feel free to take a look.

Moved changes to #465