[BUG] exception is reported for azure-messaging-servicebus sdk component. ERROR Schedulers: Scheduler worker in group main failed with an uncaught exception

Describe the bug
Implementing async subscriber to receive messages, near copy from readme example, and running from custom thread from inside Microsoft Azure Workspace Spark Synapse, raises uncaught exception. The exception is raised at the time of setting up receiver. Further on, no messages can be received from the queue and a different exception is raised.
Exception or Stack Trace
Exception #1: 20/07/16 02:10:42 ERROR Schedulers: Scheduler worker in group main failed with an uncaught exception
java.lang.NoSuchMethodError: reactor.core.scheduler.Schedulers.boundedElastic()Lreactor/core/scheduler/Scheduler;
at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.onNext(ServiceBusReceiveLinkProcessor.java:210)
at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.onNext(ServiceBusReceiveLinkProcessor.java:41)
at reactor.core.publisher.FluxRepeatPredicate$RepeatPredicateSubscriber.onNext(FluxRepeatPredicate.java:78)
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:204)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1510)
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1510)
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1510)
at reactor.core.publisher.MonoIgnoreThen$ThenAcceptInner.onNext(MonoIgnoreThen.java:296)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1510)
at reactor.core.publisher.MonoIgnoreThen$ThenAcceptInner.onNext(MonoIgnoreThen.java:296)
at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156)
at com.azure.core.amqp.implementation.ReactorSession.lambda$createConsumer$18(ReactorSession.java:402)
at com.azure.core.amqp.implementation.handler.DispatchHandler.onTimerTask(DispatchHandler.java:36)
at com.azure.core.amqp.implementation.ReactorDispatcher$WorkScheduler.run(ReactorDispatcher.java:130)
at org.apache.qpid.proton.reactor.impl.SelectableImpl.readable(SelectableImpl.java:118)
at org.apache.qpid.proton.reactor.impl.IOHandler.handleQuiesced(IOHandler.java:61)
at org.apache.qpid.proton.reactor.impl.IOHandler.onUnhandled(IOHandler.java:390)
at com.azure.core.amqp.implementation.handler.CustomIOHandler.onUnhandled(CustomIOHandler.java:39)
at org.apache.qpid.proton.engine.BaseHandler.onReactorQuiesced(BaseHandler.java:87)
at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:206)
at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:292)
at com.azure.core.amqp.implementation.ReactorExecutor.run(ReactorExecutor.java:82)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:50)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:27)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Exception #2
Exception in thread "Thread-44" java.util.concurrent.RejectedExecutionException: ReactorDispatcher instance is closed.
at com.azure.core.amqp.implementation.ReactorDispatcher.throwIfSchedulerError(ReactorDispatcher.java:93)
at com.azure.core.amqp.implementation.ReactorDispatcher.invoke(ReactorDispatcher.java:68)
at com.azure.core.amqp.implementation.ReactorReceiver.dispose(ReactorReceiver.java:169)
at com.azure.messaging.servicebus.implementation.ServiceBusReactorReceiver.dispose(ServiceBusReactorReceiver.java:177)
at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.checkAndSetTerminated(ServiceBusReceiveLinkProcessor.java:531)
at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.drainQueue(ServiceBusReceiveLinkProcessor.java:460)
at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.drain(ServiceBusReceiveLinkProcessor.java:450)
at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.dispose(ServiceBusReceiveLinkProcessor.java:355)
at com.azure.messaging.servicebus.ServiceBusAsyncConsumer.close(ServiceBusAsyncConsumer.java:80)
at com.azure.messaging.servicebus.ServiceBusReceiverAsyncClient.close(ServiceBusReceiverAsyncClient.java:1145)
at com.microsoft.azure.poc.StreamingPipeline.run(ExampleStream.scala:191)
at java.lang.Thread.run(Thread.java:748)
To Reproduce
Please find below is a sanitized version of IntelliJ IDEA project ZIP archive with code and build script prepared. Loading same from the context of Microsoft Azure Workspace Spark Synapse as Scala Job invariable runs into this exception
Project.zip
Code Snippet
please see ZIP archive above
Expected behavior
Message received from Service Bus with no issues
Screenshots
Not applicable
Setup (please complete the following information):
Not applicable to the extent I can follow
Additional context
Similar issue was reported for Microsoft Azure Event Hub Java API/SDK. Based on my understanding of developer dialogs logged in the issue, it appears that the trouble is somewhere between code as such and ability of the code to perform at high throughput.
Please find is the issue link: https://github.com/Azure/azure-event-hubs-spark/issues/363
Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

[x ] Bug Description Added
[x] Repro Steps Added
[x] Setup information Added

@hemanttanwar PTAL

+1

I am facing same issue .
ERROR 5860 --- [ctor-http-nio-2] reactor.core.publisher.Operators         : Operator called default onErrorDropped
java.lang.NoSuchMethodError: reactor.core.scheduler.Schedulers.boundedElastic()Lreactor/core/scheduler/Scheduler;
at com.azure.messaging.eventhubs.PartitionPumpManager.startPartitionPump(PartitionPumpManager.java:180) ~[azure-messaging-eventhubs-5.2.0.jar:na]
at com.azure.messaging.eventhubs.PartitionBasedLoadBalancer.lambda$claimOwnership$23(PartitionBasedLoadBalancer.java:426) ~[azure-messaging-eventhubs-5.2.0.jar:na]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[na:1.8.0_144]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) ~[na:1.8.0_144]
at com.azure.messaging.eventhubs.PartitionBasedLoadBalancer.lambda$claimOwnership$24(PartitionBasedLoadBalancer.java:426) ~[azure-messaging-eventhubs-5.2.0.jar:na]
at reactor.core.publisher.LambdaMonoSubscriber.onNext(LambdaMonoSubscriber.java:137) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1515) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1515) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:145) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFilter$FilterSubscriber.onComplete(FluxFilter.java:160) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.checkTerminated(FluxFlatMap.java:794) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.drainLoop(FluxFlatMap.java:560) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.drain(FluxFlatMap.java:540) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.onComplete(FluxFlatMap.java:426) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]

Facing same issue, I spent almost 5 hours to resolve and found here that it will be resolved in December milestone, hope so.

@ranjankumar03
But looking at the error, here is my analysis.
it seems like the setup in which ServiceBus is running is using older version of "projectreactor". The exception trace shows that it uses reactor-core-3.2.10.RELEASE.jar.
You can see that in this version Schedulers.boundedElastic() does not exists, Thus you are getting java.lang.NoSuchMethodError.
3.2.10.RELEASE API Reference: https://projectreactor.io/docs/core/3.2.10.RELEASE/api/reactor/core/scheduler/Schedulers.html
The Service Bus 7.x.x version need version "io.projectreacto:reactor-core:3.3.11.RELEASE".
We configure it here  : https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/core/azure-core/pom.xml#L87
Can you ensure that this version is used ?

@MironAtHome
I have not worked in "Microsoft Azure Workspace Spark Synapse", So I did not setup exactly same Synapse Workspace to replicate this. I will do some tutorials on how to set it up.
Thank you for sending Project.zip  Appreciate it.
But I looked at your Stream_jar.xml and it looks like your setup is using "io.projectreactor:reactor-core:3.3.7.RELEASE:jar" ? Is it possible that your setup is using older version of "io.projectreacto:reactor-core" ?
The cause for 'java.lang.NoSuchMethodError' is normally, if your setup is not using correct version of library.
The Service Bus 7.x.x version need version "io.projectreacto:reactor-core:3.3.11.RELEASE".
We configure it here : https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/core/azure-core/pom.xml#L87
Also  since your program is failing at  ' exception is raised at the time of setting up receiver', it can not be performance related issue.  To eliminate the performance issue: May be you can try reading just one message from queue and see if you can read it.

@hemanttanwar
Thanks Hemant, it's working perfect as with spring boot it was taking older version of reactor so explicitly added io.projectreacto:reactor-core:3.3.11.RELEASE in pom and works with Service Bus 7.0.0

@UmeshRajani  Happy to see it resolved for you.

Glad this is resolved for you. NoSuchMethodErrors are often caused by mismatched dependencies. When I encounter these, I check the dependency tree. It'll show you all dependencies and how it decided to resolve the one causing your error.
https://maven.apache.org/plugins/maven-dependency-plugin/examples/resolving-conflicts-using-the-dependency-tree.html
Ex: mvn dependency:tree -Dverbose -Dincludes=reactor*


Generate reference catalogs with data for all six bands

This involves modifying bin/make_reference_catalog.py to include u, z, and y bands, and then using the resulting reference catalog to generate the astrometry.net index files (as documented in the Level 2 cookbook) with band-specific fluxes for each band.  This will help ensure the zero points found in the Level 2 processing are correct for each band.  See #267.

👍
Incidentally, does this change the few percent change in the other bands that we had been looking at? The change in catsim might also help in fixing that (it was galactic extinction which was being applied wrongly).

Comparing the magnitudes I've been seeing from CatSim to the ones in the cookbook, there will be differences in the zero points in all bands (from this morning's run, r, g, and i magnitudes also differ by 0.2 mag).   I plan to re-run processEimage.py on all of the Run1.1 visits (also turning on the cosmic ray repair) to get new zero points.   I can reload my local pserv db instance with the modified data to show the outcome of these corrections.  If we decide it's desirable, we can also update the Twinkles Run1.1 tables at NERSC (or add a Run1.1_new_zp project to preserve the current data).   I'm not sure it's worth re-running all of Level 2, but it may be something we want to do, depending on the Run3 status.

The index files are here /nfs/farm/g/desc/u1/data/PhoSim-Deep-Precursor/and_files/


Add python 3.8

Based on the official Docker Python image at https://github.com/docker-library/python/blob/32b69d61f6/3.8/buster/Dockerfile in the same format as the existing metabrainz 3.6/3.7 images.

Thanks, pushed metabrainz/python:3.8 image to Docker Hub already.


Add spring-cloud-stream-modules-analytics to bom

Suggested to add this to the bom so dataflow (and presumably other consumers of these modules) would not have to declare this dependency explicity.
Look for this issue name in https://github.com/spring-cloud/spring-cloud-dataflow/blob/master/spring-cloud-dataflow-dependencies/pom.xml as an example.
Related: https://github.com/spring-cloud/spring-cloud-dataflow/issues/384

This issue was moved to spring-cloud/spring-cloud-stream-app-starters#78


[Enhancement] Login alert notifications

Login successful notification
Error alert incase of wrong password
Error alert if an unregistered user is trying to login

I can work on this issues

I can work on this issues

Kritika these issues are resolved, check PR to review the code


XGBoost GPU benchmarks

Hi, I am the author of the XGBoost GPU algorithms.
Your benchmarks of my GPU hist algorithm are simply running on the CPU. The reason for this is the 'tree_method':'hist' parameter is overriding the selection of the GPU updater. This was fixed some time ago but it seems you are using an older commit. The correct usage would now be to set 'tree_method':'gpu_hist'. I would appreciate if you can update your benchmarks, I think you might find my algorithm far more competitive.
I also noticed that the XGBoost CPU hist algorithm has not had the number of bins set correctly, so you would be comparing 256 bins for XGBoost against 63 bins for LightGBM. This was due to a mistake in our documentation regarding the naming of the parameter that I have noted in dmlc/xgboost#2567.
Thanks
Rory

@RAMitchell
Refer to search result, it seems only one dataset (04_PlanetKaggle.ipynb) has the max_bin issue:
https://github.com/Azure/fast_retraining/search?utf8=✓&q=max_bin&type= .
BTW, i think the max_bin doesn't have much impact on training time when running in CPU. But we can fix it.
@miguelgfierro I think the XGBoost hist GPU result can be updated, it truly is almost the same as CPU hist .

@RAMitchell  thank you for the feedback, I will make sure that we run the experiments again and will update both repo and blog post. To be honest, I was surprised by the results of xgb hist, so I appreciate your feedback.
Could you please tell me the exact commit of XGBoost that we could use?
Also, I will create a new branch to fix this issue. I would really appreciate if you could review that I'm using the correct parameters for xgb hist.

@RAMitchell
BTW, should we use tree_method:gpu_exact for the xgb_gpu ?

 It is from June 29 and contains the interface changes.
I agree that the max_bin parameter should not significantly affect benchmarks. Good to be sure though :).
Yes tree_method:gpu_exact would be appropriate. Note that if you use the above commit you will get a newer version of the exact algorithm modified by some Nvidia people. The speed might be marginally higher and the memory usage also marginally higher.
My 'gpu_hist' algorithm does not support 'loss_guide'. It will be hard to have an apples to apples comparison but you have the same problem benchmarking the original XGBoost algorithm against LightGBM. I think the major pitfall of benchmarking accuracy between these two algorithms is limiting by number of leaves. This would allow LightGBM to reach far greater depths and is not really fair to XGBoost.
It is not very obvious how to choose these parameters unfortunately. All I can say is that the max depth of 6 in XGBoost is very commonly insufficient to get the best results.

I should also mention that it is not possible to reproduce your experiments because the GPU VMs do not appear to be available to the public until December.

@RAMitchell not all regions have the NV VM, you can find the right region here: https://azure.microsoft.com/en-us/regions/services/
We know the original XGBoost has this problem, so we have xgb_hist+loss_guide for the accuracy comparison.  And the timing result in such setting is also comparable. It still have a apple to apple comparison in CPU.
But in GPU, all XGB only support depth-wise. We know it is less accuracy, but growing shallow trees is also faster. As result, xgb_hist GPU will have advantage in speed, but loss on accuracy.

@RAMitchell  with this commit I get an error:
[ 96%] Building NVCC (Device) object CMakeFiles/cuda_compile.dir/plugin/updater_gpu/src/cuda_compile_generated_updater_gpu.cu.o
Error copying file (if different) from "/home/hoaphumanoid/installer/xgboost_rory/build/CMakeFiles/cuda_compile.dir/plugin/updater_gpu/src/cuda_compile_generated_updater_gpu.cu.o.depend.tmp" to "/home/hoaphumanoid/installer/xgboost_rory/build/CMakeFiles/cuda_compile.dir/plugin/updater_gpu/src/cuda_compile_generated_updater_gpu.cu.o.depend".
CMake Error at cuda_compile_generated_updater_gpu.cu.o.cmake:232 (message):
  Error generating
  /home/hoaphumanoid/installer/xgboost_rory/build/CMakeFiles/cuda_compile.dir/plugin/updater_gpu/src/./cuda_compile_generated_updater_gpu.cu.o

CMakeFiles/runxgboost.dir/build.make:63: recipe for target 'CMakeFiles/cuda_compile.dir/plugin/updater_gpu/src/cuda_compile_generated_updater_gpu.cu.o' failed
make[2]: *** [CMakeFiles/cuda_compile.dir/plugin/updater_gpu/src/cuda_compile_generated_updater_gpu.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
/home/hoaphumanoid/installer/xgboost_rory/plugin/updater_gpu/src/exact/../device_helpers.cuh: In member function ‘void dh::Timer::printElapsed(std::__cxx11::string)’:
/home/hoaphumanoid/installer/xgboost_rory/plugin/updater_gpu/src/exact/../device_helpers.cuh:226:54: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 3 has type ‘int64_t {aka long int}’ [-Wformat=]
/home/hoaphumanoid/installer/xgboost_rory/plugin/updater_gpu/src/device_helpers.cuh: In member function ‘void dh::Timer::printElapsed(std::__cxx11::string)’:
/home/hoaphumanoid/installer/xgboost_rory/plugin/updater_gpu/src/device_helpers.cuh:226:54: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 3 has type ‘int64_t {aka long int}’ [-Wformat=]
/home/hoaphumanoid/installer/xgboost_rory/plugin/updater_gpu/src/device_helpers.cuh: In member function ‘void dh::Timer::printElapsed(std::__cxx11::string)’:
/home/hoaphumanoid/installer/xgboost_rory/plugin/updater_gpu/src/device_helpers.cuh:226:54: warning: format ‘%lld’ expects argument of type ‘long long int’, but argument 3 has type ‘int64_t {aka long int}’ [-Wformat=]
Scanning dependencies of target xgboost
[ 98%] Linking CXX shared library ../lib/libxgboost.so
[ 98%] Built target xgboost
CMakeFiles/Makefile2:184: recipe for target 'CMakeFiles/runxgboost.dir/all' failed
make[1]: *** [CMakeFiles/runxgboost.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

What do you think the problem is?
Alternatively, I was able to compile in the first run commit 4eb255262fd8a7172815656c9eb3c148aa0d1e68 from Jul 18, this is more recent that the commit you mentioned. Will you be happy if I use this commit?

We made a lot of improvements to cmake over that time so its hard to say what the issue is. The safest thing to do is use the most recent. I was trying to give you a commit that was a bit closer to what you had before for fairness but I don't think it matters too much.

@RAMitchell I installed the more recent commit and run the experiment  airline_GPU with these parameters:
xgb_hist_params = {'max_depth':0,
                  'max_leaves':2**8,
                  'objective':'binary:logistic',
                  'min_child_weight':30,
                  'eta':0.1,
                  'scale_pos_weight':2,
                  'gamma':0.1,
                  'reg_lamda':1,
                  'subsample':1,
                  'tree_method':'gpu_hist',
                  'updater':'grow_gpu_hist'
                 }

I got this error:
XGBoostError                              Traceback (most recent call last)
<ipython-input-22-797306538b1f> in <module>()
----> 1 xgb_hist_clf_pipeline, t_train = train_xgboost(xgb_hist_params, X_train, y_train, num_rounds)

<ipython-input-17-500123296f4f> in train_xgboost(parameters, X, y, num_rounds)
      2     ddata = xgb.DMatrix(data=X, label=y)
      3     with Timer() as t:
----> 4         clf = xgb.train(parameters, ddata, num_boost_round=num_rounds)
      5     return clf, t.interval

/anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/training.py in train(params, dtrain, num_boost_round, evals, obj, feval, maximize, early_stopping_rounds, evals_result, verbose_eval, xgb_model, callbacks, learning_rates)
    202                            evals=evals,
    203                            obj=obj, feval=feval,
--> 204                            xgb_model=xgb_model, callbacks=callbacks)
    205
    206

/anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/training.py in _train_internal(params, dtrain, num_boost_round, evals, obj, feval, xgb_model, callbacks)
     72         # Skip the first update if it is a recovery step.
     73         if version % 2 == 0:
---> 74             bst.update(dtrain, i, obj)
     75             bst.save_rabit_checkpoint()
     76             version += 1

/anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/core.py in update(self, dtrain, iteration, fobj)
    825         if fobj is None:
    826             _check_call(_LIB.XGBoosterUpdateOneIter(self.handle, ctypes.c_int(iteration),
--> 827                                                     dtrain.handle))
    828         else:
    829             pred = self.predict(dtrain)

/anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/core.py in _check_call(ret)
    128     """
    129     if ret != 0:
--> 130         raise XGBoostError(_LIB.XGBGetLastError())
    131
    132

XGBoostError: b'[13:59:09] /home/hoaphumanoid/installer/xgboost_rory/plugin/updater_gpu/src/updater_gpu.cu:66: GPU plugin exception: [13:59:09] /home/hoaphumanoid/installer/xgboost_rory/plugin/updater_gpu/src/gpu_hist_builder.cu:47: Check failed: !data.empty() DeviceHist must be externally allocated\n\nStack trace returned 10 entries:\n[bt] (0) /anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/libxgboost.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f9286b89adc]\n[bt] (1) /anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/libxgboost.so(_ZN7xgboost4tree10DeviceHist4InitEi+0x8c) [0x7f9286d92c6c]\n[bt] (2) /anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/libxgboost.so(_ZN7xgboost4tree14GPUHistBuilder8InitDataERKSt6vectorINS_9bst_gpairESaIS3_EERNS_7DMatrixERKNS_7RegTreeE+0x1e8d) [0x7f9286d9a6ad]\n[bt] (3) /anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/libxgboost.so(_ZN7xgboost4tree14GPUHistBuilder6UpdateERKSt6vectorINS_9bst_gpairESaIS3_EEPNS_7DMatrixEPNS_7RegTreeE+0x2e) [0x7f9286d9c07e]\n[bt] (4) /anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/libxgboost.so(_ZN7xgboost4tree12GPUHistMaker6UpdateERKSt6vectorINS_9bst_gpairESaIS3_EEPNS_7DMatrixERKS2_IPNS_7RegTreeESaISB_EE+0x206) [0x7f9286d7c1c6]\n[bt] (5) /anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/libxgboost.so(_ZN7xgboost3gbm6GBTree13BoostNewTreesERKSt6vectorINS_9bst_gpairESaIS3_EEPNS_7DMatrixEiPS2_ISt10unique_ptrINS_7RegTreeESt14default_deleteISB_EESaISE_EE+0x8c3) [0x7f9286c04633]\n[bt] (6) /anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/libxgboost.so(_ZN7xgboost3gbm6GBTree7DoBoostEPNS_7DMatrixEPSt6vectorINS_9bst_gpairESaIS5_EEPNS_11ObjFunctionE+0xa60) [0x7f9286c058f0]\n[bt] (7) /anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/libxgboost.so(_ZN7xgboost11LearnerImpl13UpdateOneIterEiPNS_7DMatrixE+0x22b) [0x7f9286c1f46b]\n[bt] (8) /anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/libxgboost.so(XGBoosterUpdateOneIter+0x27) [0x7f9286bdcd77]\n[bt] (9) /anaconda/envs/xgb_gpu/lib/python3.5/lib-dynload/_ctypes.cpython-35m-x86_64-linux-gnu.so(ffi_call_unix64+0x4c) [0x7f92c2785540]\n\n\n\nStack trace returned 10 entries:\n[bt] (0) /anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/libxgboost.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7f9286b89adc]\n[bt] (1) /anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/libxgboost.so(_ZN7xgboost4tree12GPUHistMaker6UpdateERKSt6vectorINS_9bst_gpairESaIS3_EEPNS_7DMatrixERKS2_IPNS_7RegTreeESaISB_EE+0x3ba) [0x7f9286d7c37a]\n[bt] (2) /anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/libxgboost.so(_ZN7xgboost3gbm6GBTree13BoostNewTreesERKSt6vectorINS_9bst_gpairESaIS3_EEPNS_7DMatrixEiPS2_ISt10unique_ptrINS_7RegTreeESt14default_deleteISB_EESaISE_EE+0x8c3) [0x7f9286c04633]\n[bt] (3) /anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/libxgboost.so(_ZN7xgboost3gbm6GBTree7DoBoostEPNS_7DMatrixEPSt6vectorINS_9bst_gpairESaIS5_EEPNS_11ObjFunctionE+0xa60) [0x7f9286c058f0]\n[bt] (4) /anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/libxgboost.so(_ZN7xgboost11LearnerImpl13UpdateOneIterEiPNS_7DMatrixE+0x22b) [0x7f9286c1f46b]\n[bt] (5) /anaconda/envs/xgb_gpu/lib/python3.5/site-packages/xgboost-0.6-py3.5.egg/xgboost/libxgboost.so(XGBoosterUpdateOneIter+0x27) [0x7f9286bdcd77]\n[bt] (6) /anaconda/envs/xgb_gpu/lib/python3.5/lib-dynload/_ctypes.cpython-35m-x86_64-linux-gnu.so(ffi_call_unix64+0x4c) [0x7f92c2785540]\n[bt] (7) /anaconda/envs/xgb_gpu/lib/python3.5/lib-dynload/_ctypes.cpython-35m-x86_64-linux-gnu.so(ffi_call+0x1f5) [0x7f92c2784ce5]\n[bt] (8) /anaconda/envs/xgb_gpu/lib/python3.5/lib-dynload/_ctypes.cpython-35m-x86_64-linux-gnu.so(_ctypes_callproc+0x3dc) [0x7f92c277c7fc]\n[bt] (9) /anaconda/envs/xgb_gpu/lib/python3.5/lib-dynload/_ctypes.cpython-35m-x86_64-linux-gnu.so(+0x9d73) [0x7f92c2774d73]\n'

Max_depth should not be 0. Num leaves is unused - only set the depth. Don't set the updater, only tree_method.

I'm using these parameters:
xgb_hist_params = {'max_depth':8,
                  'objective':'binary:logistic',
                  'min_child_weight':30,
                  'eta':0.1,
                  'scale_pos_weight':2,
                  'gamma':0.1,
                  'reg_lamda':1,
                  'subsample':1,
                  'tree_method':'gpu_hist',
                 }

and I got the following result for the big dataset:
Computed XGBoost hist with 1e+08 samples in 1118.158s with AUC=0.603
Computed LightGBM with 1e+08 samples in 1042.223s with AUC=0.857
The time is much better than the original version, this computation corresponds in the post to airline subsample size 100M, 500 rounds, xgb_hist. Then the time was 2098s and AUC=0.856.
Would the parameters be ok?
Why there is such a difference in the AUC?
NOTE: I'm away for a week, so I might not be able to follow up with this conversation until the week after

AUC seems wrong. See here for a reference: https://github.com/szilard/GBM-perf

I did a fresh run in another VM with the commit you told me. I run 01_airline_GPU.ipynb.
Using these parameters for xgboost_hist and Lightgbm:
xgb_hist_params = {'max_depth':8,
                  'objective':'binary:logistic',
                  'min_child_weight':30,
                  'eta':0.1,
                  'scale_pos_weight':2,
                  'gamma':0.1,
                  'reg_lamda':1,
                  'subsample':1,
                  'tree_method':'gpu_hist',
                 }

lgbm_params = {'num_leaves': 2**8,
               'learning_rate': 0.1,
               'scale_pos_weight': 2,
               'min_split_gain': 0.1,
               'min_child_weight': 30,
               'reg_lambda': 1,
               'subsample': 1,
               'objective':'binary',
               'device': 'gpu',
               'task': 'train'
              }

I got something similar to the other results I reported, a speed up in the algorithm but a very low performance:
{
    "lgbm": {
        "performance": {
            "AUC": 0.8422509285268153,
            "Accuracy": 0.7257980036677117,
            "F1": 0.7462596869605246,
            "Precision": 0.6653401839514281,
            "Recall": 0.849587589195607
        },
        "test_time": 46.53701155199997,
        "train_time":<PHONE_NUMBER>320002
    },
    "xgb_hist": {
        "performance": {
            "AUC": 0.5708528394163612,
            "Accuracy": 0.5599637939038674,
            "F1": 0.6275373675179221,
            "Precision": 0.5244543226440088,
            "Recall": 0.7810562854618314
        },
        "test_time": 16.03199911200045,
        "train_time":<PHONE_NUMBER>940004
    }
}

In case you want to replicate the results, you have all the code available. If in the future the xgboost team fix the issue and improve the performance, you could repeat the experiments and do a new comparison. We will be happy to see it. For now I'm going to close this issue.


Downloaded file has the wrong filename and this doesn't seem to work for personal apps?

If downloading the file the filename is set to:
'microsoft-identity-configuration.json'
While the info says it should be named:
'microsoft-identity-association.json'
Additionally, this verification does not seem to work for (old) personal apps despite it being an option under Branding and with the exact json link working in a browser.
The error is:
"Verification of publisher domain failed. The application was not found. If the application was just created, wait a few minutes and refresh the page. [62wAT]"

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 61edea45-b5d6-05b3-33c3-2265ef8cb309
Version Independent ID: c7d5813c-5cf1-44c3-88ff-3c194d3e44ad
Content: Configure an application's publisher domain
Content Source: articles/active-directory/develop/howto-configure-publisher-domain.md
Service: active-directory
Sub-service: develop
GitHub Login: @rwike77
Microsoft Alias: ryanwi

@mol Thanks for your feedback! We will investigate and update as appropriate.

Thanks @SaurabhSharma-MSFT! Let me know if you guys need anything, like the app id or something else :)

Hey @mol can you provide some more context around the error? In your issue it says :
If downloading the file the filename is set to:
'microsoft-identity-configuration.json'
While the info says it should be named:
'microsoft-identity-association.json'
I don't believe the article references downloading anything.
And in regards to the old apps section of your question.
Can you please provide a screenshot of what you're referring to and what you mean by old apps?
There is a section that refers to grandfathered apps : https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-configure-publisher-domain#grandfathered-applications

Sure, I've made this image to illustrate both issues.
The downloaded file I'm referring to is there's button on the page with the json content text, that one can click to download a "ready-made" json file - to avoid having to create the file manually and copy/pasting the json into it. But the file has the wrong name. Probably a leftover from the old system of registering apps? In the image you can see the name of the file in Explorer and the name the page says it should have (neither name works as far as the other issue is concerned btw).
The other issue of the verification not working is illustrated by a browser pointing to the file and showing the correct json but the verification still fails. I'm thinking because it's a personal app, or maybe because yes it was grandfathered. It's not a big deal but it would be nice to get it verified since we can.
I'd also love to move the app to a directory to avoid it being a personal app, but as far as I know it's not possible?

Thanks for the information! We are currently looking into this with the product team and will get back to you as soon as possible,

And here's the app listing in Azure, to explain what I mean about an old app. I masked the email because public forum :)

Great thank you for that, I have the image, you might want to delete the picture because alot of your app id is showing.

As I understand it the app id is not a secret, but you're still suggesting me to remove the images? If you have both and it's fine then I can remove them.

@mol I'm following up on this issue, this should be resolved within the next two weeks. Please let me know if this is not resolved within that time period. Also my suggestion would be to submit a support ticket and escalate this issue that way accordingly. If there's any issues with the support ticket, please let me know the support ticket number and I can follow up accordingly. Also please reopen this git issue.
Is there anything else you would like addressed within the scope of this git thread? If not, I will be going ahead and closing this github issue.

Thanks @FrankHu-MSFT sounds good, go ahead and close it.


The layers should be categorized by type in the metamodel

Created by: brollb
The layers should be categorized by type (potentially the types described at http://autumnai.github.io/leaf/leaf/layers/index.html).
The decorators should also be updated to reflect these changes

Created by: brollb
Fixed as of https://github.com/brollb/cnn-creator/commit/ff385ff785eb8e4d8125cd868412b92822649ae9


500 error on code submit

kata:
https://www.codewars.com/kata/catch-mad-mars-bar-max/train/javascript
on code submit i have this output:
500: error
Description: TypeError: First argument must be a string, Buffer, ArrayBuffer, Array, or array-like object.
at fromObject (buffer.js:287:9)
at Function.Buffer.from (buffer.js:133:10)
at new Buffer (buffer.js:112:17)
at Object.module.exports.decrypt (/server/lib/cipher.js:8:18)
at transformBody (/server/lib/routes/run.js:41:32)
at module.exports (/server/lib/routes/run.js:14:15)
at Layer.handle as handle_request
at next (/server/node_modules/express/lib/router/route.js:131:13)
at Route.dispatch (/server/node_modules/express/lib/router/route.js:112:3)
at Layer.handle as handle_request
at /server/node_modules/express/lib/router/index.js:277:22
at Function.process_params (/server/node_modules/express/lib/router/index.js:330:12)
at next (/server/node_modules/express/lib/router/index.js:271:10)
at serveStatic (/server/node_modules/serve-static/index.js:75:16)
at Layer.handle as handle_request
at trim_prefix (/server/node_modules/express/lib/router/index.js:312:13)
This error was caused due to an issue processing the web request, not because of an issue executing your code.

I'm seeing this now on the word values problem.

#1370


LLMLingua2

I am asking if you are planning to add LLMLingua2 to your toolkit. Thx for your great work BTW!

Hi, thank you for asking, we do plan to add LLMLingua 2 to our toolkit.


Update Wiki Links

The updated links in the readme lead to a redirect form wiki to a document in docs/ folder.
Also for the docs/Checking-Abilities.md, it appears that those should be links.
Anyway, in general it's a bit confusing what has more precedence - the docs folder or the wiki? Since some links lead to wiki and some lay in docs folder and some pages in docs are just copies of the ones in the wiki

@lukasbischof
The docs are now the source of truth and the wiki is now obsolete, and can be dispensed with. Alas, I don't have the permissions for it.

I updated the PR to include:

docs/ as the source of truth
Relative paths to docs folder where applicable


refer folks to install from npm, instead of git-via-npm

Updated the docs for ya!

Seriously? Not even a comment?

https://github.com/cferdinandi/smooth-scroll/issues/244#issuecomment-203116559
Please remove this from NPM, fork it, and publish it yourself.


Recent files created doesn't show in folder  #68204 persists

VSCode Version: 1.31.1
OS Version: Mojave 10.14

Steps to Reproduce:

go to folder
click "add new file"
"name file"
Shown in the Explorer "Open Editors" but not the tree underneath
only shown after restarting VS code.
checked for updates.

Does this issue occur when all extensions are disabled?: Yes

Dup of #68204


Rootfs for debian?

When using debian stretch rootfs, I always get this error:
Installing...
ERROR:Installation Failed!
HRESULT:0x80004005
Press any key to continue...

Is kind of file is that?
What command did you use to create it?

I downloaded the rootfs from here: https://github.com/DDoSolitary/LxRunOffline/wiki
using this link: https://lxrunoffline.apphb.com/download/debian/stretch

you should convert rootfs.tar.xz to rootfs.tar.gz


Google Drive SSO redirect URI 401

Preflight Checklist

[x] I agree to follow the Code of Conduct that this project adheres to.
[x] I have searched the issue tracker for a feature request that matches the one I want to file, without success.

Describe the bug
Google Drive integration's redirect URI (app.diagrams.net/google?state=[...]) returns a 401, failing to authorize.
To Reproduce
Steps to reproduce the behavior:

Go to 'app.diagrams.net'
Click on 'save' and try to use Google Drive
Complete SSO sign-in steps to authorize Draw.io with Google Drive
On final step, redirect from google domain back to app.diagrams.net/google?state=[...], which returns a 401

Expected behavior
Google Drive integration should complete with a successful redirect and then confirm the authentication in draw.io's UI, allowing the user to save draw.io documents to Google Drive
draw.io version (In the Help->About menu of the draw.io editor):

draw.io version 13.0.9

Desktop (please complete the following information):

OS: Mac OS Big Sur 11.3.1
Browser: Chrome Version 90.0.4430.93 (Official Build) (x86_64)

I am seeing this same error. I am able to create and save a drawing to my root drive, while logged in with enterprise Google credentials. However, clicking on 'open existing diagram' opens a window with 'In order to select files, you must sign in to your Google account.' - clicking on 'sign-in' pops up a window that disappears; clicking it again pops up a window that disappears, and then I get
400. That’s an error.

The server cannot process the request because it is malformed. It should not be retried. That’s all we know.

And in incognito?

The only draw.io version listed in this thread is 13.0.9, which is really old and I wouldn't expect to work. Any problems with recent versions?

I'm using from diagrams.net; it shows 14.7.6

Do "enterprise Google credentials" vary from a Google Workspace account? I wonder if Chrome and the Google account combine to add some kind of restriction.


about  me-multi-cursors

hello @abougouffa
This file contains some multi-cursor packages
After my test   evil-multiedit is my favorite
but there is a small problem
When multiple regions are selected   ' kj '  Cannot deselect region
evil-escape work well   but can`t do   evil-multiedit-abort
I don't know how to fix it please help me

Yes, I'm aware of this issue. I've tried before to fix it, but I didn't manage to.
The evil-escape package supports evil-multiedit as you can see in it source code.
However, it seems that evil-multiedit do not manage the evil-state correctly. For example, when pressing g z e m to mark all matches (evil-multiedit-match-all), when you position the cursor on one of the occurrences and M-: then evaluate evil-state, it will show normal instead of multiedit. So, with this behavior of evil-multiedit, the evil-escape-func won't work as predicted.

I tried to fix him via chatgpt but failed


Roadmap list/create/edit section

WIP - add comments

#3403

This pull request introduces 2 alerts when merging bb3d08ebef450c1d36c442db7a0245feb4746b43 into 235c630abc8dc39e475fb469e318e7e4952dbded - view on LGTM.com
new alerts:

2 for Unused variable, import, function or class

Comment posted by LGTM.com


Fail to parse complex sentence

Hello,
First of all, thanks for your code.
I tried the following example and it failed:
 Despite these accomplishments, he was never fully recognised in his home country during his lifetime due to the prevalence of homophobia at the time and because much of his work was covered by the Official Secrets Act.

['Despite these accomplishments , he was never fully recognised in his home country during his lifetime due to the prevalence of homophobia at the time', 'Despite these accomplishments , he was covered by the Official Secrets Act']

Sorry for very late reply.
Yes, the code works for simple sentence structures only. It will fail to parse complex sentences. Also, it completely depends on the parser we are using, if the parser parses the sentences differently than expected, then it will not work. And the open source parsers are not the accurate when handling complex sentences.

Closing for now.


Add tests

Let's run the tests for notebook!

Hi! This is the friendly automated conda-forge-linting service.
I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

Rebased, also seeing if the issues I am encountering on windows also occur here.

Oh, and dropped the jstests :)

Probably makes sense to bump the version number, then I would be 👍  on this PR.
Thanks @bollwyvl

Looks like real test failures on a few trivial tests on Windows. These need patches upstream. I'll look into those. In the meantime, we can either skip the tests on Windows here or hold this PR until there's another release of the notebook.

Might just need some judicious skip_ifs...
On Wed, Jun 29, 2016 at 10:31 AM Min RK<EMAIL_ADDRESS>wrote:

Looks like real test failures on a few trivial tests on Windows. These
need patches upstream. I'll look into those. In the meantime, we can either
skip the tests on Windows here or hold this PR until there's another
release of the notebook.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/conda-forge/notebook-feedstock/pull/6#issuecomment-229374134,
or mute the thread
https://github.com/notifications/unsubscribe/AACxRPM5xa_1l7oWgp6IIAUPUqvImfhHks5qQoG2gaJpZM4InzY-
.

Yeah, it's probably a skip or two and a path fix or two.

Closing this one as stale. @bollwyvl please re-open if you still think this is necessary.


feat: high-level feed api

This is draft PR as I want to hear your opinion about my approach. Once we settle on some approach I will finish the PR with test coverage.
I was thinking about how to approach this with consideration that we already have some feed components in place. I came up with an abstraction layer that I named DataFeed which supports easily setting and getting data to/from a feed. I am not sure about the naming, but since we have already "writer" and "reader" for feeds I was a bit lost and came up with this name. If you have other suggestions, shoot them!
Also introduced direct methods on Bee that matches similar API that Michelle proposed.
Closes #174

I just want to mention again my proposal which serves this use-case in an optimized way: https://github.com/ethersphere/bee/issues/1311

I am not sure if it is needed to put JSON in all the naming. Most people think about JSON as JS objects, while JSON.parse/stringify is able to handle most of other data types as well like simple strings, numbers, arrays etc.

I just wanted to mention that we also have our Data type that supports conversion to JSON, text and hex. Did you consider using that or something similar? I think it would give consistency to our API if we reused the same types in similar use-cases.

I am not sure if it is needed to put Json in all the naming. Most people think about JSON as JS objects, while JSON.parse/stringify is able to handle most of other data types as well like simple strings, numbers, null, arrays etc. which I am to support as well.

Well, I think Json is JSON regardless what devs think and the types also notate this functionality. Don't you think it describes itself and make clearer in functionality? Because you cannot pass any Data in that sense like Uint8Array.

@agazso good point about the Data. I did not really think about that. Hmmm I am not sure if having that extra layer to access the data is desired though (the extra .json() call).
I guess this brings two possible paths on how to move further with this PR.

Use Data interface as we use in other APIs. In that case, I would keep the naming as is now.
Go with JSON-only content. In that case, I would refactor the names based on @nugaon suggestion.

What do you think?

I really appreciate all of the hard work going into this, it is inspiring!  I am wrapping up with quarter end, and will be diving in deep to feeds in bee-js soon... my particular use case involves uint8 array encoding of video data, and while I will definitely need json functionality as well, I wonder if forcing all data types through json conversion is necessary for your application of this module.  However, as long as there remains a method for editing feeds that are not json formatted somewhere in bee-js, I should be just fine....

I really appreciate all of the hard work going into this, it is inspiring! I am wrapping up with quarter end, and will be diving in deep to feeds in bee-js soon... my particular use case involves uint8 array encoding of video data, and while I will definitely need json functionality as well, I wonder if forcing all data types through json conversion is necessary for your application of this module. However, as long as there remains a method for editing feeds that are not json formatted somewhere in bee-js, I should be just fine....

Hey @IxaBrjnko, thanks for the kind words!
It will be always possible to work with binary data with the low-level FeedReader/FeedWriter API.  For example a similar implementation for working with Uint8Array would look like this:
// value is of type Uint8Array
export const setUint8ArrayFeed = async (topic, value, pk) => {
    try {
        const encodedTopic = await bee.makeFeedTopic(topic)
        const swarmReference = await bee.uploadData(value)
        const feedWriter = bee.makeFeedWriter('sequence', encodedTopic, pk)
        const response = await feedWriter.upload(swarmReference)
        return response
    } catch (error) {
        console.error(error)
    }
}
export const getUint8ArrayFeed = async (topic, address) => {
    try {
        const encodedTopic = await bee.makeFeedTopic(topic)
        const feedReader = bee.makeFeedReader('sequence', encodedTopic, address)
        const feedUpdate = await feedReader.download()
        const data = bee.downloadData(feedUpdate.reference)
        return data
    } catch (error) {
        console.error(error)
    }

}

So I have taken the path of JSON-only adoption for this PR so I have refactored the naming as @nugaon suggested.


[Python] Implemented Hugging Face Model Handler

This PR adds a model handler for loading PyTorch and TensorFlow model handler from Hugging Face Hub.
It builds on this Design doc
Example Job on Dataflow with huggingface_language_modeling.py

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
[ ] Update CHANGES.md with noteworthy changes.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

Run Python 3.11 PostCommit

Run Python 3.8 PostCommit

waiting on author

@AnandInguva Robert approved the design. Can you take a final look before merging this?

Run Python 3.8 PostCommit

Run Python 3.11 PostCommit

Run Python 3.8 PostCommit

Run Python 3.11 PostCommit

Python 3.8 PC passed for huggingface_inference_it_test

Python 3.11 failure is because of https://github.com/apache/beam/issues/27643

Merging this!


Procfs v0.8.1 does not support kernel version string larger than 256

Bug description
Linux kernel version: 4.19.270. When I tried to run scaphandre (version 0.4.1) with qemu exporter, I got the following error:
Sending ⚡ metrics
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Other("Failed to parse patch version")', /home/XXX/.cargo/registry/src/github.com-1ecc6299db9ec823/procfs-0.8.1/src/lib.rs:315:34
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

To Reproduce
Directly compile the code.
Expected behavior
Qemu exporter should be started.
Environment

Linux distribution version [Ubuntu 18.04.6 LTS]
Kernel version (output of uname -r) [4.19.270]

Additional context
After a deep dive, the procfs version v0.8.1 still uses u8 as the kernel version patch string, and any number larger than 256 with cause a panic. A simple workaround for me was to compile my own procfs-v0.8.1 but change the patch to u16, following this patch. A better solution would be upgrading procfs version to v0.10.0 or newer because the patch was applied to that version and above. It's not a simple fix, as procfs changed the interface from floating point numbers to integers for CPU information. Details can be found here and here.

Closing until someone speaks up :)


Tree Style Tab has a default style on Windows

Hi, thanks a lot for providing your instructions and configs.
I do have a question, because it looks like Tree Style Tab doesn't have the expected look and behavior. Even with the custom css and config in json it still acts as a static panel, unlike the one shown in the examples. Is this a known issue?

Have you applied userChrome.css as well? None of its changes seem to have been applied in that screenshot. After doing that you'll have to restart firefox completely for it's effects to show up.

You're right, I had to move userChrome.css to a new folder called chrome within my profile folder for it to work. Cheers!

Great. Closing the issue then.

I seem to be having this same issue, except I've already moved the files to that directory. Tried restarting/resetting Firefox, but the changes still won't take effect. Also have legacy stylesheets enabled.

@zaobas Does your firefox look exactly like the screenshot in this issue? If userChrome is not taking effect at all then it's most likely problem with userChrome.css placement. Also I have updated since yesterday so you'll also need to copy vars.css to your chrome directory as well.

Yeah, it looks exactly like that; none of the changes take effect. I've tried moving both userChrome.css and vars.css to a new chrome folder within the profile folder itself, to the native chrome folder, within storage, and even within the AppData/Local profile. I've made sure the profile in the screenshot is my active profile.

OH you'll have to remove -hover from that file's name so rename it to userChrome.css. I put the -* extensions to make it easy to differentiate between all the variations of the sidebar. Restart firefox after renaming and it should work.

Ah! That worked. Thanks for the help.


Introduce -faggressive-inlining

Hi Jonathan, thanks for the tip yesterday about the inlining phase! It does make our code look much nicer.
In fact, I was thinking I would like this to be always done regardless of name and without requiring any attributes, so thought about adding this option, do you think it would be useful for others?
I could also add an attribute-based mechanism like you suggested if that's useful. If so, and if this option isn't desirable, maybe I can just have Pulse always add the attribute.

Personally, I'd rather have an attribute, since currently we're doing a bunch of hacks in HACL-rs to name some variables uu_ to avoid intermediary let-bindings that cause move-outs in Rust (see https://github.com/hacl-star/hacl-star/blob/afromher_rs/code/streaming/Hacl.Streaming.Functor.fst#L270 for instance). It would be cleaner if we could mark these let-bindings as [@inline] with a descriptive name (instead of uu_).
But I also appreciate that the "clean" solution is probably a little bit of work to propagate, so I'm fine with the flag too. Feel free to merge is that makes a significant difference for you (with the understanding that none of my projects will use this flag, so you'll have to keep an eye on it to make sure it doesn't regress).

Sounds good, let me try that and get back.

This was superseded by #470. I don't really need the flag anymore, hence closing, but I rebased it just in case it becomes useful at a point. We could also consider an F* flag to always add the attribute.


Security Issue when modifying DOM

Here is a possible security issue.
When you load a page which contains the editor and you insert the following line in the part where the toolbar is:
<a data-wysihtml5-action="change_view">switch to html view</a>

Then an attacker can insert anything into the form.

I think fixing security issues on front end side, where attacker is the owner/viewer of the browser, is already a lost cause. All browsers provide such thing as console under right click, giving owner of the browser a convenient ui for any thinkable modification or running any script he wants. This level of security has to be provided server-side unfortunately.


Update Debian GPG instructions to bring inline with best practice

As per https://wiki.debian.org/DebianRepository/UseThirdParty
"The key MUST be downloaded over a secure mechanism like HTTPS to a location only writable by root, which SHOULD be /usr/share/keyrings. The key MUST NOT be placed in /etc/apt/trusted.gpg.d or loaded by apt-key add. A sources.list entry SHOULD have the signed-by option set. The signed-by entry MUST point to a file, and not a fingerprint."
Should you require further background beyond the above Debian wiki link, see also:
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=861695
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=877012

Intresting. Thank you!

This broke site generation because & and > have to be replaced with their HTML/XML escape sequences but I understand that it is not obvious and will correct.

@michaelklishin Thanks & sorry about that !

No worries. I learned a lot after reviewing this. The instructions do look more involved now but no objections from me.
@udf2457 this will go live in a few minutes after I push my escaping fixes. I have noticed that there are .list file snippets in the guide that were not updated. Should they? or would it not make much sense to the reader unless they keep in mind how/where the key was installed?

There are also apt instructions in the Signatures guide. Ideally we should use the same
approach across the board, even if most people will use the already updated snippet. This may need a bit of wording changes to explain
what those lines do, e.g. as inline .list file comments. WDYT?

@michaelklishin I think you're right, they should have been updated.  By the same token (unless I'm misreading the page) I get the impression there is also quite a lot of repetitive duplication going on that page ?  Might be time for some extravagant use of the delete key for certain sections ?
Good spot on the signatures guide, should be updated too as you say in order to keep inline.

@michaelklishin Also, if we are going to be doing more doc updates in this area, then it would be neat if you could ensure binary pubkeys are available everywhere.  This would:

Ensure things remain inline with Debian best practice ("The reason why we avoid ASCII-armored files is that they cannot be used directly by SecureApt." from the Wiki)
Would remove/reduce the need to pipe to gpg --dearmor in the docs and so keep things neater.

i.e. https://github.com/rabbitmq/signing-keys/releases and on packagecloud too.

We can put out a new "release" in rabbitmq/signing-keys but we do not control PackageCloud or Cloudsmith keys. They are provided by those services.

So https://packagecloud.io/rabbitmq/rabbitmq-server/gpgkey is not uploaded by you ? (Apologies, I'm not familiar with the workings of packagecloud, have only ever been a consumer of the service, never a publisher on there).

@udf2457 there is some repetition because we cover multiple package services. Let's leave them as is and focus on hardening the part related to keys. We would be happy to produce a binary version of the key but it's not something I can promise to happen in a week or two in a high PTO season. I can merge a reasonable docs PR very quickly, though ;)

@udf2457 correct, PackageCloud generates a key per repository (used to be a key per org account IIRC). We do not control it.
Cloudsmith allows for custom signing keys but OSS plans do not include this feature, so we rely on GPG and RSA keys generated by the service and only can regenerate it at any time.

@michaelklishin Thanks for the clarifications.
What are your plans for a Bullseye Debian build given it was released yesterday ? (https://www.debian.org/News/2021/20210814)
Maybe I should wait for that to happen before looking at taking a bit more time to update the remaining bits of the docs ? WDYT?

Our Debian package should work just fine on Bullseye. The package is the same for all distributions. Our Erlang package has to be adapted but last time we tried it failed to build on Bullseye for reasons we could not understand.

I don't see any reason to provide Bullseye-specific docs immediately. I doubt they will be meaningfully different.
Bullseye ships with Erlang 23.2 which is sufficient for running RabbitMQ through mid-2022 (when Erlang 25 will be released), so we are not in rush to produce our Debian packages for Bullseye.

Have posted update for signatures page, will work on the Debian page separately.
Fun fact, I guess nobody uses the signatures page as you had quite a major typo in the old instructions apt-key adv  (adv vs add) !


fix GCCVER cmake define.

Summary
There are some errors in cmake's definition of GCCVER, which causes cmake to not find the definition of GCCVER when compiling. https://github.com/apache/nuttx/issues/14412 This has been fixed.
Impact
libcxx
Testing
ci test

[Experimental Bot, please feedback here]
No, this PR does not meet the NuttX requirements.
Missing Information:

Summary:

Lacks a clear explanation of the functional code changes.
Needs to specify how the GCCVER issue was fixed.

Impact:

"libcxx" is not a sufficient description.
Needs specific details on the impact across all categories (user, build, hardware, documentation, security, compatibility).

Testing:

"ci test" is not acceptable.
Provide specific build host details (OS, CPU, compiler version).
List the target architectures and boards used for testing.
Include actual testing logs (before and after) demonstrating the issue and the fix.

Recommendations:

Expand the Summary: Clearly describe the technical changes made to fix the GCCVER definition in CMake.
Complete the Impact Section:  Address each impact category with "YES" or "NO". If "YES", provide a detailed explanation. For example:

Impact on build: YES - The build process is now more robust as it will correctly identify the GCC version.

Provide Detailed Testing Information:

List all build hosts used, including OS, CPU, and compiler details.
Specify the target architectures and boards where the issue was observed and the fix was verified.
Include relevant snippets from your testing logs that clearly show:

The error messages encountered before the fix.
The successful build and execution after applying the fix.

By providing this essential information, your PR will be more likely to be reviewed and merged quickly.


Can't install on linux (Depencies file ? )

Trying to install on Linux (Ubuntu 22) , tried install different modules that were missing . however i still get some missing .
are there any apt / list  install that includes all the requirements ?
cmake .. -DCMAKE_BUILD_TYPE=Release
Compiling for ~/Qt6
Compiling pre
Compiling tcn
-- Could NOT find WrapVulkanHeaders (missing: Vulkan_INCLUDE_DIR)
-- Could NOT find WrapVulkanHeaders (missing: Vulkan_INCLUDE_DIR)
-- Could NOT find Qt6Core5Compat (missing: Qt6Core5Compat_DIR)
CMake Error at CMakeLists.txt:41 (find_package):
  Found package configuration file:

    "MY PATH"/Qt6/6.6.2/gcc_64/lib/cmake/Qt6/Qt6Config.cmake

  but it set Qt6_FOUND to FALSE so package "Qt6" is considered to be NOT
  FOUND.  Reason given by package:

  Failed to find required Qt component "Core5Compat".

  Expected Config file at
  "MY PATH"/Qt6/6.6.2/gcc_64/lib/cmake/Qt6Core5Compat/Qt6Core5CompatConfig.cmake"
  does NOT exist

  Configuring with -DCMAKE_FIND_DEBUG_MODE=TRUE might reveal details why the
  package was not found.

  Configuring with -DQT_DEBUG_FIND_PACKAGE=ON will print the values of some
  of the path variables that find_package uses to try and find the package.

-- Configuring incomplete, errors occurred!
See also "MY PATH"/Downloads/FigmaQML-3.1.0/build/CMakeFiles/CMakeOutput.log".

Have you installed Qt5Compat using the Qt MaintenanceTool ?

I plan to ditch that in the next version, but currently there is dependency to it.

Have you installed Qt5Compat using the Qt MaintenanceTool ?

yeah , that solved it , Thanks .
It would greatly help if there are a list of dependencies to track what was missing :)


unx86ing atomic functions

The functions in util/atomic.*  have a bunch of x86 assembly which is less than ideal on non-x86 hardware (like this 96 core arm64 I am playing with), and it would be nice to use some type of cross-platform atomic operations, either the ones provide in std::atomic, or a linux kernel vdso

I agree that they are less than ideal.  I have considered similar changes before (see Issue #19).
In this case, it's a little different because it opens up a new hardware platform.  Inlined assembly is an important part of what makes things work.  While I could fall back to slow primitives on non-x86, I would be hesitant to do this.  Worse would be blindly adding the arm64 code to support your platform, because its visibility semantics are different enough from x86 that I'd want to personally test and verify the changes because the memtable---a piece critical to hyperleveldb's changes and most operations---relies heavily on these changes.
I assume you may have one of the System 76 servers.  I've asked them about getting one to play with so I could support changes like this.  Unfortunately, I'm not able to personally afford one of these servers.  System76 didn't respond to my tweets, so I assume they either have no wiggle room and didn't want to disappoint or are disinterested in a one-off sale.
So while I'd love to support arm64, I cannot do so at this point.
I'll leave the issue open in case others want to chime in, or want to support me in purchasing such a server so I could port hyperleveldb to work on the platform.

I'd send you some money for a day or two with https://www.packet.net/bare-metal/servers/type-2a/ (which is what I was using), or http://shop.technexion.com/wb-imx6q-bw.html if you want something more sustainable.
Then there is also PowerPC (IBM's Power9 is going to be pretty interesting), and Risc-V, which is going to get pretty interesting in a few years. To me, this seems like having a flag for either x86-asm, or C++11 atomics would be a potentially maintainable solution. I would then use the C++11 version on x86 even if the performance was a little worse

I appreciate your perspective.  I've talked with some people close to the RISC-V effort in the past and am looking at that architecture as well.  PowerPC is less on my radar as IBM tends to be quite expensive.
For any of these architectures. there is a lot that needs to be tested.  Using std atomics might work swimmingly, but then the OS might have problems.  When I worked on Bitcoin's LevelDB bounty, the bug was related to the way the OS and LevelDB treated memory maps.  It was not a bug likely to be triggered on Linux, and possibly is dependent upon the vmm subsystem's treatment of pages.
If you would like to fork this project to add std::atomics, I would happily update the readme to make people aware of the fork.
As it stands, my primary project is Consus, for which I will probably write a new backend.  I would love to have a high-core-count server to test the project on where I have it persistently available to me for work.  HyperLevelDB is not a priority for me and adding architectures and testing it will require quite a bit of time that I would rather allocate to other projects.  If, however, I was able to get out of it some more resources to help with those other projects, I would be willing to put in the time and effort to make the changes.  I appreciate your offer to donate some resources that would be sufficient to test the implementation.  Unfortunately, it is a matter of both the resources and the time.  Adding a persistent resource to my workflow would be a bonus worth the time it would take to support arm, and I was implying that's what I was trying to get out of System 76.  Re-reading my above message makes it sound like it's only about the resources for this particular change and I'm sorry for the confusion.

I am going to see if I can lean on Cavium get a ThunderX 1k reference board (48 cores). That would make up for the bad deal us engineers got in the acquisition.
Otherwise, I'm much more familiar with PowerPC assembly, and at some point I'm going to get a power8/power9 one way or another, and Consus would be a very interesting to see on whichever platform I end up with.
To fork for std::atomics, I would want to have a little better idea of what the x86 asm code does, and what might be different in the Arm/PPC relaxed memory model as compared to x86.


Document and lock down bevy_ecs::component APIs

Objective
Document and lock down bevy_ecs::component.
Addresses #3362 for bevy_ecs::component.
Solution

Make mutative APIs on Components pub(crate). Redirect users to the APIs on World instead of Components.
Remove the Components: Default implementation. Replace it with a pub(crate) new  function. Forces Components to be initialized in the context of a World.
Rename Components::get_resource_id to resource_id to make it consistent with Components::component_id
Make ComponentId a fully opaque ID.
Make ComponentDescriptor constructors private except for the user-facing case for making components for foreign types.
Make ComponentDescriptor more of a parameter type than a storage type by removing it's accessors. Force users to use ComponentInfo and getting it via Components's fetch APIs.
Document as many of the undocumented types and APIs in the module as possible.

Changelog
Changed: Components::get_resource_id has been renamed to Components::resource_id.
Changed: ComponentDescriptor::new_with_layout has replaced ComponentDescriptor::new.
Removed: Components's Default implementation.
Removed: Components::init_component, Components::init_resource, Components::init_non_send, Components::init_with_component_descriptor.
Removed: ComponentId::new and ComponentId::index.
Removed: ComponentDescriptor::new_resource, ComponentData::type_id, ComponentDescriptor::name, ComponentDescriptor::storage_type.

@jakobhellermann's comments are excellent: I'm in agreement with all of those :)

I'm unclear of the comments from https://github.com/bevyengine/bevy/pull/6750#discussion_r1032121807 WRT the publicness of init_resouce have been addressed. I don't think it's acceptable to break / shut off dynamic resource types in this PR, but I'm not personally sure on the right path forward.

I get very worried and uneasy whenever I see PRs for "make things no longer pub, because things are excessively pub".
You always run the risk of shutting down more exotic usecases that you might not have thought anyone would want to do.
You risk causing pain to future users who have a clever idea for something they want to do, but can't, because you made the things they need private.
One of Bevy's great strengths is how easy it is for people to hack on, and how so many things can be implemented in 3rd-party crates.
Please don't "lock down" Bevy!

Please don't "lock down" Bevy!

I've stated this on Discord but I'll reiterate it here for better logging of the rationale.
I'm in general agreement here about the use of pub APIs, within reason. The primary exception lies with bevy_ecs, where the unsafe APIs are riddled with implementation specific invariants. It's borderline impossible to use them safely outside of the crate, so the few ones we do expose should be well audited, well documented, and damn near stable. Making them pub means we must take them into account inside bevy_ecs too, which can combinatorially expand the number of ways both first and third party crates and bevy_ecs itself can cause unsoundness.
Even outside of unsafe use of bevy_ecs, it's imperative we stay conscious of the API surface we're exposing. The modularity of the engine demands we expose certain APIs at crate boundaries, but that also means that we can't change it once we hit 1.0. Hyrum's Law states that if it can be observed, someone will depend on it. If we are to make more of the engine public, the interface should be more curated and deter users from footguns, especially when unsafe is involved.

@james7132 I have agreement in principle with you on these choices, but there's some outstanding controversy. This is in the 0.11 milestone but things seem to have stalled: what would you like to do with this PR?

Let's move it to 0.12. There's quite a few things that need to be addressed here and I don't think we can rush them out before the 0.11 deadline.

Closing this out as there are other PRs that got to this beforehand.


add aarch64 support. (#1)

feat: aarch64 support

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Add MIPS variant to Machine type

Closes #94 .

Coverage remained the same at 0.0% when pulling 71aa76c749b8278f0e63e92eaa589ec0db29c96a on lzutao:mips into 655dc57a26c8ebc3ca97b361e39a61b328c444d4 on gimli-rs:master.


Changelog workflow

Add a workflow that checks if we updated the changelog

You can see it working in the workflows in the commit history of https://github.com/dfns/generic-ec/pull/37


Typo "@sid"→"@sid"

bold with escape characters
https://docs.microsoft.com/en-us/sql/sql-server/install/rename-logins-matching-fixed-server-role-names?view=sql-server-2014

@hyoshioka0128 : Thanks for your contribution! The author(s) have been notified to review your proposed change.

@hyoshioka0128 Thank you!


Update dependencies

Coverage remained the same at 89.88% when pulling ef509adc12e6af87e54f3886d4301627872c5be8 on rel_branch into 379032961b1ba9f4591f6f10efb8a4e00dd291e0 on master.


testing, comparing tarballs created with zenith with result of PITR of PostgreSQL

@lubennikovaav done, I suppose?

ah, not yet merged


@JuliaRegistrator register

@JuliaRegistrator register

Registration pull request created: JuliaRegistries/General/73157
After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.
This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:
git tag -a v0.7.3 -m "<description of version>" 7a1b9471a8eb569c28ee22098f8160bf9d8f6f07
git push origin v0.7.3

Also, note the warning: This looks like a new registration that registers version 0.7.3.
Ideally, you should register an initial release with 0.0.1, 0.1.0 or 1.0.0 version numbers
This can be safely ignored. However, if you want to fix this you can do so. Call register() again after making the fix. This will update the Pull request.


Generate code for JS/TS projects

Follow up from #27
Make Buffrs able to generate some javascript and/or typescript code.

https://github.com/helsing-ai/buffrs/issues/27#issuecomment-1717351674


Feature/plan

Coverage increased (+2.4%) to 85.167% when pulling b6c41bdef0c2bab13c8f1ead508aa9f4c4dad3e7 on feature/plan into 9bbb87d14cba9cbed4d1afd1997bf8a553336823 on V2.

Coverage increased (+2.4%) to 85.167% when pulling f2a74a60f66e96021a730c2a499ac0d9273d5336 on feature/plan into 9bbb87d14cba9cbed4d1afd1997bf8a553336823 on V2.


Implement #126 Compose messages in external editor

I found it not that difficult to implement this feature that I wanted.
If there are some issues with the current implementation, I would be glad to fix them.
PR adds a new command :editor that opens "$EDITOR" in the current window or in a new one (if $EDITOR is a GUI one), after saving the file, the message will be sent immediately.
Also, current text from a buffer (if any) will be used as the initial content of the file.

after saving the file, the message will be sent immediately.

I think it's better to send the message only after leaving the editor, no? Sometimes I will save what I have and keep writing.
In any case, thanks for adding this feature.

 Hopefully, the crate I used works exactly the way you described, just didn't realize that at first, because tested it only with :wq. So I have fixed the description.

Hey, I compiled from your commit but :editor returns  Failed command: invalid command: editor.

Are you sure that you running the version from my commit? Maybe you have compiled the release but running an old debug version or vice versa.
If you are on Linux nm target/debug/iamb | grep iamb_editor  (or release) should succeed. If not, you are running the wrong binary.

@capgelka This looks great, thank you for implementing this! I added some code to force a redraw of the screen afterwards, since at least for me locally I had leftover characters on the screen after the subprocess exited.

I hope it's ok to ask about this now that the commit has been merged,
but i'd like to know what I did wrong when trying @capgelka 's commit.

I'm on linux,
ran git clone "https://github.com/ulyssa/iamb" && cd iamb
then git fetch origin pull/155/head:MASTER
git checkout MASTER
See capgelka's changes there
cargo install --force --locked iamb
cd ~/.cargo/bin && ./iamb

Thanks in advance

I also hope that it's ok to have such a conversation, though probably better to use stack overflow for such questions anyway.
To install a package from the current directory one has to use . as an argument, so you have just reinstalled iamb from crates.io

Yep, continuing commenting here is fine!
As @capgelka said, the positional argument to cargo install is the name of a crate on crates.io to install. You can use cargo install --path /path/to/iamb/repo/ to install from the directory. You can also do cargo build --release, and copy the binary from ./target/release/iamb.

I should have noticed when the installation section of the README doesn't mention cloning the repo...
Anyways, i tried again, now with --path and it works just fine.
Thank you both for the explanation and sorry for dragging this conversation.


Idea: Dash apps as JupyterLab mime renderers

This idea is not very fleshed out yet, and I'm not sure if it should even belong in this repo, but I wanted to write it out while I'm thinking about it.
JupyterLab supports the very flexible concept of Mime Renderer Extensions, which allow 3rd party extensions to create custom viewers/editors for various file types.
These are really powerful, but they involve writing the extension in JavaScript, and if the extension needs access to Python functionality there needs to be a corresponding server side extension.  See see the jupyterlab-latex for an example of an extension that does mime rendering with a Python server component.
Because of this, the creation of custom mime renderer extensions is not very accessible to Python analysts without experience with web technologies.
Dash is now a mature technology for building sophisticated react-based web apps in pure Python. What would it take to allow special Dash apps to be registered as mime renderers? I'm picturing that the Dash apps would be registered as server extensions, and then a single JupyterLab extension would look for these server extensions, route mime renderer requests to the appropriate Dash app, and display the app in an iframe.
This would give extension authors access to a Python kernel and a rich set of react components, all from Python.
Two examples come immediately to mind.

A Dash DataTable based viewer/editor for tabular datafiles.  If the file could be loaded on the Python side then it wouldn't need to be all sent to the front-end at once, making it possible to support really large files.  Could also support other tabular file types with Python readers like parquet.

An image editor based on Dash-canvas and scikit-image

cc @chriddyp @jackparmer @nicolaskruchten @gnestor @ian-r-rose

Hi @jonmmease I agree this would be really cool.
Coincidentally, @mrocklin and I spent some hours trying to do something very similar yesterday, but with bokeh dashboards. This can be found here The basic idea is as follows:

Developers of dashboard-type plots may be much more comfortable writing in Python than Javascript (in this case bokeh, but could just as easily be dash).
These dashboard plots are available via some webserver exposed via some port. The webserver is proxied under the notebook server via a server extension (in this case jupyter-server-proxy).
There is an "index" endpoint, informing the frontend what dashboards are available.
The frontend JupyterLab extension learns about the available dashboards via the index and constructs a launcher for them.
When the dashboards are launched, they are opened in an iframe in JupyterLab as one of the main area panels.

This doesn't take a mime-renderer approach, since there is a bit more logic on the frontend than those allow. With a bit of work, however, it should be possible to make the frontend mostly agnostic to who is providing the dashboards.
A similar approach would be to use the jupyterlab extension functionality that @yuvipanda has built into jupyter-server-proxy. With this you add some relatively limited metadata to an entrypoint and it automatically creates a JupyterLab launcher item which, when clicked, starts a separate server and opens it in a separate browser tab (rather than in a main area widget). This is used, for instance, to launch rstudio processes from the notebook server.

@ian-r-rose this sounds great to me! I'd welcome a PR to jupyter-server-proxy's jupyterlab extension that lets you open it in an iframe inside a JupyterLab tab. I think that, plus some more dynamism in how proxies are registered (which you need for the dask labextension anyway) would probably be a nice generic form of doing this.

Thanks for your perspective @ian-r-rose and @yuvipanda!  This isn't something I have immediate plans to work on, but I'll definitely take a look at building on/extending jupyter-server-proxy when I do.

@jonmmease If I understand you correctly, you're proposing that we create mimerender-like extensions for individual dash components (like dash-table and dash-canvas) vs. entire dashboards? So that you could open a CSV with "Dash DataTable" and PNGs with "Dash Canvas"? I really like this idea because plotly has developed so many components and JupyterLab provides a simple way to provide data to them. If these components require access to a kernel (e.g. fetching rows from a CSV on disk), then a mimerender extension would not suffice. However, I recently converted a mimerender extension to a regular extension specifically to access a document's session, so that can serve as a simple example of how to create kernel-connected renderer extensions.

@gnestor thanks for sharing the mimerenderer -> extension example, that looks like a really helpful reference.

If I understand you correctly, you're proposing that we create mimerender-like extensions for individual dash components (like dash-table and dash-canvas) vs. entire dashboards?

I wasn't thinking of necessarily restricting this to individual components. It some cases I expect it would be helpful for these mime-style renderers to contain multiple Dash components and various Python callbacks.
And my hope was that we wouldn't need to create a separate JupyterLab extension for each dashboard/filetype pairing.  I'm imagining that the dash-mimetype-renderer author would specify the mimetype registration info (e.g. file extensnion) on the Python side in some standard way. Then we would have a single dash-mimetype-renderer JupyterLab extension that would query the Python side for this info, and then register itself as a mimetype renderer for each one.  Then the extension would be responsible for dispatching "Open with" requests to the appropriate Python dash-mimetype-renderer implementation.
If possible, this would allow folks to create new file renderer extensions in pure Python. And users could install new extensions using pip/conda without needing to rebuild JupyterLab.


Update DynamicUIColor to use UIColor(dynamicProvider:)

This creates a color that will auto-update on iOS 13, and falls back to the light color on any version less.
I'm very unsure about my approach with the tests - what's best practice for testing different iOS versions behavior? Open to suggestions.

Hey @kylebshr thanks for the pull request.
My original implementation had two goals:

Supporting systemwide dark mode in iOS 13 while trying to
Offer a usable alternative to implement a custom dark mode (e.g. from app settings) in iOS <= 13.

With your changes the latter is lost. I am just thinking how to reintroduce this while keeping your  UIColor(dynamicProvider:) changes.
My best idea so far is to reintroduce the 3rd parameter with a different shape. It could be an optional closure or a closure that returns an optional mode, where absence of value could mean using the system logic.
Feel free to do any change in this direction; otherwise I will try to help later today/tomorrow!
Thanks a lot for contributing and best regards.

@kylebshr how do you like it?
I think it is a nice compromise between a nice default simple API, customisation possibilities & backwards compatibility.

Hey, sorry I didn't have time to take a look sooner - seems like a good approach 👍


doc: add Go API documentation index files for storage backends

Describe what this PR does
https://pkg.go.dev/github.com/ceph/ceph-csi/api does not list a documentation string for the cephfs, nfs and rbd packages.
Checklist:

[x] Commit Message Formatting: Commit titles and messages follow
guidelines in the developer
guide.
[x] Reviewed the developer guide on Submitting a Pull
Request
[x] Pending release
notes
updated with breaking and/or notable changes for the next major release.
[x] Documentation has been updated, if necessary.
[x] Unit tests have been added, if necessary.
[x] Integration tests have been added, if necessary.

Show available bot commands
These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

/retest ci/centos/<job-name>: retest the <job-name> after unrelated
failure (please report the failure too!)

@Mergifyio queue


New rule: no-trailing-whitespace

Will fix #596
Do I need to add new rules to lib/config/sass-lint.yml?
DCO 1.1 Signed-off-by: Alex Gyoshev<EMAIL_ADDRESS>

Coverage increased (+0.03%) to 96.899% when pulling 89c4217c7544e4849d0c81accb7b0c9dc6a394ca on gyoshev:feature/no-trailing-whitespace-rule into db4db51cb455a831a11d87b05dd9b24575cfeb75 on sasstools:develop.

@gyoshev yeah, if you could add the rule to the default config that would be great, I can review the rest of this later on :+1:

Coverage increased (+0.03%) to 96.899% when pulling e07e4b2f119e3f96fae4f02a6a3ea1dad644f994 on gyoshev:feature/no-trailing-whitespace-rule into db4db51cb455a831a11d87b05dd9b24575cfeb75 on sasstools:develop.

@DanPurdy I've added the rule to the default config, let me know if other changes are needed

@gyoshev  Ok this looks good, I've got one comment above but could you also add a comment above each example in the test files so that we know what part the spaces should be present for just for future reference when someone wants to update the tests as a few IDE's can be set to automatically remove trailing whitespace.
For example
// Two trailing spaces after .foo
.foo {
  margin: 2em;

Hopefully that makes sense, if you get those in, update with teh latest from develop and then ping me in a comment I'll merge this in for you. Thanks again! 👍

Probably worth fiddling with the .editorconfig so people using that don't have it automatically trim the whitespace.

Yeah I thought that but for the sake of two test files which are unlikely to be touched often it seemed a bit much, same issue with the line endings settings too. We're going to have to start adding extra tests for all the CRLF line endings due to Gonzales known issues with those.. Definitely worth looking at tweaking these configs in the future though.
The comments in the file for now will just make it obvious to everyone what is being tested within the file.

Coverage increased (+0.03%) to 96.945% when pulling 3523f529f6bc70d830cb88cdb5e99ffb509f11ea on gyoshev:feature/no-trailing-whitespace-rule into 9656e0aedeff9f64e719bf2204cbb92b64d82c7f on sasstools:develop.

Comments have been added, rule order is sorted, and the branch is rebased onto develop.

@gyoshev awesome! Thanks for this. I'll make an issue to investigate overwriting the editorconfig settings for these files.
👍


Autoload Grunt plugins

Use autoloading to make Gruntfile smaller

This issue was moved to consortium/hybrid-publishing-research#42


smokeping cannot restart

Describe the bug
When we reloading the smokeping systemd service, we receive the error message "ln: failed to create symbolic link '/var/lib/smokeping/css/css': Read-only file system", because the preStart section can't recreate symbolic links"
Steps To Reproduce
Steps to reproduce the behavior:
systemctl restart smokeping
Expected behavior
The smokeping restart correctely without error
Screenshots
ln: failed to create symbolic link '/var/lib/smokeping/css/css': Read-only file system

Notify maintainers

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 this path will be fetched (0.00 MiB download, 0.00 MiB unpacked):
  /nix/store/jkv35nzx1nh8g0d5p94k8i4zjg8j8i37-nix-info
copying path '/nix/store/jkv35nzx1nh8g0d5p94k8i4zjg8j8i37-nix-info' from 'https://cache.nixos.org'...
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.86, NixOS, 23.05 (Stoat), 23.05.20230420.1dc2054`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.15.0`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

I'm also seeing this breakage after the 23.05 upgrade. ln -sf is dereferencing the target (/css is a directory) and attempting to add a new symlink inside. Changing to ln -snf (adding --no-deference) should be the minimal fix.

I'm new to NixOS and got a similar problem when I just added the smokeping package into my system. I was doing a new installation and tried to configure the smokeping, but the symbol links to the css and js folders failed.
My smokeping config:
{ config, pkgs, ... }:

{
  environment.systemPackages = [
    pkgs.smokeping
  ];
  networking.firewall.allowedTCPPorts = [ some_port ];
  services.smokeping.enable = true;
  services.smokeping.webService = true;
  services.smokeping.port = some_port;
  services.smokeping.host = "<IP_ADDRESS>";
  services.smokeping.hostName = "some.domain";
}

My /var/lib/smokeping/ folder after running nixos-rebuild switch:
[root@sg-sin-s1:~]# ls -al /var/lib/smokeping/
total 20
drwxr-xr-x  4 smokeping smokeping 4096 Nov  1 14:26 .
drwxr-xr-x 10 root      root      4096 Nov  1 14:00 ..
drwxr-xr-x  3 smokeping smokeping 4096 Nov  1 09:05 cache
lrwxrwxrwx  1 smokeping smokeping   74 Nov  1 14:26 cropper -> /nix/store/mx8dxnbbgw7zhg4fr0fkd8d1bg985qfg-smokeping-2.8.2/htdocs/cropper
drwxr-xr-x  4 smokeping smokeping 4096 Nov  1 09:06 data
lrwxrwxrwx  1 smokeping smokeping   58 Nov  1 14:26 smokeping.fcgi -> /nix/store/5i1q41sd3zhc3yl0vawfaxcq4g5rmlfx-smokeping.fcgi

My nix-shell -p nix-info --run "nix-info -m" result:
[root@sg-sin-s1:~]# nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.86, NixOS, 23.05 (Stoat), 23.05.20230113.37b97ae`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.12.0`
 - channels(root): `"nixos-22.11"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

I'm still learning how to debug NixOS, and I don't know if the problem has the exact cause as the one raised in this issue.


User ability to create and post items to the marketplace

Create Product Trello Card
Location:
-CreateProduct
-index.js
-CreateProductContainer.js
-RenderCreateProduct.js
Details:

Users can now create a new product by filling out a form with all the necessary info and list it in the marketplace.

Committed Features:
1 | Product Photos Uploader.
2 | User's ability to post items to marketplace.

Thank you guys!


⚠ Bofrost.at has degraded performance

In aea14e4, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 1528 ms

Resolved: Bofrost.at performance has improved in befb0bc.


Not able to see the sparklens.Json File at mentioned Location

Hi I have recently started Learning SparkLens and trying to generate one sample json file for my SparkApplication and I'm using below spark Submit Command.
spark-submit
--conf spark.extraListeners=com.qubole.sparklens.QuboleJobListener
--conf spark.sparklens.reporting.disabled=true
--conf spark.sparklens.data.dir=/home/data/sparklens.json
--num-executors 1
--jars /home/data/SparkLensJar/sparklens-0.1.2-s_2.11.jar
--master yarn
--deploy-mode cluster
--driver-cores 1
--driver-memory 1G
--executor-cores 1
--executor-memory 1G
--supervise --class com.spark.data.Sparklens
/home/data/SparkLensJar/data-spark-utility_2.11-1.0.jar
I am able to see report/Matrics in Log but Its not creating any Json file at mentioned Location.I have also tried Hdfs location /user/username instead of localFilePath and still not able to see the Json File.

@mayurdb @normalscene. I saw one of your issue "sparklens datadirectory not found #49" It is closed now.I am also Facing  the same issue.Kindly Suggest if any other config need to add to generate fiule at HDFS location.
Thanks !!


A way to ignore query for a specific resource

Needs
It will be great to get a way to ignore a query on a ressource.
The solution I would like
Sometimes, we don't want to have an AWS S3 bucket versionned and encrypted.
#kics:ignore:568a4d22-3517-44a6-a7ad-6a7eed88722c kics:ignore:5fb49a69-8d46-4495-a2f8-9c8c622b2b6e
resource "aws_s3_bucket" "backend" {
    bucket = "backend"
    acl    = "private"
}

I kept finding this ticket instead of the docs when searching for a solution, so here's the documentation


Add plugin to find source code from storybook

Adds a way of jumping from storybook to source code
📺 As seen on design sync: https://github.rewatch.com/video/1cqzuowd7jt5a0ee-di-design-sync-march-31-2022?t=482

Super cool! So Alt+click to open the code? Is there better discoverability of the shortcut?

We don't mention any other Storybook plugins, so I'm totally fine merging this as-is.

taking the easy path and merging it 😅

Is there better discoverability of the shortcut? — @pksjce
Because it's not easily discoverable, should we mention it somewhere in the contributor docs? — @mperrotti

I need ideas on how to make this more discoverable, one option is to add a button in the top bar which activates it instead of pressing Alt. Only slightly better though 😅


New task: ESLint

Addressing #122.
Adds a task for running ESLint with a minimal API.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: Thisen sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.

Thanks so much! I would like to test this out before merging it in today :)

close and reopening to trigger new build

Shoot, the build is suppose to fail without a "change file" - would you terribly mind running this in your repo at the root: "yarn change" - it should walk you through a CLI wizard in creating a "change file" - these change file determine the published version. Given it a description and pick "minor" in this case. I need to update beachball to fail the build if it doesn't find it.

@kenotron,yarn change results in this output: "No change files are needed"


Promote admissionreview to v1

What type of PR is this?

/kind feature

What this PR does / why we need it:
This PR is part of kube-openapi upgrade: #923
all information cherry-picked from community kubernetes PR:
https://github.com/kubernetes/kubernetes/pull/78863
https://github.com/kubernetes/kubernetes/pull/80750
https://github.com/kubernetes/kubernetes/pull/79408
https://github.com/kubernetes/kubernetes/pull/80231
Which issue(s) this PR fixes:

Fixes #923
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

tested perf-test with 100 nodes, both density and load testing finished with success.

/lgtm

/approve


iOS builds are blocking non-ssl connections to local network IPs, citing an outdated iOS policy

Steps to Reproduce

Run a Flutter app on iOS (debug, say) that connects to any local network IP address without using SSL, e.g. an API call that hits http://<IP_ADDRESS>/sync or a NetworkImage etc...

Expected results:
Up until recently, these connections succeeded, allowing for easy local development.
Actual results:
Now, these connections are being denied by Flutter, which on the new network policy breaking changes page cites that since iOS 9, ATS has disallowed non-ssl connections by default.  However, that doesn't seem quite correct.  This page at Apple explains that ATS enforcement with respect to local IP addresses was reversed in iOS 10.
The workaround suggests modifying Info.plist to add in exceptions for your dev machine IP.  This seems to unnecessarily complicate cutting a production release of an iOS app.  Given warnings from others that Apple requires explanation for NSExceptionDomains, are we supposed to modify the plist file before each release, and then again when we want to start debugging locally using our dev server?  Modify our local servers to use SSL?
If I've done a dumb, let me know, but at the moment, this seems to be a somewhat unnecessary breaking change, at least for iOS.

Logs

I/flutter (28401): Bad state: Insecure HTTP is not allowed by platform: http://<IP_ADDRESS>:4000/parcels/
I/flutter (28401):
I/flutter (28401): #0      _HttpClient._openUrl (dart:_http/http_impl.dart:2435:7)
I/flutter (28401): #1      _HttpClient.openUrl (dart:_http/http_impl.dart:2342:7)
I/flutter (28401): #2      IOClient.send
package:http/src/io_client.dart:31
I/flutter (28401): #3      BaseClient._sendUnstreamed
package:http/src/base_client.dart:91
I/flutter (28401): #4      BaseClient.post
package:http/src/base_client.dart:32
I/flutter (28401): #5      ApiClient.invokeAPI
package:openapi/api_client.dart:190
I/flutter (28401): #6      ParcelServiceApi.getParcelsWithHttpInfo
package:openapi/api/parcel_service_api.dart:103
I/flutter (28401): #7      ParcelServiceApi.getParcels
package:openapi/api/parcel_service_api.dart:120

[✓] Flutter (Channel beta, 1.23.0-18.1.pre, on Mac OS X 10.15.7 19H2 x86_64, locale en-US)
    • Flutter version 1.23.0-18.1.pre at /Users/tek/Library/flutter
    • Framework revision 198df796aa (4 days ago), 2020-10-15 12:04:33 -0700
    • Engine revision 1d12d82d9c
    • Dart version 2.11.0 (build 2.11.0-213.1.beta)

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
    • Android SDK at /Users/tek/Library/Android/sdk
    • Platform android-30, build-tools 30.0.2
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 12.0.1, Build version 12A7300
    • CocoaPods version 1.10.0.rc.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 4.0)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 50.0.1
    • Dart plugin version 193.7547
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)

[✓] IntelliJ IDEA Ultimate Edition (version 2020.2.3)
    • IntelliJ at /Applications/IntelliJ IDEA.app
    • Flutter plugin installed
    • Dart plugin version 202.7319.5

[✓] VS Code (version 1.50.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.15.1

[✓] Connected device (3 available)
    • Pixel 2 (mobile) • HT7BG1A05728 • android-arm64  • Android 10 (API 29)
    • Web Server (web) • web-server   • web-javascript • Flutter Tools
    • Chrome (web)     • chrome       • web-javascript • Google Chrome 86.0.4240.80

Never mind.  Looks like ATS encourages https for all connections, including local.  For those looking for a quick and dirty solution to this, Caddy works alright...
Closed, but with protest, though mostly against Apple.

This page mentions settings (Info plist or manifest) that can be changed to allow insecure connections. However, none of those are needed for our app on platforms more recent than those mentioned, and it can still connect over http to a local IP. So I don't think Flutter should be more strict than the platforms.
In our case, we want to connect to an older IoT device that doesn't support HTTPS. Setting NSAllowsArbitraryLoads seems too much for that. (We don't want to allow arbitrary connections to the internet.)