[SPARK-32597][CORE] Tune Event Drop in Async Event Queue

What changes were proposed in this pull request?
There are scenarios where we have seen the event drop in spark, resulting in the inconsistent state for the spark Application(some time application is hung state).
For example - This can be due to the large number of parallel task processing. Producer thread keeps on adding the events to the Queue and Consumer thread is consuming the events at the slower rate compare to the Producer adding in the queue. Resulting the Queue to reach max size and events get dropped from that.
There are times if Queue Size would be little bit higher like 10 percent (or 20 percent) extra of the existing Queue size, the events can be processed preventing the event drop at that point of time.
As per the current architecture size of event Queue can be configured at start of the application by setting spark.scheduler.listenerbus.eventqueue.capacity. Once this is set there is fixed size event Queue(LinkedBlockingQueue), which cannot be changed at run time to accommodate some extra events before dropping event from the Queue
This Jira for adding a support of VariableLinkedBlockingQueue to tune the dropping of the events.
VariableLinkedBlockingQueue -> https://www.rabbitmq.com/releases/rabbitmq-java-client/v3.5.4/rabbitmq-java-client-javadoc-3.5.4/com/rabbitmq/client/impl/VariableLinkedBlockingQueue.html
As the part of this PR, we can provide the support for VariableLinkedBlockingQueue.

We need to add those Queue where we are seeing the event drop and add to spark.set.optmized.event.queue.
set --conf spark.set.optmized.event.queue=appStatus,executorManagement -> Now these two queues will be
VariableLinkedBlockingQueue and size of the queue can be increased at run time.
By default its empty string so all the Queues will use the old behavior LinkedBlockingQueue
We can set threshold of the size of the queue that we want to increase -> spark.set.optmized.event.queue.threshold

Now if we use VariableLinkedBlockingQueue, at the time when the size of the queue reach the size initialized at start of the job , Queue will try to increase the size of the Queue to threshold (spark.set.optmized.event.queue.threshold) and prevent the Event drop at that point of time.
Why are the changes needed?
Now if we use VariableLinkedBlockingQueue, at the time when the size of the queue reach the size initialised at start of the job , Queue will try to increase the size of the Queue to threshold (spark.set.optmized.event.queue.threshold) and prevent the Event drop at that point of time. Tune Event Drop in Async Event Queue
Does this PR introduce any user-facing change?
No
How was this patch tested?
Added the Unit test and tested the functionality using the spark-submit

Can one of the admins verify this patch?

Can one of the admins verify this patch?

cc @vanzin, @cloud-fan , @dongjoon-hyun , @zsxwing for review

cc @tgravescs @jiangxb1987 @squito

so one question, here, you are seeing events dropped that cause hangs in the application code?  What queue were they in?

so I definitely get the point here and it would definitely be nice to handle this better somehow, but if you are setting it be queue size + some threshold then the application has to have the memory to handle, so how is that different then just setting queue size to be larger?  The one benefit is user doesn't have to try to pick an exact size, but the downside is they don't necessarily know how it affects the memory so you ideally add memory for worst case.
The reason I asked which queue dropped is that normally the event log queue fills up but it should only affect the history. If the executor management queue filled up then could affect dynamic allocation which is definitely bad, but in my opinion we should change that not to rely on events.  The other queues ideally don't affect the application running but I am wondering if something is.

so one question, here, you are seeing events dropped that cause hangs in the application code? What queue were they in?

So we have seen the ExecutorManagement Queue , appStatus Queue, eventlog Queue.
Since there are scenarios where there is some long running jobs in Zeppelin Notebook and Jupyter notebook where the spark Application is running till the time the notebook is stopped or there is some validation that is added that will check if the application is idle based on any job running or not and if we event for jobEnd  event is dropped from appstatus queue, this will make the notebook to think spark is running the job , resulting in the Spark Job to be in hung state even the nothing is running for long period of time.
Also we have seen the impact where some stage end , stage start , task info is dropped from the executorMangment Queue resulting in taking the decision for upscaling and downscaling of executors

so I definitely get the point here and it would definitely be nice to handle this better somehow, but if you are setting it be queue size + some threshold then the application has to have the memory to handle, so how is that different then just setting queue size to be larger? The one benefit is user doesn't have to try to pick an exact size, but the downside is they don't necessarily know how it affects the memory so you ideally add memory for worst case.
The reason I asked which queue dropped is that normally the event log queue fills up but it should only affect the history. If the executor management queue filled up then could affect dynamic allocation which is definitely bad, but in my opinion we should change that not to rely on events. The other queues ideally don't affect the application running but I am wondering if something is.

So the main issue is , there is no fixed size of the Queue that can work for all the Spark Jobs and once the size of the Queue is set at the start of the application, Linked-blocking Queue can handle max those number of the events when the Queue is full, after that event is dropped. The idea here is to handle this scenario more gracefully where the queue can size can be increased in case overflow of events and once those overflow of those events are over queue size can be set to size which is the initial capacity .
This is handled on per queue if there is problem with one queue(eg appStatus Queue), then that queue size will change at run time.
Also there is future plan to make this logic dependent on the driver memory. Taking the decision to increase the size of the Queue based on certain threshold driver max heap size. This can prevent the OOM.

I am wondering if we don't want to drop any events, why don't we just set the capacity to Integer.MAX_VALUE? The LinkedBlockingQueue doesn't really allocate that much memory at the beginning.
And I find there's concern about it:
https://github.com/apache/spark/blob/643cd876e4cfc7faa307db9a2d1dd1b5ca0881f1/core/src/main/scala/org/apache/spark/scheduler/AsyncEventQueue.scala#L47-L48
But I doubt it now. Because we never throw such explicit error but only drop events when offer returns false, right?
And even if we want to throw the error, OOM is still possible when users set a huge capacity, right?
Another thought is, the default value(10000) of spark.scheduler.listenerbus.eventqueue.capacity is probably too small.

I am wondering if we don't want to drop any events, why don't we just set the capacity to Integer.MAX_VALUE? The LinkedBlockingQueue doesn't really allocate that much memory at the beginning. - This will cause Driver to OOM for long Running Jobs if there are large number of events came at point of time, since these event takes driver memory when present on queue.
But I doubt it now. Because we never throw such explicit error but only drop events when offer returns false, right? - yes thats right
And even if we want to throw the error, OOM is still possible when users set a huge capacity, right? - Generally on running production spark jobs. when we set the huge capacity of the queue, at that point of higher Driver memory is also set compared to what was there previously .
Another thought is, the default value(10000) of spark.scheduler.listenerbus.eventqueue.capacity is probably too small. - yes this is very small, with this size seen lots event drop.

I am wondering if we don't want to drop any events, why don't we just set the capacity to Integer.MAX_VALUE? The LinkedBlockingQueue doesn't really allocate that much memory at the beginning.

This will cause Driver to OOM for long Running Jobs if there are large number of events came at point of time, since these event takes driver memory when present on queue.

Could VariableLinkedBlockingQueue avoid OOM in that case?

I am wondering if we don't want to drop any events, why don't we just set the capacity to Integer.MAX_VALUE? The LinkedBlockingQueue doesn't really allocate that much memory at the beginning.

This will cause Driver to OOM for long Running Jobs if there are large number of events came at point of time, since these event takes driver memory when present on queue.

Could VariableLinkedBlockingQueue avoid OOM in that case?

Using this VariableLinkedBlockingQueue, we are increasing certain percentage threshold for size of the queue at run time, Which is less aggressive compare to keeping the Integer.MAX_VALUE, Also I am planning to add the validation of spark driver memory to increase the size of the VariableLinkedBlockingQueue

Using this VariableLinkedBlockingQueue, we are increasing certain percentage threshold for size of the queue at run time, Which is less aggressive compare to keeping the Integer.MAX_VALUE, Also I am planning to add the validation of spark driver memory to increase the size of the VariableLinkedBlockingQueue

IIUC, LinkedBlockingQueue doesn't really allocate all the memory in once by setting the capacity to Integer.MAX_VALUE.
It also increase memory usage gradually at runtime.

Using this VariableLinkedBlockingQueue, we are increasing certain percentage threshold for size of the queue at run time, Which is less aggressive compare to keeping the Integer.MAX_VALUE, Also I am planning to add the validation of spark driver memory to increase the size of the VariableLinkedBlockingQueue

IIUC, LinkedBlockingQueue doesn't really allocate all the memory in once by setting the capacity to Integer.MAX_VALUE.
It also increase memory usage gradually at runtime.

That's right, memory is allocated once the event is present in the Queue. In case when huge number of events flowing ,on keeping Integer.MAX_VALUE and there is no control on the number of events on the event queue, it keeps the large number of events queue(might be 100k or more), So there is chance of OOM is here if the size is not controlled and driver memory is not sufficient.
VariableLinkedBlockingQueue is similar to LinkedBlockingQueue with the functionality of setting the size of the queue at run time. So we can make use of this VariableLinkedBlockingQueue and adjust the size based on various conditions

sorry I'm still not seeing any difference here then increasing the size of the current queue?  If both are not really allocating memory for the entire amount until runtime then either way you have to set memory of driver to be the maximum amount used.    why not set the queue size to size + size * spark.set.optmized.event.queue.threshold?
If you look at the driver memory used, I don't think that is very reliable. They could change very quickly.

sorry I'm still not seeing any difference here then increasing the size of the current queue? If both are not really allocating memory for the entire amount until runtime then either way you have to set memory of driver to be the maximum amount used. why not set the queue size to size + size * spark.set.optmized.event.queue.threshold?
If you look at the driver memory used, I don't think that is very reliable. They could change very quickly.

Ok If we take out consideration of VariableLinkedBlockingQueue and Driver Memory validation to handle this issue, than I can think of below 2 approach

Either to Set Queue size as  size + size * spark.set.optmized.event.queue.threshold -> The problem with this approach is that event drop can still happen after increasing the size if the flow of incoming events is at higher rate compare to consumer rate . In this case also critical events can be dropped from the queue (executor Mangament Queue,Appstatus Queue etc). But this will help in the scenario where there is high rate of event at one point of time and on increasing with spark.set.optmized.event.queue.threshold can help in preventing the overflow the queue here.

Make the size of the queue as Unbounded -> The only problem with this approach is Driver getting OOM. But If the event are really important and we cannot avoid dropping the event while running Spark Job. For example -
a) For some user while running the Spark Job they cannot avoid dropping of events from Appstatus Queue like in case of notebooks(jupyter/Zeepline)
b) For other user they want to have the dynamic allocation of resources (upscaling/ down scaling) needs to be consistent.
c) For other user they want to have all the events in the eventlog File, which they used later for ML models to analyse the Spark Jobs.
There is need to provide higher driver memory in this case but some spark jobs does not require this much driver memory. We cannot make default high driver memory since some application can run fine with default driver memory 1 gb.

We can think of making it per queue basis any such approach based on the user requirement like I have already done in this PR using the conf(spark.set.optmized.event.queue) for VariableLinkedBlockingQueue  at the start of application.
Happy to discuss any other idea for tuning these Queues
@Ngone51  @tgravescs

We can think of making it per queue basis any such approach based on the user requirement like I have already done in this PR using the conf(spark.set.optmized.event.queue) for VariableLinkedBlockingQueue at the start of application.

I don't get your point here. it can already be configured on a per queue basis (set "spark.scheduler.listenerbus.eventqueue.$name.capacity") and if user sets spark.set.optmized.event.queue why not just set spark.scheduler.listenerbus.eventqueue.$name.capacity to be larger to match whatever driver memory set at?
I'm not necessarily against a change here, I get the issues with dropping events, but this just feels like extra code to do what user can already do. If I'm missing something and you have specific use case I'm missing, please explain in more detail.
So the executor management queue is definitely a problem and if we keep seeing issues personally I think we should integration dynamic allocation manager into scheduler and get away from using messages.  The app status queues, I hadn't thought about things like zepplin using those messages, but that makes sense, in just a normal app they shouldn't affect normal application runs.
The only time I've seen issues with these queue is generally when dynamic allocation is enabled and you are running huge number of tasks and dynamic allocation is requests lots of executors and generally task time is low. Basically the startup point when we request executors, we get messages from them starting, we get task start and end messages all going at the same time.  I'd be curious to hear what conditions are causing these for you. Part of it is to make sure we haven't regressed and something is taking a long time to process events. its not ideal but we have fixed certain components that did this and it has helped in the past.
I don't really see a solution to this problem unless you can dynamic change it and reliable get the memory of the driver. The problem is (depending on GC and tunings being used) heap will use memory up to a certain point even if it didn't need to necessarily, getting those metrics I don't think is really reliable.  I know in the past things like the runtime stats don't increment all the time, it has to have a certain number allocated.  The problem with this also is that is heap is used a lot you can't dynamically change so then it goes back to user to increase heap size, which at the same time they might as well increase queue size.  So even this is not ideal.
Another thing we can do is if there are certain events critical we can look at putting them into its own queue as well.  Perhaps another question is, is your driver just running out of CPU to process these fast enough?

I was thinking that though spark has many queues and in many cases may be not all queues are used used to the same level at the same time.. I mean, when some queues are heavily used, some might be consume those events faster.
If this is true, instead of having separate queue sizes for each and making it rigid, what if its a pool, from which the queue can loan event holders. and when the events are processed, they are given back to the pool.
This pool is not memory units stuff, rather its just a counter (atomic probably). Let's say if we say for a driver memory of X GB, we allocate N event holders, which needs to be used by all queues. N then is the pool size.
When an event needs to be enqueued in a queue, ask the pool if an event placeholder can be used. If it says ok ( based on the current value of usage) then the queue can enqueue event. if no, the event is dropped.
The idea here is that it is a middle ground between restricted queue size and infinite capacity queue. So here the queues are not statically bound to a size but rather more flexibility of given to grow. Also it has a softer high water mark ( which is N) beyond which it cant grow.
Let me know what you think...  @SaurabhChawla100  @Ngone51  @tgravescs

Part of the problem with that is that in most cases some queues have higher priority, which is why they were split apart. you really want the executor management queue to work if dynamic allocation is on so you wouldn't want to take the chance of another queue using all of its size.  If its off then its not as big of a deal and you could "share" that size.  Beyond that I don't really know the stats on the number of message the queues usually see.  It would make configuring individual queues much harder but I don't know how much people do that.

It would make configuring individual queues much harder but I don't know how much people do that.

Much harder? IIUC, if users have some experienced stats of the queues of the applications, I guess they could set the individual queues more accurately and we don't need such "pool" at all.

I don't get your point here. it can already be configured on a per queue basis (set "spark.scheduler.listenerbus.eventqueue.$name.capacity") and if user sets spark.set.optmized.event.queue why not just set spark.scheduler.listenerbus.eventqueue.$name.capacity to be larger to match whatever driver memory set at? I'm not necessarily against a change here, I get the issues with dropping events, but this just feels like extra code to do what user can already do. If I'm missing something and you have specific use case I'm missing, please explain in more detail.

Yes I agree we can use "spark.scheduler.listenerbus.eventqueue.$name.capacity" to set the value. Entire idea is to make Queue to perform well in situation where the Queue overflow the initial capacity(can be set using conf spark.scheduler.listenerbus.eventqueue.$name.capacity) and some additional threshold capacity helps in reducing the event drop and there will be event drop after it crosses the threshold value. This is the best case scenario where the event drop handled at the time by the threshold capacity and no event is dropped. For handling this I used VariableLinkedBlockingQueue so at run time this can be taken care by increasing the initial capacity at run time with some validation of keeping the driver memory into the consideration.
This can be done with LinkedBlockingQueue by setting the queue size to add this extra threshold when creating the Queue at the start of the Spark Application.
Since this is also true that "There is no fixed size of the Queue which can be used in all the Spark Jobs and even for the Same Spark Job that ran on different input set on daily basis". So this extra threshold can helps in best case scenario where the initial Queue size is set. This will prevent abrupt behaviour of Spark application due to event drop for the critical Queues(appStatusQueue, executormangmentQueue etc).
If something fails / abrupt behaviour after this extra threshold, then consider some manual effort in tuning the queue size which we will do incase of setting "spark.scheduler.listenerbus.eventqueue.$name.capacity". If we consider only this "spark.scheduler.listenerbus.eventqueue.$name.capacity" and every time we need to change the size of the queue when the application performs abruptly which requires more manual efforts even to same Job compare to the cases which is already handled by using the threshold.
So this is one of the Real World scenario of Multi-Tenant environment, where multiple Spark Application running on same Spark Cluster with fixed number of max nodes that can come and there is fixed number of the cores and memory available. Some of the abrupt behaviour that we have seen in this case

Scenario when dynamic Allocation is enabled and event drop has happened in executor management queue and can result in not downscaling of the executor for one of the Spark Application, Even though Executor is idle for long time. So this is holding a resource and some other application wants some more  resource and did not get since there is limited number of resources available in the Cluster.

Also seen the scenario where there is dependency Zeppelin notebook on AppStatus Queue and event drop prevented the spark application to go down after the idle timeout, If non of the spark Jobs running for some time. Here also we found the resource is holded one resource is driver and other min number of executors that we set, which can be used by some other Spark Application

Also there are many scenarios that can impact the spark Application due to Event Drop.
This is just to reduce manual intervention on changing the conf to some extent.

Another thing we can do is if there are certain events critical we can look at putting them into its own queue as well. Perhaps another question is, is your driver just running out of CPU to process these fast enough?

Completely Agree on this.

Yeah so you have described the affects of it dropping the events, which I know. The thing I want to know is why it dropped the events in your cases.
I'm not sure the circumstances you are running or seeing these issues, is this just individuals running their own clusters. If it's a centralized cluster, why not have some cluster defaults and just raise the default values there?   This is essentially just doing that.

Much harder? IIUC, if users have some experienced stats of the queues of the applications, I guess they could set the individual queues more accurately and we don't need such "pool" at all.

@Ngone51  if configuring the event sizes was so easy, then I am fine. Guessed, it would have been easier to configure 1 number than 3 or 4... thanks.

@itskals  Ah..no. I was not saying configuring individual event sizes is easy. I was just confused about @tgravescs 's comment.

I was just saying, if you had one pool you took from you can't control it per queue. Unless of course you put in some sort of minimum or something per queue. In most of the applications I've seen the executor management queue is most important as it will mess up dynamic allocation and can results in application hangs, wasted resources etc.  The other queues generally just drop events and lose history information. Obviously the use cases here for jupyter/zeppelin might be different though.

Yeah so you have described the affects of it dropping the events, which I know. The thing I want to know is why it dropped the events in your cases.

There are scenarios where in stage there are large number of task and all task completed in very small time resulting in the queue to fill up very fast and dropping for event after some time from those Queues (executormanagement Queue, AppStatus Queue, even in Event Log Queue). For us executormanagement and AppStatus matters most since we have dynamic allocation enabled and most of the jobs are on the Notebooks( jupyter/zeppelin)

I'm not sure the circumstances you are running or seeing these issues, is this just individuals running their own clusters. If it's a centralized cluster, why not have some cluster defaults and just raise the default values there? This is essentially just doing that.

We are already setting some value in  cluster defaults . But as I said earlier also "There is no fixed size of the Queue which can be used in all the Spark Jobs and even for the Same Spark Job that ran on different input set on daily basis"
Thats the reason we are looking for some dynamism in this process queue and some way to reduce human efforts to change the Queue size. Also IMO changing the driver memory(increasing / decreasing the driver memory) is easier compare to setting the Queue size(also there are multiple Queues).
@tgravescs @Ngone51 @itskals

We are already setting some value in cluster defaults . But as I said earlier also "There is no fixed size of the Queue which can be used in all the Spark Jobs and even for the Same Spark Job that ran on different input set on daily basis"

So what is your plan to set these configs at?  unlimited?  If there is no set size for all applications then how does this help, you are adding a config to add a set size to a set size config.
lets say I make my event queue size 10000 for the entire cluster. My spark.set.optmized.event.queue.threshold is 10% 10000 * 10% = 1000.  This means my event queue size is 11000.
If users are already changing the event queue size from 10000 to say 30000, then there is no reason they can't make it 33000 (the same as having  spark.set.optmized.event.queue.threshold=10%.

lets say I make my event queue size 10000 for the entire cluster. My spark.set.optmized.event.queue.threshold is 10% 10000 * 10% = 1000. This means my event queue size is 11000.
If users are already changing the event queue size from 10000 to say 30000, then there is no reason they can't make it 33000 (the same as having spark.set.optmized.event.queue.threshold=10%.

So the idea here is whatever be initial capacity is set, have some threshold on top that initial capacity that can prevent the event drop in the best case scenario.So that there is less human effort is applied in changing the conf for setting queue capacity.
if users are already changing the event queue size from 10000 to say 30000, then there is no reason they can't make it 33000 (the same as having spark.set.optmized.event.queue.threshold=10%. - Yes they can make but in next run , after they see some abrupt behaviour (application hung/ Resource wasted).But what if due to this extra threshold size there is no event drop. Might be that extra size (33000 with 10% or 36000 with 20%) prevented the user to change the conf more frequently.
Anyways If we are the only one impacted by this event drop problem and others are fine with setting it manually. Then I believe the current behaviour is fine.
We can revisit this problem if there is some ask for this in future.
Thank you every one for their valuable feedback on this issue

Yes they can make but in next run , after they see some abrupt behaviour (application hung/ Resource wasted).But what if due to this extra threshold size there is no event drop. Might be that extra size (33000 with 10% or 36000 with 20%) prevented the user to change the conf more frequently.

I don't understand this, how does it prevent the user from changing the conf?  They either change the percentage to be 20% or the event queue size to 36000.  Either way they change a conf.

I don't understand this, how does it prevent the user from changing the conf? They either change the percentage to be 20% or the event queue size to 36000. Either way they change a conf.

@tgravescs what I understand from the discussions here is that, let us assume 30,000 is a good number in many cases for a certain workload. But in some scenarios, like slight change in input data pattern etc, the number of events generated increases. In that case, this 20% can be helpful. How? This 20% will accommodate, hopefully, the extra burden w/o losing the events and user manually changing it to 36,000. ( could be seen this way, "instead of changing the value twice, change two values once")
Other things that I see is that, the queue size need not increase to the config value( 20%) in one step. It can be gradual and multi step.  Also, it can shrink back to original value when event queue is back to "acceptable" usage.
Also I see that the increase in the additional queue size is memory pressure aware and tries to keep itself as best effort. In such cases, setting this value to a higher value say 100% might not harm.
Anyways, it feels good if the system detects that need to increase the event size and does it automatically, while at the same time being resource aware + giving flexibility to user to control it (if needed).
@Ngone51  @SaurabhChawla100

In that case, this 20% can be helpful.

The problem is why you would like to set it to 20% in first place? Why not 10% or 30%? If one exactly know he/she would expect 20%  more space, I think he/she would/could also set it to 36,000, especially when he/she has no background of this PR.
Instead of just increasing the queue size(percentage or fixed number), I'd prefer to improve the utilization of the current queue. The idea of “pool” might be a good inspiration. And as @tgravescs mentioned above, if we worry about some queues, we could set the minimum size to avoid aggressive "sharing". But I think the most important thing here is that we need to investigate the statistics of the queues of common applications and see if the "pool" could really help to ease the problem.

what I understand from the discussions here is that, let us assume 30,000 is a good number in many cases for a certain workload. But in some scenarios, like slight change in input data pattern etc, the number of events generated increases. In that case, this 20% can be helpful. How? This 20% will accommodate, hopefully, the extra burden w/o losing the events and user manually changing it to 36,000. ( could be seen this way, "instead of changing the value twice, change two values once")

@itskals  - Thanks for explaining this is detail

The problem is why you would like to set it to 20% in first place? Why not 10% or 30%? If one exactly know he/she would expect 20% more size, I think he/she would/could also set it to 36,000, especially when he/she has no background of this PR.

We can use it as default value like 10% in the conf and user has fixed the value of some 30,000 only. And there is slight increase in the load while processing the job, and we are getting 32,000 task completed in small interval of time and causing the overflow of the queue even after 30,000 . But if we have this extra threshold of default value 10% which is 33000 than those event drops is prevented and job can be prevented from the abrupt behaviour due to the event drop. And also prevented user to change the value from 30000 to some other number less frequently.
This is the best case scenario example and the proposal is to handle this best case scenario, which prevents the event drop.

Queue capacity is an approximation to bound memory usage in event queues; and tends to be conservative - event loss is fine (except DRA) - but driver OOM causes entire app to fail.
@SaurabhChawla100 I understand the motivation, but if the 'x'% more required to avoid dropping event could have been predicted - it also could have been very well used to bump up capacity of queue to begin with along with corresponding driver memory increase.
Since a reasonably simple solution exists in this case (by leveraging queue capacity configs), based on the current usecases/discussion in the PR, I agree with @tgravescs - I dont see a good reason to introduce the additional complexity.

I think we can't 100% guarantee no events drop, whatever we do is just a best effort. That said, the consumer side of the event queue should not be sensitive to event drop. If your use case is to check if a Spark application is still running jobs, can we query the spark driver status via REST API directly?
I get the intention of this PR to make the queue size more dynamic, as the peek number of events is unpredictable. But I don't see how this patch solves it, as users still need to predicate it and set the new config.

I think the direction to proceed for us, is to get to an agreement that this is worthy problem to solve.... few questions from my side, which can help:

Is the impact of event drops critical for the reliability of spark ?
[Note: By spark here I mean the base offering core+Dynamic allocation+App status etc]
How common is event drop issue that one has got a complaint of from ones experience?
What is general remedy one offers? ( it could be general strategy Databricks (or your company) offers to its customers)

For the solutions part [if the issue is critical to solve]:

what should be the general direction to solve such problems? [auto-detect=> auto-adjust=> auto-heal etc]
How much user involvement is acceptable?

If we have a quorum, we can decide what can be next steps......

From my experience:

For the dynamic allocation it definitely is, but in my opinion the right solution there is to pull the dynamic allocation manager into the core scheduler and not rely on the messages and listener queues.  I have not seen direct issues with others.
In my previous experience it was very uncommon and adjust queue sizes worked. But I also had increased the default size of the queue along with the default memory of the driver to be larger at a cluster config level.


Tag Classification with Image and Information

This is a set of steps for tag classification with image and IMU/GPS information.

[ ] Implement the ModalityType, which is the input type of ImageBind, to be able to convert IMU/GPS information to an embedding vector.
[ ] Find a way to combine the image embedding vector and the IMU/GPS information embedding vector.
[ ] Tag classification test with image and IMU/GPS information

In the tag classification with image and information experiment, the IMU/GPS information from BDD100K is converted to an embedding vector and combined with the image vector to perform classification.

Scene classification with inertial information

Classification is performed with embeddeding vector: <image feature> + <imu feature>


error:FetchError: invalid json response body at https://api.nftport.xyz/v0/files reason: Unexpected token < in JSON at position 1

Has somenone figured this out?
I'm getting the error when i try to upload the images to IPFS:
error:FetchError: invalid json response body at https://api.nftport.xyz/v0/files reason: Unexpected token < in JSON at position 1
error:FetchError: invalid json response body at https://api.nftport.xyz/v0/files reason: Unexpected token < in JSON at position 1
error:FetchError: invalid json response body at https://api.nftport.xyz/v0/files reason: Unexpected token < in JSON at position 1
error:FetchError: invalid json response body at https://api.nftport.xyz/v0/files reason: Unexpected token < in JSON at position 1
...and so on, for each image.
I've checked the code many times but I'm still missing something, please help if you can.

fixed with latest code:
https://github.com/codeSTACKr/video-source-code-create-nft-collection/tree/fix-uploads-mints


feat: add jcaigitlab/asdf-git

What I want
Please revert remove asdf-git  and add plugin for git to registry.
Reason
I use this git plugin in mise on *nix machines, along with the all_compile = true setting, to compile newer versions of git than what comes with the distribution. The benefits are:

Ease of administration, especially on remote machines. One mise install gets me a full dev environment everytime.
Using git@latest ensures I encounter the exact same version of git across all machines, all with exact same behavior. This allows full automation of configuration, with no expected differences in behavior across machines. I just copy config files from a remote, and that's it.
Makes git administration much easier in environments such as WSL2 boxes, where git administration used to be a major pain point. Now it's just mise install and some cp commands to copy configuration files.

Workarounds
I manually injected this plugin on my latest install with mise plugins install git https://gitlab.com/jcaigitlab/asdf-git.git when I got a No repository found for plugin git error, but the added overhead makes the experience tedious for a bit.
I now have to remember to install this plugin manually, instead of having it in my config.toml file and just using mise install to set up an environment on a new machine.

I think including a git plugin will make the experience of using mise much better and smoother.
I am happy to write a PR myself if approved.

git is a prerequisite for using mise, it does not make sense to make it available as a plugin and doing so will result in an infinite loop for plugins that call git when using shims, so no, I won't be considering putting it back in


Support for namespaced attributes?

I have this element: <locRef xlink:href="AT-07-14-01"/>, how can I read its value? It looks like @Attribute doesn't support namespaces?

Yeah, unfortunately it does not. You could provide custom a CodingKeys enum. I we'd benefit a lot from supporting Macros


Element numbering not being assigned to grid

Opened a new issue regarding this
@akaszynski
Hello, I have a problem to read the result when I read the nodes from a txt file.
But when I set the all_nodes a few nodes,it can work.
By the way, the result.principal_nodal_stress(0) read all the nodes result and I can't assign the the specific nodes.

Originally posted by @1990chs in https://github.com/pyansys/pymapdl-reader/issues/33#issuecomment-865938698
Would it be possible to post the result file?

sorry, the result file in my company computer. It impossible to export result file. The nodes which stored in txt file have midnodes , does it influence the function?

It shouldn't,  I'll take a look at this error locally.  Effectively, the element number aren't being exported to the internal grid, and I can't tell why not.  Midside nodes shouldn't be contributing to this bug.

Ok. I will check it again. Maybe the nodes are not corrected. I need check.
principal_nodal_stress(rnum) this API can't assign the nodes numbers . It needs update.

Thanks for letting me know.  I'll get working on this.

It seems that there are some problems to read nodes ID from txt file. I input the nodesID in python file directly, it can work, but there is a warining .

the error solved , and I will open an issue #52 about the suggestion features

Why can't this issue  close?

@1990chs I'll close the issue for you since it appears to be resolved!


null error in ionic.io.bundle.min.js

I'm getting this error in a new app I just made that uses check():
[Error] TypeError: null is not an object (evaluating 'u.IonicPlatform.getMain().onReady')
e (ionic.io.bundle.min.js, line 2)
(anonymous function) (app.js, line 21)
(anonymous function) (ionic.bundle.js, line 48811)
onPlatformReady (ionic.bundle.js, line 2447)
onWindowLoad (ionic.bundle.js, line 2428)

This is most likely caused by a missing call to Ionic.io();, I am removing the need to call taht manually in the next update to the platform client..


Fix the storage command sample notebook and fix a bug in list expansi…

…on in
storage magics.
This notebook was not in the automated notebook tests as originally it was so
far out of date the automated test failed on (almost?) every cell as all the
new outputs were different. Unfortunately I didn't remember that it had been
skipped so when testing the notebook updates recently it was overlooked.

LGTM. If I remember correctly "UDFs using Code in Cloud Storage" may need to be updated too.

I checked the UDF notebook; it looks fine (it uses Python, not magics).


Update quick_install.sh

Add set -e to exit on first failure
What this PR does / why we need it:
Easier debugging on install / avoids potentially running commands out of order if early command fails.

This change is

/ok-to-test

/retest

/retest

/retest

/lgtm
/approve

@alexwennerberg thanks for adding this!


sqlite unsupported type NULL of column #... compiler error when it should return a value

Bug Description
The call sqlx::query!(r#"select * from Ads"#) gives a compiler error that NULL of column #... This behavior doesn't happen when using SQLite console. I can run SELECT * from Ads and get some or no rows back. sqlx doesn't return some or no rows back it instead it does a compiler error.
(the table I'm using is in the code sample below)
Minimal Reproduction
to set this up please use actix-web = "4.2.1" and create an .env file with DATABASE_URL=sqlite:sqlite.db and finally another file called sqlite.db in the same directory
main.rs
#[actix_web::main]
async fn main() -> std::io::Result<()> {
    let pool = SqlitePool::connect("sqlite.db").await.unwrap();

    let mut conn = pool.acquire().await.unwrap();

    sqlx::query!(
        r#"
        CREATE TABLE IF NOT EXISTS Ads (
            company,
            position,
            remote,
            apply_link,
            description,
            paid
        )
    "#
    )
    .execute(&mut conn)
    .await
    .unwrap();

   // insert partial data here if you wish e.g. insert into Ads(company) values ('netflix');

    // uncomment this once you've run the code at least once so the table can be created
    // let res = sqlx::query!(r#"select * from Ads"#) // <- compiler error
    //    .fetch_all(&pool)
    //    .await
    //    .unwrap();
   Ok(())
}

glad to provide any more information if needed.
Info

SQLx version:  "0.6.2"
SQLx features enabled: "sqlite", "runtime-actix-native-tls"
Database server and version: SQLite 3.39.4
Operating system: OS X
rustc --version: rustc 1.63.0 (4b91a6ea7 2022-08-08)

Hi @sasacocic,
Missing type information makes cause type-checked queries (ie. the query! macro) to panic when they fail to find that type information.
So depending your preferences, there are 3 ways to make your error go away:

Option A, add type declarations to the columns within your table create statement.  If you want strong typing guarantees, you may want to look at sqlites 'strict' table option too.
Option B, insert some data into the database file before compilation, so that sqlx can sample the first row of the query result.
Option C, use the sqlx::query function instead of the sqlx::query! macro (the function does not perform a compile time type check)

ahhhh ok that makes sense thanks for the clarification

@tyrelr I wanted to follow up on this, because I think I've maybe found a bug, but it might also possibly just me being a sqlx noob. Basically when I try to create to create a prepared statement with an OFFSET the following query always returns []
I've checked that my database actually has values in it as well.
    let p = sqlx::query_as::<_, Ad>("SELECT * FROM ads LIMIT 25 OFFSET ?")
        .bind(1)
        .fetch_all(&pool)
        .await
        .unwrap();

    dbg!(p);

@tyrelr actually turns out I wrote the wrong SQL and isn't an issue with sqlx 🙃  I just needed to adjust offset correctly.

One other way you may be able to solve the problem is by adding type information in your select query. For example: Select company "company: String" From Ads. You would need to specify each column in your query, but if you didn't want to change the table creation SQL, this may help. Just thought this may help someone!


add some comments for xds proxy

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Please check any characteristics that apply to this pull request.

[X] Does not have any user-facing changes. This may include CLI changes, API changes, behavior changes, performance improvements, etc.

@ramaraochavali: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio
e6b5b2bea220bbd8e124b03203b4be4cb56eb724
link
true
/test release-notes_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/test release-notes_istio


Flaw inside testsuite_mpi

Description

Type: Bug
Priority: Major

Bug
eCos
mbed TLS build:
Version: 2.6.0
It seems like one mbedtls_mpi testsuite fails when executed in a loop. To be more specific the function that fails is:
Test mbedtls_mpi_is_prime #3
depends_on:MBEDTLS_GENPRIME
mbedtls_mpi_is_prime:10:"2833419889721787128217599":0

It seems that it always fails on the eleventh run inside the miller rabin test as the loop counter exceeds 30, which is an error condition (returns MBEDTLS_MPI_NOT_ACCETABLE). I understand that the miller-rabin test requires randomness which is retrieved from std_rand at the moment. I'd suggest that you try to run the test in a loop for several times and if the error doesn't occur we could have a look at the random numberes produced.

ARM Internal Ref: IOTSSL-2084

Based on the description, the test may be failing due to #1897. But I haven't checked the values.


az cli commands haaaaaang

Describe the bug
As of late, all az cli commands I issue from either the native PowerShell terminal or the integrated VS Code terminal hang...for like 5 mins before executing.
Related command
az login
Errors
The command is taken, but hangs for minutes before executing/completing.
Issue script & Debug output
NA
Expected behavior
That az cli commands execute similar to the time it takes other CLI commands.
Environment Summary
azure-cli: 2.51.0
Powershell:
Name                           Value
----                           -----
PSVersion                      7.3.4
PSEdition                      Core
GitCommitId                    7.3.4
OS                             Microsoft Windows 10.0.22621
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           <IP_ADDRESS>
WSManStackVersion              3.0

VS Code:
Version: 1.84.2 (user setup)
Commit: 1a5daa3a0231a0fbba4f14db7ec463cf99d7768e
Date: 2023-11-09T10:51:52.184Z
Electron: 25.9.2
ElectronBuildId: 24603566
Chromium: 114.0.5735.289
Node.js: 18.15.0
V8: <IP_ADDRESS>-electron.0
OS: Windows_NT x64 10.0.22621

Additional context
Quite a blocker when trying to script or do things in a timely fashion.

Thank you for opening this issue, we will look into it.

The az upgrade command in version 2.51.0 contains a bug. Please uninstalls the 32 bit and install 2.54 x64 manually.
Is the execution time in PowerShell normal, how about this command az version?

Please see a similar issue https://github.com/Azure/azure-cli/issues/27796.


Error while SSO Delegation" error="no service account rule matches"

Hi,
I've followed the docs for SSO RBAC namespace delegation https://argoproj.github.io/argo-workflows/argo-server-sso/#sso-rbac-namespace-delegation and getting an error as below -
Error while SSO Delegation" error="no service account rule matches"

argo v3.3.5
emissary executor v3.3.5
Cluster-install
Cluster roles and cluster role bindings

Currently, we have 3 service accounts - read-only (default), argo-admin and external-team
read-only (default)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: argo-read-only
  namespace: argo
  annotations:
    workflows.argoproj.io/rbac-rule: "true"
    workflows.argoproj.io/rbac-rule-precedence: "0"

argo-admin
apiVersion: v1
kind: ServiceAccount
metadata:
  name: argo-admin
  namespace: argo
  annotations:
    workflows.argoproj.io/rbac-rule: "'admin' in groups"
    workflows.argoproj.io/rbac-rule-precedence: "1"

external-team
namespace - external-team
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-team
  namespace: external-team
  annotations:
    workflows.argoproj.io/rbac-rule: "'external-team' in groups"
    workflows.argoproj.io/rbac-rule-precedence: "1"

Updated the above changes with all the required cluster roles and cluster role bindings were added
I'm (Sandeep) part of admin group and able to submit all worklfows

Argo server logs on argocd
time="2022-07-11T19:15:36.044Z" level=info msg="Error while SSO Delegation" error="no service account rule matches"
time="2022-07-11T19:15:36.044Z" level=info msg="selected SSO RBAC service account for user"<EMAIL_ADDRESS>loginServiceAccount=argo-admin serviceAccount=argo-admin ssoDelegated=false ssoDelegationAllowed=true subject=xxxxxxx.................................
time="2022-07-11T19:18:39.367Z" level=info msg="Alloc=12697 TotalAlloc=661075 Sys=69073 NumGC=145 Goroutines=102"
time="2022-07-11T19:23:39.367Z" level=info msg="Alloc=11727 TotalAlloc=662318 Sys=69073 NumGC=147 Goroutines=102"
time="2022-07-11T19:28:39.367Z" level=info msg="Alloc=11448 TotalAlloc=663600 Sys=69073 NumGC=150 Goroutines=102"

Cluster role binded for default service account argo-admin
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argo-server-cluster-role
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - get
      - watch
      - list
  - apiGroups:
      - ""
    resources:
      - secrets
    verbs:
      - get
      - create
  - apiGroups:
      - ""
    resources:
      - pods
      - pods/exec
      - pods/log
    verbs:
      - get
      - list
      - watch
      - delete
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - watch
      - create
      - patch
  - apiGroups:
      - ""
    resources:
      - serviceaccounts
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - argoproj.io
    resources:
      - eventsources
      - sensors
      - workflows
      - workfloweventbindings
      - workflowtemplates
      - cronworkflows
      - clusterworkflowtemplates
    verbs:
      - create
      - get
      - list
      - watch
      - update
      - patch
      - delete

In the above service accounts, the rbac-rule-precedence is having same value for two service accounts - is it causing the issue ?
or something else ?

Message from the maintainers:
Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

https://github.com/argoproj/argo-workflows/blob/57bac335afac2c28a4eb5ccf1fa97bb5bba63e97/server/auth/gatekeeper.go#L307
Should I modify my cluster roles ??

Hi @basanthjenuhb - could you help me on this ? I'm stuck on this. Thanks :)

I'm also not able to authenticate with other service accounts. It only works with one service account, which happens to match the first group in the list. See error:
time="2022-07-13T20:34:41.051Z" level=info msg="Error while SSO Delegation" error="no service account rule matches"

Was able to figure out the issue from @basanthjenuhb's  comment in another issue. The service account I needed was missing.

@hbrewster-splunk Best guess,

Either argo server service account doesn't have permission to list service account in the namespace that you are using. or 2. There are no service accounts in the namespace.

Ohh that's nice! In my case, I do have a service account in the namespace (not where Argo is installed) but still it's giving the same error to the users in the external team group (see above)

I think the service accounts need to be in the same namespace where argo-workflows is installed. The role and role binding would be in the target namespace.
https://github.com/argoproj/argo-workflows/blob/master/docs/argo-server-sso.md#sso-rbac

@seif-raafat

That's the whole point of SSO RBAC delegation right ? Service accounts should be in their respective namespaces
The link which you sent is for default routing into the Argo server itself 😄

Yeah you're right 😁 I'll need to reconfigure my setup and test it out. I'll let you know how it goes.
https://github.com/argoproj/argo-workflows/blob/master/docs/argo-server-sso.md#recommended-usage

@sandeepitachi
I think your setup looks correct. Not sure what is the issue. Needs debugging.
Can we connect on zoom or something for 15 mins offline to debug this ?

@basanthjenuhb Hey basanth! Thanks for the response. Sure let me know when's your availability ? Anytime works for me

I have updated the service accounts to be deployed to each namespace and can confirm that it works. Rule matches are done correctly. One thing that is confusing to me is the log entry has loginServiceAccount referencing the default account in the installation namespace while serviceAccount correctly points to the namespace service account. See below:
time="2022-07-14T16:15:13.843Z" level=info msg="selected SSO RBAC service account for user"<EMAIL_ADDRESS>loginServiceAccount=user-default-login serviceAccount=my-namespace-admin ssoDelegated=true ssoDelegationAllowed=true subject=xxxxxxx.................................

Thanks for the explanation! It makes sense to me now. I found the page <argo-workflows-server-url>/userinfo to be pretty helpful when debugging user groups.

@basanthjenuhb
Sorry fo the incovienience. It's working :) Our org AD team groups sync policy is a bit slow. I can see, multiple groups under the user info tab now. Though I had the correct configuration - That's what I was wondering what was causing an issue - I checked like 10 times - haha 😄

@sandeepitachi what prevents a user going into the Argo Workflows UI and submitting a new workflow yaml that has another team's serviceAccoutName and namespace in it? ie https://github.com/argoproj/argo-workflows/blob/147730d49090348e09027182dcd3339654993f41/examples/cron-backfill.yaml#L51

when you say "your service account" do you mean my personal user from IDP is somehow being restricted to only use a particular serviceaccount or namespace? I'm not sure where that restriction happens otherwise I could just reference someone else's serviceaccount and namespace in the yaml

Yes. You could verify your service account on the user info tab on Argo workflows UI. Users are binded to the service account


/code command prefixes code with whitespace

I confirm (by marking "x" in the [ ] below: [x]):

[x] This is not a troubleshooting question. Troubleshooting questions go here: http://www.mattermost.org/troubleshoot/.
[x] This doesn't reproduce on web browsers (such as in Chrome). If it does, issue reports go to the Mattermost Server repository.
[x] I have read contributing guidelines.

Summary

Environment

Operating System: Windows 10 Pro Build 19044.1503
Mattermost Desktop App version: 5.0.2
Mattermost Server version: 6.2.1

Steps to reproduce
Type "/code foo"
Expected behavior
Renders "foo"
Observed behavior
Renders " foo" <- notice the prefixed space character!

Possible fixes
Not currently known

This is a webapp issue, transferring to server repo.


Onsite - race condition in passing along VA profile id

Value Statement
The Onsite call was nested into a Celery Chain, which calls one after another. The celery chain is called with apply_async(). Onsite is dependent on the return value from the lookup_va_profile_id method. This may be the simplest method, but please review and determine the best route to move forward.
Checklist

[ ] The Onsite method gets the va profile id (if there's one to get)
[ ] Modifications to tests if needed

Additional Info/Resources

https://docs.celeryq.dev/en/v4.4.7/userguide/index.html
https://docs.celeryq.dev/en/v4.4.7/userguide/calling.html

Started drilling into this. Part of our issue was misunderstanding of immutable signatures. This is chaining without immutable:

Chaining with immutable:

Immutable signatures cannot have args populated from previous values returns in the chain.

All testing seems to point to it being related to the immutable signature. As can be seen below it utilizes the different workers to get the expected results of chaining together methods.

If you follow the worker identifier it can be seen that both a_list and b_list flip the worker usage around, so this should be the solution. Going to increase dev to two celery tasks and give it a shot.

@EvanParish assigning to you per Kyle to pick up where he left off.

It looks like a fairly simple change. I'll get it tested in Dev and Staging once we have the issues there sorted out.
I was doing my own verification of Kyle's testing above, it looks like apply_async() just makes it so the app isn't waiting on the current chain to finish. The chain seems to operate as expected, passing the values from the previous task to the next. If the task's signature is immutable, denoted with .si then that task will not accept values passed from the previous one and must have all of its own variables defined. These were observed by Kyle previously, just reiterating that I've verified these points as well.

While talking with Kyle we identified a place where the Onsite notification would not have been sent most of the time when an email is sent using the email address and made changes to correct that.
I found out that testing the other aspects of this change is hard to do in Dev because we need to have valid inputs for MPI to recognize, or it will fail before even trying to send the Onsite notification. We can't successfully test the other ways of sending Onsite notifications unless we have valid values other than the va_profile_id. So we would need things like a valid ICN, BIRLSID, or EDIPI that can be validated through MPI in our Dev environment.

To alleviate any confusion with the flow of onsite a flowchart has been added to our Glossary.

Tom Harrison provided some possible ICN values to be used in the Dev environment. It looks like I'm getting the 422 error I would expect to see in Dev when trying to use them on our API.

I had to fix a bug causing one of the unit tests to fail. I added the new Onsite celery task to the email path previously, but since it's no longer necessary I removed it. The code that was checking for it was still in the unit test however, causing it to fail. I removed the checks for the Onsite task and that took care of the problem.


extra statistics

seems like these would be nice:
https://code.google.com/p/memcached/issues/detail?id=290
https://code.google.com/p/memcached/issues/detail?id=289

I think get_expired is meaningless when lru_maintainer is turned on, but if not, it's really useful.

some super nice person implemented this, which is merged into the next branch. closing this out.


kn func run:  for Python does not show any message for the user to know that it is running

As a Function Author, when I am using command kn func run for python language/runtime, I should be able to discern that server is started so that I know the function is ready to be used.
GIVEN I am running newly create Python function
WHEN I issue 'kn func run' command
THEN I see that my function is running and ready to be called.

You should see Serving on http://<IP_ADDRESS>:8080/ in the logs, which is the standard message from the wsgi server (waitress: https://docs.pylonsproject.org/projects/waitress/en/latest/runner.html) see this code


request: add a line break to the end of modules

I'm sure everyone has gotten used to it but as a beginner to elixir it feels more clean to have some separation:
status quo:
defmodule HelloWeb.PageController do
  use HelloWeb, :controller

  def index(conn, _params) do
    render(conn, "index.html")
  end
end

my radical proposal:
defmodule HelloWeb.PageController do
  use HelloWeb, :controller

  def index(conn, _params) do
    render(conn, "index.html")
  end

end

well... I guess it doesn't look that much better but it feels better when there is more than one def. This is probably a controversial opinion so I am fine to keep using:
  #
end

I just want to present this alternative idea

I get it you mean format on save formatting style. If so, elixir-ls uses standard builtin elixir formatter and we have no plans to change it. If you would like to see any changes to it, please use elixir mailing list for feature discussions.


rosdep install fails with Cannot locate rosdep definitions on Jetson Xavier

Hi
I am step by step setup create3 but command rosdep install --from-path src --ignore-src -yi have some issues.

Use environment

Error:

ros2@ros2-desktop:~/create3_examples_ws$ rosdep install --from-path src --ignore-src -yi
ERROR: the following packages/stacks could not have their rosdep keys resolved
to system dependencies:
create3_teleop: Cannot locate rosdep definition for [teleop_twist_joy]
create3_lidar: Cannot locate rosdep definition for [slam_toolbox]
what can i do

Hi,
what version of ROS 2 did you install and how did you do that?

Hi alsora,
It's ROS 2 Galactic. I am follw this
link. https://iroboteducation.github.io/create3_docs/setup/ubuntu2004/

Those instructions are for a desktop (x86_64) build of ROS 2.
I think you are running on a Jetson Xavier, so those instruction are likely not correct.

Hi alsora,
But create3 documentation can use on jenson.
like this https://iroboteducation.github.io/create3_docs/setup/jetson/
Am i missing something?

Yes, but note that these instructions are different from those you linked before.
Did you follow the jetson specific instructions ?
And FYI, this problem is not rekated to Create 3 software, but rather "how to setup ROS 2 on NVidia jetson".

Also, to chime in here: Jetson support is via the community, and this is issue is not with Create 3 examples; I am moving this to a discussion, and we can continue it there with the community.


Missing ) after argument list on windows

basedir=$(dirname "$(echo "$0" | sed -e 's,\\,/,g')")
          ^^^^^^^
SyntaxError: missing ) after argument list

I have this issue when running tests on Windows (AppVeyor).

It seems like you're running the bash tap shim that's created for the benefit of windows bash and Cygwin.
How are you running the command?


[WM-1667] Don't submit parameter_value when input source is changed to None

This PR fixes the behavior mentioned in the ticket.
Before: The payload being sent for input switched to "None" source included parameter_value
{
  "input_name": "fetch_sra_to_bam.Fetch_SRA_to_BAM.docker",
  "input_type": {
    "type": "optional",
    "optional_type": {
      "type": "primitive",
      "primitive_type": "String"
    }
  },
  "source": {
    "type": "none",
    "parameter_value": "quay.io/broadinstitute/ncbi-tools:<IP_ADDRESS>"
  }
}

Now
{
    "input_name": "fetch_sra_to_bam.Fetch_SRA_to_BAM.docker",
    "input_type": {
      "type": "optional",
      "optional_type": {
        "type": "primitive",
        "primitive_type": "String"
      }
    },
    "source": {
      "type": "none"
    }
}

Closes https://broadworkbench.atlassian.net/browse/WM-1667

@mspector Yes I though about some other way to write this test but I wasn't able to since the behavior I wanted to test depends on state changes which I wasn't sure how to do in a smaller unit testable way.


Insert at cursor does not work

Reported by @jooyoungseo, Insert at Cursor in both quick chat and the panel chat do not work. ATM, clicking the button does nothing and using the keybinding opens the Copilot Suggestions panel.

Works fine for me. Please check for errors

Hm, it's working now, but only when I have an editor open. Shouldn't the action be disabled when I have no editors open?

Probably. We don't have the right context keys right now for the right set of requirements

should it open a new document in the case of no active text editor?


leak references for safety in PyWeakRefMethods::upgrade_borrowed

Note: this is a patch fix for 0.22.4, the PR is targeting that branch, not main.
As per discussion in #4528, borrowing from a weakref is definitely unsafe and high risk for use-after-free; any arbitrary Python code could in theory remove the last strong reference.
In 0.23 we will straight-up remove these unsound methods, but in 0.22.4 we can't really do that so the mitigation proposed here is to switch to leak the upgraded reference when "borrowing" from the weakref. This avoids use-after-free in favour of the more gentle memory leak. The methods are also all deprecated as a nudge for users to shift away from them ASAP.

To get on with shipping 0.24 I will merge this now; I can't see a way we can do anything reasonable other than leak.

It may be the lesser evil in this case, but I have found out the hard way that python lacks overflow checking on it's reference counts. So a reference count leak, repeated a sufficiently large number of times on the same object, can lead to a use after free.
"sufficiently large number of times" is unlikely to happen on 64-bit platforms but is far more plausible on 32-bit platforms.

Hmm, that's a good point. I mean this also just straight-up ensures a memory leak before the overflow too, so it's definitely unpleasant for users and they should move to the owned methods which don't have any of this mess.


UVI Request

--- UVI JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from v3.2 to before v5.15.5",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=6a2968aaf50c7a22fced77a5e24aa636281efca8",
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=ed35e950d8e5658db5b45526be2c4e3778746909"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "6a2968aaf50c7a22fced77a5e24aa636281efca8",
      "note": "introduced"
    },
    {
      "type": "commit",
      "value": "ed35e950d8e5658db5b45526be2c4e3778746909",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "NFC: add NCI_UNREG flag to eliminate the race\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v5.15.5 by commit ed35e950d8e5658db5b45526be2c4e3778746909, it was introduced in version v3.2 by commit 6a2968aaf50c7a22fced77a5e24aa636281efca8. For more details please see the references link."
}
--- UVI JSON ---

This issue has been assigned UVI-2021-1002312


Can't find cl.exe

From windows 11 enterprise

I'm using windows 10 and I get the "unable to find program cl.exe" error


 The error I get in the logs is -
Unable to establish connection: Data source 'Other Databases (JDBC)' has not been licensed.
I can see the license is indeed valid in Tableau Desktop and I am able to query the same datasource using JDBC in tableau desktop. I looked into the past issues and lowered the MaxThreads to 1 as well but error still remained.
I am using the following -

MacOS Monterey
Tableau Desktop 2021.4
Python version 3.9
tdvt build from master branch
Staples and cast_calcs datasets

Command -<EMAIL_ADDRESS>-m tdvt.tdvt run mydb
Running tdvt suite [mydb]: tested sql [False]: tested tuples [True]: tested error [False]: output dir [/tmp/3ea43f67ea98e10abd0397eeb0767caae100232a46a65411833e4f25fpra1wci]: logical [True]: config file [StaplesConnectionTestmydb]: override [-DLogLevel=Debug]: tds [Staples.mydb.tds]: thread [6] tdvt thread id: 5

2022-02-11 17:27:57,645 Thread-[5] Running test:StaplesConnectionTestmydb
2022-02-11 17:27:57,653 Thread-[5]  calling /Applications/Tableau Desktop 2021.2.app/Contents/MacOS/tabquerytool --query-file-list /tmp/3ea43f67ea98e10abd0397eeb0767caae100232a46a65411833e4f25fpra1wci/StaplesConnectionTestmydb/tests.txt -d /Users/kharekartik/Documents/Packages/tableau-tests/tds/Staples.mydb.tds --combined --output-dir /tmp/3ea43f67ea98e10abd0397eeb0767caae100232a46a65411833e4f25fpra1wci -DLogDir=/tmp/3ea43f67ea98e10abd0397eeb0767caae100232a46a65411833e4f25fpra1wci/StaplesConnectionTestmydb -DOverride=ProtocolServerNewLog -DLogLevel=Debug -DLogicalQueryRewriteDisable=Funcall:RewriteConstantFuncall -DInMemoryLogicalCacheDisable
2022-02-11 17:28:05,993 Thread-[5] CalledProcessError: Return code: 17 LoadDatasource TableauException: Unable to establish connection: Data source 'Other Databases (JDBC)' has not been licensed.

No data source specified

2022-02-11 17:28:05,994 Thread-[5] Command line output:LoadDatasource TableauException: Unable to establish connection: Data source 'Other Databases (JDBC)' has not been licensed.

Hi Kartik,
It looks like your TDVT tests are running against Other JDBC, not your new connector. You should re-generate your Calc and Staples TDS files by connecting to each table using your new connector, right-clicking on the connection in the Data pane, and selecting "Add to saved Datasources".
Let us know if the above, coupled with setting MaxThreads to 1 as you've already done, doesn't solve the issue. Thanks!

Hi Kartik, we have reproduced this issue internally with Tableau 2021.4. As a workaround until this is fixed, try using 2021.3 or earlier.

@KKcorps
Can you try the following steps?

Run command atrdiag  -enableATRFeature
Run command atrdiag  -deleteAllATRs
Start Tableau, you might see dialog says licensing changed, need restart, just let it restart
Run TDVT again.

If you still see issue, please share the log with us.

Thanks a lot. I'll update you in a day if this works or not.

@KKcorps Any update?

@KKcorps I am closing this old ticket.  Feel free to reopen if you still have issue.


Suggest using a JSON config file for secrets

https://github.com/kausrini/Mini-ScienceDMZ/blob/cc11d81d7ed5c5757b766466d4885e5739e0a949/log_email/sendStatus.py#L21-L24

Any particular reason why JSON is better than python other than interoperability?

The primary reason would be to avoid arbitrary code paths that are unintended. Anyone can place a python module in the location injected to sys.path and that would take precedence over packaged modules (say pi_settings.py). Even if that possibility is ruled out, any code can be placed in secrets.py will be executed as part of this application.
If a JSON config is used, it is relatively simpler & secure. Also, it provides a clear distinction between user config and application code.


Fixed SCons decider bug

I had a problem where SCons would always build the entire project regardless of the changes I made. This lead to builds taking several minutes for just a small change. It turns out that the decider function was misbehaving, so I replaced it with a custom function which does practically the same thing.

It would be better to fix this directly upstream, so that we don't have to maintain our decider if it's the upstream one which misbehaves. See https://github.com/SCons/scons
CC @bdbaddog

Ok, good news. It turns out that this is fixed in the latest version of SCons (3.0.5). I was using 3.0.4.


PartialMessage of message with oneof field requires that oneof field be non-partial

Calling .create() on a MessageType instance allows partial oneof fields at runtime, but at compile-time they are required to be non-partial. TS Playground
// pulled from https://github.com/timostamm/protobuf-ts/blob/master/packages/runtime/src/message-type-contract.ts
export type PartialMessage<T extends object> = {
    [K in keyof T]?: PartialField<T[K]>
}; type PartialField<T> =
     T extends (Date | Uint8Array | bigint | boolean | string | number) ? T
   : T extends Array<infer U> ? Array<PartialField<U>>
   : T extends ReadonlyArray<infer U> ? ReadonlyArray<PartialField<U>>
   : T extends { oneofKind: string } ? T
   : T extends { oneofKind: undefined } ? T
   : T extends object ? PartialMessage<T>
   : T ;

// example
interface Bar {
    arr: number[];
    other?: number;
}

interface Foo {
    args: {
        oneofKind: "bar";
        bar: Bar;
    } | {
        oneofKind: undefined;
    }
}

function create(i: PartialMessage<Foo>) {}

create({
    args: {
        oneofKind: 'bar',
        bar: { // ts-2741: Property 'arr' is missing in type '{ other: number; }' but required in type 'Bar'.
            other: 1,
        }
    }
})

I don't think there's actually a way to get this working given the existing types. This is another instance where the updated oneof representation would make this trivial: : T extends { kind: string; value: infer U } ? { kind: T['kind']; value: PartialField<U> }.

Good call. Would you like to open a PR against the 3.x branch?

I believe I have a solution that can work for the existing v2 representation. I know this PartialMessage is depended upon heavily so it's possible I'm not properly covering some case.
type SetOneof = { oneofKind: string };
type SetOneofKind<T extends SetOneof> = T extends { oneofKind: string & keyof T }
    ? T['oneofKind']
    : never;

type PartialOneofValueMap<T extends SetOneof, K extends SetOneofKind<T> = SetOneofKind<T>> = {
    [k in K]: { oneofKind: k } & {
        [p in k]: T extends { oneofKind: k } ? PartialField<T[k]> : never;
    };
};
type PartialOneof<
    T extends SetOneof,
    K extends SetOneofKind<T> = SetOneofKind<T>,
    M extends PartialOneofValueMap<T, K> = PartialOneofValueMap<T, K>
> = T extends { oneofKind: keyof M } ? M[keyof M] : { oneofKind: undefined };

export type PartialMessage<T extends object> = {
    [K in keyof T]?: PartialField<T[K]>
}; type PartialField<T> =
     T extends (Date | Uint8Array | bigint | boolean | string | number) ? T
   : T extends Array<infer U> ? Array<PartialField<U>>
   : T extends ReadonlyArray<infer U> ? ReadonlyArray<PartialField<U>>
   : T extends { oneofKind: string & keyof T } ? PartialOneof<T>
   : T extends { oneofKind: undefined } ? T
   : T extends object ? PartialMessage<T>
   : T ;


[BUG] QR code is not scannable

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the bug
QR code is not scannable for<EMAIL_ADDRESS>I wonder if the short LN address makes the QR less detailed, and therefore the icon hides too much of the QR.
Screenshots [optional]

Steps To Reproduce
Try scan the above screenshot, or hardcode the app to use leo's LN address.
Expected behavior
QR should be scannable.
Alby information
3.2.1
Device information
No response
Additional context
No response
Are you working on this?
None

@rolnz I would like to work on this issue, can you assign this to me.

I feel using higher error correction level would be a best way, currently we are using medium level but using quartile level will fix this issue (works fine for 1 character lightning address names too)

sounds good @Rithvik-padma

@rolznz I'M INTRESTED WORK ON THIS TECHNOLOGY CAN YOU ASSING ME PLEASE ?


Unknown error response code: 404

Hi Guys,
I´m having this error when trying to execute:  client.getRecentMediaTags(tag)
Don´t know if Instagram changed something on the url´s that the lib is calling.
It is also ocurring on the demo:
http://jinstagram-sachinhandiekar.rhcloud.com/BasicWebDemo/search.jsp
Trying to lookup some tags like: formandasfisioipa or comendaveraprata
Anybody had this problem yet?

I also have this error when I try to call instagram.getRecentMediaNextPage(pagination)

Related to #110

Me and @giomendo found out what was the problem. It was an encoding problem when sending the String that we wanted to monitor. The string contained some bad characters on the end and sometimes also at the begging and that just made things break. Sorry for the inconvenience.

Closing the ticket... Re-open if needed


Update REST.js

@pablopaul just out of curiosity, you don't get the pull request template for new PR?

@PeterNgTr I got it, but ignored it, the lint error will be fixed soon :)


Stacked Services - issue with navigateToView

This function will give us an error:
 /// Pushes [view] onto the navigation stack
  Future<dynamic> navigateToView(Widget view, {dynamic arguments}) {
    return Get.to(view, arguments: arguments);
  }

Get.to function doesn't support arguments anymore.

@elhe26 in v2.13.1 It's still available. I what version are you using?

Sorry. My fork has Get: 2.5.2 for stacked_services.


Fix building with 1.72

The reported MSRV of this project is 1.72, but rustc 1.72 fails to compile it:
error[E0446]: private type `CommonMarkOptions` in public interface
   --> src/parsers/pulldown.rs:57:5
    |
57  | /     pub fn show(
58  | |         &mut self,
59  | |         ui: &mut egui::Ui,
60  | |         cache: &mut CommonMarkCache,
...   |
63  | |         populate_split_points: bool,
64  | |     ) {
    | |_____^ can't leak private type
    |
   ::: src/lib.rs:240:1
    |
240 |   struct CommonMarkOptions {
    |   ------------------------ `CommonMarkOptions` declared as private

This PR fixes the compilation error, and enforces testing with 1.72 using a rust-toolchain file, which the CI will heed, as will any local cargo check.
This should probably have a patch release.

The struct is not supposed to be public. Instead the function in the pulldown parser should be pub(crate). The newer compiler must have gotten smart enough to understand that the functions are within a private module. I'll make an alternative fix

Patch has been released


Transformer now adds environmentProperties to TaskLaunchRequest

Updated task dependencies so that they will work with new Task-Stream 2.0 SNAPSHOT
Now sets content type to application/json so downstream users won't have to set content type for task launchers.

resolves #7

@olegz Using the following stream definition it appears that text/plain is the output contentType for the transformer.
http --server.port=9000 | tlr-transform2 --application-name=foofun --uri=maven://org.springframework.cloud.task.app:timestamp-task:1.2.0.RELEASE --command-line-arguments='--spring.datasource.url=jdbc:mariadb://localhost:3306/practice --spring.datasource.username=root --spring.datasource.password=password --spring.datasource.driverClassName=org.mariadb.jdbc.Driver' | mytl

This doesn't work when applied to stream above.

I found the problem.
Looks like Spring Integration doesn't always honor not propagated headers. The AbstractMessageProcessingTransformer has explicit code like:
if (result instanceof Message<?>) {
	return (Message<?>) result;
}
return getMessageBuilderFactory().withPayload(result).copyHeaders(message.getHeaders()).build();

I think we can overcome the issue here with the @ServiceActivator instead of @Transformer meanwhile we are fixing the violation on the Spring Integration side.
@cppwfs ,
please, raise a JIRA on the matter: https://jira.spring.io/browse/INT
That was a good team work though!
Although with earlier appropriate comments we would come to the solution more quickly 😉

Well, never mind.
That was an issue of old jars in the IDEA caches.
After refreshing it starts working. However still with the mentioned change from the @Transformer to @ServiceActivator.
The new test should expect isEqualTo(MediaType.APPLICATION_JSON) instead of plain string.
And there is also UsingNoURIIntegrationTests failing. But that is just because we don't produce MessageTransformationException anymore.


Freezing Backbone Layers

I am working on Fine-Tuning the Owl-Vit models and an interested in freezing the "Clip Backbone" and only training the final heads.
Is there an easy way to set this up in the config?
I imagine that I can set the learning rate to 0, but that would not mark them as frozen AND avoid computing the gradient (to save time).
  sched_txt.re = '(backbone/clip/.*)'
  sched_txt.lr_configs = ml_collections.ConfigDict({
      'learning_rate_schedule': 'compound',
      'factors': 'constant',
      'base_learning_rate': 0,
  })

Any recommendations on the best practice for this would be appreciated

I think you can set sched_txt.lr_configs = None to freeze these parameters. This should then exclude these parameters from optimization here.
You could also try adding a jax.lax.stop_gradient between the backbone and the heads, if the above does not save compute.
Note that the code checks that every parameter is optimized by exactly one schedule, so you'll need to change the regular expression for the non-frozen parameters to the exact inverse of the frozen ones (e.g. using a negative lookahead expression).

In reference to the first approach, I've tried to set-ups and they both break.
I tried
sched_backbone = ml_collections.ConfigDict()
sched_backbone.re = '(backbone/clip/.*)'
sched_backbone.lr_config = None

and got the error
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/scenic/scenic/projects/owl_vit/main.py", line 47, in <module>
    app.run(main=main)
  File "/home/ubuntu/scenic/scenic/app.py", line 65, in run
    app.run(functools.partial(_run_main, main=main))
  File "/home/ubuntu/.venv/scenic/lib/python3.9/site-packages/absl/app.py", line 308, in run
    _run_main(main, args)
  File "/home/ubuntu/.venv/scenic/lib/python3.9/site-packages/absl/app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "/home/ubuntu/scenic/scenic/app.py", line 101, in _run_main
    main(rng=rng, config=FLAGS.config, workdir=FLAGS.workdir, writer=writer)
  File "/home/ubuntu/scenic/scenic/projects/owl_vit/main.py", line 37, in main
    trainer.train(
  File "/home/ubuntu/scenic/scenic/projects/owl_vit/trainer.py", line 224, in train
    schedule = scenic_optax.make_schedule(config.get('schedule'))
  File "/home/ubuntu/scenic/scenic/train_lib/optax.py", line 161, in make_schedule
    schedule = [(re, name, create_schedule(lr_configs))
  File "/home/ubuntu/scenic/scenic/train_lib/optax.py", line 161, in <listcomp>
    schedule = [(re, name, create_schedule(lr_configs))
  File "/home/ubuntu/scenic/scenic/train_lib/optax.py", line 153, in create_schedule
    fn = get_learning_rate_fn(
  File "/home/ubuntu/scenic/scenic/train_lib/lr_schedules.py", line 287, in get_learning_rate_fn
    raise ValueError(
ValueError: `base_learning_rate` has to be defined in the lr_config.

Attempt 2
sched_backbone = ml_collections.ConfigDict()
sched_backbone.re = '(backbone/clip/.*)'
sched_backbone.lr_configs = ml_collections.ConfigDict()

and got the error trace
  File "/home/ubuntu/scenic/scenic/app.py", line 101, in _run_main
    main(rng=rng, config=FLAGS.config, workdir=FLAGS.workdir, writer=writer)
  File "/home/ubuntu/scenic/scenic/projects/owl_vit/main.py", line 37, in main
    trainer.train(
  File "/home/ubuntu/scenic/scenic/projects/owl_vit/trainer.py", line 224, in train
    schedule = scenic_optax.make_schedule(config.get('schedule'))
  File "/home/ubuntu/scenic/scenic/train_lib/optax.py", line 161, in make_schedule
    schedule = [(re, name, create_schedule(lr_configs))
  File "/home/ubuntu/scenic/scenic/train_lib/optax.py", line 161, in <listcomp>
    schedule = [(re, name, create_schedule(lr_configs))
  File "/home/ubuntu/scenic/scenic/train_lib/optax.py", line 153, in create_schedule
    fn = get_learning_rate_fn(
  File "/home/ubuntu/scenic/scenic/train_lib/lr_schedules.py", line 287, in get_learning_rate_fn
    raise ValueError(
ValueError: `base_learning_rate` has to be defined in the lr_config.

I traced through the code around the step you linked and the problem is that:

frozen_mask, masks, scheds = _split_frozen(masks, scheds)  identifies frozen layers based on things where the sched is None.
The line above it, does pattern matching on the param names + finds the right schedule function based on whats encoded in our input config.
All the schedule functions are created here - but it is impossible to create a "None" schedule here - this function does not have a case for 'lr_config' being None.

Am I missing a simple workaround here?
And could you clarify what you mean by jax.lax.stop_gradient I am not sure where I would add that in

It's possible that we have never tested the freezing feature and that there's a bug. @AlexeyG , have you ever tried setting sched_txt.lr_configs = None to freeze params?
Regarding stop_gradient: This stops the flow of gradients during backprop, so adding image_features = jax.lax.stop_gradient(image_features) and query_embeddings = jax.lax.stop_gradient(query_embeddings) to the main model here will stop backpropagation into the backbone and might save compute.

Looks like the stop_gradient method worked for now...but
I believe there is in fact a bug here, I was able to partially fix it, but do not have insight to fix it entirely.
** Fix 1**
https://github.com/google-research/scenic/blob/261bb131ebe5119d09d63d27be76c517c0de64cc/scenic/train_lib/optax.py#L151
+    if lr_configs is not None:
+        fn = get_learning_rate_fn(
+            ml_collections.ConfigDict({'lr_configs': lr_configs}))
+    else:
+        return None

** Fix 2**:
https://github.com/google-research/scenic/blob/261bb131ebe5119d09d63d27be76c517c0de64cc/scenic/train_lib/optax.py#L178
  not_frozen_mask = jax.tree_util.tree_map(operator.not_, frozen_mask)
  schedule_fns = [fn_base for fn_base, _ in (scheds or [])]
  schedule_base_lr = [base_lr for _, base_lr in (scheds or [])]

The problem at this point is that the "frozen masks" end up of the wrong shape and break during the update phase. So it be great if you could revisit the optmax set-up

I think the right approach is to use stop_gradient and set the learning rate to 0.0 for the weights that should be frozen. That should save the computation of the gradients for the frozen weights and produce the right Optax state. Please re-open if this doesn't work.


Spanish translation

I don't know where to put this and this is my first time at github. Excuse me if this not has to be here.
I've done a translation to spanish for the Web-UI. Maybe it can be improved, but for now most of the translate job is done.
How can I provide you the file?. Again, is my first time here. I don't know exactly what I'm doing it.
Regards!.

Hey @Hu5ker
thanks for your work!
Basically to add your contributions, you'll do the following:

Fork this repository via click on the "Fork" button top right
Clone your fork to your local file system, using git
Add your language file in the file tree of your local repository
Commit your addition
Push your changes to your fork repository
Send a pull request to transfer the changes from your repository into ours

Please check the following links:
https://try.github.io/levels/1/challenges/1
https://help.github.com/articles/using-pull-requests/
If you still have issues getting this done, you can also send me the file and i'll commit in your name to add it. Let me know

Let me know if you made any progress on this :)


Iceberg product tests

Depends on https://github.com/prestosql/docker-images/pull/64

prestodev/spark3.0-iceberg:31 is live now

Thanks!


Pagespeed phpmyadmin problem

Hello,
I'm using apache with nginx as reverse proxy.
If I include Page Speed, the ../phpmyadmin page loads and becomes blank.
If I exclude Page Speed phpmyadmin working properly.
How can I fix the problem?
Thank you

Disallowing the path may help:
pagespeed Disallow */phpmyadmin *

I would love to know a bit more about the problem though.
What does the html source of the blank page look like? And the response headers?
Are there any warnings or errors in the browser dev console when you receive the blank page?
Or in nginx's error logs?

Hi, thanks for answer,
"What does the html source of the blank page look like? And the response headers?" phpmyadmin is visible for only a second then it disappear, white page and clean console, no errors
Nginx error.log:
2017/02/24 16:21:44 [error] 11210#11210: *2 FastCGI sent in stderr: "Unable to open primary script: /var/www/html/index.php (No such file or directory)", client: MYIPADDRESS, server: www.MYDOMAIN.it, request: "GET /icons/ubuntu-logo.png HTTP/1.1", host: "www.MYDOMAIN.it", referrer: "http://www.MYDOMAIN.it/"
If i add pagespeed Disallow */phpmyadmin * in pagespeed.conf, i get the following erro:
nginx[10997]: nginx: [emerg] invalid number of arguments in "pagespeed" directive in /etc/nginx/pagespeed.conf:8

Ugh sorry, that should have been
pagespeed Disallow */phpmyadmin*

I accidentally put in whitespace.

Still the same error:
nginx[11585]: nginx: [emerg] "pagespeed" directive "Disallow" not recognized or too many arguments in /etc/nginx/pagespeed.conf:8
But on line 8 i have this:
    pagespeed FileCachePath /var/cache/pagespeed_cache;
This is strange

The configuration line should end with ;
Do you get the FastCGI sent in stderr: "Unable to open primary script"... error line without ngx_pagespeed?
A quick search for it shows that it is likely a permissions problem.

Thank you bro.. that was the ; I am embarrassed for this.
No more erros in error.log
Phpmyadmin working good.

At the moment I cannot relate the log message to anything specific to ngx_pagespeed. Therefore I would like to assume this issue resolved. If the FastCGI error message starts up showing more frequently we can re-open this.


Rename ob_space and ac_space to observation_space and action_space

representations 'ob_space' and 'ac_space' are confusing when using original gym env and GymEnv in the same code.

#231


Adding maven.compiler.release property

Adding maven.compiler.release property to ensure binary compatible cross-compiles.

Codecov Report

Merging #200 (791e7a9) into master (6de48f6) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #200   +/-   ##
=========================================
  Coverage     98.76%   98.76%
  Complexity     3748     3748
=========================================
  Files           195      195
  Lines         10462    10462
  Branches       1544     1544
=========================================
  Hits          10333    10333
  Misses           12       12
  Partials        117      117

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


is this ready for opencv3

Hi @erogol , yes, there is a branch called 'opencv3', please see:
https://github.com/andrewssobral/bgslibrary/tree/opencv3

thanks for return


Allow team owners to merge accounts

Fix for https://github.com/gratipay/gratipay.com/issues/4479.

Ready for review, @rohitpaulk. 🙇

Rebased, was c9f3ecea620ec5488689ea7d08355e3bf5d3db1e.


Compaction fails

Hi -- really liking medea so far, in spite of the mythological allusions :-)
I'm writing some simple example code, first to create some entries, next to list some, and finally to delete some entries.
I was surprised to see that if I delete a key 'foo', then the next time I list all the keys it's there, with a value of 'undefined'.  Is this expected behavior?
So I thought I'd try compacting the database after deleting a key, and got the following error:
% node delete_some_entries.js
Key 'foo' removed.
Error on medea compact: { Error: EBADF: bad file descriptor, fsync
    at Error (native) errno: -9, code: 'EBADF', syscall: 'fsync' }
Compact done.

This is using node v6.2.0 on OS X El Capitan.
Thanks!

Update: I thought I'd try removing the db.compact call from the code which does the deletes, and put it in its own file.
I deleted the storage directory contents, re-created the data, and ran:
% node compact_db.js
Compact done.
% node list_entries.js
Key 'foo' has val undefined (?)
Key 'bar' has val 'bar_value'
Key 'baz' has val 'baz_value'

So no error (yay!) but I still see that the key deleted ('foo') is still in the list of keys.  Is this expected behavior?
Thanks!

Hey, @csterritt!  Having no error and a value of undefined is the equivalent of a "Not Found." You can see how medeadown handles it here: https://github.com/medea/medeadown/blob/master/medeadown.js#L122

Hey @kevinswiber thanks for the reply!
I guess I assumed that database compaction would remove a key with no value.  Nope?

I'd also like to know the answer to this.
Compactions should remove deleted keys (the equivalent of an AOF rewrite)


Fix esbmc command

The --function argument was not being passed to ESBMC when the user invoked LSVerifier with the --function-prioritized option

The code has been successfully validated and is functioning as expected.


Respect network-mode being set to none for nodes on the docker runtime.

This is useful for selectively preventing docker's automatic default kernel route injection, as well as disabling connection to the management network.
This makes it easier to test isolation scenarios, control lab egress, and prevent confusion over unexpected default kernel route injection.

Closing to redo PR from a different branch.


Deal with non-standards in Normative References

https://github.com/Spatial-Web-Foundation/SWF-Corpus_and_IEEEP2874-D2/issues/1482
Also, note that no real title was provided in that reference:
* [[[dtc,1]]],
span:organization[Digital Twin Consortium].
span:type[misc]
Available at: span:uri[https://www.digitaltwinconsortium.org].

and that is making the citation grab "Available at:" as the only title it can see:

For most standards, and not just IEEE, normative references are always referenced by their identifier, and they simply don't foresee a case where a normative reference does not have one. What I think is happening here is, in the absence of an identifier, it is grabbing the title to use as a reference, and it thinks that "Available at" is part of the title.
References must have titles to be cited at all, span:title[], and absent a title, I would be plugging in "[Untitled]" (although I'm not sure I even do that yet.) Metanorma is being overzealous in assuming any text not marked up is the title, but... it needs to be. I'll put in a note in the documentation that if you're going to mark text up in bibliographic entries, you really should specify what the title is, otherwise Metanorma is just going to grab all the text it sees, with unpredictable results.
A reference without a title... is a foolhardy thing, in my opinion. The organisation is not really a substitute for the resource title. I'll look into that as well, but try providing a title (even if it is just "Digital Twin Consortium"); and remove "Available at", since that is automatically provided by Metanorma.

@opoudjis thank you for checking this and for the explanation on how to change this. if i were to go through and add span:title[ ] to each of those entries would that take care of things?  would it be better to change span:organization[ ] to span:title[ ] instead of adding another span with duplicated content?  i think it is probably better that i make changes to my document source then requiring time for you to update/add a special case in the software.  if adding the span:title[ ] in these cases will solve the "available at" issue then i think that is the best way, and it follows your advice about "references without titles"...
i will report back here after i make those updates. could be a few days.  thanks again

I misunderstood the issue. Normative references render fine, it's the bibliographic references that don't:

[[1]](file:///Users/nickn/Documents/Arbeit/upwork/ribose/metanorma-ieee/a.html#dtc1) [normative reference]

[Digital Twin Consortium. Available at: . [B1]](file:///Users/nickn/Documents/Arbeit/upwork/ribose/metanorma-ieee/a.html#dtc) [bibliography]

There is no title extracted at all; absent a title, Metanorma is falling back on the formatted reference, which is the entire string <formattedref format="application/x-isodoc+xml">Digital Twin Consortium.   Available at: <link target="https://www.digitaltwinconsortium.org"/>.</formattedref>. And it is not clever enough to render the link at that point.
Changing the markup from organization:[] to title:[] does indeed render the crossreference as:

[Digital Twin Consortium [B1]](file:///Users/nickn/Documents/Arbeit/upwork/ribose/metanorma-ieee/a.html#dtc)

—which is what we want.
The only real action here is

to update the documentation to insist on title in markup
to suppress n.d. for "no date provided" not only for standards, but also for websites (as discussed in the source tcket).


Add a by-reference option to C++ MethodResult::Success

Currently Success takes a pointer, since the return value is optional. However, that makes the case where there is a return value annoying if it's not a complex EncodableValue that was already constructed, since you have to write things like:
EncodableValue response(true);
result->Success(&response);

We should add an overload that would allow for the more natural
result->Success(EncodableValue(true));

Since having reference and pointer versions of the same API is confusing, I opted for deprecating the pointer versions. All that's left here is removing those (which I'll do once internal usage has been updated).


adding hovering effect to the logo of shopping basket .

Is there an existing issue for this?

[X] I have searched the existing issues

Feature Description
hey @Tejashri-Taral ,
i want to add hovering effect to the shopping basket .
currently it is :

can you please assign me this issue ?
Use Case
it will add more effect to the website .
Benefits
.
Priority
High
Agreements

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSoC contributor
[ ] I'm a Hacktoberfest contributor
[X] I want to work on this issue
[X] I'm willing to provide further clarification or assistance if needed.

@Tanushree084 someone is already working on it.


can not run quantum_transfer_learning example

Dear Xingyao Wu,
I meet troubles when running the quantum_transfer_learning, as follows
errors: System error: Ray has not been started yet. You can start Ray with 'ray.init()'.
tools: py3.8, torch.1.11, cuda 11, RTX3090

I am using python 3.8.10, torch=1.11.0, ray=1.6.0, and I don't have any problems of running all the examples. Have you install "ray" ?

您好，感谢回复。我有安装ray，应该就是JDtensorPath出错了，不知道为什么。
---Original---
From: @.>
Date: Wed, Jun 29, 2022 00:45 AM
To: @.>;
Cc<EMAIL_ADDRESS>Subject: Re: [JDEA-Quantum-Lab/TeD-Q] can not run quantum_transfer_learningexample (Issue #1)
I am using python 3.8.10, torch=1.11.0, ray=1.6.0, and I don't have any problems of running all the examples. Have you install "ray" ?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: @.***>

Hi, could you provide the error log? Let's find out what the problem is.


Emoji-Picker Popup lost on window resize

The Emoji Picker popup is lost on changing the window size of the browser even though the compose box and its contents remain same.
To reproduce : open emoji-picker while composing, Then try to resize the browser window. You should now be able to see that the popup is lost.

This is sorta intentional, but mostly because it's a pain to preserve the picker when the context around which it was popped changes (e.g. the parameters used to size it).  I'm going to close this as wontfix, in that I don't think this is going to have a good effort/value tradeoff to try to change this.

Thank you for explaining :smile: