Error when handling column names which use reserved keywords

Thanks for working on a spark connector for Exasol. It works well so far. Though I stumbled on an issue regarding reserved keywords. If the query is returning a column, which is a reserved keyword in Exasol (e.g. "CONDITION" in the example below), I get following stacktrace:
Caused by: java.sql.SQLException: syntax error, unexpected CONDITION_ [line 1, column 61] (Session:<PHONE_NUMBER>207481827)
	at com.exasol.jdbc.ExceptionFactory.createSQLException(ExceptionFactory.java:175)
	at com.exasol.jdbc.EXASQLException.getSQLExceptionIntern(EXASQLException.java:50)
	at com.exasol.jdbc.AbstractEXAStatement.execute(AbstractEXAStatement.java:468)
	at com.exasol.jdbc.EXAStatement.execute(EXAStatement.java:278)
	at com.exasol.jdbc.AbstractEXAStatement.executeQuery(AbstractEXAStatement.java:601)
	at com.exasol.spark.rdd.ExasolRDD.compute(ExasolRDD.scala:123)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
...

In the query it doesn't matter whether the column is referenced via "CONDITION" (proper escaping via quotes) or with the table alias, e.g. products.condition - it results in the same error.
I don't see this error when using JDBC for the same query (we've been using the same query for more than a year via JDBC). The error only shows up when using the spark-exasol-connector. Which is strange, as reading the stacktrace the error comes from the JDBC driver. Probably the JDBC driver has some special handling for such column names (?)
However, there's a simple workaround for this issue by renaming the respective column in the query, e.g. "CONDITION" AS product_condition. Then the error disappears and I get the result. The column can be renamed later in the data frame.

Hello @beatbull ,
Thanks for reporting this! Please keep providing your feedback!
I am going to add this as todo for time being. The connector does not obey the quoting yet; they are put as plain Scala strings when doing column-pruning.
private[this] def enrichQuery(columns: Array[String], filters: Array[Filter]): String = {
    val columnStr = if (columns.isEmpty) "*" else columns.mkString(", ")
    s"SELECT $columnStr FROM ($queryString)"
}

However, I am going to look into it in the coming weeks.

Hello @morazow ,
thanks for the quick response. Indeed the snippet you posted is causing the issue with the reserved keywords - qouting the columns or prefixing them with a table alias (e.g. columns.map(col => s"a.$col").mkString(", ") in combination with a table alias in the query) would fix the issue easily.

I could also push a PR but seems I lack permissions to do so.

Hello @beatbull,
Thanks for the proposal! The pull requests are very much welcome!
I will check and ask to update the permissions if people cannot send a pull request.
Meanwhile, I have pushed a branch with your proposal of adding a prefix to selected columns.
Please let me know if that solves your issue.

Hi @morazow ,
I tried the suggested fix, it works fine when using reserved keywords as column names in queries. Thanks for the quick fix.

Hey @beatbull ,
Great to hear it works! Please let us know if there are any issues!
I am going to close this issue and merge other branches to master.


AttributeError: 'NoneType' object has no attribute 'shape'

I'm getting a AttributeError: 'NoneType' object has no attribute 'shape' error when trying to generate gradcam visualizations.  This is with PyTorch Version:  1.13.1+cu117 and Torchvision Version:  0.14.1+cu117.
Any idea what I'm doing wrong or how to get past this error?
from PIL import Image
from torchvision import datasets, models, transforms

# Load resnet18 model
model = models.resnet18(pretrained=False)
model.load_state_dict(torch.load("path_to_model.pt))

# Apply transform to input image
transform = transforms.Compose(
    [
        transforms.Resize(size=(299, 299)),
        transforms.ToTensor(),
        transforms.Normalize([0.2858, 0.2105, 0.1557], [0.3035, 0.2469, 0.2060])
    ]
)
image = Image.open("path_to_image.jpg")
input_tensor = transform(image)

# Create batch
input_tensor = input_tensor.unsqueeze(0)

# Choose target layers - This is what is recommended for resnet18
target_layers = [model.layer4[-1]]
cam = GradCAM(model=model, target_layers=target_layers, use_cuda=True)

# Display gradcam visualization
grayscale_cam = cam(input_tensor=input_tensor, targets=None)
grayscale_cam = grayscale_cam[0, :]
visualization = show_cam_on_image(image, grayscale_cam, use_rgb=True)

Error message:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[12], line 41
     23 cam = GradCAM(model=model, target_layers=target_layers, use_cuda=True)
     25 # You can also use it within a with statement, to make sure it is freed,
     26 # In case you need to re-create it inside an outer loop:
     27 # with GradCAM(model=model, target_layers=target_layers, use_cuda=args.use_cuda) as cam:
   (...)
     39 # You can also pass aug_smooth=True and eigen_smooth=True, to apply smoothing.
     40 # grayscale_cam = cam(input_tensor=input_tensor, targets=targets)
---> 41 grayscale_cam = cam(input_tensor=input_tensor, targets=None)
     43 # In this example grayscale_cam has only one image in the batch:
     44 grayscale_cam = grayscale_cam[0, :]

File /opt/miniconda/miniconda3/envs/youth_eye_detection/lib/python3.8/site-packages/pytorch_grad_cam/base_cam.py:188, in BaseCAM.__call__(self, input_tensor, targets, aug_smooth, eigen_smooth)
    184 if aug_smooth is True:
    185     return self.forward_augmentation_smoothing(
    186         input_tensor, targets, eigen_smooth)
--> 188 return self.forward(input_tensor,
    189                     targets, eigen_smooth)

File /opt/miniconda/miniconda3/envs/youth_eye_detection/lib/python3.8/site-packages/pytorch_grad_cam/base_cam.py:84, in BaseCAM.forward(self, input_tensor, targets, eigen_smooth)
     81     self.model.zero_grad()
     82     loss = sum([target(output)
     83                for target, output in zip(targets, outputs)])
---> 84     loss.backward(retain_graph=True)
     86 # In most of the saliency attribution papers, the saliency is
     87 # computed with a single target layer.
     88 # Commonly it is the last convolutional layer.
   (...)
     93 # use all conv layers for example, all Batchnorm layers,
     94 # or something else.
     95 cam_per_layer = self.compute_cam_per_layer(input_tensor,
     96                                            targets,
     97                                            eigen_smooth)

File /opt/miniconda/miniconda3/envs/youth_eye_detection/lib/python3.8/site-packages/torch/_tensor.py:488, in Tensor.backward(self, gradient, retain_graph, create_graph, inputs)
    478 if has_torch_function_unary(self):
    479     return handle_torch_function(
    480         Tensor.backward,
    481         (self,),
   (...)
    486         inputs=inputs,
    487     )
--> 488 torch.autograd.backward(
    489     self, gradient, retain_graph, create_graph, inputs=inputs
    490 )

File /opt/miniconda/miniconda3/envs/youth_eye_detection/lib/python3.8/site-packages/torch/autograd/__init__.py:197, in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    192     retain_graph = create_graph
    194 # The reason we repeat same the comment below is that
    195 # some Python versions print out the first line of a multi-line function
    196 # calls in the traceback and some print out the last line
--> 197 Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    198     tensors, grad_tensors_, retain_graph, create_graph, inputs,
    199     allow_unreachable=True, accumulate_grad=True)

File /opt/miniconda/miniconda3/envs/youth_eye_detection/lib/python3.8/site-packages/torch/nn/modules/module.py:62, in _WrappedHook.__call__(self, *args, **kwargs)
     60     if module is None:
     61         raise RuntimeError("You are trying to call the hook of a dead Module!")
---> 62     return self.hook(module, *args, **kwargs)
     63 return self.hook(*args, **kwargs)

File /opt/miniconda/miniconda3/envs/youth_eye_detection/lib/python3.8/site-packages/flashtorch/saliency/backprop.py:217, in Backprop._register_conv_hook.<locals>._record_gradients(module, grad_in, grad_out)
    216 def _record_gradients(module, grad_in, grad_out):
--> 217     if self.gradients.shape == grad_in[0].shape:
    218         self.gradients = grad_in[0]

AttributeError: 'NoneType' object has no attribute 'shape'

After taking a closer look at the stacktrace, looks like another library ("flashtorch") crept in.  I am removing all references of flashtorch and retrying.  Please disregard and sorry for the noise!


Social — Change Sort Order on Relations View

The Relations view currently lists accounts alphabetically,  which can result in a lot of inactive accounts filling the top of the screen. This should be sorted like:

personas I follow
personas who follow me
follow date

This will simplify the mental parsing of the relations.

Done and deployed. Closing Ticket.


feat: Read current version for a NPM project

[X] Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
[X] Ensure that the pull request title represents the desired changelog entry
[X] Please describe what you did
[X] Link to relevant issues in GitHub or Jira
[X] Link to relevant pull requests, esp. upstream and downstream changes
[X] Ensure you have provided tests - that demonstrates feature works or fixes the issue

Fix #18
The version attribut is get in the package.json.

@ADI10HERO & @garethjevans ok to review again, there is quality failure but it's from io.jenkins.plugins.conventionalcommits.utils.MakeProjectType class after my merge from main :
HIGH:
Found reliance on default encoding in io.jenkins.plugins.conventionalcommits.utils.MakeProjectType.getCurrentVersion(File, ProcessHelper): new java.util.Scanner(File)

@philippart-s we've fixed this in another PR, and added a GitHub check to make sure that they don't happen again. A rebase should solve this which I can run from the GH UI.


Stand up blockchain program on web with heroku

I integrated the server with Heroku so that it solves issue #26. It can now be deployed on the web for free.

@hskang9 thanks for this, but I think we'll need to harden this:

Run it behind uWSGI / gunicorn
Turn off Debug Mode, by environment variable

Okay.

I added uWSGI and turned off debug mode.

Why are we using both gunicorn and uWSGI? We only need one I think 😀

I would use gunicorn then 😀

@bhaveshAn I added instruction for installation of python buildpack too

This is pretty decent @hskang9 thanks. Have you been able to test it out on Heroku?

I was confused with my project based on your source after testing. Now the source code and README is fixed.


Broken with Alternate Player for Twitch

Looks like #9 was resolved but seems it is broken again with Alternate Player for Twitch (https://addons.mozilla.org/en-US/firefox/addon/twitch_5/) on Firefox.
Firefox 120.0.1 (64-bit)
SoundFixer 1.4.0
Alternate Player 2<IP_ADDRESS>

It wasn't resolved, I think I've just closed that issue because I don't think there's anything I could do (if it doesn't work with some player, that's because the browser's security model prevents it from working). Sorry!


Infinite increase channels count on RabbitMq with many clients

Hello.
I discovered the bug.
Client use ClientRequest for request data from a server. Server receives request, creates SendEndpoint and creates channel on connection to RabbitMq for send response. Client receives response and it's Ok.
But each client have unique temporary queue for receiving responses, and server for each client creates new channel on connection. After closing the client and it's connection, channels from server to RabbitMq stay alive until close server connection.
This leads to an inifinite increase channels count on RabbitMq and memory leak on server (server stores in memory all these endpoints).
Expected behavior: endpoints with them's channels must be closed automatically, for example, by time interval without activity.
I tested it on https://github.com/MassTransit/Sample-RequestResponse with MT 3.5.7 and MT 5.1.2. Each start of Client (and request to server) initiates new channel from RequestService to RabbitMq.
I use RabbitMq v.3.7.5 and Erlang v.20.3.

It shouldn't be infinite, after 1 minute those channels should start to close and cleanup. The minimum lifetime for a cached send endpoint is 1 minutes, up to a maximum of 10,000 endpoints.
Those numbers might need to be adjusted, but that was the baseline chosen.
Also, why is your client creating a new response queue for every request? The bus endpoint should be used to handle the responses. It's significantly faster than connecting a new receive endpoint for every request.

It shouldn't be infinite, after 1 minute those channels should start to close and cleanup. The minimum lifetime for a cached send endpoint is 1 minutes, up to a maximum of 10,000 endpoints.

Yes, I looked at the code and saw the use GreenCache with parameters: capacity 10000, min time 1 min, max time 24h. And I decided to check it. I expected that channels should start to close after reaching a quantity limit (10000), but to my regret, this didn't happen. I was able to create 20000 channels and they are still alive.
I attach screenshots:
Connetion with many channels on RabbitMq (after idle more than 12h)
Snapshot in DotMemory of RequestService (after idle more than 12h)

Also, why is your client creating a new response queue for every request? The bus endpoint should be used to handle the responses. It's significantly faster than connecting a new receive endpoint for every request.

No, I don't create new response queue for every request, I use bus endpoint for each client. But each new client creates new bus endpoint for handle responses, and server creates new send endpoint for handle requests and send reponses. After closing all clients send endpoints on server still alive.

Are you creating a new bus per request? Again, you’re wasting system resources by not sharing a single bus and leaving it connected.
Yes, GreenCache will exceed the limit if the age of a cached item hasn’t passed the minimum. It’s by design.
I’d suggest you stop creating a new bus for each request. Create one and start it at process startup, then use it for all the requests.

I'm afraid that you misunderstood me.
No, I don't create a new bus per request. I use one bus per client-side application and leaving it connected. But my application is client-server application, I have one server-side application and many client-side applications for many people. Each client-side application can do requests to server. And as a result I have many channels on one connection at server-side, them still alive after closing all clients.

Yes, GreenCache will exceed the limit if the age of a cached item hasn’t passed the minimum. It’s by design.

Yes, I understand it. And I saw that in code minimum is a minute. But channels count has not changed in a minute, nor in 12 hours.

What can you say about this situation? It's easy to reproduce with Sample-RequestResponse. Our project already working in production but we met with this bug only after the update of RabbitMq (in latest versions of RabbitMq parameter max_channels by default is 2047, in previous versions it is 65535).
Can I hope that this bug will be fixed in the future?

I’d suggest you stop creating a new bus for each request. Create one and start it at process startup, then use it for all the requests.

I am also getting too many channels created. However, I am creating new bus for each request. In my scenario, I cannot create a single bus and share it in subsequent connections, since my code is injected to an external application, and I do not have control over it.
What should I do in such scenario?
Details are present at https://groups.google.com/forum/#!topic/masstransit-discuss/AhE3disvQ3k

I run to exactly same problem.

Hi, I have the same problem. How can I solve that? It's urgent, I'm in production environment

If it's urgent, you need to change your application so that it doesn't create a new temporary queue for every request - use the same bus instance for each request and start/stop it with the application lifecycle.

What can we do to help fix the issue?

@GreenKn1ght

If it's urgent, you need to change your application so that it doesn't create a new temporary queue for every request - use the same bus instance for each request and start/stop it with the application lifecycle.

We use single bus per entire client application and stop the bus at application closing.

@GreenKn1ght then how are you running out of channels?

I have no idea and thats the problem.
Let's start from scratch. As @ChipsetSV said, we able to reproduce this issue on https://github.com/MassTransit/Sample-RequestResponse.
If you run this example and start send requests to service, you'll see that number of channels in Channels tab was incresed. If you wait for one minute, number of connections will decrease to initial value even if bus was not stop. This is expected behavior and there is no issue.
But if you update MassTransit from 3.5.0 to 3.5.7 (same with 5.1.5) and repeat steps above, number of channels will not decrease after a while and even if you stop the bus. For now we can clear the channels only by closing server application.

I found the commit that brought this behavior: 789bffe234b460af986b906dbc5af34797521ee2

So I finally took the time to reproduce the issue (with 5.1.5) and have it fixed. But it requires an update to GreenPipes in addition to changes I have in my local repository for MassTransit, which means I need to update a few other projects as well. I may try to get to it this weekend. But I did resolve the issue to where channels are discarded after they are no longer being used.

Nice to hear it, thanks!

Great news, thanks!

thanks! in advance for ur help

@phatboyg, hello.
When we can see a new version with this bugfix?

By the end of the year it should be released. You can test with a develop version to see if it resolves your issue and help verify the fix.

That's great )

I tested this case in test app. It's ok, channels are closing.

I'm still seeing this on 5.2.3, unless what I'm seeing is different to the bug described. Each new client causes MT to create a new channel per response that never gets closed.
See gist reproducing. The new client requirement is simulated by creating a new bus per request. In reality though this is new GUI sessions querying the servers.
Playing around on develop and your 'Serious Transport Refactoring' commit seems to fix it.....
https://gist.github.com/thomastomanek/76f6e3c301cabc104c159aced49461cb

Well, 5.3.1 was just released yesterday and has all of the latest bits.
And realize, the channels don't release immediately, they are released after a minute when a new channel is requested. The soft limit is 1000, after which they should hover around that value as the channel usage shifts in response to new requests.

This is an issue in 5.5.3. I don't know if it has regressed or just not resolved. My issue is exactly like those described. I have a long running host with individual clients. Every app uses a single bus instance.
Each new client instance causes one more new channel on the host when a Request/Response happens. We are approaching 2000 channels, well above the suggested 1000 limit. The channels are not expiring.

When you're approaching 2000 channels, how old he is the old channel at that point? Because the lowest age, I believe, is 1 minute. Not positive, but pretty sure.

Days old. Each new channel is getting created by an app that runs every 10
minutes.
On Wed, Aug 7, 2019, 8:09 PM Chris Patterson<EMAIL_ADDRESS>wrote:

When you're approaching 2000 channels, how old he is the old channel at
that point? Because the lowest age, I believe, is 1 minute. Not positive,
but pretty sure.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/MassTransit/MassTransit/issues/1165?email_source=notifications&email_token=ABOGMU6IWPUGPYH663KEQSDQDNP3FA5CNFSM4FKS54DKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD32BKHI#issuecomment-519312669,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABOGMUZZGCTNNDCT4BQBAZ3QDNP3FANCNFSM4FKS54DA
.

https://gist.github.com/nick-randal/6baeb89d07854cc5866203c54975fc9d
I named the files to simulate what our production system is doing. Host is a long running process which uses a single bus instance.
Client is an application that starts up periodically, uses a single bus instance but does do a Request / Response. Each time the client runs a channel gets orphaned in the host connection. These channels are never expiring and are going past 1000. We actually incurred an outage because we hit an internal limit in rabbit of 2047 channels.

I made a comment on your Gist, I don't think it's MT.
Also, on the other issue, I posted a Gist that I used to try to reproduce the issue, without success.
https://gist.github.com/phatboyg/eff85c26d5da1f4f56b6f729847dd3ad

Yes, I see the comment but that doesn't explain the exact same behavior in production where there is not interaction with the console. The console interaction is only in Linqpad.

How is the code hosted in production?

Windows Service is the host, on service start it opens a connection to the bus and maintains that single connection.
We have different worker applications that start up, do some work and exit. Each worker for the life of the application maintains a single connection to the bus. All other Publish / Consume methods are working as expected. This is specifically related to Request / Response.
We see that at the first request / response cycle a fresh channel is created on the host to respond to the request. However, at the point of the client application closing and stopping its bus. The host channel that was used to send is not closing. At seems that the host send channel is unaware that its consumer has closed its connections so I believe RabbitMQ's channel.Close() is not begin called.
Only in Linqpad do we see the 1000 channel limit but to me it begs the question, why aren't the channels closed when the consumer is no longer there?
On our production server we are seeing the channel creep exceeding 1000 and going all the way to the limit of 2047, at which point clients start getting exceptions like

operation none caused a connection exception not_allowed: "number of channels opened (22) has reached the negotiated channel_max (22)
as noted in https://www.rabbitmq.com/channels.html#lifecycle.

The host channel for the send endpoint that is used to deliver the response to the client bus instance is a channel on the host connection. It has no correlation whatsoever to the client channel on the client connection. The connections and channels are between the broker and the host or client, the client and host are not in any way directly connected. So there isn't a way for the client closing the channel to signal the host that the channel for that send endpoint should be closed.
The channels are closed when cached send endpoints are removed from the cache, which happens after one minute -- if the cached endpoint limit (1000) is reached, If it's under the limit, those channels may not be closed.
If you're still seeing channels increase, your host is either running a seriously old version of MassTransit that may have had the bug, or something else is blocking the host from closing those cached send endpoints. Like maybe something in your system is still using that endpoint, or holding a reference to it.
For real, there is nothing I can do if I can't reproduce the issue. And so far, everything works as expected on Mac or Windows, .NET Core or .NET Framework, RabbitMQ 3.6 or 3.7.

As for the cache lifetime, it's set here:
https://github.com/MassTransit/MassTransit/blob/develop/src/MassTransit/Transports/SendEndpointCache.cs#L29
And it's actually 10 seconds now, instead of 60.
    static SendEndpointCacheDefaults()
        {
            Capacity = 1000;
            MinAge = TimeSpan.FromSeconds(10);
            MaxAge = TimeSpan.FromHours(24);
        }

We are using 5.5.4.
I understand that all connections are to Rabbit and not to each other.  The channels for request / response are not created until the first time that a Request / Response is started. Just as the channels are inflated from that initiation, my expectation would be along the lines of sending a, I'm leaving message so that the host can explicitly close that channel out instead of have the blind 1000 pool.
Couldn't this be an explicit call by choice to send a close out or just make client IDisposable and send a "Close it" command based on the type?
Otherwise the only way I can see to do this cleanly, is to go back to a basic RPC pattern with Publish on both ends.
Something like
https://gist.github.com/nick-randal/00eebecfc34d363a33fbaa85bd21ffcc
Send() seems like the preferred way to do this instead of having the Publish() fanout behavior, but I don't see a way to determine a URI at runtime for a connecting client to be able to call
GetSendEndpoint()

You really should figure out why your instance isn't behaving like it should, because honestly, every single combination of runtime and OS I've tried works as expected. Something in your code is preventing the endpoints from being cleaned up.

Since we have the same setup and I have tried both a linux and windows instance of rabbit, and various .net frameworks, it occurred to me that there is one distinct difference. Your example works for both you and us, however the timing is really the biggest difference. Blasting channels over a short period of time does hover at 1000. However, our request / response leak is happening once every ten minutes. So it takes days to reach that mark but then it doesn't stop at 1000. I think there may be an issue in the cache logic when the leaks are creeping along.

So how often is the host receiving a request from a new endpoint address?

It depends on the endpoint. At a minimum 1 request is made on a new endpoints startup but it might make more requests.
Based on your suggestion of using direct exchanges with routing keys, we have a working solution I am happy with. There are no channel leaks or accumulation. The only only has 1 channel per unique request / response type.
It is working but do you see anything that we might have done incorrectly or could have improved?
https://gist.github.com/nick-randal/0bf53d5659bd4a365aac18a042be2270

I'll look at it again shortly, but I think you can still use the request client on the client side, but publish from the server side with the routing key. I'll work up an example after a bit, once i get a chance to sit down and think about it.

FYI, with v6.2 (actually I think 6.1 maybe) there is no longer a channel increase with RabbitMQ using request/response. Responses are now sent on the same channel as the inbound message receiver.

I was having the same issue on v6.0 running on asp.net core 2.2 and not working as expected in terms of the number of tcpConn&channel. After upgrading it to latest version (v7.0.4) it disappeared.


[HUDI-6416] Completion markers for handling execution engine (spark) …

…retries
Change Logs
During spark stage retries, spark driver may have all the information to reconcile the commit and proceed with next steps, while a stray executor may still be writing to a data file and complete later (beyond reconcile step, before the JVM exit).
Extra files left on the dataset, excluded from reconcile commit step could show up as data quality issue for query engines with duplicate records.
This change brings completion markers which tries to prevent the dataset from experiencing data quality issues, in such corner case scenarios.
This change, would prevent the second/subsequent tasks/executors from creating additional files (with a different write token) and reuse the successfully completed files, by storing the serialized write status in the completion marker file.  During the second and subsequent attempts, write status stored in the completion marker is returned so that the same data file created by the first attempt can be reused.
Impact
Improved reliability, data quality due to infrastructure related failures, resulting in stage/task retries.
Risk level (write none, low medium or high below)
Low/Medium:  This change has been in production for about a year now at Uber.
Documentation Update
ENFORCE_COMPLETION_MARKER_CHECKS - Allows configuring whether to fail the job or continue with retries, when an already completed file is being retried.   With a planned change, this would allow the second/subsequent attempt to create a file to succeed using the previously created copy of data.
ENFORCE_FINALIZE_WRITE_CHECK - Allows configuring whether to fail the job if commit reconciliation step has been completed and the write stage is retried (say a block from writeStatus RDD is found to be lost, when iterating over write statues for record index update).   Single executor writing to multiple files (data spilling over to more than one file) and stage failure post reconciliation results in data quality issues.  This flag is helpful in failing the job, instead of creating incorrect commit.
Contributor's checklist

[x ] Read through contributor's guide
[x] Change Logs and Impact were stated clearly
[x] Adequate tests were added if applicable
[x] CI passed

Thanks for the contribution @nbalajee , In general I'm confused why we need two marker files for each base file, before the patch, we have in-progress marker file and write status real paths, we can diff out the corrupt/retry files by comparing the in-progress marker file handles and the paths recorded in writestatus.
And we also have some instant completion check in HoodieFileSystemView, to ignore the files/file blocks that are still pending, so why the reader view could read data sets that are not intented to be exposed?

Thanks for your review @dannyhchen and @nsivabalan for the review.

Thanks for the contribution @nbalajee , In general I'm confused why we need two marker files for each base file, before the patch, we have in-progress marker file and write status real paths, we can diff out the corrupt/retry files by comparing the in-progress marker file handles and the paths recorded in writestatus.
And we also have some instant completion check in HoodieFileSystemView, to ignore the files/file blocks that are still pending, so why the reader view could read data sets that are not intented to be exposed?

Following diagram summarizes the issue.
(a) when a batch of records given to an executor for writing, spills over to multiple data files (split into multiple parts due to file size limits, f1-0_w1_c1.parquet, f1-1_w1_c1.parquet etc)
(b) A spark stage is retried as a result all tasks are retried (some of the tasks from previous attempts could still be on-going).  Mainly happens with spark fetchfailed exception.

With respect to the partial files, when the feature flag is enabled, it will reduce the possibility of a partial data files getting created.  Since second and subsequent attempts, in the presence of of completed marker, would return success (with previously completed writeStatus) we are minimizing multiple tasks attempting to create files.
Deleting the .hoodie/.temp/  marker folder, after the finalizeWrite, ensures that any new task attempting to create a data file (that would show up as a partially written file, when the Job/JVM exits) will fail at the time of writeHandle creation, if the marker folder is absent (and not create a partial file).

@nbalajee is this can resolve the orphan file which product by spark speculation execution, which create maker file after commit submit


user-guide/get-started: reorganize "Understanding DVC" section

Files currently located in /static/docs/understanding-dvc/
Remove all the Understanding DVC docs and just rescue all that is unique and valuable into the get started and user guide docs.

Will you please give an example of this for better understanding? @jorgeorpinel

@ryokugyu for example:
How it works - should be reviewed, simplified. made actionable. move to the user guide
What is DVC - reviewed, moved to the user-guide
etc.

Can I take this up?

Hi @Naba7 thanks for the interest. Will you be dropping issue #463 after all?

@Naba7 I agree with @jorgeorpinel let's do issues one by one, or at least tow in parallel. Quality is important, not speed here.

Yes, I agree @jorgeorpinel @shcheklein . It will be better to focus on one issue rather than fumbling over many. Sure, I will work on #463 .

I am appending here my ideas about how to reorganize "Understanding DVC", but it is clear that those pages need to be heavily rewritten and updated (not just moved around or relocated on the menu).

User Guide

[ ] Introduction

Why DVC

Problems in DS/ML
https://dvc.org/doc/understanding-dvc/collaboration-issues

Existing Tools
https://dvc.org/doc/understanding-dvc/existing-tools
https://dvc.org/doc/understanding-dvc/related-technologies

What is DVC
https://dvc.org/doc/understanding-dvc/what-is-dvc

DVC Concepts
https://dvc.org/doc/understanding-dvc/what-is-dvc
https://github.com/iterative/dvc.org/issues/53#issuecomment-485588486

DVC Features
https://dvc.org/doc/understanding-dvc/core-features
https://dvc.org/doc/understanding-dvc/how-it-works

Other Resources
https://dvc.org/doc/understanding-dvc/resources
Maybe some of the resources can be referenced from the other sections in
the Introduction, and this section be removed.

I am appending here my ideas about how to reorganize "Understanding DVC", but it is clear that those pages need to be heavily rewritten and updated (not just moved around or relocated on the menu).
* User Guide

  * [ ]  Introduction

    * Why DVC

      * Problems in DS/ML
        https://dvc.org/doc/understanding-dvc/collaboration-issues
      * Existing Tools
        https://dvc.org/doc/understanding-dvc/existing-tools
        https://dvc.org/doc/understanding-dvc/related-technologies
    * What is DVC
      https://dvc.org/doc/understanding-dvc/what-is-dvc

      * DVC Concepts
        https://dvc.org/doc/understanding-dvc/what-is-dvc
        [#53 (comment)](https://github.com/iterative/dvc.org/issues/53#issuecomment-485588486)
      * DVC Features
        https://dvc.org/doc/understanding-dvc/core-features
        https://dvc.org/doc/understanding-dvc/how-it-works
    * Other Resources
      https://dvc.org/doc/understanding-dvc/resources
      Maybe some of the resources can be referenced from the other sections in
      the Introduction, and this section be removed.

This structure looks good :+1:  I was wondering whether we can move The Privacy Policy structure as discussed in #987 and #144(comment) under one of these sections?

Actually that proposal while great when it was done may be outdated right now. The goal now is to reduce the amount of pages and absorb the unique information from Understanding DVC pages into existing User Guide pages as much as possible.

I was wondering whether we can move The Privacy Policy structure as discussed in #987 (comment) and #144(comment) under one of these sections?

[ ] Good finds! (possibly removing GAPP from footer and from nav sidebar, which is now possible BTW). However, this is all unrelated to this issue.

Also, @srishti-nema as you're already assigned to #540 please focus on that for now.

I am appending here my ideas about how to reorganize "Understanding DVC", but it is clear that those pages need to be heavily rewritten and updated (not just moved around or relocated on the menu).
User Guide
Introduction
Why DVC
Problems in DS/ML
https://dvc.org/doc/understanding-dvc/collaboration-issues
Existing Tools
https://dvc.org/doc/understanding-dvc/existing-tools
https://dvc.org/doc/understanding-dvc/related-technologies
What is DVC
https://dvc.org/doc/understanding-dvc/what-is-dvc
DVC Concepts
https://dvc.org/doc/understanding-dvc/what-is-dvc
#53 (comment)
DVC Features
https://dvc.org/doc/understanding-dvc/core-features
https://dvc.org/doc/understanding-dvc/how-it-works
Other Resources
https://dvc.org/doc/understanding-dvc/resources
Maybe some of the resources can be referenced from the other sections in
the Introduction, and this section be removed.

I have created a similar indexed version of absorbing DVC. I have incorporated certain facts that make it seem more in place, developing from how ML evolved to asking the right questions now. @jorgeorpinel, @shcheklein, if it's okay, could I create a PR?

@VANRao-Stack sounds cool! could you outline the changes first?

@VANRao-Stack sounds cool! could you outline the changes first?

Yeah, definitely! I began with creating a two subsections within understanding the problem, where I began with issues in current ML practices, and then asking the right questions in order to resolve the same. I provided a couple of historical pointers to show how it was a similar thinking that lead to the growth of ml/dl in the first place.
NOTE: I did not change the questions posed, as I thought they were sufficient here.
I then created another section, containing two other sub-sections, with one explaining about the technologies currently in the market (comparison to existing technologies), then followed by a section describing what Experiment software Management meant.
NOTE: I'm still not sure about existing technologies and am having a little difficulty editing it. I don't think this section should be placed here, but instead towards the end right before core features or before Resources, I could instead here make a section on the technology today isn't good enough for development.
The next is a section on what DVC is and a couple of its key features, and as mentioned in #53, I noticed that there wasn't a keyword on DVC files and remote, so I added the same.
The next is core features, and as I mentioned in the previous note, I think we can move the exiting technologies here as clearly, a comparison with the existing technologies would be suitable here along with a couple of extra notes.
And finally how-it-works and resources, both of which I kept unedited, since a change I don't think was in order.

@VANRao-Stack while some of the things you mention seem to make sense, I don't think your message outlines the suggested changes.
Can you try to describe the changes you are proposing in a very clear way? A list, a table, etc. something visual and easy to grasp as opposed to a recollection of your thought process combined with specific actions. Please help us understand your plan here 🙂
Thanks!

@VANRao-Stack while some of the things you mention seem to make sense, I don't think your message outlines the suggested changes.
Can you try to describe the changes you are proposing in a very clear way? A list, a table, etc. something visual and easy to grasp as opposed to a recollection of your thought process combined with specific actions. Please help us understand your plan here 🙂
Thanks!

Yeah sure!

Started with collaboration issues, review it and place it in the beginning after giving it a little more description with respect to the context.

Move on to asking the right question, similarly review it and add or delete content with respect to the context.

Next section, show how the current technologies aren't enough

After a brief description of what an experiment management software is, formally define what dvc is, add the rest of keywords taken from #53 here.

Another round of comparison between the existing technologies and dvc, and briefly explain its core features.

How it works section was asked to be reviewed and made more actionable. I do not understand this, so I have left it as it is for now.

The rest of the user guide remains pretty much the same, with an extra section on Resources at the very end.

I hope this clearly outlines my idea, if this isn't what was planned please let me know about it, any suggestions or opinions are also welcome.

Thanks @VANRao-Stack. I still don't see a specific plan of what parts of Understanding DVC will go where in other existing doc sections. We'd like to see such a "map" as an initial plan. You also seem to have the intention to improve some of the content in question but that's not part of this issue, just deciding which parts to preserve, moving them, and changing their wording as needed to fit the new context...
How about this, just focus on one page (for example https://dvc.org/doc/understanding-dvc/collaboration-issues) and try to detail where you would move all that content into other sections of the docs, of which parts to just remove and why. You can submit a PR directly if that's easiest for you, or just post another comment with the plan here.

Thanks @VANRao-Stack. I still don't see a specific plan of what parts of Understanding DVC will go where in other existing doc sections. We'd like to see such a "map" as an initial plan. You also seem to have the intention to improve some of the content in question, which isn't bad, but that's not within the scope of this task, just deciding which parts to preserve, moving them, and changing their wording as needed to fit the new context...
How about this, just focus on one page (for example https://dvc.org/doc/understanding-dvc/collaboration-issues) and try to detail where you would move all that content into other sections of the docs, of which parts to just remove and why. You can submit a PR directly if that's easiest for you, or just post another comment with the plan here.

Oh, I had actually used the plan of where everything would be moved according to a previous comment by @dashohoxha  that has been marked as resolved right now. I will create a PR so you can better see what I intended.

I see. Yeah, sorry that plan is really old. I mentioned we needed a new one in https://github.com/iterative/dvc.org/issues/425#issuecomment-631716646. Please focus on a single page of Understanding DVC for now and lmk when your plan for that content is ready to check. Thanks

I see. Yeah, sorry that plan is really old. I mentioned we needed a new strategy in #425 (comment) above. Again, please focus on a single page of Understanding DVC for now and lmk when your plan for that content is ready to check. Thanks

Alright, since I am only supposed to work with one page, I propose we move related technologies to the end of the use cases section. After a couple of articles on workflows and other such scenarios, comparing it to related technologies would directly show how DVC brings something new to the table.
@jorgeorpinel , @shcheklein What do you think?

@VANRao-Stack I think it might be confusing to be honest. My take on this that it should become a part of the User Guide in some way. But the first question to answer - what value do expect/why do we have this section in the first place?

@VANRao-Stack I think it might be confusing to be honest. My take on this that it should become a part of the User Guide in some way. But the first question to answer - what value do expect/why do we have this section in the first place?

Well the answer to the question you posed about why we have this section in the first place is, I think, with a first glance DVC kind of seems just like another tool like Git, but however it isn't, and this section shows how DVC differs from the existing technologies . The whole point of having a use case section was to make people familiar with the different cases in which DVC could be deployed, hence having a comparison between the existing technologies and DVC would help make this point firm in the readers' minds.

move related technologies to the end of the use cases section. After a couple of articles on workflows and other such scenarios...

@VANRao-Stack Use Cases already talk about the benefits of DVC so I'm not sure about your reason either. Besides, this issue specifies the goal is to absorb Understanding DVC into Get Started or (most likely) User Guide.

what value do expect/why do we have this section in the first place?

This is a tricky question actually. The reason for Understanding DVC as a whole was probably originally something that is no longer a priority because it's covered by other sections like Get Started (or the glossary/ proposed Basic Concepts) better.
The Related Technologies page seems to me like it had the intention to attract users from those tools who find them limited when dealing with data, or/and to answer FAQ's like "isn't this just like Git-LFS?". Another message sent by that page, as Abhijith mentioned, is a summary of the features that DVC covers, which currently would require a bunch of other tools to put together. So perhaps this one page could be moved into a new User Guide indeed, titled something like "Migrating from Related Tools" or absorbed into the proposed How-to section (see #899 epic).

I think this discussions reveals that at least parts of this issue are not very introductory so I removed the good-first-issue label. Maybe you should try another one @VANRao-Stack... sorry for the back and forth. If you can find something much more defined, with a specific description that is crystal clear as to what should be done, that would be best for now I think.

OR if you want a small subtask that helps advance this issue, how about this:

[ ] Go over each page and delete content that is 100% redundant with the other existing sections.

For example the first bullet in https://dvc.org/doc/understanding-dvc/how-it-works is pretty much covered in a few places: https://dvc.org/doc/tutorials/get-started/initialize and https://dvc.org/doc/command-reference/init
That should be a well-scoped and relatively easy PR to review.

OR if you want a small subtask that helps advance this issue, how about this:
Go over each page and delete content that is 100% redundant with the other existing sections.
For example the first bullet in https://dvc.org/doc/understanding-dvc/how-it-works is pretty much covered in a few places: https://dvc.org/doc/tutorials/get-started/initialize and https://dvc.org/doc/command-reference/init
That should be a well-scoped and relatively easy PR to review.

Sounds good! I'll start working on it!

So I was working on the Core Features page, as you had suggested. Clearly most of the points mentioned is redundant:

DVC works on top of Git repositories and has a similar command line interface and Git workflow.

DVC is Programming language agnostic: Python, R, Julia, shell scripts, etc. as well as ML library agnostic: Keras, Tensorflow, PyTorch, Scipy, etc.

DVC supports cloud storage (Amazon S3, Microsoft Azure Blob Storage, Google Cloud Storage, etc.) for data sources and pre-trained model sharing.

These points have been mentioned several times in the docs before that, the only new point that I'm seeing is probably, DVC is open source and self serve. What should I do here? @jorgeorpinel

These points have been mentioned several times in the docs before that

Maybe you're onto something here, yes. But let's check on a case-by-case basis. Can you list where each of these points are mentioned in other docs? Also you only listed 3 out of the 6 bullets in that doc, let's check each one. You're probably right about some of them at least.
Core features are probably important to near the top of introductory docs, so hopefully these other mentions are not buried in very specific areas.

These points have been mentioned several times in the docs before that
Maybe you're onto something here, yes. But let's check on a case-by-case basis. Can you say where each of these points are mentioned in other docs? Also you only listed 3 out of the 6 bullets in that doc, let's check all. You're probably right about some of them at least.
Core features are probably important to near the top of introductory docs, so hopefully these other mentions are not buried in very specific areas.

I'll mention which bullet and the links where its has been previously mentioned:

The second bullet about reproducibility is covered very clearly in the deep dive tutorial : https://dvc.org/doc/tutorials/deep/reproducibility#reproducibility

Similarly, the large data file versioning is covered in https://dvc.org/doc/user-guide/large-dataset-optimization#large-dataset-optimization

Its was mentioned that DVC was agnostic here https://dvc.org/doc/tutorials/get-started/agenda#agenda

DVC's support for cloud storage is clearly explained here: https://dvc.org/doc/tutorials/get-started/configure#configure

Sorry for the delay @VANRao-Stack I'm pretty busy this week and weekend. Feel free to submit PRs meanwhile.

reproducibility (2) is covered very clearly in the deep dive tutorial
large data file versioning (3) is covered in ...#large-dataset-optimization

Those are now introductory docs, except for Get Started. Again: "Core features are important to have near the top of introductory docs, so hopefully these other mentions are not buried in very specific areas." (from https://github.com/iterative/dvc.org/issues/425#issuecomment-634390312)
But maybe this concept is also repeated in a more introductory doc?

DVC was agnostic (4) here .../get-started/agenda#agenda

Yes! But I think this note should be added directly to the docs home, https://dvc.org/doc, instead.

DVC's support for cloud storage (6) is clearly explained here: .../get-started/configure#configure

Agree about this one.
You're still missing points 1 and 5...


Add a "clear" button for the subset selection

Fix #245

Coverage increased (+0.004%) to 99.602% when pulling 4dca475746a595e08e91421ef9c56446263a1184 on clear-selection into 8ef057c5f74127bcf12d4de49932f2e0ec48aaa6 on master.


Implement verification of smart-contracts

Epic: #200
Implement verification of smart-contracts
Need to implement smart-contracts bytecode verification via source code

Sequence diagram:


[LLRP] Discovery command on RRFID gateway gives error with status code 202

🐞 Bug Report
Affected Services [REQUIRED]

 The issue is located in:  RFID device service
Is this a regression?

 Not sure
Description and Minimal Reproduction [REQUIRED]
Discovery command on RFID gateway UI gives below error:
'Request failed. Status code of 202'
Repro steps:

Setup RFID gateway hardware as per documentation

Deploy the RFID device and app-inventory service with latest V3 code.

Go to http://<IP_ADDRESS>:59711/ RFID gateway UI
a)  Right click on webpage, click on 'inspect' to open browser console
b) If CORS is not enabled on consul (by default CORS is disabled), notice that it gives CORS related errors on console but no
errors on UI.
c) Enable CORS on consul and set corsallowed origin on consul. Find attached consul screesnhtos.
d) Restart rfid-device service
e) Refresh RFID gateway UI
f) Open browser console
g) Click on 'Discovery'
Actual behavior:  It gives request failed error with status code of 202 on UI and also on browser console.
Expected behavior: The UI should not complain about the 202 status code
In the debug logs it shows 'Discover triggered' message correctly,
Find attached debug logs, UI screenshots before and after enabling CORS.

_edgex-device-rfid-llrp_logs (2).txt

🔥 Exception or Error

🌍  Your Environment
Deployment Environment:
EdgeX Version [REQUIRED]:
Anything else relevant?

HTTP 202 means Accepted and is a success response. Likely we just are checking for 200 OK only, and instead should check for 2xx or 200 and 202.


Problem when using crane with bevy

I have a problem by using crane with bevy. If I run my quickstart template with bevy using nix run I get following error:
thread 'main' panicked at 'Failed to create root directory "/nix/store/xc03jppghc27vffpvi4w303026x4ilv9-pong-rs-0.1.0/bin/assets" for file asset reader: Os { code: 30, kind: ReadOnlyFilesystem, message: "Read-only file system" }', /nix/store/eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0/bevy_asset-0.12.0/src/io/file/mod.rs:48:13
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

The bevy executable tries to create the /assets folder in the install directory.
Sorry if this a kind of noob question, however how can I deal with this.

Also I got this problem:
/nix/store/irzb5aj5ifp5r921kjfm8pi6mspwvhmh-pong-rs-0.1.0/bin/pong-rs: error while loading shared libraries: libudev.so.1: cannot open shared object file: No such file or directory
I can work around this while manually setting LD_LIBARY_PATH with the path that pkg-config returns, however it still doesn't work like it should

Hi @LDprg! When Nix builds a derivation it will put the results in /nix/store/... which is readonly meaning you will not be able to create or update any files there after installation.
It's easy to create that directory as part of the build, you'd need to add another phase which can create it. One example is to use postInstall:
craneLib.buildPackage {
  # other args omitted...

  postInstall = ''
    mkdir -p $out/assets
  '';
}

For LD_LIBRARY_PATH you can use something like wrapProgram or makeWrapper to create a wrapper script which updates LD_LIBRARY_PATH with the appropriate dependencies before executing the actual program

Hi @LDprg, I managed to get Bevy building with this Flake:
{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
    crane.url = "github:ipetkov/crane";
    crane.inputs.nixpkgs.follows = "nixpkgs";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, crane, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = nixpkgs.legacyPackages.${system};
        craneLib = crane.lib.${system};

        buildDeps = (with pkgs; [
          pkg-config
          makeWrapper
          clang
          mold
        ]);

        runtimeDeps = (with pkgs; [
          libxkbcommon
          alsa-lib
          udev
          vulkan-loader
          wayland
        ] ++ (with xorg; [
          libXcursor
          libXrandr
          libXi
          libX11
        ]));

        my-crate = craneLib.buildPackage rec {
          pname = "bevy-flake-template";
          src = ./.;

          nativeBuildInputs = buildDeps;
          buildInputs = runtimeDeps;

          postInstall = ''
            wrapProgram $out/bin/${pname} \
              --prefix LD_LIBRARY_PATH : ${pkgs.lib.makeLibraryPath runtimeDeps} \
              --prefix XCURSOR_THEME : "Adwaita"
            mkdir -p $out/bin/assets
            cp -a assets $out/bin
          '';
        };
      in
      {
        checks = {
          inherit my-crate;
        };

        packages.default = my-crate;

        devShells.default = craneLib.devShell {
          checks = self.checks.${system};

          RUST_SRC_PATH = "${pkgs.rustPlatform.rustLibSrc}";
          LD_LIBRARY_PATH = "${pkgs.lib.makeLibraryPath runtimeDeps}";
          XCURSOR_THEME = "Adwaita";
        };
      });
}

I have a full template on my Github here, hopefully this helps you or anyone else looking :slightly_smiling_face:.


Ollama is not using GPU to run model on Windows 11

What is the issue?
version: 0.1.31
gpu: nvidia qradro p2000 (5GB GPU memory)
os: win 11
cuda:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Nov__3_17:51:05_Pacific_Daylight_Time_2023
Cuda compilation tools, release 12.3, V12.3.103
Build cuda_12.3.r12.3/compiler.33492891_0
Command: ollama run tinydolphin

the 2nd column is CPU (20%), the 3rd one is GPU (0%)
OS
Windows
GPU
Nvidia
CPU
Intel
Ollama version
0.1.31

Hi @maxithub can you update to version 0.1.32 of Ollama ans tell us if you reproduce the same issue?

@igorschlum thank you very much for the swift response. I just upgraded to 0.1.32, and noticed there is a new process named ollama_llama_server created to run the model. Unfortunately, the problem still persists.

And i can't find any log in the %userprofile%/.ollama folder, like it was mentioned in other issues.

我也出现了同样的问题，它只占用我的cup算力，显卡无法使用

Ohh, finally got it working now after install the latest CUDA version cuda_12.4.1_551.78_windows.exe

Really love the simplicity offered by Ollama! One command and things just work! Thank you so much for the brilliant work!

谢谢，已通知富健，他会及时查看。


View Release Notes button links to wrong Github URL

Bug description
The UI dialog that opens when clicking to see info on the new version contains a "View release notes" button. This took me to https://github.com/podman-desktop/podman-desktop/releases/tag/vv1.14.1 which has an incorrect tag. It should be v1.14.1 and not vv1.14.1.

Operating system
mac OS 15.1.1
Installation Method
Installer from website/GitHub releases
Version
1.13.2
Steps to reproduce
No response
Relevant log output

Additional context
No response

Thanks for your report @sdemjanenko
It's a dup of https://github.com/podman-desktop/podman-desktop/issues/9447 fixed in 1.14.x


Update description of Azure IoT Toolkit

@formulahendry,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot


Soundcloud Not Working - 401 HTTP Error 401: Unauthorized

Hi,
All SoundCloud URLs are failing with 401 Unauthorized:
youtube-dl --verbose http://soundcloud.com/m28xxlover/nervo-live-at-electric-zoo-2015-new-york-05-sep-2015
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'http://soundcloud.com/m28xxlover/nervo-live-at-electric-zoo-2015-new-york-05-sep-2015']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2015.09.22
[debug] Python version 2.7.10 - Linux-3.16.0-4-amd64-x86_64-with-debian-stretch-sid
[debug] exe versions: ffmpeg N-45620-gcf16b45-, ffprobe N-45620-gcf16b45-
[debug] Proxy map: {}
[soundcloud] m28xxlover/nervo-live-at-electric-zoo-2015-new-york-05-sep-2015: Resolving id
[soundcloud] m28xxlover/nervo-live-at-electric-zoo-2015-new-york-05-sep-2015: Downloading info JSON
ERROR: Unable to download JSON metadata: HTTP Error 401: Unauthorized (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 326, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1865, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

The problem seems to be that all API calls with that client id will fail (such as the first one, to get the JSON from http://api.soundcloud.com/resolve.json?url=http://soundcloud.com/m28xxlover/nervo-live-at-electric-zoo-2015-new-york-05-sep-2015&client_id=b45b1aa10f1ac2941910a7f0d10f8e28) because the b45b1aa10f1ac2941910a7f0d10f8e28 client ID is exceeding the bandwidth or maybe it has simply been banned.
I've tried using client id 986b39d4513a5b501d57d973318715f0 (from https://github.com/soundcloud/soundcloud-javascript/issues/12) and the URL returns a valid response.

Thanks for debugging. It's already fixed in eb11cbe8674705647d6bd2947a44e08543663633.

Ouch, I deserve that for searching only the open issues and not bothering with the closed ones. Sorry!


tools/syz-runtest: switch to the generic program execution

syz-runtest effectively implemented the same execute program/return result
mechanism we use now for normal fuzzing.
So extend the general mechanism to allow collecting output/errors,
repeating program, and executing a precompiled binary as a test.
And switch syz-runtest to the general mechanism.
This removes another chunk of code from syz-fuzzer.

Is this PR meant to replace https://github.com/google/syzkaller/pull/4723?

Is this PR meant to replace #4723?

I don't know. It's a change on top that PR b/c it's all changing the same code.
We can either review/merge #4723 first, then I rebase this one.
Or, consider this a replacement (but I have more changes coming on top :)).
What do you prefer?

Rebased on top of master. This is now ready for review.

This removes another chunk of code from syz-fuzzer.

Not sure about removes, rather reshuffles the code within syz-fuzzer :) But it's very cool to get rid of a few more superfluous RPC calls.

Well, yes. The main removed chunk was the 3rd implementation of generic program runner we have in syz-fuzzer/testing.go.


Slider: make button target bigger, make the right/unselected half gray

Target is bigger, but does not have gray for unselected portion.


Mypy checking of Qcodes fails with hypotesis>=3.62

At this stage it's not clear if this is an issue with hypotesis (an upstream bug) or specific to the way we use mypy

Worked around in #1163

We should wait for https://github.com/HypothesisWorks/hypothesis/issues/1359 to be fixed before investigating further. That bug seems to at lease be part of the issue

This has been fixed in a later hypothesis release


Reland: "Add example and troubleshooting comment for showSnackBar"

Description
Reland https://github.com/flutter/flutter/pull/102073, which was reverted in https://github.com/flutter/flutter/pull/104843
Related issues
Fixes https://github.com/flutter/flutter/issues/84263

Hey @Piinks!
I have rebased the PR, it seems to be ok.

I appreciate your patience. I try to migrate customers for one change at a time to keep it sane and organized. :)

I really appreciate your reviews too! You do such a great job: working on your own PRs, reviewing relentlessly other contributors' PRs, addressing customers migration and also participating in many Flutter team great videos! :blue_heart:

Thanks very much! I am running against internal tests now.

Thanks for your patience! All of the customers are fixed. 🎉


Initial draft of PBJ types for token state

Fixes https://github.com/hashgraph/hedera-services/issues/5804
Fixes https://github.com/hashgraph/hedera-protobufs/issues/264
Adds initial draft of PBJ types Account, Token, Token-relation for token-service implementation.
Currently same fields in MerkleAccount, MerkleToken and MerkleTokenRelStatus are used in the protobuf messages. They are subject to change in the future.

I'm not sure about where we should put things. Ideally I like moving it out of services just so we don't have confusion about states vs. HAPI/gRPC, but maybe right now we have no choice but to put it where you have it in this PR. I had a lot of comments and suggestions, but I don't want to hold up development for them right now, but they are questions we will need to have answered before we're ready for production.

Where to put things?
Do we need to encode shard and realm as well?
Should we reuse HAPI objects? (If we do reuse them, it makes it almost impossible to fork these objects if we need more state or something else later)
What should the java package be for generated code? And what module runs the compiler to build it?

As per the discussion

We will retain them under services/state
We will need to add shard and realm to the objects. Need to check with platform team before adding them. So will leave for now on this PR.
We will reuse HAPI objects. But the PBJ compiler will keep generated objects under different directories.
Decided on pbj.java_package =  "com.hedera.hapi.node.state.token" and java_package = "com.hederahashgraph.api.proto.java.state"


Add async variants of all commands

Passthrough functions for all commands that calls the ELF variant with .get().

@swift-server-bot please test

@swift-server-bot add to allowlist

@0xTim thanks! I'm seeing errors in CI seemingly unrelated to this PR -

error: redundant conformance of 'Gauge' to protocol 'Sendable' in Swift 5.7 🤔
some Swift 6 errors on nightly

Any ideas what those might be? Can we drop 5.7 support?

@joshuawright11 yeah I think we can drop 5.7 since NIO doesn't support it any more.
Nightly errors can potentially be ignored as well

@0xTim this is embarrassing but I'm not super familiar with Jenkins - it seems like configs aren't actually in this repo and someone with access to https://ci.swiftserver.group/ would need to disable the 5.6?
It's also throwing the same error as nightly on the api breakage pipeline - not sure what the solution is there.

@fabianfett got it - thanks for the reply, my bad for jumping on this PR instead of waiting for a response to my issue!
Glad to hear this repo is still maintained and I look forward to a fully async await version someday 😊


🌐(backend) add missing translation in contract template

Purpose
We forgot to wrap some strings into a trans tag
Proposal

[x] Add missing translated string into contract definition template

Should we use translation everywhere ?

Do you mean should we use translate over trans everywhere ? Otherwise I do not understand what you mean.

Should we use translation everywhere ?

Do you mean should we use translate over trans everywhere ? Otherwise I do not understand what you mean.

Oh, yes, sorry. According to django docs trans has been replaced with translate since version 3.1

Should we use translation everywhere ?

Do you mean should we use translate over trans everywhere ? Otherwise I do not understand what you mean.

Oh, yes, sorry. According to django docs trans has been replaced with translate since version 3.1

Oh ok. So Let's use translate


[Suggestion] Why no mention of Cadence prettier tool?

I was searching if there are any existing code formatting tools for Cadence, so I figured this repo is the one where such a tool would probably live.
That wasn't the case, so after some more searching I found that the current prettier tool lives here atm: https://github.com/turbolent/prettier
Should we at least mention or link to that repo within this repo's README file to make it easier to discover?
But maybe that tool isn't "production ready" yet and there is still more work to be done before making it available to Cadence users (based on https://github.com/onflow/cadence/issues/209).

Sorry for the confusion here: Though the code formatting functionality is being developed in the Cadence codebase / repository, there's no tool for it yet that a developer could use to format their Cadence code.
https://github.com/turbolent/prettier is a Go library for formatting, it is not a tool, it is only a building block that is used in Cadence to implement the code formatting functionality.
The closest to a tool we have to try out the code formatting functionality is the internal https://github.com/onflow/cadence/tree/master/tools/pretty. As the README says, for now it is just there to assist contributors working on the implementation of the Cadence code formatter, it is not meant for Cadence developers. However, once the code formatting functionality is complete, we can take this internal tool, move it into this repo, and turn it into something a typical Cadence developer could use, and finally integrate it into the CLI and other tools like the language server and VS Code extension.


[pilot] mark Hash options with is_string:false

In #16294 we discovered that PILOT_BETA_APP_REVIEW_INFO cannot be set via the pilot distribute cli. The options are listed as STRING in pilot distribute --help incorrectly.

I am surprised this had the desired effect. AFAIK, is_string is deprecated and should be avoided. @joshdholtz does this ring a bell?

Ah, so is_stringis starting to slowly get removed from all tools and I believe its removed in pilot already. The beta_app_review_info is a Hash because it supports localized keys so I don’t remember exactly how or if it works with CLI.
It might be a String via CLI because CLI can’t really natively support a Hash. I think it might handle the input as a JSON object thought?
I could be wrong though and that might need to be added. 😬

Totally did not realize this was a PR and thought this was an issue 🙃 I think the actual solution there should not be setting is_string: false but by changing the logic in the fastlane option / CLI generation logic to handle Hash properly 🤔

Thanks for explanation. Closing as this is not the right approach.


Update partialuser.py

some partial user objects returned by roblox dont have hasVerifiedBadge.

would be nice if you could specify which ones

would be nice if you could specify which ones

The only one I am aware of is the one returned by https://users.roblox.com/v1/users/authenticated


id_token kid value is not compliant with OpenID Connect RFC

Description:
The id_token header must contain a kid claim with an ID to be validated against the jwks URL. The ID Token kid claim generated, contains an additional "_RS256" at the end which is conflicting with the response from the jwks URL, therefore the client application can't validate the token properly.
Suggested Labels:
Suggested Assignees:
Affected Product Version:
5.7
OS, DB, other environment details and versions:
any
Steps to reproduce:
generate an ID token
Related Issues:

We too were able to observe a similar issue with some IS 5.7.0 wum level. In those wum levels, there is a mismatch between the kid value obtained from JWKS endpoint and the JWT token.
We already identified this issue and fixed this in the latest Identity server  5.7.0 WUM update. If you take a wum update in your environment, you can resolve this issue.
We did an improvement[1] related to the kid value in a wum update.  Before this improvement, when signing with the same key but using different algorithms like RS256, RS348 and RS512 we had the generated the same kid for all the keysets. Hence in order to differentiate them, we had appended the algorithm to the kid and generated as a new kid. So after this fix[1], the kid will have the algorithm appended in the kid value.
Eg: if RS256 is used, then  kid value will be "ZmE4OGM5ZWEyNmZlNjg2ZGUwY2M5OGMyMDhlNjA0MDQ4YTZkOTExMQ_RS256"
With the latest wum updated pack, you will get the 'kid' value in the JWT token and JWKS endpoint will have the algorithm appended to it. Please update your environment also with by taking a 5.7.0 wum update.
This fix is available with the IS5.1.0-beta2 also.
[1]https://github.com/wso2/product-is/issues/5244

I will try it tomorrow and let you know.

Hi Team, I have downloaded updates but the problem persists:
JWKS
{"keys":[{"kty":"RSA","e":"AQAB","use":"sig","kid":"NTAxZmMxNDMyZDg3MTU1ZGM0MzEzODJhZWI4NDNlZDU1OGFkNjFiMQ","alg":"RS256","n":"luZFdW1ynitztkWLC6xKegbRWxky-5P0p4ShYEOkHs30QI2VCuR6Qo4Bz5rTgLBrky03W1GAVrZxuvKRGj9V9-PmjdGtau4CTXu9pLLcqnruaczoSdvBYA3lS9a7zgFU0-s6kMl2EhB-rk7gXluEep7lIOenzfl2f6IoTKa2fVgVd3YKiSGsyL4tztS70vmmX121qm0sTJdKWP4HxXyqK9neolXI9fYyHOYILVNZ69z_73OOVhkh_mvTmWZLM7GM6sApmyLX6OXUp8z0pkY-vT_9-zRxxQs7GurC4_C1nK3rI_0ySUgGEafO1atNjYmlFN-M3tZX6nEcA6g94IavyQ"},{"kty":"RSA","e":"AQAB","use":"sig","kid":"NTAxZmMxNDMyZDg3MTU1ZGM0MzEzODJhZWI4NDNlZDU1OGFkNjFiMQ_RS256","alg":"RS256","n":"luZFdW1ynitztkWLC6xKegbRWxky-5P0p4ShYEOkHs30QI2VCuR6Qo4Bz5rTgLBrky03W1GAVrZxuvKRGj9V9-PmjdGtau4CTXu9pLLcqnruaczoSdvBYA3lS9a7zgFU0-s6kMl2EhB-rk7gXluEep7lIOenzfl2f6IoTKa2fVgVd3YKiSGsyL4tztS70vmmX121qm0sTJdKWP4HxXyqK9neolXI9fYyHOYILVNZ69z_73OOVhkh_mvTmWZLM7GM6sApmyLX6OXUp8z0pkY-vT_9-zRxxQs7GurC4_C1nK3rI_0ySUgGEafO1atNjYmlFN-M3tZX6nEcA6g94IavyQ"}]}
ID TOKEN - HEADER
{
"x5t": "NTAxZmMxNDMyZDg3MTU1ZGM0MzEzODJhZWI4NDNlZDU1OGFkNjFiMQ",
"kid": "NTAxZmMxNDMyZDg3MTU1ZGM0MzEzODJhZWI4NDNlZDU1OGFkNjFiMQ_RS256",
"alg": "RS256"
}
MacBook-Joao:summary joaoemilio$ grep 5244 *
Binary file update-summary-wso2is-km-5.7.0+1548762622360.full.pdf matches
Binary file update-summary-wso2is-km-5.7.0+1562045447740.full.pdf matches
ID TOKEN
eyJ4NXQiOiJOVEF4Wm1NeE5ETXlaRGczTVRVMVpHTTBNekV6T0RKaFpXSTRORE5sWkRVMU9HRmtOakZpTVEiLCJraWQiOiJOVEF4Wm1NeE5ETXlaRGczTVRVMVpHTTBNekV6T0RKaFpXSTRORE5sWkRVMU9HRmtOakZpTVFfUlMyNTYiLCJhbGciOiJSUzI1NiJ9.eyJhdF9oYXNoIjoiUXRZUnhMdDRHTk9jV0hSLTN0VGxtUSIsImF1ZCI6IjRoZmRNR1BPbHFrMDFQQ0NKelAxOG8ydGE2SWEiLCJzdWIiOiJhZG1pbiIsIm5iZiI6MTU4MTk1NDIwNSwiYXpwIjoiNGhmZE1HUE9scWswMVBDQ0p6UDE4bzJ0YTZJYSIsImFtciI6WyJwYXNzd29yZCJdLCJpc3MiOiJodHRwczpcL1wvbG9jYWxob3N0Ojk0NDNcL29hdXRoMlwvdG9rZW4iLCJleHAiOjE1ODE5NTc4MDUsImlhdCI6MTU4MTk1NDIwNX0.hFm0UCmj478GInjOAnU4moB6adqI86-NPUT64LPyAwykjNX8Yg-jWJDzP8Tl9WSrkJLPw_pLu4Skjp33geJzSfdGT5mZtIbxikX2mhbJi2uw7r81McoFkAB443Uv6bed_FgnXdGhgHIyvbG8ebjjgqV1dknwoEniq0yEeJ9rKAQ_P-DUaIiM74rklmV9Cyhhj9qLpNjFxmKNtjIo3f4uPixSXCYnNV7ol1pavqgJAx6jLgQPzDaZdNvfVUTj8zO24XlawP67VW7b5YIMin-ctN4xOcMjIS4-PnfmfitsLPjRQZogzTRa7LINMu6zTV-4GlQq3G-RhPhInpW1vozbng

Hi jao,
In the JWKS endpoint, you pointed above have the key set with the kid that comes in your ID token.
{
"keys":[
{
"kty":"RSA",
"e":"AQAB",
"use":"sig",
"kid":"NTAxZmMxNDMyZDg3MTU1ZGM0MzEzODJhZWI4NDNlZDU1OGFkNjFiMQ",
"alg":"RS256",
"n":"luZFdW1ynitztkWLC6xKegbRWxky-5P0p4ShYEOkHs30QI2VCuR6Qo4Bz5rTgLBrky03W1GAVrZxuvKRGj9V9-PmjdGtau4CTXu9pLLcqnruaczoSdvBYA3lS9a7zgFU0-s6kMl2EhB-rk7gXluEep7lIOenzfl2f6IoTKa2fVgVd3YKiSGsyL4tztS70vmmX121qm0sTJdKWP4HxXyqK9neolXI9fYyHOYILVNZ69z_73OOVhkh_mvTmWZLM7GM6sApmyLX6OXUp8z0pkY-vT_9-zRxxQs7GurC4_C1nK3rI_0ySUgGEafO1atNjYmlFN-M3tZX6nEcA6g94IavyQ"
},
{
"kty":"RSA",
"e":"AQAB",
"use":"sig",
"kid":"NTAxZmMxNDMyZDg3MTU1ZGM0MzEzODJhZWI4NDNlZDU1OGFkNjFiMQ_RS256",
"alg":"RS256",
"n":"luZFdW1ynitztkWLC6xKegbRWxky-5P0p4ShYEOkHs30QI2VCuR6Qo4Bz5rTgLBrky03W1GAVrZxuvKRGj9V9-PmjdGtau4CTXu9pLLcqnruaczoSdvBYA3lS9a7zgFU0-s6kMl2EhB-rk7gXluEep7lIOenzfl2f6IoTKa2fVgVd3YKiSGsyL4tztS70vmmX121qm0sTJdKWP4HxXyqK9neolXI9fYyHOYILVNZ69z_73OOVhkh_mvTmWZLM7GM6sApmyLX6OXUp8z0pkY-vT_9-zRxxQs7GurC4_C1nK3rI_0ySUgGEafO1atNjYmlFN-M3tZX6nEcA6g94IavyQ"
}
]
}
Normally JWKS endpoint can have multiple keysets and have to identify the suitable keyset with the KID. As long as the spec there is no restriction about how to create the KID, the only thing is it has to be unique.

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


publish<EMAIL_ADDRESS>

Requested by: @adinauer
Merge target: 6.4.x
Quick links:

View changes
View check runs

Assign the accepted label to this issue to approve the release.
Leave the #retract comment under this issue to retract the release (original issuer only).
Targets

[x] symbol-collector
[x] maven
[x] github
[x] registry

Targets marked with a checkbox have already been executed.  Administrators can manually tick a checkbox to force craft to skip it.

I'm going to reset 6.4.x so that this merges cleanly.


WIP: Search integration

Depends on PR #18 (first commit)
Depends on inveniosoftware/invenio-search#39
Depends on inveniosoftware/invenio-pidstore#4
Depends on inveniosoftware/invenio-rest#16

@lnielsen this is the PR for search integration

@egabancho @drjova This is the PR for invenio search REST API

@jirikuncar the access control commits are coming from PR #18.


Rewrite settings

This PR rewrites settings store to use Easy Peasy a reactive store library that uses a battle tested library immer-js under the hood.
The library does a lot of the heavy lifting resulting in cleaner and significantly less code.
Updating state is a breeze you don't need ugly reducer and spread operators everywhere.
Previous approach was also writing this to a store.bin file which is not the recommended way for a reactive store.
easy-peasy uses the built-in browser LocalStorage which is the recommended way and more secure since it's sandboxed as opposed to storing secrets in a plain file in home folder.
This effectively fixes all race conditions and other bugs.
/closes #955
/claim #955

@louis030195
Could you review this again?
It’s fully backwards compatible now
I attached demo on discord

is screenpipe-js still working?
like pipe.settings features etc
why is this better?
how do i know it will not cause the same issue when two components change the settings in the same time?
tbh this seems like you just asked chatgpt "make the settings better" and applied

This is better because it doesn't relies on hacky useEffects and useStates to achieve true reactivity
Using a global state management library is the industry practice.
There are lot of libraries but easy-peasy is the simplest.
I've personally used this library for multiple projects.
One of my project AssetAlert also uses it https://assetalert.io/
Library ensures clean mutations to global store and immer-js removes all the dirty spread operators everywhere so no more unintended side effects and race conditions.

ill try later


Versioning: checks added

This PR is a suggestion of versioning explanations addins.
Check if the version is higher than the development version: mention some incompatibilities.
If the launching script is not found (Ans.Dpf.Grpc.bat), launch it "manually" (setting paths) (for 2021 R1).

Hi @anslpa! Is this PR still active?


fix(region): avoid clean all rules when sync secgroups

这个 PR 实现什么功能/修复什么问题:

[ ] 功能、bugfix描述
[ ] 冒烟测试
[ ] 单元测试编写

是否需要 backport 到之前的 release 分支:

release/3.4

/area region
/cc @zexi

Codecov Report

Merging #9973 (dc49aff) into master (54823a6) will increase coverage by 9.94%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master    #9973       +/-   ##
===========================================
+ Coverage    5.97%   15.91%    +9.94%
===========================================
  Files         849      124      -725
  Lines      132219    12840   -119379
===========================================
- Hits         7896     2044     -5852
+ Misses     123621    10605   -113016
+ Partials      702      191      -511

Flag
Coverage Δ

aFlag
15.91% <ø> (+9.94%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/cloudcommon/db/tenantcache.go

pkg/compute/models/vpcs_mapped_address.go

pkg/multicloud/ucloud/ufile.go

pkg/compute/models/cloudproviderregions.go

pkg/multicloud/esxi/mobase.go

pkg/monitor/alerting/scheduler.go

pkg/compute/models/storagecaches.go

pkg/mcclient/options/k8s/role.go

pkg/cloudcommon/db/lockman/bug.go

pkg/compute/models/usage.go

... and 709 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 54823a6...dc49aff. Read the comment docs.

Codecov Report

Merging #9973 (dc49aff) into master (54823a6) will increase coverage by 9.94%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           master    #9973       +/-   ##
===========================================
+ Coverage    5.97%   15.91%    +9.94%
===========================================
  Files         849      124      -725
  Lines      132219    12840   -119379
===========================================
- Hits         7896     2044     -5852
+ Misses     123621    10605   -113016
+ Partials      702      191      -511

Flag
Coverage Δ

aFlag
15.91% <ø> (+9.94%)
:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

pkg/cloudcommon/db/tenantcache.go

pkg/compute/models/vpcs_mapped_address.go

pkg/multicloud/ucloud/ufile.go

pkg/compute/models/cloudproviderregions.go

pkg/multicloud/esxi/mobase.go

pkg/monitor/alerting/scheduler.go

pkg/compute/models/storagecaches.go

pkg/mcclient/options/k8s/role.go

pkg/cloudcommon/db/lockman/bug.go

pkg/compute/models/usage.go

... and 709 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 54823a6...dc49aff. Read the comment docs.


If you open Communities & Collections directly,  the result list is empty.

If you open Communities & Collections directly (go to https://dspace7-demo.atmire.com/community-list), the result list is empty.
Originally posted by @saschaszott in https://github.com/DSpace/dspace-angular/issues/899#issuecomment-711028738
May or may not be related to #899

I expect this will be due to the zone.runOutsideAngular call in datasource. SSR will think the page is done rendering as soon as that call occurs, and send the result to the client.
While runOutsideAngular can improve the responsiveness of the UI on the client, we may need to remove it here to prevent this issue.
I expect this and #899 together will take 3h to fix.
Atmire is interested in claiming these tasks

Sounds good @artlowel .  I've assigned this back to you for your team & moved to "in progress".  Also added #899 to the beta5 board and assigned to you as well.


Spotbugs Project

@gleclaire
Spotbugs is a community replacement of findbugs.  Information can be followed over on spotbugs.
A couple of months ago, there was some talk about what to do with findbugs maven plugin since spotbugs is drop in place replacement.  Since I already had a fork, the decision was made to use a branch there and get that released.  Everything is in RC status currently with spotbugs.  In order to be as compliant or close to findbugs maven plugin as possible I raised a few pull requests to apply additional work I was doing.  Two versions have been released for spotbugs at this point with second one ironing out some issues the updates I had been working for newer maven presented to findbugs plugins (in new PR today against this repo).  Now quite a number of projects are using spotbugs maven plugin or spotbugs variation (checkstyle, fbcontrib, findsecbugs).  The first official release of spotbugs-maven-plugin occurred on 6/29.  Already there are 200 downloads shown on central statistics which I suspect is greatly going to grow as word gets out.
Yesterday all my current pull requests were merged into findbugs maven plugin.  I have since rebased and sent another pull request while releasing another spotbugs maven plugin picking up other changes in recent week to this project.
Yesterday I was invited into the spotbugs project and asked to donate the code to them or keep versions sync'd.  So my thought initially was that I wasn't getting any real traction here and probably would completely flip the repo over to them but still keep it showing it's original roots forked from here.  At that point the master there would become all about spotbugs.  Now that my PRs are merged, I still think that is appropriate but before doing anything wanted to get some feedback if it makes sense to have two separate repos for the needs here where any issues on legacy at least for some time are addressed here at findbugs maven plugin then consumed over on spotbugs maven plugin, or if this project should just entirely flip over to spotbugs.
I did release spotbugs-maven-plugin today aligning with their version scheme.
Thoughts?

having them plugin as part of the big project is a good idea.

@gleclaire : you are welcome if you want to join SpotBugs project. Just drop a mail to me.

@gleclaire : may I ask you for help with https://github.com/spotbugs/spotbugs/issues/8? See last comment from @hazendaz. It would be great to have you on board in SpotBugs project. We are preparing first public SpotBugs release, and the maven part relly need some polishing.

@gleclaire My intentions at the moment are to provide continued support for the original findbugs maven plugin thus trying to get fixes and such up there.  It would be awesome if I could get committer access on this repo so I can speed that up a bit when I need to.  On the spotbugs project, there is intended to mostly address spotbugs new features.  By doing this I have the spotbugs-maven-plugin tied to the master from findbugs-maven-plugin and separate spotbugs branch for spotbugs needs.    Hope that helps to understand my motivations at the moment.

I think we can close this.

All good.  Thanks
Get Outlook for Androidhttps://aka.ms/ghei36

From: Garvin Leclaire<EMAIL_ADDRESS>Sent: Tuesday, October 24, 2017 12:03:28 PM
To: gleclaire/findbugs-maven-plugin
Cc: Jeremy Landis; Mention
Subject: Re: [gleclaire/findbugs-maven-plugin] Spotbugs Project (#67)
I think we can close this.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/gleclaire/findbugs-maven-plugin/issues/67#issuecomment-339042140, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AA7ho54N1V5pb2rK2TAzMs1qOHddPJD9ks5svgpOgaJpZM4Ond3X.


AMMFactory and MarketFactory belong together

Their love can no longer be denied.

In addresses.ts, put the AMMFactory under the MarketFactory so every market factory has one ammfactory.
Deploy one AMMFactory for each market factory
Make it so that AMMFactories only work with one MarketFactory.

These changes are needed for rewards and for a smoother upgrade procedure and for performance when there have been several upgrades.

This is a bigger architectural discussion that hopefully we can have in the future. For now the AMMFactory will keep a list of the blessed market factories (after rewards hits dev).


Fix a typo in distributed_tree_driver

retest this please

So, switching to gnupg.net produces this blob:
gpg: key A2C794A986419D8A: public key "Tom Stellard<EMAIL_ADDRESS>imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg: Signature made Wed Jan  9 00:40:22 2019 UTC
gpg:                using RSA key A2C794A986419D8A
gpg: Good signature from "Tom Stellard<EMAIL_ADDRESS>[expired]
gpg: Note: This key has expired!

So maybe there is just something wrong with the key itself.

And again, a ton of "new" warnings.

Lets rebase this PR on master to make it pass. @dalg24 If you want, we can move the gnupg server check out in a separate one, but I don't think this is necessary.

Done.

Retest this please.


Have precision of threshold be customizable

Currently, we hard code the precision of threshold as 5 in here. Having this customizable will allow us to use the linear tree model when the number that are used are smaller in general. My suggestion is have another parameter that defaults to 5 and when people wants to use the model with smaller number, they could do it by set this parameter to the number they desired.
Let me know if this change is good and I could create a PR for it. I'm open for discussion.

Hi,
the code reference you are pointing out is for "visualization" purposes. In other words, when you call the summary or model_to_dot methods the threshold values are rounded to 5 digits. This does not influence the training behavior as you see (here, here and here).
If u support the project, don't forget to leave a star ;-)

Hi @cerlymarco,
I'm trying to get the tree details (threshold, loss, children) and use it elsewhere. it seems that I shouldn't use self._nodes directly as it seems that you intend it to be a private member. How about adding another API to return the _nodes so that there's a way to get these information with complete precision?

@cerlymarco hello, is there any solution for this?

@dawidjaja
self._nodes is not a private method


hackgreenville-com-436 Update laravel and add composer dev command

hackgreenville-com-436 Update laravel and add composer dev command
Package Update

resolves #436

We cannot merged this as we run php 8.1 in prod, and this needs min php 8.2

I lazily updated my local asdf php to 8.3 and ran update. my local PHP 8.1 install was broken I resolved the issue last night. I'll re-run the update and re-create this PR with the proper PHP version used. Sorry about that @allella and @bogdankharchenko


Armature Import Issue, Wont work with Blend files or GLTF

Godot version
4.0 beta1
System information
Windows 10 Pro, GLES3 Intel HD Graphics, Blender 3.3
Issue description
Every time I Import from Blender to Godot no matter what format GLTF/.Blend/FBX the Skeleton3D is messed up. I tried importing animation without armature from Blender and it works fine. When I add armature I have a track for animation in Godot and all the nodes are there, but I can't play the Animation and the Skeleton is incorrect scale or I can't even see it for some reason.
I don't know if there's settings I need to change or if it's my system, but all file formats are producing the same result and I can't even get any rigged/animated models working because of it.
Steps to reproduce

Create a Blender3D Rig and Animation then export GLTF or just save to .blend file in the Godot project folder.
Open Godot and import Blender3D Rig and Animation.
Try to play it.

Minimal reproduction project
Import Test.zip

Pain points.

your blend file has a skeleton at the root. Godot has some trouble with this.
your skeleton has three roots
make the two roots be parented with the first bone
then, make sure the godot engine imports influences set to be 4 influences

Import Test.zip Here's an example.

Here is a video of the project I uploaded.
https://user-images.githubusercontent.com/32321/204073281-b6ab54c9-3f11-419d-9376-f3f714897278.mp4

Thanks for your help @fire . Unfortunately, I tried with another very simple Model and still  get the same thing. I used a simple cube and used Armeture. Mad an Empty the scene root, but still no animation and  the bones are overlapping each other.

It seemed to have worked on your system can you try to post your project again and I'll try to open it in my system. I don't know if it's OpenGL or something else. But I made one root bone and then also set to 4 influences, changed the scene root to something besides a skeleton.

Import Test Renamed.zip
Try this one

I think my case is related. 4.0beta6 blender 3.3.1
blender

godot

surprisingly everything works well, animations, deformations and positions...
but the skeleton and the meshes are rotated 90º

@barrrettt What is your up and forward axis set to in Blender?

everything works fine. look the bone attackment:

it may be an error when the gizmos cube is render:

Thinking maybe this is not the right place for my issue. if you need more information, please ask. i can't think of anything else that would be helpful. Maybe just skeletons and meshes have different functions for the change of base?

@barrrettt Yes, likely open a new issue, though it's almost certainly user error on your end (as evidenced by the screenshots, Blender uses Z as up and Godot uses Y as up, thus explaining the 90 deg rotation).
You should have an axis setting when exporting in Blender, try fixing it to conform to the Godot convention

I say this because I have the blender defaults

This is literally the core of your problem. Blender defaults differ from Godot convention

In that case I would suggest to put some kind of warning or option in the blender file importer.

I cannot reproduce this behavior. it must be a local problem. good

@fire Send me a video of you fixing the problem because as it stands for me i can see the skeleton but it still wont animate or pose. I used all the settings you mentioned and something is still wrong. The FPSARMS.tscn you sent says it has corrupted and won't work. Is anyone else running into this issue. I don't get why it's so hard for me to implement one of the key aspects of game animation. Please someone fix this. I can't even use Godot 4 at this point(just been playing with the animation settings since downloaded it). I've been in 3.5 because I am stuck. I don't know if it's OpenGL or what.

Thank you
Hopefully someone can tell me whats going on

For anyone wondering here's a simple animation and armature same problem optimize is even off
https://user-images.githubusercontent.com/14103171/206635703-9aa9ffd8-98a7-4196-b3b0-8a4828a43f99.mp4

I just got Godot  4 beta 8 and it works!!! Thank you Everyone


Question: Concat Protocol

Hi there,
Had a question. Is there a filter for using the Concat Protocol rather than the Concat Demuxer (which is what I think the current ConcatFilter class does)?
Looking at Concat Protocol here: https://trac.ffmpeg.org/wiki/Concatenate

Not support and i not recommend use CP (concat protocol)
You can add as multi inputs and concat in filters.

If you still want CP, try this
new VideoFileInput("concat:file1|file2")

oh i was write it here https://github.com/tqk2811/FFmpegArgs/blob/2d8245b1b9d26c59ee9896cfcf2bfe0279d0f834/FFmpegArgs.Inputs/ImageFilesConcatInput.cs#L15

oh i was write it here, you can custom yourself for video
https://github.com/tqk2811/FFmpegArgs/blob/2d8245b1b9d26c59ee9896cfcf2bfe0279d0f834/FFmpegArgs.Inputs/ImageFilesConcatInput.cs#L15

Does the Concat Protocol allow us to do this? From https://trac.ffmpeg.org/wiki/Concatenate
" using intermediate files
If you have MP4 files, these could be loss lessly concatenated by first transcoding them to MPEG-2 transport streams. With H.264 video and AAC audio, the following can be used:
ffmpeg -i input1.mp4 -c copy -bsf:v h264_mp4toannexb -f mpegts intermediate1.ts
ffmpeg -i input2.mp4 -c copy -bsf:v h264_mp4toannexb -f mpegts intermediate2.ts
ffmpeg -i "concat:intermediate1.ts|intermediate2.ts" -c copy -bsf:a aac_adtstoasc output.mp4 "

Yes, but I don't recommend it.

Yes, but I don't recommend it.

Alright. But just for our own sake, if we had to do it, how would we go about doing it with the ConcatProtocol method?

Write same as
https://github.com/tqk2811/FFmpegArgs/blob/2d8245b1b9d26c59ee9896cfcf2bfe0279d0f834/FFmpegArgs.Inputs/ImageFilesConcatInput.cs#L15

VideoFilesConcatInput.ConcatProtocol( filesList );

VideoFilesConcatInput.ConcatProtocol( filesList );

Awesome. This method uses temporary streams for the inputs and then concatenates those?

VideoFilesConcatInput.ConcatProtocol( filesList );

Awesome. This method uses temporary streams for the inputs and then concatenates those?

It same as -i "concat:intermediate1.ts|intermediate2.ts"


server configuration fails to make directory

Describe the bug

When doing a server configuration it fails because a directory can't be made?
To Reproduce
Steps to reproduce the behavior:

Fresh install
daytona server configure
Accept defaults
See error

Expected behavior
I expected it would create the directories. It didn't I checked and also they were not there before.
Desktop (please complete the following information):

OS: Mac 14.4.5
Daytona Version: v0.16

More context: The issue is caused by https://github.com/daytonaio/daytona/blob/main/pkg/views/server/configure.go#L153 which attempts to create a directory on the host machine while the configuration could be edited for a remote server. Mkdir should be executed on the server and not on the CLI.

/bounty $20

/attempt #618

/attempt #618