[Bug]: REST client doesn't store header, parameter and matrix parameter fields for new transforms

Apache Hop version?
SNAPSHOT-20241108
Java version?
openjdk version "17.0.11" 2024-04-16 LTS
Operating system
macOS
What happened?
header, parameter and matrix parameter fields are picked up correctly for existing transforms, but aren't stored for new transforms.
Issue Priority
Priority: 2
Issue Component
Component: Transforms

.take-issue


[Serve] Introducing GPU sampler for CUDA

This PR introduces the GPU sampler for CUDA only. The GPU sampler makes use of the GPU sampling ops introduced in apache/tvm#16575.
We will follow up to benchmark the performance of the GPU sampler over CPU sampler.

Marked as draft since it depends on apache/tvm#16703.

Special thanks for @yongwww for the great contribution on the sampling operators, which work smooooothly!


feat: bump php dependencies

Lock file operations: 0 installs, 55 updates, 0 removals
  - Upgrading badges/poser (v2.0.4 => v2.1.0)
  - Upgrading composer/xdebug-handler (2.0.1 => 2.0.2)
  - Upgrading doctrine/annotations (1.13.1 => 1.13.2)
  - Upgrading friendsofphp/php-cs-fixer (v3.0.0 => v3.1.0)
  - Upgrading http-interop/http-factory-guzzle (1.0.0 => 1.2.0)
  - Upgrading monolog/monolog (2.2.0 => 2.3.2)
  - Upgrading php-http/client-common (2.3.0 => 2.4.0)
  - Upgrading php-http/discovery (1.13.0 => 1.14.0)
  - Upgrading php-http/message (1.11.1 => 1.12.0)
  - Upgrading phpstan/phpstan (0.12.89 => 0.12.96)
  - Upgrading phpstan/phpstan-phpunit (0.12.19 => 0.12.22)
  - Upgrading phpstan/phpstan-symfony (0.12.36 => 0.12.42)
  - Upgrading sentry/sentry (3.3.0 => 3.3.2)
  - Upgrading sentry/sentry-symfony (4.1.3 => 4.2.1)
  - Upgrading symfony/asset (v5.3.0 => v5.3.4)
  - Upgrading symfony/browser-kit (v5.3.0 => v5.3.4)
  - Upgrading symfony/cache (v5.3.0 => v5.3.7)
  - Upgrading symfony/config (v5.3.0 => v5.3.4)
  - Upgrading symfony/console (v5.3.0 => v5.3.7)
  - Upgrading symfony/debug-bundle (v5.3.0 => v5.3.4)
  - Upgrading symfony/debug-pack (v1.0.9 => v1.0.10)
  - Upgrading symfony/dependency-injection (v5.3.0 => v5.3.7)
  - Upgrading symfony/dom-crawler (v5.3.0 => v5.3.7)
  - Upgrading symfony/dotenv (v5.3.0 => v5.3.7)
  - Upgrading symfony/error-handler (v5.3.0 => v5.3.7)
  - Upgrading symfony/event-dispatcher (v5.3.0 => v5.3.7)
  - Upgrading symfony/filesystem (v5.3.0 => v5.3.4)
  - Upgrading symfony/finder (v5.3.0 => v5.3.7)
  - Upgrading symfony/flex (v1.13.3 => v1.15.1)
  - Upgrading symfony/framework-bundle (v5.3.0 => v5.3.7)
  - Upgrading symfony/http-client (v5.3.0 => v5.3.7)
  - Upgrading symfony/http-foundation (v5.3.0 => v5.3.7)
  - Upgrading symfony/http-kernel (v5.3.0 => v5.3.7)
  - Upgrading symfony/monolog-bridge (v5.3.0 => v5.3.7)
  - Upgrading symfony/options-resolver (v5.3.0 => v5.3.7)
  - Upgrading symfony/password-hasher (v5.3.0 => v5.3.7)
  - Upgrading symfony/phpunit-bridge (v5.3.0 => v5.3.7)
  - Upgrading symfony/polyfill-intl-grapheme (v1.23.0 => v1.23.1)
  - Upgrading symfony/polyfill-mbstring (v1.23.0 => v1.23.1)
  - Upgrading symfony/polyfill-php80 (v1.23.0 => v1.23.1)
  - Upgrading symfony/process (v5.3.0 => v5.3.7)
  - Upgrading symfony/profiler-pack (v1.0.5 => v1.0.6)
  - Upgrading symfony/psr-http-message-bridge (v2.1.0 => v2.1.1)
  - Upgrading symfony/routing (v5.3.0 => v5.3.7)
  - Upgrading symfony/runtime (v5.3.0 => v5.3.4)
  - Upgrading symfony/security-core (v5.3.0 => v5.3.7)
  - Upgrading symfony/stopwatch (v5.3.0 => v5.3.4)
  - Upgrading symfony/string (v5.3.0 => v5.3.7)
  - Upgrading symfony/twig-bridge (v5.3.0 => v5.3.7)
  - Upgrading symfony/twig-bundle (v5.3.0 => v5.3.4)
  - Upgrading symfony/var-dumper (v5.3.0 => v5.3.7)
  - Upgrading symfony/var-exporter (v5.3.0 => v5.3.7)
  - Upgrading symfony/web-profiler-bundle (v5.3.0 => v5.3.5)
  - Upgrading symfony/webpack-encore-bundle (v1.11.2 => v1.12.0)
  - Upgrading symfony/yaml (v5.3.0 => v5.3.6)

I see that many vendors changed minor versions in composer.lock
I think it would be nice to raise constraints in composer.json, too.
For example: we're upgrading badge/poser from v2.0.4 to v2.1.0, we should change constraint from ^2.0 to ^2.1.

done!


Azure Service Bus: dial tcp xx.xxx.xx.xx:5671: i/o timeout occuring

Bug Report

import path of package in question github.com/Azure/azure-sdk-for-go/sdk/messaging/azservicebus
SDK version 1.4.0
go version: go version go1.19.7 darwin/amd64

What happened?
I have the code below, it is continually timing out when I attempt to ReceiveMessages
client, err := creds.ServiceBus(namespaceName)
	if err != nil {
		ch <- halt(
			cctx,
			logger,
			err,
			"connectivity test",
			`An error occured when connecting to Service Bus namespace endpoint.
			 Azure Service Bus implements these endpoints as unique subdomains.`,
		)
		return
	}
	receiver, err := client.NewReceiverForQueue(
		queueName,
		&azservicebus.ReceiverOptions{ReceiveMode: azservicebus.ReceiveModePeekLock},
	)

	if err != nil {
		ch <- halt(
			cctx,
			logger,
			err,
			"connectivity test",
			`Received an error when attempting to configure the Azure Service Bus client.`,
		)
		return
	}
	logger.Info("servicebus_queue_receiving")
	var receiveErr error = nil
	for {
		innerCtx, cancel := context.WithTimeout(cctx, 60*time.Second)
		messages, err := receiver.ReceiveMessages(innerCtx, archaius.GetInt(cfg.AzureEdhPrefetchMsgs, 50), nil)
		// this is where the timeout occurs
		if err != nil {
			if err.Error() != "context deadline exceeded" {
				logger.Error(err.Error())
				receiveErr = err
				cancel()
				break
			} else {
				logger.Debug("timeout exceeded with no messages, polling again.")
			}
		}

These are the logs I am seeing:
    {
            "msg": "servicebus_queue_receiving",
    },
    {
        "@message": "[azsb.Auth] (queue-name) refreshing claim",
    },
    {
        "@message": "[azsb.Conn] Creating new client, current rev: 0",
    },
    {
        "@message": "[azsb.Conn] Links closing (permanent: false)",
    },
    {
        "@message": "[azsb.Receiver] (receiveMessages.getlinks) Retry attempt 0 was cancelled, stopping: dial tcp xx.xxx.xx.xx:5671: i/o timeout",
    },
    {
        "@message": "[azsb.Receiver] Stopped message releaser with ID 'empty'",
    },
    {
        "@message": "[azsb.Conn] Links closing (permanent: true)",
    }

This only occurs once I attempt to retrieve messages, I can seemingly instantiate the client without error's being returned but I get this dial timeout that causes the context to expire resulting in nothing being returned. Is there an error being suppressed when I create the client that I can't see? It's strange to me that I can create a client but not use it.
What causes this timeout and is someone able to explain to me how Azure connects to your service bus via the sdk as I have confusion around the networking there?
Thanks

Great questions and points to me needing to write some extra comment text.
So, to the first question:

Is there an error being suppressed when I create the client that I can't see? It's strange to me that I can create a client but not use it.

The oddness you're seeing here is because clients in the Azure SDK are lazily-initialized - the underlying TCP connection isn't actually created until you perform an operation (any function that takes a context.Context).
So no error is being suppressed, but we only got an error when we tried to actually create the TCP connection.

What causes this timeout and is someone able to explain to me how Azure connects to your service bus via the sdk as I have confusion around the networking there?

That error is coming back from the underlying Go TCP library, which is just opening a TCP connection (on port 5671) to your Service Bus instance. There's actually an HTTP service on the Service Bus host (used for administration) which you can just open in a browser if you want to check if you can reach it via HTTP, at least, just to see if network connectivity works. For example, https://<your-sb-instance>.servicebus.windows.net.
Sometimes the port trips people up because it's not allowed through their firewall. This doc describes that a bit more: AMQP ports.
Now, if you're not allowed to connect out using 5671 you can use websockets which will tunnel you out via TLS on port 443. We have an example here: using websockets and proxies.


UI: add constant mapping capability

Provide UI support for mapping a constant value to a target model field.  Required capabilities:

Define a new constant
Edit/remove a constant
View defined constants
Assign a constant to a target field

+1

It seems like we could approach this a few different ways.  For one, it's not clear to me whether it's necessary to really maintain a set of constants.  It obviously depends on the real use cases, but it seems like most of the time I've seen the need to provide constants in a mapping, the constants only apply to the one field being populated; other fields that require constants will often require different values.  Would it be better for the user to just click on a target field in the target model and press a button to indicate a constant mapping should be created, followed by some type of dialog or dynamically displayed field to enter that constant?  If not, then there is also a scope issue to address.  Bearing all that in mind, here are the four options as I see them I'd like to get feedback on:

Do not maintain a set of constants, rather allow the user to enter the constant each time a constant mapping is created.
Maintain a set of constants that are scoped to each Dozer config
Maintain a set of constants that are scoped to each project
Maintain a set of constants that are scoped to the workspace

All but the first option will require the constants to be persisted within the project explicitly within a visible file or implicitly via its state location.  Thoughts?  Other options?

I feel we can start with option 1 and build to option 2.  A few of the reasons why I think option 2 is where we want to go long term:

Even with option 1, we will need to extend Dozer configuration to have the concept of a literal value for the "from" side of the mapping.  So we are storing the config somewhere anyway.
Having the idea of a constant pool in the mapper is a bit more flexible for mappings as the value can be changed in one place if it's used in multiple locations.  This is particularly important if a constant is referenced in a custom mapping as those could be tough to track down.
IMO, being able to define a constant and drag that to the target preserves our general mapping flow whereas starting on the target side is a bit of a change-up.

Starting with option 1 seems like a nice first step as it's not a huge investment of time and it will definitely satisfy the requirement.  I think we should keep option 2 in the picture though, even if it gets bumped to a future milestone.

Transformations I have seen to date have included many constants that get reused a lot.  They tend to be static, but it's critical that the values used in each place are identical.  The risk with having the user enter values each time is the user making a mistake.  Now, there should be tests they write to catch that but who knows.

BTW, I think there could be a smooth transition/interplay between options 1 and 2.  Once we make the constant pool visible in the editor, the user would still have the choice of:
a) Going the option 1 route and right-clicking on a target node.  At this point, they would be prompted for the constant name and value.
b) Adding the ability to add a constant and then view the constant pool which could serve as a source for a mapping (e.g. drag constant to target field).


Relabel: adult respiratory distress syndrome

Mondo term (ID and Label):
adult respiratory distress syndrome
Suggested new label:
Make Acute respiratory distress syndrome the primary label and make adult respiratory distress syndrome a narrow synonym.
Optional: Any additional information (like supporting evidence, PubMed ID, etc.)
https://www.ebi.ac.uk/ols/ontologies/ncit/terms?iri=http%3A%2F%2Fpurl.obolibrary.org%2Fobo%2FNCIT_C3353
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6709677/
@pnrobinson @mellybelly @DNAhotline @jmcmurry  - please let me know if you have any objections. I will try to get this change in for the April release (hopefully within the next week).
Related to #1341

@ShahimEssaid and I discussed this.
It appears that the respiratory distress classes in Mondo should be reviewed.

I am not going to take action on this before the April release.

I think this is straightforward, the adult form should be a subclass; add childhood as a sib?
the ordo equiv should go with the adult subclass:
https://www.ebi.ac.uk/ols/ontologies/ordo/terms?iri=http://www.orpha.net/ORDO/Orphanet_70578


Junk slice written to zarr with append_dim option

I often write to zarr using the append_dim option. But I find that  sometimes I get a junk slice in the first index of the zarr file along the dimenstion I am appending along. This seems to be related to the way coordinates are assigned along this dimension. Here's a minimal working example:
import xarray as xr
import numpy as np

target='test_zarr.zarr'
for i in range(0,10):
    test_slice = xr.DataArray(np.random.rand(10,20),dims=['j','k'])
    test_slice = test_slice.assign_coords({"time": i*10}) #this line is necessary for this example to fail
    test_slice = test_slice.expand_dims('time')
    test_ds = test_slice.to_dataset(name='u')
    test_ds.to_zarr(target, append_dim='time')

When I open the zarr file:
test_opened = xr.open_zarr('test_zarr.zarr')
test_opened.time

gives
array([140624802912264,               0,              10,              20,
                    30,              40,              50,              60,
                    70,              80,              90])

Environment:
Output of xr.show_versions()
INSTALLED VERSIONS
commit: None
python: 3.9.4 | packaged by conda-forge | (default, May 10 2021, 22:13:33)
[GCC 9.3.0]
python-bits: 64
OS: Linux
OS-release: 3.10.0-1160.11.1.el7.centos.plus.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: None
libnetcdf: None
xarray: 0.18.2
pandas: 1.2.4
numpy: 1.20.3
scipy: 1.6.3
netCDF4: None
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: 2.8.1
cftime: None
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2021.06.0
distributed: 2021.06.0
matplotlib: 3.4.2
cartopy: None
seaborn: None
numbagg: None
pint: None
setuptools: 49.6.0.post20210108
pip: 21.1.2
conda: None
pytest: None
IPython: 7.24.1
sphinx: None

Thanks for the report! This looks like a similar issue to https://github.com/pydata/xarray/issues/5511


invalid indent corrected

Description
Fix for invalid helm chart.
Changes proposed in this pull request:

Proper indent for annotations in helm template for trial cleaner job

Related issue(s)

/test pre-control-plane-components-kyma-environment-broker


ML 8.9 : Object Classification (V)

Description
📌  Issues for Week 8
Welcome to 'ML' Team, good to see you here
:arrow_forward: This issue will helps readers in acquiring all the knowledge that one needs to know about ----
Object Classification.
:red_circle: To get assigned to this issue, add your Batch Numbers mentioned in the spreadsheet of "Machine Learning", the approach one would follow and choice you prefer (Documentation, Audio, Video). You can go with all three or any number of options you're interested to work on.
✔ Domain : Machine Learning
:red_circle::yellow_circle: Points to Note :

The issues will be assigned on a first come first serve basis, 1 Issue == 1 PR.

"Issue Title" and "PR Title should be the same. Include issue number along with it.

Changes should be made inside the Machine Learning/Machine_Learning/ML_for_Seqential_Data_&_Visual_Data Branch.

Follow Contributing Guidelines & Code of Conduct before start Contributing.

This issue is only for 'GWOC' contributors of 'Machine Learning' domain.

:white_check_mark: To be Mentioned while taking the issue :

Full name
Batch Number
GitHub Profile Link
Which type of Contribution you want to make :

[ ] Video

Happy Contributing 🚀
All the best. Enjoy your open source journey ahead. 😎
Domain
Machine Learning
Type of Contribution
Video
Code of Conduct

[X] I follow Contributing Guidelines & Code of conduct of this project.

I am M. Ashish Reddy, I would like to work on this issue.
Batch Number: 2
Serial Number: 260
GitHub Profile Link: https://github.com/ashish-reddy-20-08
Approach: A detailed explanation with examples.
@abhisheks008

Assigned to @ashish-reddy-20-08


[Bug]: Angular CLI v11.2.12 freezes Dev Container creation

Feature id and version
ghcr.io/devcontainers-contrib/features/angular-cli:2
Devcontainer base Image
mcr.microsoft.com/devcontainers/typescript-node:14-bullseye
What happened?
I added ghcr.io/devcontainers-contrib/features/angular-cli:2 feature with "version": "11.2.12", but the extension freezes when trying to build the Dev Container because there is an interactive question regarding sharing anonymous usage data with Angular Team (see log output).
If I remove the feature and put npm i -g<EMAIL_ADDRESS>in the postCreateCommand, the same message shows up but I can interact with it and proceed with container creation. The workaround is to add export NG_CLI_ANALYTICS=ci before the npm install command (source).
I have "cli": { "analytics": false } on my angular.json, however, I am using the dev container in a parent folder with two angular projects on subfolders, so there is no angular.json file in the root folder of the dev container.
Relevant log output
? Would you like to share anonymous usage data with the Angular Team at Google under
Google’s Privacy Policy at https://policies.google.com/privacy? For more details and
how to change this setting, see https://angular.io/analytics. (y/N)

This project is not associated with the devcontainer project and appears to have been abandoned https://github.com/devcontainers-contrib/features/issues/628 . The official project has removed all devcontainer-contrib features from their listing. https://github.com/devcontainers/devcontainers.github.io/issues/451. There is a fork that is being worked on to update and maintain the features https://github.com/devcontainers-extra/features


[BUG]Unable to delete RKE2 cluster from Rancher UI when the underlying Harvester cluster is inaccessible/deleted

Describe the bug
If a Rancher RKE2 cluster is created via Rancher node driver, there's no way to delete it from Rancher UI after the imported Harvester cluster is de-registered and no longer accessible. Deletion stuck at "Waiting for machine [Harvester VM name] to delete".
To Reproduce
Steps to reproduce the behavior:

Import Harvester cluster into Rancher.
Create a new RKE2 cluster via Rancher node driver.
Delete the Harvester cluster from Rancher.
Now try to delete the RKE2 cluster after Harvester cluster is deleted and no longer accessible.

Expected behavior
There should be a way to easily cleanup the RKE2 cluster from Rancher UI, even after the Harvester cluster is no longer accessible.
Support bundle

Environment:

Harvester ISO version: latest master
Underlying Infrastructure (e.g. Baremetal with Dell PowerEdge R630): N/A

Additional context
Add any other context about the problem here.

I can reproduce this issue, but we cannot fix this issue from Harvester side if Rancher already can't access Harvester cluster. I think we should separate this issue into two cases:

Harvester is inaccesible: this may be caused by network issue, so we can do best effort to delete RKE2 cluster. Also, we can provide documents to manually delete VMs in Harvester and CR objects in Rancher.
Harvester is deleted: we should have a webhook to prevent users delete management.cattle.io/v3/cluster CR in Rancher if there is a RKE2 cluster running on Harvester.

For different cases, I can get different results.

Harvester is shutdown and inaccessible: we will get error message in deleting machine job if we try to remove RKE2 cluster.

Harvester is up but its CR is deleted in Rancher: we can remove RKE2 cluster in Rancher, but VMs will still remain in Harvester.

I think we need to separate this issue into 3 cases:

Harvester is inaccessible (network issues, harvester shutdown): in this case, we will get errors in removing RKE2 machine jobs like the following. I think it is fine. We can just do best effort to remove the RKE2 cluster because we can't connect to the Harvester cluster.

Cloud Credential is deleted: in this case, we will get error messages in Rancher. Also, all providers have this issue, we need a validation webhook to prevent it.

2022/01/27 03:15:33 [ERROR] error syncing 'fleet-default/rke2-in-aws-pool1-7c09ec84-8jv8n': handler machine-provision-remove: secrets "cc-k4t8n" not found, requeuing

Harvester CR is deleted in Rancher: Harvester is a special provider because it is also a cluster in Rancher. If we create an RKE2 cluster on Harvester, it is like a cluster in Rancher relying on another cluster in Rancher. From Harvester side, although this is a special case in Rancher, I would prefer to have a validation webhook for it. Also, even it's fine to delete Harvester CR in Rancher, we still have a running cluster-agent deployment in Harvester. Not sure whether we also need to clean this in Rancher handler function.

cc @guangbochen @gitlawr WDYT? Thank you!

Tested in Rancher v2.6-1d2b7468851553f4b89f8af99d74305af9d1b594-head on docker instance and Harvester master-9ecb4f9d-head in single node KVM environment.
Verification Steps

Imported Harvester into Rancher
Created cloud credentials
Created RKE2 cluster
Waited for RKE2 to provision
Deleted Harvester cluster
Deleted RKE2 cluster


Use Ubuntu Precise for Travis unit test jobs

Travis CI seems to have updated its default base image to use Ubuntu Trusty today, the Druid tests are failing with what might be memory usage errors:
https://docs.travis-ci.com/user/common-build-problems/#My-build-script-is-killed-without-any-error
This PR reverts the two unit test Travis jobs to use Ubuntu Precise.
The integration test still runs with Trusty (required for docker support).

This PR should unblock the Travis CI failures occurring today.
I think we could merge this to unblock PR acceptance for now and get the tests working with Travis's Trusty image in a later PR.

👍


An error occurred while processing your request in AWS Elastic Beanstalk

I deployed the default example template of Angular2 to an AWS Elastic Beanstalk server. However, loading the site only shows an error without no further details.

Where can I look for further logs (AWS gave no error logs, event viewer does not show error logs)?

Ill just explain how i got my app on my EC2 instance and hopefully it will help you out.
Since asp.net core is cross platform, go ahead and set up a Linux, I recommend the ubuntu server, EC2 instance. You can find lots of stuff online on how to do this if you are unfamiliar with how to do this( this video on youtube is basically what we will be doing but using asp.net core instead of node.js Node.js on AWS) .
Once you have your instance running, go ahead and install all the packages that you need to run an asp.net core app, Angular2 and whatever you are going to use to push your project to the server (git, ssh, ftp, etc).

npm
webpack
mono
dotnet

Finally once your instance is now able to run your project, push your project to the instance, build it and run it.   Things like rerouteing the IP tables, running your project in the background, ect. If you have any questions or problems, let me know.
I had an issue similar to this and @SteveSanderson  helped me fix it.
#299

@AndrewKralovec thanks man. i appreciate your reply :) however, for now, i dont have the luxury of time to learn and setup a Linux environment. I need to have a running application as soon as possible so I decided to deploy it first in Windows. But I will follow your advice as soon as i get it running.
Fortunately, I solved the issue. I forgot my EC2 instance doesn't have a node.js installed. After installing that, everything worked fine :)


feat(services/azdls): list start from and sas token support

Which issue does this PR close?
Closes #.
Rationale for this change
This changes proposes implementation of the start_after workaround used by Hadoop to improve listing in ADLS from a particular point.
This is important for table formats like Delta Lake that make use of list_with_offset in object store to get the latest version of a table.
In my testing listing directories with > 10k records similar to what Delta Lake would do improves times from seconds to milliseconds (~3seconds to 60ms).
Also this adds SAS token support which could be split into its own PR but needed it for testing against my company's ADLS accounts.
What changes are included in this PR?
Changes to azdls module to include list with start_after functionality. This includes a module crc64 which computes crc for use in the continuation token generated for start after
Are there any user-facing changes?
Yes, start_after will function for azdls, and SAS token support.

@Xuanwo I'm opening this as a draft to get initial feedback. This is not directly supported by Microsoft, or I should say documented. In pursuit of them adding this capability to the azblob endpoints, they were the ones that directed me to Hadoop example, so I have confidence in it but could see if its something they would officially support/document.
One caveat as well is I believe (per the hadoop code) the change is will only work if hierarchical namespace (xns) is enabled which isn't something I believe we can tell when the list request is being made. It could be the user has to opt in and inform if xns is enabled. Hadoop code does another request to figure that out dynamically which I'd personally want to avoid.

hey @Xuanwo as I was refactoring the code I wrote a test against one of our actual storage accounts. I then executed the list call with recursive =  true as the object_store integration does. However when I do that list result from azure contains everything.
I'm trying to figure out how recursive could be impacting the azdls list but I don't see it used by azdls service.

Ok so I found the FlatLister does not retain the OpList args.
Adding modifying FlatLister to take in the original args and pass them to the inner lister seems to work for this case, but I'm not sure how recursive listing and start_after should interact for all edge cases.
The underlying Azure REST API does have a recursive parameter, but when actually using that:

the recursive listing performance in general is not great on azdls
start_after continuation token causes 500 when recursive = true
so I am hesitant to enable it

Thank you @alexwilcoxson-rel for catching this.

Adding modifying FlatLister to take in the original args and pass them to the inner lister seems to work for this case, but I'm not sure how recursive listing and start_after should interact for all edge cases.

It should be fine as long as we handle the capability correctly.

The underlying Azure REST API does have a recursive parameter, but when actually using that:

the recursive listing performance in general is not great on azdls
start_after continuation token causes 500 when recursive = true
so I am hesitant to enable it

That seems to be a problem. Would you like to raise a separate issue for this? We can track it and switch to azdls's own recursive when available.

The underlying Azure REST API does have a recursive parameter, but when actually using that:

the recursive listing performance in general is not great on azdls
start_after continuation token causes 500 when recursive = true
so I am hesitant to enable it

That seems to be a problem. Would you like to raise a separate issue for this? We can track it and switch to azdls's own recursive when available.

[ ] Yes I will create a follow up issue to switch to use recursive parameter available on azdls list. Mentioning though the seeming incompatibility with this start_after workaround

This feature is undocumented, so it's best not to enable it by default. How about adding a new flag called enable_list_with_start_after, where we set the capability list_with_start_after to true?

[ ] gate behind feature

Hey @Xuanwo just letting you know I probably won't get back to this until later in the week. I will share though that I am using this in our fork of version ~0.48 I think. We're using it along size the base object_store azure store (since those have all the put support for delta), and it has gone well!
Looking forward to getting this in along with the put if-not-match changes that are in progress as well!


Fixes to work with collection-2021* conda envs

These fixed made the startup files to start successfully against the collection-2020-2.0rc8 conda env. Done with @dmgav during today's deployment visit at HXN.
xref #17 and https://github.com/NSLS-II-HXN/hxnfly/pull/3 and the latest tag of https://github.com/NSLS-II-HXN/hxntools.
Requires:

[ ] https://github.com/NSLS-II-HXN/hxnfly/pull/8

CI fails because change-xspress3-imports branch of hxnfly was merged.

I manually added changes from https://github.com/NSLS-II-HXN/profile_collection/pull/22 to avoid long rebase.

This fails CI but it is what the beamline is actually running on. Merging.


Add DLNA support

This PR adds DLNA support, fixing #186

What is a cheap DLNA device that I can buy to test this out?

Also, before we merge this, can you get rid of the async dependency in dlnacasts?
async is bloated and includes 50+ useless functions, plus all of lodash. I've never needed more than a couple async functions, just parallel and series are adequate 99% of the time. Can you swap out your async.parallel call with run-parallel please?
Cheers!
@dcposch Can you review this too?

Cheap DLNA device? any cheap android TV stick can handle DLNA these days.
Even more, if to have a Samsung or Phillips 'smart tv' ia m pretty sure it supports it.
You can even install a DLNA renderer in linux (apt-get install gmediarender) in virtualbox

Will swap async with run-parallel and made all the changes you suggested

Cool, will test out on a Samsung Smart TV later tonight.
Will merge after you've made the changes.

changes updated

Okay, I just had a chance to test this out with an Xbox.
It worked for an .mp4 file, but not a .webm file. Nothing we can do about that if the Xbox doesn't support .webm, I suppose.
But we need better error handling. We need to let the user know that the streaming failed and give a reason, if we can.
I'll merge this now, but let's try to improve it over the coming days and weeks.

I didn't catch it because in my tests with android tv sticks (2) and TVs (2), webm was supported (all of them are >2014).
@feross: completely agree with you, we need to let user know something went wrong.
In fact, its not DLNA specific. If chromcast fail to connect, it silently fails and revert back to internal player. In airplay case, its added to state.errors
        state.errors.push({
          time: new Date().getTime(),
          message: 'Couldn\'t connect to Airplay'
        })

but not notified to the user neither.
I can catch the unsupported media error and notify the user, leading me to the question. Is there a notification system in webtorrent-desktop? I was thinking something like Android's Toast, or more complex, use some kind of native notification, but i think this short, program related notifications should be handled locally.

today i just need to read the code and not asking questions, status.error was the native solution.
Updating PR.

I didn't catch it because in my tests with android tv sticks (2) and TVs (2), webm was supported (all of them are >2014).

It's great that most devices support .webm these days.

If chromecast fail to connect, it silently fails and revert back to internal player.

I'll fix that in a PR shortly.
The rest of your changes look good.

@dcposch On second thought, I think the changes in this PR are fine to merge before the release :)

@grunjol Thanks for this, and all your other, great PRs! I just added you as a GitHub collaborator. Congrats!
Please read the contributing guidelines so you know what responsibilities collaborators have. Cheers! :beers:

@feross this broke all casting as well as seeking for me. every time lazyLoadCasts() runs, it throws this error:
module.js:338 Uncaught Error: Cannot find module 'mime'

No chooser yet, but I think that's fine.
Looking at the mime error.

Oh, I see what happened. dlnacasts doesn't have mime in the package.json, but mime got hoisted to the top level by npm dedupe for me, which is why I didn't notice it.

@dcposch Fixed on master.

fixed in dlnacasts 0.0.3


remove S25|P65 (reals are size continuum) as redundant

New pi-Base feature: https://topology.pi-base.org/spaces/S000025/properties/P000065 shows why this assertion is redundant with the others.
(note to self: should refactor that feature so it appears more like the list on a deduced trait)

One question about redundancy is that, should we remove ALL the redundancy, or just like something T106 that, we would keep some redundancy for simplicity? What do you think @StevenClontz

I'm open to discussion; we've decided we're okay with redundant theorems, should we preserve certain traits as well?

In this case, it does not seem to be a good idea to me to remove the fact that the cardinality of $\mathbb R$ is the continuum.
It's a basic result that does not need any topology, and it would be replaced by a chain of theorems, some of which are nothing obvious.
But often when we see something like this, it can go both ways between groups of properties.  Namely, we can ask:
Can we keep this trait and instead remove some of the other traits that could be derived from P65 in combination with some of the other traits?

I guess if we can't find anything to replaced it with, I would not object to removing P65.

In this case, it does not seem to be a good idea to me to remove the fact that the cardinality of R is the continuum. It's a basic result that does not need any topology, and it would be replaced by a chain of theorems, some of which are nothing obvious.

Since it's the definition of Cardinality = c, I vote to keep the 'By definition' trait file. It's nice that the trait file also displays a derivation from the other traits, though.

I do think I want to tweak the UI so it doesn't appear to be a warning, but instead just a "heads up". I think it will be particularly useful during the review of new PRs.

Another parallel thought.  Would there be a benefit in hiding that warning by default, and have a button somewhere near the bottom of the screen somehow to show this derivation if it exists? That way, ordinary users who don't care about redundancy of traits would no be bothered by that.
Or maybe we could have a very subtle indication, like the button at the bottom would be in a different color, or even better, the button would only appear if the trait is redundant, and then a reviewer click on that to check it.

Yeah, having it pop-out like our Markdown button makes sense.

(Edit: I might try to brush up on my git skills a bit to look through the graveyard of deleted traits to see if any might be worth reviving, if that seems like a worthwhile thing to do.)

Nobody will stop you, but I am not sure we want to spend time reviving deleted traits at this point.  It can happen organically as people do changes to a particular space and notice something about their favorite trait.

I couldn't find any other asserted property I could weaken to require cardinality, but I think I agree it seems silly to remove such an easily seen property, so I'm closing this and leaving the redundancy.


ERROR WHEN RUN PREPROCESS

When i run prprocess folder, it always error not found {save path}/mimic3/descemb_whole/labevents.pkl' and not found {save_path}/mimic3_mv/npy/input_ids.npy'.., Can you help me ?

Can you provide your environment variable? (The result of $env) ?

It seems that your command is incorrect.
When you execute the command
$ bash preprocess_run.sh {MIMIC-III} {MIMIC-IV} {eICU} {save_path}
you should replace {save_path} with the directory where you want to save the results.
Also, before executing the command, make sure that {save_path} points to an existing directory.
If you have any questions, feel free to ask.
And thank you for your interest in our research!

i did as you said, but there is no descemb_whole folder in mimic3 and no input_ips.npy file in /mimic3_mv/npy
Vu-Truong-An-TopCV.vn-241024.00040.pdf
HUST_PPT_template_2022_RED_4x3.pptx
Uploading IEEE_BrandedPresentationPPT_Template_A.pptx…

i did as you said, but there is no descemb_whole folder in mimic3 and no input_ips.npy file in /mimic3_mv/npy

The files are not initialized which leads to the error as I mentioned in the top issue.

It seems like the first step of preprocessing (main_step1.py) was incorrectly terminated.
Can you share the full log, after doing the below steps?

Remove all contents within the save_dir.
Execute bash preprocess_run.sh {MIMIC-III} {MIMIC-IV} {eICU} {save_path}

(unifl) backne@backne-Precision-3640-Tower:~/physionet.org/files/UniFL/preprocess$ bash preprocess_run.sh {MIMIC-III} {MIMIC-IV} {eICU} data
--2024-10-25 11:56:44--  https://www.hcup-us.ahrq.gov/toolssoftware/ccs/Multi_Level_CCS_2015.zip
Resolving www.hcup-us.ahrq.gov (www.hcup-us.ahrq.gov)... <IP_ADDRESS>, <IP_ADDRESS>, 2600:1f18:6dc:c282:d4de:b071:7a42:c950, ...
Connecting to www.hcup-us.ahrq.gov (www.hcup-us.ahrq.gov)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://hcup-us.ahrq.gov:443/toolssoftware/ccs/Multi_Level_CCS_2015.zip [following]
--2024-10-25 11:56:45--  https://hcup-us.ahrq.gov/toolssoftware/ccs/Multi_Level_CCS_2015.zip
Resolving hcup-us.ahrq.gov (hcup-us.ahrq.gov)... <IP_ADDRESS>, 2600:1f18:6c7b:6b00:6e76:46c6:d93b:8cce
Connecting to hcup-us.ahrq.gov (hcup-us.ahrq.gov)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 111731 (109K) [application/zip]
Saving to: ‘data/cache/Multi_Level_CCS_2015.zip’
Multi_Level_CCS_2015.zip                           100%[===============================================================================================================>] 109,11K   111KB/s    in 1,0s
2024-10-25 11:56:47 (111 KB/s) - ‘data/cache/Multi_Level_CCS_2015.zip’ saved [111731/111731]
--2024-10-25 11:56:47--  https://data.nber.org/gem/icd10cmtoicd9gem.csv
Resolving data.nber.org (data.nber.org)... <IP_ADDRESS>
Connecting to data.nber.org (data.nber.org)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2766948 (2,6M) [text/csv]
Saving to: ‘data/cache/icd10cmtoicd9gem.csv’
icd10cmtoicd9gem.csv                               100%[===============================================================================================================>]   2,64M   309KB/s    in 11s
2024-10-25 11:56:59 (257 KB/s) - ‘data/cache/icd10cmtoicd9gem.csv’ saved [2766948/2766948]
--2024-10-25 11:56:59--  https://www.cms.gov/Medicare/Coding/ICD10/Downloads/2018-ICD-10-CM-General-Equivalence-Mappings.zip
Resolving www.cms.gov (www.cms.gov)... <IP_ADDRESS>, 2405:4800:10b:198::28a, 2405:4800:10b:182::28a
Connecting to www.cms.gov (www.cms.gov)|<IP_ADDRESS>|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1114989 (1,1M) [application/zip]
Saving to: ‘data/cache/2018-ICD-10-CM-General-Equivalence-Mappings.zip’
2018-ICD-10-CM-General-Equivalence-Mappings.zip    100%[===============================================================================================================>]   1,06M  93,6KB/s    in 9,8s
2024-10-25 11:57:09 (111 KB/s) - ‘data/cache/2018-ICD-10-CM-General-Equivalence-Mappings.zip’ saved [1114989/1114989]
create MIMIC3 ICU start!
length of PATIENTS.csv  :  46520
length of ICUSTAYS.csv  :  61532
length of DIAGNOSIS_ICD.csv  :  61532
length of ADMISSION.csv  :  58976
length of temp  : 49500
readmission value counts : readmission
0              43687
1               2708
dtype: int64
HOME HEALTH CARE             25.956867
HOME                         23.841349
SNF                          16.034753
REHAB/DISTINCT PART HOSP     13.261188
IN_ICU_MORTALITY              7.238516
LONG TERM CARE HOSPITAL       4.896795
IN_HOSPITAL_MORTALITY         3.562707
DISC-TRAN CANCER/CHLDRN H     1.292959
SHORT TERM HOSPITAL           0.904815
DISCH-TRAN TO PSYCH HOSP      0.863687
HOSPICE-HOME                  0.837982
LEFT AGAINST MEDICAL ADVI     0.583503
HOSPICE-MEDICAL FACILITY      0.336735
HOME WITH HOME IV PROVIDR     0.133666
OTHER FACILITY                0.123384
ICF                           0.102820
DISC-TRAN TO FEDERAL HC       0.025705
SNF-MEDICAID ONLY CERTIF      0.002570
Name: DISCHARGE_LOCATION, dtype: float64
Index(['DISC-TRAN CANCER/CHLDRN H', 'DISC-TRAN TO FEDERAL HC',
'DISCH-TRAN TO PSYCH HOSP', 'HOME', 'HOME HEALTH CARE',
'HOME WITH HOME IV PROVIDR', 'HOSPICE-HOME', 'HOSPICE-MEDICAL FACILITY',
'ICF', 'IN_HOSPITAL_MORTALITY', 'IN_ICU_MORTALITY',
'LEFT AGAINST MEDICAL ADVI', 'LONG TERM CARE HOSPITAL',
'OTHER FACILITY', 'REHAB/DISTINCT PART HOSP', 'SHORT TERM HOSPITAL',
'SNF', 'SNF-MEDICAID ONLY CERTIF'],
dtype='object')
No Discharge                 51.983138
HOME                         16.402334
HOME HEALTH CARE             15.754569
SNF                           7.636943
REHAB/DISTINCT PART HOSP      4.079377
LONG TERM CARE HOSPITAL       1.475465
DISCH-TRAN TO PSYCH HOSP      0.596355
DISC-TRAN CANCER/CHLDRN H     0.529522
SHORT TERM HOSPITAL           0.395856
LEFT AGAINST MEDICAL ADVI     0.395856
HOSPICE-HOME                  0.372722
HOSPICE-MEDICAL FACILITY      0.131095
OTHER FACILITY                0.084826
ICF                           0.084826
HOME WITH HOME IV PROVIDR     0.064262
DISC-TRAN TO FEDERAL HC       0.010282
SNF-MEDICAID ONLY CERTIF      0.002570
Name: imminent_discharge, dtype: float64
Index(['DISC-TRAN CANCER/CHLDRN H', 'DISC-TRAN TO FEDERAL HC',
'DISCH-TRAN TO PSYCH HOSP', 'HOME', 'HOME HEALTH CARE',
'HOME WITH HOME IV PROVIDR', 'HOSPICE-HOME', 'HOSPICE-MEDICAL FACILITY',
'ICF', 'LEFT AGAINST MEDICAL ADVI', 'LONG TERM CARE HOSPITAL',
'No Discharge', 'OTHER FACILITY', 'REHAB/DISTINCT PART HOSP',
'SHORT TERM HOSPITAL', 'SNF', 'SNF-MEDICAID ONLY CERTIF'],
dtype='object')
average length:  6.250340590699946
dx freqeuncy [   1  403 1878 3428 5099 5930 5481 4779 3902 3180 2243 1427  738  301
98   15]
max length:  15
min length:  0
The final MIMIC3 cohort pickle is saved at: data/mimic3_cohort.pkl
create eICU ICU start!
Unique patient unit stayid :  200859
final acuity class category dict_items([(0, 'Home'), (1, 'IN_HOSPITAL_MORTALITY'), (2, 'IN_ICU_MORTALITY'), (3, 'Nursing Home'), (4, 'Other'), (5, 'Other External'), (6, 'Other Hospital'), (7, 'Rehabilitation'), (8, 'Skilled Nursing Facility')])
Home                        59.131458
Skilled Nursing Facility    12.999186
IN_ICU_MORTALITY             5.487500
Other External               4.482644
Rehabilitation               4.327030
Other                        4.080340
Other Hospital               4.011581
IN_HOSPITAL_MORTALITY        3.432552
Nursing Home                 1.059742
Name: hospitaldischargelocation, dtype: float64
Index(['Home', 'IN_HOSPITAL_MORTALITY', 'IN_ICU_MORTALITY', 'Nursing Home',
'Other', 'Other External', 'Other Hospital', 'Rehabilitation',
'Skilled Nursing Facility'],
dtype='object')
Home                        48.671552
No Discharge                30.948460
Skilled Nursing Facility     8.062366
Other                        3.055581
Other External               2.908411
Other Hospital               2.670768
Rehabilitation               2.375222
Nursing Home                 0.629693
Name: imminent_discharge, dtype: float64
Index(['Home', 'No Discharge', 'Nursing Home', 'Other', 'Other External',
'Other Hospital', 'Rehabilitation', 'Skilled Nursing Facility'],
dtype='object')
diganosis unique :  3736
Number of diagnosis with only ICD 10 code: 85
Number of more than one icd10 codes : 11
Number of icd10key_error_list : 41
Number of dx we have to find...786
Number of dx we have to find...684
Number of dx we have to find...567
Number of dx we have to find...508
Number of dx we have to find...0
The final eicu cohort pickle is saved at: data/eicu_cohort.pkl
create MIMIC4 ICU start!
HOME HEALTH CARE                25.868050
HOME                            24.760928
SKILLED NURSING FACILITY        18.662705
REHAB                            7.968566
CHRONIC/LONG TERM ACUTE CARE     6.538659
IN_ICU_MORTALITY                 6.342574
IN_HOSPITAL_MORTALITY            3.807053
HOSPICE                          2.037769
PSYCH FACILITY                   1.104106
ACUTE HOSPITAL                   0.838638
AGAINST ADVICE                   0.817521
OTHER FACILITY                   0.425352
ASSISTED LIVING                  0.126701
HEALTHCARE FACILITY              0.025642
Name: DISCHARGE_LOCATION, dtype: float64
Index(['ACUTE HOSPITAL', 'AGAINST ADVICE', 'ASSISTED LIVING',
'CHRONIC/LONG TERM ACUTE CARE', 'HEALTHCARE FACILITY', 'HOME',
'HOME HEALTH CARE', 'HOSPICE', 'IN_HOSPITAL_MORTALITY',
'IN_ICU_MORTALITY', 'OTHER FACILITY', 'PSYCH FACILITY', 'REHAB',
'SKILLED NURSING FACILITY'],
dtype='object')
final acuity class category dict_items([(0, 'ACUTE HOSPITAL'), (1, 'AGAINST ADVICE'), (2, 'ASSISTED LIVING'), (3, 'CHRONIC/LONG TERM ACUTE CARE'), (4, 'HEALTHCARE FACILITY'), (5, 'HOME'), (6, 'HOME HEALTH CARE'), (7, 'HOSPICE'), (8, 'IN_HOSPITAL_MORTALITY'), (9, 'IN_ICU_MORTALITY'), (10, 'OTHER FACILITY'), (11, 'PSYCH FACILITY'), (12, 'REHAB'), (13, 'SKILLED NURSING FACILITY')])
No Discharge                    63.801321
HOME                            13.241727
HOME HEALTH CARE                11.431717
SKILLED NURSING FACILITY         6.247549
REHAB                            1.663700
CHRONIC/LONG TERM ACUTE CARE     1.009080
PSYCH FACILITY                   0.549036
HOSPICE                          0.538478
AGAINST ADVICE                   0.444961
ACUTE HOSPITAL                   0.242843
OTHER FACILITY                   0.206643
ASSISTED LIVING                  0.070892
HEALTHCARE FACILITY              0.007542
Name: imminent_discharge, dtype: float64
Index(['ACUTE HOSPITAL', 'AGAINST ADVICE', 'ASSISTED LIVING',
'CHRONIC/LONG TERM ACUTE CARE', 'HEALTHCARE FACILITY', 'HOME',
'HOME HEALTH CARE', 'HOSPICE', 'No Discharge', 'OTHER FACILITY',
'PSYCH FACILITY', 'REHAB', 'SKILLED NURSING FACILITY'],
dtype='object')
average length:  7.867100063350327
dx freqeuncy [  20  262 1136 2562 4071 5697 7280 8280 8831 8676 7366 5919 3796 1804
533   63    2]
max length:  16
min length:  0
The final MIMIC4 cohort pickle is saved at: data/mimic4_cohort.pkl
INFO: Pandarallel will run on 8 workers.
INFO: Pandarallel will use Memory file system to transfer data between the main process and workers.
src :  mimic3 , table_name :  LABEVENTS
[0] Raw data shape:  (27854055, 9)
[1] Exclude useless columns:  (27854055, 7)
[2] Rename columns to ID, TIME
[3] Map ITEMID into descriptions:  (27854055, 7)
/home/backne/physionet.org/files/UniFL/preprocess/preprocess_utils.py:103: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
df['ID'] = df['ID'].astype('int')
/home/backne/physionet.org/files/UniFL/preprocess/preprocess_utils.py:104: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
df['TIME'] = pd.to_datetime(df['TIME'])
/home/backne/physionet.org/files/UniFL/preprocess/preprocess_utils.py:110: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
time_fil['TIME'] = (time_fil['TIME'] - time_fil['INTIME']).astype('timedelta64[m]')
/home/backne/physionet.org/files/UniFL/preprocess/preprocess_utils.py:111: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
time_fil.drop(columns=['INTIME', 'INTIME+12hr'], inplace=True)
[4] Filter ID,TIME by ICUSTAY ID,TIME:  (2151467, 7)
[5] Filter null columns:  (2151467, 7)
[6] Filter rows where ITEMID == nan:  (2151467, 7)
src :  mimic3 , table_name :  PRESCRIPTIONS
/home/backne/physionet.org/files/UniFL/preprocess/main_step1.py:33: DtypeWarning: Columns (11) have mixed types. Specify dtype option on import or set low_memory=False.
df = pd.read_csv(os.path.join(src_path, table_name) + ".csv")
[0] Raw data shape:  (4156450, 19)
[1] Exclude useless columns:  (4156450, 13)
[2] Rename columns to ID, TIME
[3] Map ITEMID into descriptions:  (4156450, 13)
/home/backne/physionet.org/files/UniFL/preprocess/preprocess_utils.py:103: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
df['ID'] = df['ID'].astype('int')
/home/backne/physionet.org/files/UniFL/preprocess/preprocess_utils.py:104: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
df['TIME'] = pd.to_datetime(df['TIME'])
/home/backne/physionet.org/files/UniFL/preprocess/preprocess_utils.py:110: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
time_fil['TIME'] = (time_fil['TIME'] - time_fil['INTIME']).astype('timedelta64[m]')
/home/backne/physionet.org/files/UniFL/preprocess/preprocess_utils.py:111: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
time_fil.drop(columns=['INTIME', 'INTIME+12hr'], inplace=True)
[4] Filter ID,TIME by ICUSTAY ID,TIME:  (367389, 13)
[5] Filter null columns:  (367389, 13)
[6] Filter rows where ITEMID == nan:  (367389, 13)
src :  mimic3 , table_name :  INPUTEVENTS
/home/backne/physionet.org/files/UniFL/preprocess/main_step1.py:28: DtypeWarning: Columns (7,9,17,20,21) have mixed types. Specify dtype option on import or set low_memory=False.
pd.read_csv(os.path.join(src_path, table_name) + "_CV.csv"),
[0] Raw data shape:  (21146926, 38)
[1] Exclude useless columns:  (21146926, 29)
[2] Rename columns to ID, TIME
[3] Map ITEMID into descriptions:  (21146926, 29)
/home/backne/physionet.org/files/UniFL/preprocess/preprocess_utils.py:103: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
df['ID'] = df['ID'].astype('int')
/home/backne/physionet.org/files/UniFL/preprocess/preprocess_utils.py:104: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
df['TIME'] = pd.to_datetime(df['TIME'])
preprocess_run.sh: line 8: 34475 Killed                  python main_step1.py --mimic3_path $mimic3_path --mimic4_path $mimic4_path --eicu_path $eicu_path --save_path $save_path
Traceback (most recent call last):
File "/home/backne/physionet.org/files/UniFL/preprocess/main_step2.py", line 531, in
main()
File "/home/backne/physionet.org/files/UniFL/preprocess/main_step2.py", line 452, in main
df = complete_dataframe(src, table_names, config, args.save_path)
File "/home/backne/physionet.org/files/UniFL/preprocess/main_step2.py", line 23, in complete_dataframe
lab = pd.read_pickle(
File "/home/backne/Tools/anaconda3/envs/unifl/lib/python3.9/site-packages/pandas/io/pickle.py", line 187, in read_pickle
with get_handle(
File "/home/backne/Tools/anaconda3/envs/unifl/lib/python3.9/site-packages/pandas/io/common.py", line 795, in get_handle
handle = open(handle, ioargs.mode)
FileNotFoundError: [Errno 2] No such file or directory: 'data/mimic3/descemb_whole/LABEVENTS.pkl'
Traceback (most recent call last):
File "/home/backne/physionet.org/files/UniFL/preprocess/split_for_FL.py", line 126, in
main()
File "/home/backne/physionet.org/files/UniFL/preprocess/split_for_FL.py", line 43, in main
fold_mimic3 = pd.read_csv(
File "/home/backne/Tools/anaconda3/envs/unifl/lib/python3.9/site-packages/pandas/util/_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "/home/backne/Tools/anaconda3/envs/unifl/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 680, in read_csv
return _read(filepath_or_buffer, kwds)
File "/home/backne/Tools/anaconda3/envs/unifl/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 575, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "/home/backne/Tools/anaconda3/envs/unifl/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 934, in init
self._engine = self._make_engine(f, self.engine)
File "/home/backne/Tools/anaconda3/envs/unifl/lib/python3.9/site-packages/pandas/io/parsers/readers.py", line 1218, in _make_engine
self.handles = get_handle(  # type: ignore[call-overload]
File "/home/backne/Tools/anaconda3/envs/unifl/lib/python3.9/site-packages/pandas/io/common.py", line 786, in get_handle
handle = open(
FileNotFoundError: [Errno 2] No such file or directory: 'data/mimic3/fold/fold_100.csv'
processing: data/mimic3_mv/npy/input_ids
Traceback (most recent call last):
File "/home/backne/physionet.org/files/UniFL/preprocess/split_npy.py", line 47, in
main()
File "/home/backne/physionet.org/files/UniFL/preprocess/split_npy.py", line 37, in main
arr = np.load(
File "/home/backne/Tools/anaconda3/envs/unifl/lib/python3.9/site-packages/numpy/lib/npyio.py", line 390, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: 'data/mimic3_mv/npy/input_ids.npy'

preprocess_run.sh: line 8: 34475 Killed python main_step1.py --mimic3_path $mimic3_path --mimic4_path $mimic4_path --eicu_path $eicu_path --save_path $save_path

This indicates that the Python process running main_step1.py was terminated without a stack trace.
It seems to be an out-of-memory (OOM) issue.
Could you share your hardware specs?
If reproducing the preprocessing on your hardware seems unlikely, I'll share the preprocessing results.

maybe my capacity is overloaded, please share the preprocessing results, please!!!

I'm now executing the preprocessing codes.
Can you share your email address?

my email<EMAIL_ADDRESS>

I'll send it after it finished.
Again, thank for your interest in my research!


Remove the need to specify the RedeemerTag

This reduces friction for writing applications.
Also the proposed way of specifying a redeemer tag = MINT for minting inputs is actually invalid. Mint scripts can only be invoked without datums, by specifying them as minting scripts.
Fixes #177

Codecov Report

Merging #178 (afd8c6f) into main (ab01bec) will decrease coverage by 0.16%.
The diff coverage is 71.42%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@            Coverage Diff             @@
##             main     #178      +/-   ##
==========================================
- Coverage   86.38%   86.22%   -0.16%
==========================================
  Files          26       26
  Lines        2761     2766       +5
  Branches      654      655       +1
==========================================
  Hits         2385     2385
- Misses        284      287       +3
- Partials       92       94       +2

Impacted Files
Coverage Δ

pycardano/txbuilder.py
90.62% <60.00%> (-0.92%)
:arrow_down:

pycardano/plutus.py
92.61% <100.00%> (+0.03%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Thank you @nielstron for finding and removing this unnecessary complexity! Your change makes sense and it looks good to me.


Python 3.10, 3.11, 3.12 images broken for us

We recently upgraded our base images from, for example,
python:3.11-bookworm@sha256:f7543d9969bdc112dd9819ca642e14433fdacfe857f170f6b803392fc7e451ad

to
python:3.11-bookworm@sha256:157a371e60389919fe4a72dff71ce86eaa5234f59114c23b0b346d0d02c74d39

and that broke all our Python images. I'm trying to understand what exactly changed (and then how to fix it). I saw in the issues that setuptools and wheel were drop as well as something with ensurepip?
What I've noticed in the image itself so far is that the name of the python binary seems to have changed - my symlink to /usr/local/bin/python3.11 is dead, there's only /usr/local/bin/python.

I'm getting build errors for an already-deployed, successfully building commitd, since yesterday.
Base image is pinned to :3.12-slim.
Googling the error mostly pop setuptools issues, but nothing about that on my Dockerfile, AFAIK (at least no change on that). So, @black-snow's guess looks reasonable to me (with my limited knowledge).
The error pops when building Django staticfiles, no code nor deps changed:
#12 [stage-0 7/9] RUN   DEFAULT_FROM_EMAIL=dummy   FEEDBACK_EMAIL=dummy   EMAIL_URL=consolemail://   DATABASE_URL=sqlite://:memory:   SECRET_KEY=dummy   STATICFILES_STORAGE=django.contrib.staticfiles.storage.StaticFilesStorage   python manage.py collectstatic --no-input --verbosity 2
#12 0.426 Traceback (most recent call last):
#12 0.426   File "/app/manage.py", line 15, in <module>
#12 0.426     execute_from_command_line(sys.argv)
#12 0.426   File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
#12 0.427     utility.execute()
#12 0.427   File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 416, in execute
#12 0.427     django.setup()
#12 0.427   File "/usr/local/lib/python3.12/site-packages/django/__init__.py", line 24, in setup
#12 0.427     apps.populate(settings.INSTALLED_APPS)
#12 0.427   File "/usr/local/lib/python3.12/site-packages/django/apps/registry.py", line 91, in populate
#12 0.427     app_config = AppConfig.create(entry)
#12 0.427                  ^^^^^^^^^^^^^^^^^^^^^^^
#12 0.427   File "/usr/local/lib/python3.12/site-packages/django/apps/config.py", line 193, in create
#12 0.427     import_module(entry)
#12 0.427   File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
#12 0.427     return _bootstrap._gcd_import(name[level:], package, level)
#12 0.427            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#12 0.427   File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
#12 0.427   File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
#12 0.427   File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
#12 0.427   File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
#12 0.427   File "<frozen importlib._bootstrap_external>", line 995, in exec_module
#12 0.427   File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
#12 0.427   File "/usr/local/lib/python3.12/site-packages/rest_framework_simplejwt/__init__.py", line 1, in <module>
#12 0.427     from pkg_resources import DistributionNotFound, get_distribution
#12 0.427 ModuleNotFoundError: No module named 'pkg_resources'
#12 ERROR: process "/bin/sh -c DEFAULT_FROM_EMAIL=dummy   FEEDBACK_EMAIL=dummy   EMAIL_URL=consolemail://   DATABASE_URL=sqlite://:memory:   SECRET_KEY=dummy   STATICFILES_STORAGE=django.contrib.staticfiles.storage.StaticFilesStorage   python manage.py collectstatic --no-input --verbosity 2" did not complete successfully: exit code: 1
------

In addition, same commit builds successfully with :3.9-slim.

Regarding my particular issue I'm not quite sure where it stems from as there's a bunch of moving pieces involved (python base image, minimal debian base, copying things around, poetry, venvs, ...) - but for now I'll just fix it with
RUN ln -s /usr/local/bin/python /usr/local/bin/python3.11.
I'm still curious about what exactly changed. Could we perhaps get a changelog we could check before bumping the digest? It's rather difficult to manually diff what's changed in between versions ;)

This looks very similar to #967

See also https://github.com/docker-library/python/issues/957 (and other linked/related issues/PRs) :eyes:

Hi
These are two completely separate issues (in general it's best to open separate issues rather than re-using an existing open issue when the symptoms aren't the same):

@black-snow your issue is presumably due to the subtle change in shebang lines described in #967, that was a side-effect of #955, and changed in the images of all supported Python versions. Most apps will work with both the old and new shebang lines, but if you are performing unusual manual symlinking/moving files around (like the use-case in #967 was), then you may need to adjust your symlinks/manual steps. The path you mentioned (/usr/local/bin/python3.11) isn't in these images, so it must be something you are creating manually.
@pataquets your issue is due to setuptools no longer being installed in the Python 3.12+ images as of #964. This removal will only affect broken packages that don't correctly specify their dependencies. It seems you are using an old version of djangorestframework-simplejwt that doesn't include the fix from https://github.com/jazzband/djangorestframework-simplejwt/commit/19cf38e4a440fd648e82a94a7998ac857d48a28d. Upgrade to a newer djangorestframework-simplejwt version or else manually install the setuptools package in the meantime.

Setuptools was distributes with 3.121-4 but adruptedly stopped in 3.12.5 ... Is this something that permanent or is it just something that was done in the middle of a version chain by accident?

@klausmyrseth This was answered/linked to already above. Please read:

https://github.com/docker-library/python/pull/954

ye but still quite interesting to do that on a minor verison update if you read my message it was a try to not be to harsh about the version usage and the fact that this is a breaking change for many. and to be completely sure as the update is HUGE and if someone suddenly brings it back that would break the builds again.

The reason I linked to that PR, is that your message asked if this change was done by accident - and the PR title and description make it clear that it wasn't by accident. It sounds like instead maybe you are (quite reasonable :-)) asking "why was this change made deliberately", in which case there's more detail on the reasoning why in:
There is more background on why the change was made in:
https://github.com/docker-library/python/issues/964#issuecomment-2340617118

Thanks @edmorley !
Breaking changes suck and they are rather hard to avoid when the tag doesn't reflect a SemVer of the image but the underlying Python version (and who would want a 3.11-bookworm-1.17?).
So the remaining open question for me is if we could start providing a changelog. I pin all my digests and updates are deliberate, but it's veeeery cumbersome to find out what changed between two digests. I'll rename the issue to reflect that.

I'm not sure a changelog would be very easy to implement in a useful way. For example, how would you track the changes, when there isn't a version per se? By date? Also, the rate of commits on this repo is generally very low, so skim reading https://github.com/docker-library/python/commits/master/ should typically be sufficient to figure out exactly what changed in the last week or so.
In general if you ever have a problem with any open source project, I'd recommend checking the GitHub releases section, followed by the commit history, then searching recently updated issues and PRs. In this case a search for the word "symlink" would have found the issue filed 5 days ago, describing exactly your issue:
https://github.com/docker-library/python/issues?q=is%3Aissue+symlink

What about bringing the links and names to the pull requests into a changelog that is easy accessible.
For me that would help me to see what's going on. For my sake I was very aware of the change python wise) but it can be hard to see what's going on with the docker. We had some code refering distutils in the old libs so I knew what to look for when the services refused to start suddently, but the first time I met the error it was hard to find the reason.
Also our thought process to remove the change when it was in v3.12.1 we thought you would not remove it until 13 as the version association brings that association to most users (like me)

@edmorley I disagree with a bunch of that ;)
Ofc I found similar issues, but do I want to rely on that someone else has already run into a similar problem and filed an issue for me to find before I bump the digest? Certainly not.
Do I want to identify all diff commits and read them in order to see if anything relevant might have changed? Nope.
I get that its somewhat difficult and many docker images are in a rather bad shape regarding this, yet writing a simple CHANGELOG.md when substantial things change seems easy enough to me.

ye but still quite interesting to do that on a minor verison update if you read my message it was a try to not be to harsh about the version usage and the fact that this is a breaking change for many. and to be completely sure as the update is HUGE (for many) and if someone suddenly brings it back that would break the builds again.

Very interesting to do this in a minor version indeed. Just another reminder that we should never use third-party docker images on non-local environments

ye but still quite interesting to do that on a minor verison update if you read my message it was a try to not be to harsh about the version usage and the fact that this is a breaking change for many. and to be completely sure as the update is HUGE (for many) and if someone suddenly brings it back that would break the builds again.

Very interesting to do this in a minor version indeed. Just another reminder that we should never use third-party docker images on non-local environments

At least you shouldn't roll 'em out without proper testing and scanning.


FDG-9170 Site Outage Messaging - Create site outage page

Jira: https://federal-spending-transparency.atlassian.net/browse/FDG-9170
Test Coverage:
all files (% lines): 90.97%

page-unavailable-text.tsx: 93.75%

unavailable/index.tsx: 100%

the image overflows into the footer around 1200px


Using PolySharap with Visual Studio Build Tools 2022

Is it possible to build PolySharp-enabled project using only Visual Studio Build Tools?. I'm on 17.5.4, yet it complains:
packages\PolySharp.1.13.1\build\PolySharp.targets(45,5): Error : The PolySharp source generators have been disabled on the current configuration, as they need Roslyn 4.3 in order to work. PolySharp requires the source generators to run in order to process shaders, so the library cannot be used without a more up to date IDE (eg. VS 2022 17.3 or greater) or .NET SDK version (.NET 6.0.400 SDK or greater).

I suspect it won't work without full IDE? Using Visual Studio Installer, I cannot find component "C# and Visual Basic Roslyn compilers" (Microsoft.VisualStudio.Component.Roslyn.LanguageServices). Can .NET6/7 somehow be used for language services when targeting .NET Framework? Or is it just for targeting older .NET Core versions? I have installed .NET6/7 on the host too, but the error is still there.
Perhaps if Build Tools are not supported, should be documented as it would help to decide to use PolySharp or no. it happens that my devops agent uses build tools and not the full IDE.

Actually cannot confirm if this is working/not working with build tools - it happened that azure devops agent VSBuild task was still finding VS 2019, I did install VS2022 IDE version and remove VS2019, now it won't complain. But I suspect build tools would be still left out as I wasn't able to find "C# and Visual Basic Roslyn compilers" (Microsoft.VisualStudio.Component.Roslyn.LanguageServices) there.
Can anyone confirm PolySharp is compatible with Visual Studio Build Tools 2022 or not?

The required MSBuild and Roslyn compiler versions are part of the .NET SDK.  As long as you have the current (7.0) SDK version, you can build from command line.
The latest SDK is only supported in the latest Visual Studio version, as listed on the download page, but to be clear, VS is not required at all to use PolySharp.

Ok @saucecontrol, that would imply I'd have to migrate to sdk-style project. Thanks for answer.

"I'd have to migrate to sdk-style project"

You don't have to. PolySharp works just fine with legacy-style .csproj projects too.
The only requirements are that:

You need to have the .NET 6+ SDK installed (as you'll need Roslyn 4.3+)
You must be using <PackageReference> to reference PolySharp (packages.config isn't supported)

You must be using  to reference PolySharp (packages.config isn't supported)

That could be it. I did have .NET 6 and 7 installed. But turns out, project wasn't using PackageReferences. But I could still build this project with full IDE and  "C# and Visual Basic Roslyn compilers" (Microsoft.VisualStudio.Component.Roslyn.LanguageServices enabled (via Visual Studio Installer).
But I cannot test now the case as I moved on by installing full IDE on build server.


VSCode can't detect Gulp tasks

I'm on WIndows 10
I'm running Visual Studio Code 1.22.1
I have node v8.10.0 installed
I have npm 5.8.0 installed
I have<EMAIL_ADDRESS>installed
Based on Visual Studio Codes instructions: Step 3: Run the gulp default task I have followed this to the T, but VSCode is failing on me when I click F1 and then Tasks:Run Task
The following output is produced in the output window...
Auto detecting gulp for folder scss failed with error: Error: Command failed: gulp --tasks-simple --no-color

In addition to this error, what the heck is task-simple--no-color??????????????

This works for me without any problems.

Can you provide me with a GitHub repository that I can clone to reproduce your problem.
--task-simple & --no-color are command line args for gulp to list available task in a simple format with coloring.

Thanks for confirming that @dbaeumer.  I'll have time on Monday to create a repository for you to clone. I wish I knew what I was doing wrong...but this is a good start.

Hi @blachawk
What version of Gulp are you using. Recently I had the same issue and found there were 2 version of gulp installed.
You can look over the issue here #30570

You hit the mark!  can't believe I overlooked that as a reason.  Thsnk you so much @sethstha

@sethstha thanks for helping here.
@blachawk OK to close the issue?

Great, That works. Thanks :)

Closing the issue then.


Move templates into own root level.

Solves https://github.com/propelorm/Propel2/issues/1634
The idea:
Template paths are by convention in

templates/
besides src/ folder (which is required to detect root path)

Any extensions should follow the same convention or pass their custom path to renderTemplate()
BC Impact:

BC breaking for all template extensions used in user land code
Must be outlined in changelogs.

If needed, we could add a shim to lookup also in old "folder" - if not found in new primary location.
This would mitigate the impact.

Codecov Report

Merging #1848 (5142f55) into master (94069d2) will decrease coverage by 20.63%.
The diff coverage is 70.00%.

@@              Coverage Diff              @@
##             master    #1848       +/-   ##
=============================================
- Coverage     87.75%   67.11%   -20.64%
- Complexity     7761     7771       +10
=============================================
  Files           282      226       -56
  Lines         21291    20960      -331
=============================================
- Hits          18684    14068     -4616
- Misses         2607     6892     +4285

Flag
Coverage Δ

5-max
67.11% <70.00%> (-20.64%)
:arrow_down:

7.4
67.11% <70.00%> (-20.64%)
:arrow_down:

agnostic
67.11% <70.00%> (+0.19%)
:arrow_up:

mysql
?

pgsql
?

sqlite
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...nerator/Builder/Om/TableMapLoaderScriptBuilder.php
0.00% <0.00%> (-100.00%)
:arrow_down:

src/Propel/Generator/Command/InitCommand.php
0.00% <0.00%> (-69.24%)
:arrow_down:

src/Propel/Generator/Manager/MigrationManager.php
0.00% <0.00%> (-95.35%)
:arrow_down:

.../Propel/Generator/Builder/Om/AbstractOMBuilder.php
86.51% <80.00%> (-10.38%)
:arrow_down:

src/Propel/Common/Util/PathTrait.php
95.23% <95.23%> (ø)

src/Propel/Generator/Model/Behavior.php
92.59% <100.00%> (-1.16%)
:arrow_down:

src/Propel/Runtime/ActiveQuery/PropelQuery.php
0.00% <0.00%> (-100.00%)
:arrow_down:

...el/Runtime/Validator/Constraints/DateValidator.php
0.00% <0.00%> (-100.00%)
:arrow_down:

...ggregateColumn/AggregateColumnRelationBehavior.php
0.00% <0.00%> (-97.30%)
:arrow_down:

... and 185 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 94069d2...5142f55. Read the comment docs.

@mringler @nederdirk
What do you think about it? Any concerns or input?

I will merge now.
Let me know if we need to enable BC path lookup shim for the beta release - if needed.


Blender 4.2 install has python error

My windows download the cats-blender-plugin-master.zip file，and use Blender 4.2 to install it, but I get this error:

This is my python version and numpy version:
C:\Users\manzh>python --version
Python 3.10.6
C:\Users\manzh>pip install numpy
Requirement already satisfied: numpy in c:\users\manzh\appdata\local\programs\python\python38\lib\site-packages (1.23.0)
[notice] A new release of pip is available: 23.3.2 -> 24.2
[notice] To update, run: c:\users\manzh\appdata\local\programs\python\python38\python.exe -m pip install --upgrade pip
I guess is my python version problem ?

So you ignored my post which states: "READ FIRST PLEASE - THIS GITHUB IS NOT MONITORED BY ANYONE SO DON'T BOTHER POSTING AN ISSUES, READ THIS INSTEAD!" which answers your question?
Please read this post and close your issue: https://github.com/absolute-quantum/cats-blender-plugin/issues/687


Fix fee for USDT transfers

Tested by:

Moonbeam -> Statemint
Statemint -> Acala
Statemint -> Bifrost
Statemint -> HydraDX


ci: move build ignore script to next app dir and reverse exit commands

Description

Moves the script so it can be found by Vercel.
Reverses the exit commands so builds actually build when they should.

How to test

The preview build in this branch should be working.

Screenshots
How it should now look:

:tada: This PR is included in version 1.9.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Trimming a TS file

Is your feature request related to a problem?
I want to be able to trim a TS file, for example remove X seconds in the beginning and remove X seconds/minutes/hours from the end. Clean and simple
Proposed solution:
Be able to set in and out using timecode similar to ffmpeg
Alternatives:
ffmpeg has this ability however it won't retain the SCTE-35 track correctly so most likely a bug in ffmpeg.

Hi @abacklundtvmedia
TSDuck processes a TS, it processes TS packets, not the embedded audio and video (except for a few specialised usages). TSDuck can remove packets at the beginning and the end of a file. See the plugins skip and until. You need to know the number of packets to remove. If the TS is CBR or not-too-variable VBR, it is easy to convert the duration into packets using the TS bitrate. I have done this many times in scripts. First, let tsanalyze evaluate the bitrate. In a script you can collect it using grep and sed with "normalized" output or using jq with JSON output. Then, convert the requested durations into packet counts and remove them.

ffmpeg has this ability however it won't retain the SCTE-35 track correctly so most likely a bug in ffmpeg.

Precisely because ffmpeg is very good at audio/video processing and very bad at TS processing. The strengths of ffmpeg are the weaknesses of TSDuck and vice-versa.
That being said, since TSDuck is often used in combination with ffmpeg and since ffmpeg is known to destroy everything it does not understand in the TS, there is a way to encapsulate and restore data PID's to preserve them through ffmpeg processing. See the example in section 5.2.28 of the TSDuck user's guide. It preserves the EIT's across ffmpeg processing. You can use a similar principle to preserve SCTE-35 or any other data PID.

Thank you for the suggestion and explanation 🙏🏻
Are there by the way any real-world examples (that are described somewhere) out there how people tend to use ffmpeg in combination with TSduck?


Puzzles folder needs to be included in release packages

Presently the puzzles folder is missing from all the binary packages, and needs to be manually downloaded and put in the right place.  It should be included in the packages.

Fixed in 5c02b84 / v0.2


Handler unloading

Description
Allows the user to close/unload ClientBoundPacketHandler implementations.
Why?
Dynamic handler loading seems like a novel idea.

I don't necessarily disagree with adding an option to remove handlers, but why would we need a close method? If you have the reference to the handler instance you should be able to handle anything else as a result of removing the handler yourself.

The user could certainly call there own close method after removing a handler. However, it looks like all of the mods can access each-other handlers currently, and this close method followed that expectation.

The handlers registered aren't currently accessible, the only way to add a handler is to provide the handler itself. Other mods don't have access to the handler added.

Other mods should be able to access it through mixin, reflection, or access wideners. However, at that point you could probably write your own injection into unregisterHandler anyway to write a close method so I don't really care if it gets kept.

I think with the current goals and implementation of the Mod API, methods to unregister handlers are unnecessary


feat: add ResolveAttribute

Issue #, if available:
Description of changes:
Add ResolveAttribute
Also change build to preferentially use MavenLocal over MavenCentral
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

LGTM pending comments from @lavaleri .


Itp test

Add a test that fails:
https://github.com/XENONnT/base_environment/runs/4483785976?check_suite_focus=true

Okay I reverted https://github.com/XENONnT/straxen/pull/757 feel free to merge it again with master to see if the test passes now. Can merge afterwards.

@WenzDaniel thanks, actually I would like to see this test fail. I also feel like we should have a few extra test for this function. This test is the bare minimum of course 😉

The current ones running should fail hopefully :D

okay, they just did, I'm updating with master now and merge if they succeed now

Hi, @JoranAngevaare I think it is caused by one of my mistakes, a typo in the s2 correction causing a metadata field to be considered as a map. I will change the file later but I think it has been excluded from default tests since https://github.com/XENONnT/private_nt_aux_files/pull/129, so the test should pass now.

@zhut19 , ah okay, good to know thanks for fixing it, I guess fixing the map is one method.
I did see this issue later than https://github.com/XENONnT/private_nt_aux_files/pull/129 (I think) but okay, if you are fixing the map, there is no issue :)


Skip config verification for 1/1 federations

For a 1/1, this screen isn't necessary

I wonder if we would do this on the frontend or the backend?

Closed by #142


Binding for ListAdapter

Is your feature request related to a problem?
Provide binding for ListAdapter, because it more efficient for recyclerview too
Thank you

Him @fjr619 !
BindingListAdapter is added in the 1.0.6-SNAPSHOT.
Thanks for your suggestion!


backend/handlers: fix err handling in getExchangeBuySupported

If getExchangeBuySupported was called after an account was closed it was returning a generic error, causing the frontend to display the generic error popup when disconnecting the bitbox every now and then.
This makes the endpoint to return an empty list of supported exchanges if the account is not available.

There is a CI error, but looks unrelated to the PR

There is a CI error, but looks unrelated to the PR

yeah sorry, it's our new translation check and will be fixed with https://github.com/BitBoxSwiss/bitbox-wallet-app/pull/2794