Experiments on real bulk data

Hi Kevin,
I conducted experiments on real bulk data (by setting the train_datasets as 'data6k,data8k,donorA,donorC', and setting test_dataset as 'sdy67'). I noticed that the 'sdy67' dataset is already included in the "pbmc_data.h5ad" file. Should I directly use these data without preprocessing? I got unsatisfying result (which is much lower than the metrics you reported in the paper):

Can you suggest what is wrong with my experiments? Thanks a lot.
Best,
Xiaohan

Hi @hathawayxxh ,
yes that doesn't seem right. Could you expand a little on what you have been doing? I would need the exact command you used to run Scaden and also how you calculated those output metrics.
You can use the pbmc_data.h5ad file also for evaluating on sdy67, as long as you don't specify this dataset as training dataset.
Cheers,
Kevin

Hi Kevin,
Thanks for your reply. When I run the Scaden, it always reports the following error:

So I rewrote this algorithm to a pytorch version. Experiments on the simulated data are correct (e.g., train_datasets = [data8k,donorA,donorC], test_dataset = [data6k].), resulting in similar results with your paper.
However, the result is incorrect when I use all simulated data for training and use the sdy67 dataset (included in the pbmc_data.h5ad) for testing. In this case, I set the train_datasets = [data6k,data8k,donorA,donorC], test_dataset = [sdy67]. Other parts are kept unchanged with the simulation experiments.

Hi Xiaiohan,
Okay, could you still send me the exact commands you used when trying to run the TF version of Scaden? Because there shouldn't be any error.
Nice that you implemented a pytorch version! Thought about doing this myself as I like it more than TF but stayed away from it for now. It's impossible for me to tell what the reason for that failing is if you used your own code, sorry. There could be a lot of things going wrong here. If you could point to your code I might have a look at it and see if I can find some issues.
Cheers,
Kevin

Hi Kevin,
My command for simulation experiment is:
scaden train "processed_for_donorC.h5ad" --train_datasets 'data6k,data8k,donorA' --steps 5000 --model_dir model
Then, it reports the error：

Best,
Xiaohan

Thanks - and the scaden process command?
Also, did you use the pip version or the container? What system are you running on?

Hi Kiven,
The problem might be caused by the tf version. which version have you used in your experiments?

In the initial implementation, Tensorflow 1.10.0 was used, but I have updated to Tensorflow 2 since. So everything from > v2.0.0 should work actually.

I found the former error was caused by tf version == 2.3.0:

Then I degraded the tf version to 2.0.0. The scaden runs correctly, but reported another error when saving models:
ValueError: Model <tensorflow.python.keras.engine.sequential.Sequential object at 0x7f0ec82b6400> cannot be saved because the input shapes have not been set. Usually, input shapes are automatically determined from calling .fit() or .predict(). To manually set the shapes, call model._set_inputs(inputs).

Hi Kevin,
I want to know did you perform any preprocessing before experiments on real bulk data?
I have checked the genes of PBMC2 (GSE107011) cohort, and find that only 9547 genes overlap with the genes in the simulated data. Should I use preprocess to match the genes before training the network?

Hi @hathawayxxh ,
yes please :) that's why I was asking you about the process steps you have performed.
If you don't run scaden process, it won't work. This steps assures that the genes overlap with the prediction data (among other things).

Yes, I understand the experiment with GSE107011 should perform preprocess, but the SDY67 dataset is already contained in the "pbmc_data.h5ad" file. I guess the genes in SDY67 is already matched with the simulated data, so I guess it does not need the preprocessing procedure?

That's correct, however you should still pre-process it. The preprocessing also normalizes the count data, wich is essential. The data stored in this training dataset is not normalized.

Thanks, I used the GSE107011 to pre-process the file and get preprocessed.h5ad.
scaden process "/apdcephfs/share_1364273/shared_info/xiaohanxing/sc_deconv_datasets/pbmc_data.h5ad" "/apdcephfs/private_xiaohanxing/sc_deconv/bulk_data/GSE107011_exp.txt"
In this process, all samples in the .h5ad file will be normalized (including the SDY67 file). However, using this file, the test performance on SDY67 is still unsatisfactory (CCC=0.15).
Did I miss anything?

GSE107011 is the Monaco et al. dataset. SDY67 is a different dataset. So I'm guessing you're labels are wrong?

The labels are correct.

I'm sorry but you are saying the test performance is bad on SDY67 and at the same time you say you use GSE107011 for processing, which is clearly the wrong file for processing if you want to evaluate performance on SDY67. If you say the labels are correct then what are you using for processing?

I'm getting pretty confused here. It would really help if you give me a proper description of what you have been doing/are doing, as I have asked for now a few times. That is:

what system are you running on
how did you install it

Hi Kevin,
Thanks for your help. I still cannot run your codes correctly. The detailed information is:

ubantu system,  CUDA Version: 10.1. tf_version: 2.3.0. python: 3.6.8.

I installed the scaden by pip3 install scaden.

For the simulated data experiments, I prepared data by scaden process "/apdcephfs/share_1364273/shared_info/xiaohanxing/sc_deconv_datasets/pbmc_data.h5ad" "/apdcephfs/share_1364273/shared_info/xiaohanxing/sc_deconv_datasets/test_data/donorC_500_samples.txt".

used the command scaden train "processed.h5ad" --train_datasets 'data6k,data8k,donorA' --steps 5000 --model_dir model

However, the model runs into an error:

Then I degraded the tf version to 2.0.0. The scaden runs correctly, but reported another error when saving models:
ValueError: Model <tensorflow.python.keras.engine.sequential.Sequential object at 0x7f0ec82b6400> cannot be saved because the input shapes have not been set. Usually, input shapes are automatically determined from calling .fit() or .predict(). To manually set the shapes, call model._set_inputs(inputs).

For the real bulk data, I preprocessed the data by scaden process "/apdcephfs/share_1364273/shared_info/xiaohanxing/sc_deconv_datasets/pbmc_data.h5ad" "/apdcephfs/private_xiaohanxing/sc_deconv/bulk_data/GSE107011_exp.txt"

then use the command scaden train "processed.h5ad" --train_datasets 'data6k,data8k,donorA,donorC' --steps 5000 --model_dir model. It runs into the same error as the simulated experiments.

Using the preprocessed data, I did experiments with my pytorch codes. On the simulated data, it gotcomparable performance as your codes. However, on the real bulk data, the performance is bad (SDY67 CCC = 0.15, GSE107011 CCC = 0.40).

so my question is:

do you know how to figure out the errors when I run your codes? I have searched a lot of information on the interenet. It seems like something wrong with the tf and cuda version. But I have tried many methods but failed to solve this problem.
maybe I should use the docker you provided? so it will not be affected by the local environment.
It is strange that my pytorch codes achieve similar performance with your codes on simulated data but much worse results on the real bulk data. I guess there might be something wrong with my data preprocessing. Could you give me a guidance on the commands for real data experiments?

Thanks a lot.

Hi @hathawayxxh ,
thanks for the explanation, that clears up things quite a bit!
Those errors are weird and I will see if I can reproduce them. But I haven't encountered them before. Definitely try the docker image in the mean time.
I will absolutely try and see if I can somehow repeat this and hopefully fix this for you. Might take a while though.
Regarding your pytorch implementation - yes it is weird that you get good performance on the simulated data but not on the bulk data. But it is impossible for me to tell you why this is the case without looking at the code and trying to reproduce it. There are countless points where it could have failed, from data pre-processing, training, or even some differences between pytorch and tensorflow (although that shouldn't make such a difference). The implementation might just be different. So I'm afraid I can't help you out on that one!
But I will look into the issues with my package.

Hi Kevin,
Thanks for your reply and help.
Best,
Xiaohan

Hi Kevin,
We re-run the bulk data experiments with the CPU docker that you provided. The procedures are :

download the training pbmc dataset for testing from https://figshare.com/s/e59a03885ec4c4d8153f in current workspace.
preprocess training data (pbmc_data.h5ad) and test data (GSE107011_exp.txt), delete some genes through the variance on the test data, and then normalize the log_min_max of the train data), and save the reserved gene_names in the model_dir/genes.txt file, so that you can only select the expression levels of these genes in the predict stage. This operation will save the preprocessed training data in processed.h5ad.
scaden process "pbmc_data.h5ad" "GSE107011_exp.txt"
train the network with all simulated cohorts or simulated cohorts + PBMC1.
scaden train "processed.h5ad" --train_datasets'data6k,data8k,donorA,donorC' --steps 5000 --model_dir model
or scaden train "processed.h5ad" --train_datasets'data6k,data8k,donorA,donorC,sdy67' --steps 5000 --model_dir model
test on GSE dataset, and generate the prediction file 'scaden_predictions.txt'
scaden predict --model_dir model GSE107011_exp.txt

Then, we computed the metrics for the predictions. For the model trained by simulated datasets, the CCC on PBMC2 dataset is 0.6038 (while the result in your paper is 0.68). For the model trained by simulated datasets and PBMC1, the CCC on PBMC2 dataset is 0.7273 (while the result in your paper is 0.86).
Could you help check the preprocess and training procedure? Is the training steps set as 5000 in your paper? Is there any procedures that I neglected?
Beside, the PBMC2 dataset I used in the experiments is attached. Could you help check whether this data file is the same as the data that you used in the paper?
GSE107011_exp.txt
Thanks a lot and look forward for your reply.
Best,
Xiaohan

Hi Xiaohan,
that looks much better, although it still doesn't reach the performance we could get in the paper ...
On first glance, this looks alright. Did you also include the monaco (PBMC2) dataset for prediction on PBMC1?
I'm pretty busy this week so I'll probably won't be able to look into this. Hopefully next week though!
Best,
Kevin

Ah sorry I misread you did only test on the PBMC2 (monaco) dataset and didn't get the matching results. Okay, I'll have a look at this when I find some time.
The results are not terribly off, so I'm currently assuming it might just be some slight pre-processing difference. Which doesn't mean that's ideal  - it should be more robust. I'll have a look at the file you provided. Thanks!

Hi, I had a quick look, it's not exactly the same, the file that I used had more genes in it:
monaco_samples.txt
could you try with this?
Make sure to modify nothing manually, just run scaden process pbmc_data.h5ad monaco_samples.txt
The results won't be exactly the same, I introduced code to make it fully reproducible only lately (slight variability during training and prediction is expected due to non-reproducible operations on the GPU and random keys), but it should be very close.

Hi Kevin,
Thanks a lot for your reply. I will try with this data.
Best,
Xiaohan


ServiceWorker errors: inactive tab, keepAlive

Issue and Steps to Reproduce
Log in to your app;
Turn on 'Preserve log' setting of your js console;
Switch to another tab for some minutes;
Go back, see errors in console before you will be redirected to auth provider;
Versions
Current latest;
Screenshots

For better search posting here are errors:
initWorker.js:105          GET http://localhost:3000/OidcKeepAliveServiceWorker.json net::ERR_FAILED

bundle.js:23846          Uncaught (in promise) TypeError: Failed to fetch
    at keepAlive

OidcServiceWorker.js:262 Uncaught (in promise) TypeError: Cannot read properties of null (reading 'access_token')
    at handleFetch (OidcServiceWorker.js:262:80)

hi @inomn , thank you again for your issue :)
What does look like your oidc configuration?
Which OS and which browser do you use?
Fetch error are normal but I have to do it cleanner, but I'am interrested about the session lifetime problem.
How many time do you keep the tab inactive? does your computer turn in sleep mode ?

hi @guillaume-chervet thanks again for the answer :)
MacOS 12.4, Chrome 105.0.5195.125 (Official Build) (arm64)
Inactive between 10 and 30 minutes, with no sleep mode turning on or something.
Working all the time, using the same browser but other tabs.
Config file:
export const oidcConfiguration: OidcConfiguration = {
  client_id: 'frontend',
  authority: config.oidc.authority,
  redirect_uri: `${config.host}/authentication/callback`,
  scope: '',
  service_worker_relative_url:'/OidcServiceWorker.js',
  service_worker_only: false,
  authority_configuration: {
    authorization_endpoint: `${config.oidc.authority}/realms/test/protocol/openid-connect/auth`,
    token_endpoint: `${config.oidc.authority}/realms/test/protocol/openid-connect/token`,
    userinfo_endpoint: `${config.oidc.authority}/realms/test/protocol/openid-connect/userinfo`,
    end_session_endpoint: `${config.oidc.authority}/realms/test/protocol/openid-connect/logout`,
    revocation_endpoint: `${config.oidc.authority}/realms/test/protocol/openid-connect/revoke`,
    check_session_iframe: `${config.oidc.authority}/realms/test/protocol/openid-connect/login-status-iframe.html`,
    issuer: `${config.oidc.authority}/realms/test`,
  },
};

I still didn't get it, why Fetch error is normal? This whole bunch of errors will flood Sentry :) and overall looks not 'good' to have 'normal' errors for a such usual use case.
Btw one more issue I have: logout method (keycloak) fails.
http://localhost:8080/realms/test/protocol/openid-connect/logout?id_token_hint=undefined&post_logout_redirect_uri=http%3A%2F%2Flocalhost%3A3000%2Fbookings

Bad request: Invalid parameter: id_token_hint

Hi @inomn, for for logout you need to add "openid" scope.
The client app trigger some fetch in order to keep alive the service worker alive the longer as possible (generally it can be many hours).
ServiceWorker increase a lot security but session lifetime is link to the serviceworker lifetime.
You can increase a lot the lifetime by using silent login: silent_redirect_uri: window.location.origin+'/authentication/silent-callback',
But now because browser start to block cookie tier, it is better to have a OIDC Server under the same domain than the application.

One of the problem come from safari cookies tier blocked by default :
https://infi.nl/nieuws/spa-necromancy/
https://github.com/IdentityModel/oidc-client-js/issues/1074
I need a mac to reproduce it :/

Silent signin is still an option if your oidc server share the same domain name. We do it at AXA.
For example:
App.axa.fr
Oidcserver.axa.fr
With an access token 2 minute lifetime, you may add "offline_access" scope to retrieve the refreh_token 😀

We also got the same error and we had to disable service workers for our project.
Now we have a problem with the silent login being slow when the tab becomes active again and the API calls gets a expired token from the access token hook.
@guillaume-chervet have you considered adding a promise function like getAccessTokenSilently from Auth0's sdk?

Thank you again all for your feedbacks.
These error with the service worker is a normal behavior. I need to find a way to not display it.
@mlsmaycon , do you use the fetch given by react oidc or do you inject access token yourself?
Do you have a video or something that can help me to understand the behavior?
I have done years of test with the silent signin, so i have implemented the best behavior i have found. I make all fetch with token wait for silent signin end when token are invalid before them to make a request when we are back to the tab. I have disable silent signin when the tab is inactive because it was failing. I think, It may still be enhanced.
I will give access to silentsignin methods though vanillaoidc methods. It is my next main task but it require some internal refactoring and clean up. I am pretty busy this month 😯

Hi @guillaume-chervet we are injecting the key ourselves. See the video with the problem:
https://user-images.githubusercontent.com/7747744/190903296-b9d638ef-82f9-4624-82d5-7caa59c8af86.mp4

Thank you very much @mlsmaycon  for the video.
You may for that case wait the end of the synchronisation like done here => https://github.com/AxaGuilDEv/react-oidc/blob/bfd3784f626e03c3700a0058b27a913410580e8e/packages/react/src/oidc/FetchToken.tsx#L31
I have not found better solution by now :/

We also got the same error and we had to disable service workers for our project.
Now we have a problem with the silent login being slow when the tab becomes active again and the API calls gets a expired token from the access token hook.
@guillaume-chervet have you considered adding a promise function like getAccessTokenSilently from Auth0's sdk?

I second this issue very much! We use React Query in our project, and we allow for refetch on window focus. By the time we switch tabs, the token could get expired, and the renew request might take time to be satisfied before the app is usable again. Is there a way for background refetch of the token when the user is on a different tab/application? Where would you suggest using the synchronization code you have attached?
Thank you for all your help!

Another related thing: how to set the logging level to errors only?

I set the token lifetime to days. Now after logging in I see this error:
OidcServiceWorker.js:263 Uncaught (in promise) DOMException: Failed to execute 'respondWith' on 'FetchEvent': The event handler is already finished.
    at handleFetch (http://localhost:3000/OidcServiceWorker.js:263:27)

Hi @tielushko , you may use the fetch provided by useFetch hook given by react oidc. It is compatible with react query and will wait for tokens to be sync before making the request.

Hi @inomn, the logs come from appauthjs used behind the seen. I have start to dig how to remove it but it is hidden. I am thinking more more to remove it and to ask oidc org to certify react oidc.

Hi @tielushko ,
here how i use fetch=> https://github.com/AxaGuilDEv/ml-cli/blob/master/src/Ml.Cli.WebApp/ClientApp/src/Server/withCustomFetch.js
I inject fetch in my component as a dependency (to make code testable)
And i inject it by composition => https://github.com/AxaGuilDEv/ml-cli/blob/0ceccf0f6503437a83cd16a84aaf5497c20456a7/src/Ml.Cli.WebApp/ClientApp/src/Server/Dataset/Add/New/New.container.js#L201

Hi @guillaume-chervet , back to my initial message; So since it's errors, it will be captured by Sentry.
Same issue when you visiting inactive tab, first request can be without token \ with invalid token - error, and only then ServiceWorker will do his stuff. What's the recommended way to avoid ANY errors with using this lib? Thanks.

Yes i think i can hide the error.

I just published a new fix version 6.6.5. It seem to work. @inomn

Hi @guillaume-chervet thanks, I'm now on v6.6.6, looks like error was just replaced with log?
But the fetch error is still there

This happens after a really short inactivity time, like 2 minutes.
Here's a page refresh (not hard) after just 2 mins without activity

yeah @inomn thank you, I am seraching how to fix this. it is a magic error :/

Hi @guillaume-chervet - related to this conversation, our application has the need to continue polling the server even when the tab is in the background (a type of heartbeat). I've noticed the lib skips fetching a new token on token expiry if document.hidden is true.
Is there a way to at least put an option in to allow the token to continually refresh in the background, even if the tab / page is not visible?  It would probably also fix the issues above where you come back to a tab and the token is now expired. Thanks.

Hi @beanovar , It is a browser limitation. When a tab is inactive javascript have very few time to execute (that make fetch fail). I have to try to do it inside service worker to improve to lifetime.

Hopefully we are talking about the same line, it's this one here: https://github.com/AxaGuilDEv/react-oidc/blob/bbb63160f255eba573b14e88453ad4d7fe24f259/packages/react/src/oidc/vanilla/oidc.ts#L946
My understanding is that timers are simply throttled for background tabs, but fetch requests should still work. Is it ok to provide an option to override this and allow it to try and continue fetching tokens when the document is hidden? I would be quite interested to see the behaviour of that...

Yes i can make the behavior better for classic renew, but after a certain amount of time the browser does not give enough time to javascript to fetch. The silent sigin never work when the tab is inactive. It require to much operation.

Thanks. I’m curious where you found that chrome will not give enough time for http fetches to successfully complete when the tab is in the background ? I’ve done digging and all I can find is that chrome will throttle your timers to only execute every second (even if they are scheduled to run more frequently) https://developer.chrome.com/blog/timer-throttling-in-chrome-88/
I can’t find anything that describes the amount of time you have to execute your code once the timer does trigger is limited ?

I found it somewhere. As soon as find it again i will give it here. I will try to enhance the behavior to sync token the maximum as possible.

Hi @guillaume-chervet - related to this conversation, our application has the need to continue polling the server even when the tab is in the background (a type of heartbeat). I've noticed the lib skips fetching a new token on token expiry if document.hidden is true. Is there a way to at least put an option in to allow the token to continually refresh in the background, even if the tab / page is not visible? It would probably also fix the issues above where you come back to a tab and the token is now expired. Thanks.

i have the same view i found a workaround that's work for me now :  Object.defineProperty(document, "hidden", {   configurable: true,   get: function() { return true; } });  that's override document.hidden to be always true

Hopefully we are talking about the same line, it's this one here:
https://github.com/AxaGuilDEv/react-oidc/blob/bbb63160f255eba573b14e88453ad4d7fe24f259/packages/react/src/oidc/vanilla/oidc.ts#L946
My understanding is that timers are simply throttled for background tabs, but fetch requests should still work. Is it ok to provide an option to override this and allow it to try and continue fetching tokens when the document is hidden? I would be quite interested to see the behaviour of that...

@beanovar  i have the same view i found a workaround that's work for me now :
Object.defineProperty(document, "hidden", {       configurable: true,       get: function() { return false }     });
that's override document.hidden to be always true

Hi @beanovar @chaimabenkaddour
Thank you very much for the information,  i will work on it tomorrow.
Are you using serviceworker mode?

Hi @beanovar @chaimabenkaddour
Thank you very much for the information, i will work on it tomorrow.
Are you using serviceworker mode?

Hi @guillaume-chervet no i'm not using the service worker and i'm using the v6.10.9

Hopefully we are talking about the same line, it's this one here:
https://github.com/AxaGuilDEv/react-oidc/blob/bbb63160f255eba573b14e88453ad4d7fe24f259/packages/react/src/oidc/vanilla/oidc.ts#L946
My understanding is that timers are simply throttled for background tabs, but fetch requests should still work. Is it ok to provide an option to override this and allow it to try and continue fetching tokens when the document is hidden? I would be quite interested to see the behaviour of that...

Hi @guillaume-chervet
same as the need of @beanovar out application use timers to fetch some data and should still working even on document hidden, so please do you see if there is a way to improve this behaviour
browser: Chrome
React 17.0.0
Axios 0.21.2
react-oidc 6.10.9 without service worker mode

Hi @MidoAhmed , @chaimabenkaddour , @beanovar,
The version 6.10.16, now continue fetches refresh token in background.
It work also with mobile and laptop back after a sleep mode.

fix(oidc): token not refreshed when hidden tab #902

@guillaume-chervet  thank you so mush for your reactivity

Indeed thankyou @guillaume-chervet ! Look forward to trying it. Have you had any luck finding out what can be done about the failed keep alive fetch logs at all?

Hi @beanovar , i do not find for the moment how to catch its error.

All the failed requests to the keep-alive endpoint mentioned as part of this should in theory be resolved as part of https://github.com/AxaGuilDEv/react-oidc/pull/967.


Get image from items inside folder

Hello,
i want to list items from an folder including the first image of each entry. Is this possible, an upcoming feature or extendable by a plugin?
Regards
Chris

Hi Chris,
unfortunately this is not easily possible right now, but it is an important feature, so I will add it to release 1.3.5. Release 1.3.4 with a new media-library will be published tomorrow probably.
Just as a implementation-reminder for me:

Add field for hero-image to meta-tag
For each image that is published on a page, check if hero-image is set already
if not, set it.

That's it basically, but requires an image field-type for the form-builder...

If this is very urgent, then there might be another (more dirty) workaround:

I have to add an event onPublishArticle (which is planned anyway)
Create a plugin that listens to the event.
When page is published, parse it, search for image and write it to the yaml-file of that page.

I could add that event to the release tomorrow and you could create your own plugin then.

Thank you for the great response! Its not urgent but important for daily use. I try to add a custom field in meta where i put the preview image url. Very curious about the next release!
Regards
Chris

Yes, that is a simple and clever workaround! :)

Ok but i dont get the metatabs content outside of the detail page?

Yes, you can, it is described here: https://typemill.net/theme-developers/helper-functions#list-articles
I am not super sure, what you want to achive. Generally, you can use the meta-content of pages inside a folder wherever you want. An example of this can be found in the dev-theme or in the emergency theme: https://themes.typemill.net/
This only works with pages inside a folder. If you want to get the meta-information of one special page: That is not possible right now but I can implement that in the next release, such a feature makes sense...
I will implement an image-fied-type for the next release, too (or try to) ...

or better said: the function getPageMeta(settings, article) is there already, but you need to get the item-object (= article) of the page. I am in a hurry right now and have to look into the code if there is some workaround already ...

Very nice getPageMeta(settings, article) helped. Its not that intuitive at first but performance driven.

Preview of the next version 1.3.6, it provides a new field-type for images that you can use in meta-tabs:

published with version 1.3.6 today


Python Problem

Preconditions

[X] Requirements fulfilled
[X] The known issues did not answer my question
[X] The question has not been asked in the past or the answer does not satisfy me
[ ] There is currently no other opened issue that matches my question
[ ] I cannot find any answer to my question in GitHub discussions

Current Situation
I recently downloaded the Apple TV enhanced Plugin but I get a this error message and I don't know what should I do.
I am using a windows 11 machine using Oracle VM Machine.

Logs
[8/30/2024, 11:06:03 AM] [Apple TV Enhanced] [I] Update check: You are using the latest version of AppleTV Enhanced (excluding betas): 1.6.0
[8/30/2024, 11:06:03 AM] [Apple TV Enhanced] [I] Python check: Starting python check.
[8/30/2024, 11:06:03 AM] [Apple TV Enhanced] [I] Python check: plugin dir exists.
[8/30/2024, 11:06:03 AM] [Apple TV Enhanced] [I] Python check: Python 3.8.10 is installed and supported by the plugin.
[8/30/2024, 11:06:03 AM] [Apple TV Enhanced] [I] Python check: Virtual python environment is not present. Creating now ...
[8/30/2024, 11:06:03 AM] [Apple TV Enhanced] [E] Python check: virtualenv python module is not installed. If you have installed homebridge via the apt package manager, update the homebridge apt package to 1.1.4 or above (this applies for installations based on the Raspberry Pi OS image as well). When using the official docker image, update the image to version 2023-11-28 or above. Otherwise install the python virtualenv module manually.

Configuration
{
    "name": "Apple TV Enhanced",
    "mediaTypes": [
        "music",
        "tv",
        "video"
    ],
    "deviceStates": [
        "loading",
        "paused",
        "playing"
    ],
    "deviceStateDelay": 0,
    "avadaKedavraAppAmount": 15,
    "disableVolumeControlRemote": false,
    "absoluteVolumeControl": false,
    "setTopBox": true,
    "discover": {
        "multicast": true
    },
    "forceVenvRecreate": false,
    "logLevel": 3,
    "updateCheckLevel": "stable",
    "updateCheckTime": 3,
    "deviceSpecificOverrides": [
        {
            "overrideMediaTypes": false,
            "overrideDeviceStates": false,
            "overrideDeviceStateDelay": false,
            "deviceStateDelay": 0,
            "overrideRemoteKeysAsSwitch": false,
            "overrideAvadaKedavraAppAmount": false,
            "avadaKedavraAppAmount": 15,
            "overrideCustomInputURIs": false,
            "overrideDisableVolumeControlRemote": false,
            "disableVolumeControlRemote": false,
            "overrideAbsoluteVolumeControl": false,
            "absoluteVolumeControl": false,
            "overrideSetTopBox": false,
            "setTopBox": false
        }
    ],
    "platform": "AppleTVEnhanced"
}

Operating System
Linux
Operating System: Bits
64-bit
Operating System: Distribution
Windows Virtual Machine (Oracle VM)
Operating System: Distribution Version
20.04.5
Docker
no
Docker Image
None
Docker Image Tag
bash: docker: command not found
Homebridge Version
1.8.4
Homebridge Config UI Version
4.57.1
Homebridge Storage Path
/var/lib/homebridge
Homebridge Apple TV Enhanced Version
1.6.0
Node Version
v20.17.0
NPM Version
10.8.2
Python Version
3.8.10
PIP Version
20.0.2
HDMI Hops
Apple OTT/ Philips Home Theater
Audio Output
yes
Same Subnet
yes
Additional Context
No response

Which OS are you running inside the VM? I guess Ubuntu based on the version 20.04.5 that you have provided. If you have installed homebridge via apt, just update to the latest apt package which will install all requirements automatically.

Which OS are you running inside the VM? I guess Ubuntu based on the version 20.04.5 that you have provided. If you have installed homebridge via apt, just update to the latest apt package which will install all requirements automatically.

No I installed Home bridge using the windows installation guide with the iso image
https://github.com/homebridge/homebridge/wiki/Install-Homebridge-on-Windows-10-Using-Hyper-V
I got the iso image from this link

Which OS are you running inside the VM? I guess Ubuntu based on the version 20.04.5 that you have provided. If you have installed homebridge via apt, just update to the latest apt package which will install all requirements automatically.

when I tried to download virtualenv it still doesn't want to work

Which OS are you running inside the VM? I guess Ubuntu based on the version 20.04.5 that you have provided. If you have installed homebridge via apt, just update to the latest apt package which will install all requirements automatically.

No I installed Home bridge using the windows installation guide with the iso image
https://github.com/homebridge/homebridge/wiki/Install-Homebridge-on-Windows-10-Using-Hyper-V
I got the iso image from this link

I wasn't aware that there is such an image. I will take a look at the image and check what is required to get the plugin up and running with that image when I get the time to do it. I'll give you an update here.

Which OS are you running inside the VM? I guess Ubuntu based on the version 20.04.5 that you have provided. If you have installed homebridge via apt, just update to the latest apt package which will install all requirements automatically.

No I installed Home bridge using the windows installation guide with the iso image
https://github.com/homebridge/homebridge/wiki/Install-Homebridge-on-Windows-10-Using-Hyper-V
I got the iso image from this link

I wasn't aware that there is such an image. I will take a look at the image and check what is required to get the plugin up and running with that image when I get the time to do it. I'll give you an update here.

Sure man take your time

You may want to take a look at this issue that I have just created: https://github.com/homebridge/homebridge-vm-image/issues/5
As an alternative, you can of course just update your VM / docker container.

I don't think we are getting an answer on that issue I opened within the repository of the VM image. However, you can install the virtualenv package by running pip install venv.

Closing that issue for now. If you need further advice feel free to reopen.

Got the same issue after updating the plugin. Upgraded python and venv but issue did not resolve until downgrading plugin first. After a successful venv build on the downgraded plugin, upgrading to the current plugin version worked.

Got the same issue after updating the plugin. Upgraded python and venv but issue did not resolve until downgrading plugin first. After a successful venv build on the downgraded plugin, upgrading to the current plugin version worked.

I have the same problem, also running HB on a virtualbox in Win11 pro.
Whatever update of python i do in HB Terminal, it succesfully install it, but it always show that pythong version is 3.8.
Mind to share to which plugin version number you downgraded?
First downgrade plugin version, then retry pything/venv upgrade from terminal, then upgrade to new plugin version?
Thanks

Got the same issue after updating the plugin. Upgraded python and venv but issue did not resolve until downgrading plugin first. After a successful venv build on the downgraded plugin, upgrading to the current plugin version worked.

Honestly I made a partition in my laptop and downloaded and installed Ubuntu Server on it which fixed the issue and Apple TV enhanced plugin works great

Got the same issue after updating the plugin. Upgraded python and venv but issue did not resolve until downgrading plugin first. After a successful venv build on the downgraded plugin, upgrading to the current plugin version worked.

You can try installing Ubuntu Server on a virtual box and downloading HomeBridge on it if you really want windows


some doc formatting issues in passes/synthesis/unitary_synthesis.py

By accident I ran into some small formatting and spelling issues in qiskit/transpiler/passes/synthesis/unitary_synthesis.py.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.04%) to 84.58%

Files with Coverage Reduction
New Missed Lines
%

src/sabre_swap/layer.rs
2
98.95%

Totals

Change from base Build<PHONE_NUMBER>:
0.04%

Covered Lines:
63053

Relevant Lines:
74548

💛  - Coveralls


Add a Gitter chat badge to README.md

Financial-Times/polyfill-service now has a Chat Room on Gitter
@JakeChampion has just created a chat room. You can visit it here: https://gitter.im/Financial-Times/polyfill-service.
This pull-request adds this badge to your README.md:

If my aim is a little off, please let me know.
Happy chatting.
PS: Click here if you would prefer not to receive automatic pull-requests from Gitter in future.

@JakeChampion Is this something you've discussed within the Origami team?  If we are going to offer support on a gitter channel, we need to ensure that it's regularly checked or staffed, and that we cover its existence and use in our documentation.  It is an extra responsibility which I'm cautious about and I wouldn't want people to get frustrated with the project because they were trying to talk to us in gitter and we're not there.

No, I didn't even notice this. I just created a Gitter account and this was a recommended channel alongside other repositories I contribute to. ¯_(ツ)_/¯


CON-691: Added more context to iam-password-policy-number.

Modified Dgraph query to include more context.

Going to reject this because it's PR'd into master, but please view the comment I left.


stat: fix panic message

Please take a look.
Closes #252.

I'm seeing a number of failures (panics) on go master. They seem to be related to shadow detection. I will look into it.


Feature: Convert a series of PNG images to video

Lots of good stuffs in the repo, but still miss the feature "Convert a series of PNG images to video".
I guess it will be similar to http://bigflake.com/mediacodec/EncodeAndMuxTest.java.txt, but it is good to have a working example.
Note: It is quite easy to do this in iOS with so many code examples around...

Or maybe anyone can point me the right direction? For example, step 1, step 2, when to use a new thread, when to call stuff in the main ui thread, etc.


Update semantic_versioning.yml

:tada: This PR is included in version 1.3.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


Fix #4 and better exception handling

Saw your post on the The Programmer's Hangout discord.
Really like your script and it's scope :)

Thanks for the improvements.


Back-end logs and monitoring

Integrate back-end with the services setup in issue #9 and #10.
Hopefully you can just use an library.

Basic loging and monitoring added via Application Insight


Citation Optimizer result flawed

The Citation Optimizer returned using Resourceful, Saboteur, and Tactician to 100% pull Shroomies.
When I chose those citations in the game, it actually narrowed the pool down to Shroomies and Axum as potential solutions.
I ended up finding another character to pull since I didn't have any citations that would narrow down to Shroomies, but I wanted to let you know that this occurred.

Can you please doublecheck this for me? Looking at the Shroomies page on DataCore (https://datacore.app/crew/shroomies_crew/), the polestars needed to retrieve Shroomies are Resourceful, Saboteur, and Villain (or Hunter, Saboteur, and Tactician).


Power Rename simply does nothing

Microsoft PowerToys version
Vo.62.1
Running as admin

[X] Yes

Area(s) with issue?
PowerRename
Steps to reproduce
I installed the tools and trying out resizing which worked ok, trying renaming multiple files but nothing happens. Apply (Rename) button is not responding.

✔ Expected Behavior
Click on Rename and rename the selected image files.
❌ Actual Behavior
Inactive rename button.
Other Software
No other software.

You "Search" input is empty. No search → no replace

Thank you Jay. How do I rename *.jpg , replace everything before the file extension? Replacing certain search strings is ok but doesn't help to rename everything before the file extension. Like in MacBook. A simple task.

I suppose that depends on how you want the file names to end up. Short answer is:
Search: .* (with RegEx checked; literally means "any characer for any amount")
Replace: <up to you>
and select "File name only"

Note
This will give all files the same name

A much simpler method would be to select the files in File Explorer and press F2 to Rename them all at once. Explorer will add the (1), (2) suffix automatically.

Thank you.

 I don't want to search. I've already selected the files I want to rename. I then right-click and select PowerRename. Obviously the files I selected are the files I want to rename!!!! To build the proper search in a folder that contains 100's of files is absurd.


fix: while I < data.len to let data_len = data.len()

Resolves #208

Added the "let bytes_len = bytes.len();" before the loop.

Replaced bytes.len() with bytes_len.

Hey, thanks for the PR. There are a lot of spots in the shinigami cairo code need this same change done. Could you optimize all the loops that have this pattern?

Hey, thanks for the PR. There are a lot of spots in the shinigami cairo that code need this same change done. Could you optimize all the loops that have this pattern?

@b-j-roberts I've optimized all the loops in that pattern


Ensure submodules are included in the build

Fixes #26

Not sure why github actions aren't running. let's just merge and see what happens


Use SFB SDK in my enterprise app for chat service (messaging and presence as of now)

Hi There,
We are planning the integrate iOS SFB SDK in my enterprise app for chat service (messaging and presence as of now) between users in community.
Can any one explain or route to a documentation what is the step by step procedure to register my enterprise app and use SFB App SDK service.
Once registration of my enterprise app is successful, what are the steps to achieve the below services from the App SDK.

My Enterprise app should be able to get all my conversation which is shown in SFB iOS App.
My Enterprise app should be able to get all the previous messages of a particular conversation(when ever I open a conversation in the new screen).
My Enterprise app should be able to send a message and should be able to get the participants status (online/offline/away)

I have checked the git samples (Banking App),  it fails when I Initiate  chat service or video call in BankingApp.
Thanks,
Vinodh

Hello,
All informations about App SDK are on MSDN (https://msdn.microsoft.com/en-us/skype/appsdk/skypeappsdk)
BankingApp needs configuration to work. All steps are listing on Github (in the App SDK repository)
For your information, as described on this webpage (https://msdn.microsoft.com/en-us/skype/appsdk/embedskypeb2ccomms), you can only use ios SDK for B2C scenarios where an anonymous user joins a SfB Meeting
Regards

Thanks @alexisconia !!
So as per the above reply, I can make use of iOS SfB SDK in Client App,

Join SfB meeting, provided you have a meeting URL
Once user has joined the meeting, user can either have audio/video call or chat

Kindly add if I have missed something ?
Second part of my query is,
As mentioned earlier we want to build a SfB backend for our iOS client App like,

Get all my conversation which is shown in SfB iOS App.
Get all the previous messages of a particular conversation(whenever user open a conversation in the new screen).

I believe this is not possible only from the client iOS app. How much work would be involved from our App server side ? (since it would be talking to SfB server)
Could you please guide me with some reference, for the above mentioned queries ?
Thanks


python311Packages.garth: 0.4.27 -> 0.4.29

Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 257674 run on x86_64-linux 1

8 packages built:

python310Packages.garminconnect
python310Packages.garminconnect.dist
python310Packages.garth
python310Packages.garth.dist
python311Packages.garminconnect
python311Packages.garminconnect.dist
python311Packages.garth
python311Packages.garth.dist

Result of nixpkgs-review pr 257674 run on x86_64-linux 1

8 packages built:

python310Packages.garminconnect
python310Packages.garminconnect.dist
python310Packages.garth
python310Packages.garth.dist
python311Packages.garminconnect
python311Packages.garminconnect.dist
python311Packages.garth
python311Packages.garth.dist


feat(parser): add support for Pydantic v2

Issue number: #2672
Additional issue: #2427
Summary
Changes
After an extensive week of investigation, experimentation, and implementation, I'm excited to open this pull request, which introduces support for Pydantic v2 in the parser feature. I wanted to provide you with an overview of the changes and considerations made during this process:
Support for Pydantic v1 and v2:

The parser utility has been enhanced to ensure compatibility with both Pydantic v1 and v2.
Users now have the flexibility to choose their preferred version of Pydantic for parse utility.

Powertools and the support for Pydantic:

Historically, Powertools has considered Pydantic v1 as the official library for parse utility, given its long-standing compatibility and robustness within our code.
However, with this update, we are embracing the advancements and enhancements introduced in Pydantic v2, allowing users to leverage the latest features and improvements.
It's important to highlight that while we have introduced support for Pydantic v2, our official version for Powertools remains Pydantic v1. This decision has been made to prioritize compatibility and provide a seamless experience for our existing users. By maintaining Pydantic v1 as our official library, we ensure that our ecosystem and the applications built with it continue to function as expected without any disruptions.

No-breaking changes were made
NOTE: We haven't detected any breaking changes, but please let us know as soon as possible if you find any.

Importantly, this update does not introduce any breaking changes to the existing functions within the Powertools framework. The functions that were already in use remain intact and fully functional, ensuring compatibility with the existing codebase.
However, it's worth noting that with the support for Pydantic v2, some Pydantic functions used in Powertools are now considered deprecated. This deprecation serves as a signal to users that these functions will be removed in a future major release, specifically Powertools v3.

Compatibility table

Pydantic v1
Pydantic v2
V2 deprecation
V2 removed
In use?
Code change required

@validator
@field_validator
:heavy_check_mark:

:heavy_check_mark:
:heavy_check_mark:

@root_validator
@model_validator
:heavy_check_mark:

:heavy_check_mark:
:heavy_check_mark:

.parse_obj()
model_validate()
:heavy_check_mark:

:heavy_check_mark:

.json()
model_dump_json()
:heavy_check_mark:

:heavy_check_mark:

.dict()
model_dump()
:heavy_check_mark:

:heavy_check_mark:

.parse_raw()
model_validate_json()
:heavy_check_mark:

:heavy_check_mark:

To ensure a smooth transition and minimize disruptions for our users, we have temporarily suppressed the PydanticDeprecatedSince20 and PydanticDeprecationWarning warnings (related to these functions). This allows existing applications to continue functioning as expected without outputting warnings.
If needed, you can enable the warnings yourself with something like the code below. Reference: https://docs.python.org/3/library/warnings.html

from aws_lambda_powertools.utilities.parser import event_parser, BaseModel, envelopes
from aws_lambda_powertools.utilities.parser.models import (
    SqsModel,
)

from aws_lambda_powertools import Logger
import pydantic

import warnings
warnings.simplefilter('default')

How to test before we merge this PR
To facilitate the testing of Pydantic v2 before merging this pull request, we have created a separate branch called poc/pydanticv2 where you can find the relevant code changes. Here are the steps to follow in order to test it effectively:
If using sourcecode:
- Clone Powertools repository
- Fetch the poc/pydanticv2 branch
- Check out the branch locally by running the command: git checkout poc/pydanticv2.
If using pip:
Create or modify the requirements.txt file and add the following content:
git+https://github.com/aws-powertools/powertools-lambda-python@poc/pydanticv2#egg=aws_lambda_powertools
pydantic>2

Then install by running: pip install -r requirements.txt.
If using Poetry:
poetry add git+https://github.com/aws-powertools/powertools-lambda-python#poc/pydanticv2
poetry add pydantic>2

Note for documentation
Since the Powertools currently relies on Pydantic v1 as its official dependency, you need to change the aws-lambda-powertools[tracer] or aws-lambda-powertools[all] installation to aws-lambda-powertools or aws-lambda-powertools[validation,tracer,aws-sdk] and add Pydantic v2 as an extra dependency.
Missing in this PR

[ ] Documentation
[ ] Workflow in develop branch to ensure that any changes will work in Pydanticv2
[ ] Community feedback

User experience

Please share what the user experience looks like before and after this change

Checklist
If your change doesn't seem to apply, please leave them unchecked.

[x] Meet tenets criteria
[x] I have performed a self-review of this change
[x] Changes have been tested
[ ] Changes are documented
[x] PR title follows conventional commit semantics

Is this a breaking change?
RFC issue number:
Checklist:

[ ] Migration process documented
[ ] Implement warnings (if it can live side by side)

Acknowledgment
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

hooorraay

Hi Powertools customers and community at large! We have the PR for Pydanticv2 ready for testing and would love to hear your feedback. Our goal is to merge this PR on July 21st and it would be nice to hear something before then.
Tagging people we know are directly interested in this support: @ran-isenberg @djfurman @half2me @Wurstnase @taharah @wickeat @waveFrontSet

@leandrodamascena is there a pypi PR version I can consume in my project and run the tests?

So far there isn't @ran-isenberg! But you have an alternative to install from your local folder or from GitHub. Do these alternatives not work for you? Does it really need to be a package on Pypi?
I ask because we considered the idea of having a package on Pypi in alpha release, but we chose to leave the installation directly from the GitHub branch/local folder, as this facilitates any changes we have to make.

If you could share the proper toml definition for a github branch that would be amazing and save me some chatgpt time :)

So, it's working and it didnt break me. Well done @leandrodamascena and the entire team!

Same here!
I've got two comments that might be helpful to others testing out the poc (or generally migrating to pydantic v2).

There's a code transformation tool called bump-pydantic that tremendously helped me doing the upgrade. It could transform some things automatically and left TODO markers wherever it couldn't.
After fixing the deprecation warnings in our library, I was still left with deprecation warnings from aws-lambda-powertools when running pytest (even though @leandrodamascena mentioned that these deprecation warnings are filtered out). This appears to be intended behaviour of pytest always showing deprecation warnings when running tests.

So, it's working and it didnt break me. Well done @leandrodamascena and the entire team!

Same here!

That's amazing to hear @waveFrontSet! Thanks for spending time validating this.

I've got two comments that might be helpful to others testing out the poc (or generally migrating to pydantic v2).

There's a code transformation tool called bump-pydantic that tremendously helped me doing the upgrade. It could transform some things automatically and left TODO markers wherever it couldn't.

Yes, I've tried that, but in our case, that's not very useful because we don't want to change the code to only support v2. We are maintaining support for v1 while adding support for v2.

After fixing the deprecation warnings in our library, I was still left with deprecation warnings from aws-lambda-powertools when running pytest (even though @leandrodamascena mentioned that these deprecation warnings are filtered out). This appears to be intended behaviour of pytest always showing deprecation warnings when running tests.

Good point about pytest error messages. I understand that we can suppress messages in pytest, but I'm not sure if we're missing to suppress these messages in other pieces of our codebase or if it's just pytest behavior.

Good point about pytest deprecation messages. I understand that we can suppress messages in pytest, but I'm not sure if we're missing to suppress these messages in other pieces of our codebase or if it's just pytest behavior. Can you please share one of that messages here? Please remove any confidential data.

Sure, here you go:
 .../lib/python3.10/site-packages/aws_lambda_powertools/utilities/parser/parser.py:163: PydanticDeprecatedSince20: The `parse_obj` method is deprecated; use `model_validate` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.2/migration/
    return model.parse_obj(event)
 .../lib/python3.10/site-packages/pydantic/main.py:941: PydanticDeprecatedSince20: The `parse_obj` method is deprecated; use `model_validate` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.2/migration/
    warnings.warn('The `parse_obj` method is deprecated; use `model_validate` instead.', DeprecationWarning)

I'm pretty sure you're already catching this warning, though: If I run the same code as a script the warning won't show up.

Sure, here you go:

 .../lib/python3.10/site-packages/aws_lambda_powertools/utilities/parser/parser.py:163: PydanticDeprecatedSince20: The `parse_obj` method is deprecated; use `model_validate` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.2/migration/

    return model.parse_obj(event)

 .../lib/python3.10/site-packages/pydantic/main.py:941: PydanticDeprecatedSince20: The `parse_obj` method is deprecated; use `model_validate` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.2/migration/

    warnings.warn('The `parse_obj` method is deprecated; use `model_validate` instead.', DeprecationWarning)

I'm pretty sure you're already catching this warning, though: If I run the same code as a script the warning won't show up.

Thanks for sharing this, it confirmed what I was suspecting. I'm away today, but I'll sort it out by Monday and let you know.

I'm pretty sure you're already catching this warning, though: If I run the same code as a script the warning won't show up.

Hi @waveFrontSet, can you test again?

Hi @leandrodamascena, updated to commit 1774a1c, still the same behaviour. I'm seeing the same warnings while running pytest, but not while executing it as a script.
.../lib/python3.10/site-packages/aws_lambda_powertools/utilities/parser/parser.py:163: PydanticDeprecatedSince20: The `parse_obj` method is deprecated; use `model_validate` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.3/migration/
    return model.parse_obj(event)
.../lib/python3.10/site-packages/pydantic/main.py:944: PydanticDeprecatedSince20: The `parse_obj` method is deprecated; use `model_validate` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.3/migration/
    warnings.warn('The `parse_obj` method is deprecated; use `model_validate` instead.', DeprecationWarning)

Hi @leandrodamascena, updated to commit 1774a1c, still the same behaviour. I'm seeing the same warnings while running pytest, but not while executing it as a script.
.../lib/python3.10/site-packages/aws_lambda_powertools/utilities/parser/parser.py:163: PydanticDeprecatedSince20: The `parse_obj` method is deprecated; use `model_validate` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.3/migration/
    return model.parse_obj(event)
.../lib/python3.10/site-packages/pydantic/main.py:944: PydanticDeprecatedSince20: The `parse_obj` method is deprecated; use `model_validate` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.0.3/migration/
    warnings.warn('The `parse_obj` method is deprecated; use `model_validate` instead.', DeprecationWarning)

Hi @waveFrontSet! I could reproduce the problem in my local environment and fixed it. Can you check again?
Thank you.

Hi @leandrodamascena,
this has done the trick! No warnings when running pytest. Great job. 👍


Update storage-faq-for-disks.md

How can I encrypt my existing data that were created before June 10, 2017?
As of June 10, 2017, new data written to the underlying storage accounts of managed disks is automatically encrypted. We are also planning to encrypt existing data, and the encryption will happen asynchronously in the background. If you must encrypt existing data now, create a copy of your disk. New disks will be encrypted.  This encryption is a Storage Service Encryption.  To encrypt the disks attached to the VMs, this has to be manually done by following the steps here (https://docs.microsoft.com/en-us/azure/security/azure-security-disk-encryption)

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

@sprab : Thanks for your contribution to the Azure documentation! The author, @robinsh, has been notified to review your proposed change.

hi @sprab. I was looking at your PR. First, we moved all the files around, and the article doesn't exist in that folder any more. There are now two copies, and they are here:
/articles/virtual-machines/windows/faq-for-disks.md
/articles/virtual-machines/Linux/faq-for-disks.md
These are identical except for the metadata at the top, so whatever change you make to one, you can make the same change to the other one.
Both files need to be updated in their new location. If the PR Mergers try to merge yours, it will cause merge conflicts. So I would close this PR and open a new one against those two files.
Also, you've added a paragraph that I think you mean to replace the paragraph above it - just go ahead and edit the paragraph you want to replace, or you'll end up with both paragraphs in the final doc.
If you refer to SSE, it would be great if you could hyperlink it like this:
To perform the encryption, this uses Storage Service Encryption.
The PM for managed disks would like to talk to you about your edit and maybe add some clarity that will help. @ramankumarlive -- his e-mail address is<EMAIL_ADDRESS>Can you please e-mail him and include a link to this PR?
Thanks,
@robinsh

@robinsh - To resolve this pull request, please add an appropriate comment so that automation can take the corresponding action. Thanks.

@githubber17 I am waiting for @ramankumarlive to talk to the customer to see what he wants to do. It can't be merged because it's pre-TOC-refactoring. It has to be closed and the changes made to the new version, or we can make the change. Will let you know as soon as I figure it out. @ramankumarlive Do you have an opinion?

@sprab - We need to close this PR since it cannot be merged. Would you be interested in re-applying the changes in the new locations?
/articles/virtual-machines/windows/faq-for-disks.md
/articles/virtual-machines/Linux/faq-for-disks.md
Sorry for the inconvenience.


Confusing warning Config sweep parameters are ignored in the "remote" mode

I  found this warning which is very confusing for the remote mode. Is that something that should be removed?:
https://github.com/triton-inference-server/model_analyzer/blob/main/model_analyzer/triton/server/server_factory.py#L167
[Model Analyzer] WARNING: Config sweep parameters are ignored in the "remote" mode because Model Analyzer does not have access to the model repository of the remote Triton Server.
That's doesn't seem to be true, I checked the triton endpoint for the config of the model and indeed the model_analyzer is changing the dynamic_batch and instance count parameters. The only thing that users need to do as mentioned here https://github.com/triton-inference-server/model_analyzer/blob/main/docs/launch_modes.md#remote is to enable --model-control-mode explicit in triton.

@nv-braf

Hi @nv-braf do you have any updates regarding this question? Is it true that in the remote mode, the model analyzer will still change the configuration of the remote tritonServer?
Thanks

Apologies for the delay @juanma9613. @nv-braf is dealing with some medical issues and this slipped through the cracks.
Yes, in remote mode Model Analyzer can change the configuration on the remote Triton Server. The warning that is printed is incorrect. There is a ticket to clean that up along with a few other incorrect warnings.


Lockscreen Controls on EMUI

On devices like honor or huawei using emui 3.1+ (API 21+),
there is no lockcreen controller as the Play Music app from google handle it,
do you have an idea how to fix ?

I met the same problem, and i try to find a way to set the system media lock,but there is no system settings,so I guess there is no system media lock screen, they change the source code to support the pictorial 。

Open

Sorry, but I'm not sure how lockscreen controls work on EMUI, and AOSP doesn't include "lockscreen" controls any longer.
I'd recommend asking your question on Stack Overflow.


More MCJIT speedups

MCJIT is a lot slower than old JIT. Part of that is inherent, and not addressed by this patch. But part is because MCJIT doesn't do lazy compilation of the bitcode in the module -- it does it all, and because we stuff a ton of things in the module (via llvm_ops.cpp - a large library of functions the shader might need), compiling the whole module is a large fixed per-shader-group cost.
This patch combats this in two ways:

A whole ton of stuff that is used rarely, is too big to inline, or is so expensive that inlining won't help appreciably, is moved OUT of llvm_ops.cpp and statically compiled, then just referenced by ordinary function call in the IR. The only point of having them written out in the module is to give them the opportunity to be inlined. So we remove the stuff where that's pointless.

Of what remains, we carefully traverse the module's functions and for each one, change its linkage designation from ExternalLinkage to LinkOnceODRLinkage. This has the effect of allowing it to still compile (or inline) if used within the module, but to understand that it won't be referenced from outside the module, and so it's free to be discarded if not actually used.

So these GREATLY reduce the "fixed overhead" of using MCJIT to very very close what we had for old JIT with lazy compilation.
(Unfortunately, we still need to compile the shader, and MCJIT is slower than old JIT, so for big shading groups, it takes quite a bit longer to JIT it. I'll address that separately with some more work. And also for this reason, old JIT is still used by default for LLVM versions old enough to still support it.)

A few of these moves seem like they would affect old JIT as well. I assume the before and after benchmarks on old JIT stays comparable?

I've seen very little time difference when using old JIT, as this stuff has moved around. Within the timing noise. I think that the old JIT was faster at compiling stuff in llvm_ops.cpp, but worse at inlining it (I think the lazy bitcode stuff interfered with the inlining), and those two effects very nearly cancel each other out.

LGTM


npm doesn't work through https_proxy

I'm trying to use npm to install packages on a machine which doesn't have full internet connectivity (we connect to it via a VPN).
For most (non-npm) usages, ssh proxying our office's squid port over ssh to localhost:3129 works fine.
(We use the command 'ssh -R <IP_ADDRESS>:3129:squid_server_ip:3128<EMAIL_ADDRESS>As I test, I am successfully able to wget https://www.google.com over this proxy.
The npm docs say it should work: (http://npmjs.org/doc/config.html)

https-proxy
http://npmjs.org/doc/config.htm>Default: the HTTPS_PROXY or https_proxy or HTTP_PROXY or http_proxy environment variables.
Type: url
A proxy to use for outgoing https requests.

I have pasted the console below.
Thanks,
Chris.
P.S. I have also tried export https_proxy=https://<IP_ADDRESS>:3129 and export HTTPS_PROXY=https://<IP_ADDRESS>:3129, but it doesn't help.
root@server:~# export http_proxy=http://<IP_ADDRESS>:3129
root@server:~# export https_proxy=http://<IP_ADDRESS>:3129
root@server:~# export HTTP_PROXY=http://<IP_ADDRESS>:3129
root@server:~# export HTTPS_PROXY=http://<IP_ADDRESS>:3129
root@server:~# wget --no-check-certificate https://www.google.com
--10:32:12--  https://www.google.com/
=> `index.html.2'
Connecting to <IP_ADDRESS>:3129... connected.
WARNING: Certificate verification error for www.google.com: unable to get local issuer certificate
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]
[ <=>                                 ] 10,583        --.--K/s

10:32:12 (164.01 KB/s) - `index.html.2' saved [10583]
root@server:~# npm search forever
npm WARN Building the local index for the first time, please be patient
npm http GET https://registry.npmjs.org/-/all
npm ERR! Error:<PHONE_NUMBER>:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:583:
npm ERR!
npm ERR!     at CleartextStream._puller (tls.js:489:24)
npm ERR!     at CleartextStream._pull (tls.js:420:19)
npm ERR!     at SecurePair.cycle (tls.js:677:20)
npm ERR!     at EncryptedStream.write (tls.js:122:13)
npm ERR!     at Socket.ondata (stream.js:38:26)
npm ERR!     at Socket.emit (events.js:67:17)
npm ERR!     at TCP.onread (net.js:347:14)
npm ERR! Report this entire log at:
npm ERR!     http://github.com/isaacs/npm/issues
npm ERR! or email it to:
npm ERR<EMAIL_ADDRESS>npm ERR!
npm ERR! System Linux 2.6.24-27-server
npm ERR! command "node" "/usr/local/bin/npm" "search" "forever"
npm ERR! cwd /root
npm ERR! node -v v0.6.7
npm ERR! npm -v 1.1.0-beta-10
npm ERR! message<PHONE_NUMBER>:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:583:
npm ERR! message
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     /root/npm-debug.log
npm not ok
root@server:~# cat /root/npm-debug.log
info it worked if it ends with ok
verbose cli [ 'node', '/usr/local/bin/npm', 'search', 'forever' ]
info using<EMAIL_ADDRESS>info using<EMAIL_ADDRESS>verbose config file /root/.npmrc
verbose config file /usr/local/etc/npmrc
verbose config file /usr/local/lib/node_modules/npm/npmrc
WARN Building the local index for the first time, please be patient
verbose raw, before any munging /-/all
verbose url resolving [ 'https://registry.npmjs.org/', './-/all' ]
verbose url resolved https://registry.npmjs.org/-/all
http GET https://registry.npmjs.org/-/all
ERR! Error:<PHONE_NUMBER>:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:583:
ERR!     at CleartextStream._puller (tls.js:489:24)
ERR!     at CleartextStream._pull (tls.js:420:19)
ERR!     at SecurePair.cycle (tls.js:677:20)
ERR!     at EncryptedStream.write (tls.js:122:13)
ERR!     at Socket.ondata (stream.js:38:26)
ERR!     at Socket.emit (events.js:67:17)
ERR!     at TCP.onread (net.js:347:14)
ERR! Report this entire log at:
ERR!     http://github.com/isaacs/npm/issues
ERR! or email it to:
ERR<EMAIL_ADDRESS>ERR!
ERR! System Linux 2.6.24-27-server
ERR! command "node" "/usr/local/bin/npm" "search" "forever"
ERR! cwd /root
ERR! node -v v0.6.7
ERR! npm -v 1.1.0-beta-10
ERR! message<PHONE_NUMBER>:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:583:
ERR! message
verbose exit [ 1, true ]
root@server:~#

I'm also having problems with npm over a proxy. $ npm get proxy gives me what I have set for HTTP_PROXY in env, and I tried adding "http://" to it (with npm set proxy=http://proxyserver...), but no dice.

Also having this issue. I have https_proxy set in the environment, and tried https-proxy in npm config.. just get the error message from above.

It seems that the work around right now is to redirect to the non-ssl url:
npm config set registry=http://registry.npmjs.org/

Thanks for the workaround @codereflection.
I think this issue should be reopened. There is no reason to not use the environment variables.

@atuljangra I'd have to look at the release notes, but more recent versions of npm (most definitely including the versions of npm bundled with Node.js 0.12.0 and io.js 1.1.0+) take advantage of request's support for the HTTP_PROXY, HTTPS_PROXY, and NO_PROXY environment variables.

@othiym23 that could be it. I'm on Ubuntu 14.04 and I've nodejs: 0.10.25 and npm: 1.3.10.
Thanks for the clarification.

The work around from @codereflection  worked for me ! Thanks !

Bumping this issue.  Like its been said, setting registry to http works but not when it goes to github or bitbucket for the package.  There should be a switch --no-check-certificate like wget.

Seems  npm 1.4.21 recognizes the environment variables:
$ npm --version
1.4.21
$ npm get registry
https://registry.npmjs.org/

$ export http_proxy="http://webcache"
$ export https_proxy="https://sslproxy"
$ npm get proxy
http://webcache/
$ npm get https-proxy
https://sslproxy/

When hitting HTTPS proxy https://sslproxy/ I am hit with that error.  I simply changed the proxy url to http and it worked fine:
- https_proxy=https://sslproxy/
+ https_proxy=http://sslproxy/
                  ^

:+1: @hashar, thanks dude, this worked for me too!

This issue breaks our build environment.
/ # npm version
{ npm: '2.15.11',
  ares: '1.9.0-DEV',
  http_parser: '1.2',
  modules: '11',
  node: '0.10.48',
  openssl: '1.0.1u',
  uv: '0.10.37',
  v8: '<IP_ADDRESS>',
  zlib: '1.2.8' }

/ # node --version
v0.10.48

Can it be reopened?


Fix for the overlapping of workspace scrollbar onto the blocks drawer. (fixes issue no. #3132)

I've implemented functionality to detect whether the drawer within the workspace is opened. If it is open, I've ensured that the scrollbar gets hidden to provide a seamless user experience to interact with the last block. (changes made in the hideChaff() function).

I've added code to set the scrollbar visibility upon zooming in or out if the user wants to position the existing blocks properly. (changes made in the onMouseWheel_() function).

BEFORE:
Link: https://github.com/mit-cml/appinventor-sources/assets/116882464/88a26769-242b-409d-ae9b-5da3c93f941f
**AFTER: **
Link: https://github.com/mit-cml/appinventor-sources/assets/116882464/42a95582-9173-4ee6-ac9e-fc66a12203fc
Would love to hear any other possible better solutions or suggestions.

Can one of the admins verify this patch?

@AppInventorWorkerBee ok to test


SPFx Provisioning SharePoint assets - managed metadata

What type of issue is this?
Question
What SharePoint development model, framework, SDK or API is this about?
💥 SharePoint Framework
Target SharePoint environment
SharePoint Online
What browser(s) / client(s) have you tested

[ ] 💥 Internet Explorer
[ ] 💥 Microsoft Edge
[X] 💥 Google Chrome
[ ] 💥 FireFox
[ ] 💥 Safari
[ ] mobile (iOS/iPadOS)
[ ] mobile (Android)
[ ] not applicable
[ ] other (enter in the "Additional environment details" area below)

Additional environment details

SPFx version: 1.12.1

Issue description
Hi, I am following these steps: https://docs.microsoft.com/en-us/sharepoint/dev/spfx/web-parts/get-started/provision-sp-assets-from-package
I am correct in thinking that we cannot create a managed metadata column as part of a solution meant to be deployed in different tenants because we need to specify the sspid (which is tenant-dependent) in the element.xml? or we can put some kind of token there as we can using PnP?
            <Property>               <Name>SspId</Name>               <Value xmlns:q1="http://www.w3.org/2001/XMLSchema" p4:type="q1:string"                 xmlns:p4="http://www.w3.org/2001/XMLSchema-instance">[TERM STORE ID]</Value>             </Property>
In my tests, the column is created but it is not linked to the termset.

Correct - I believe that managed metadata fields can't be created through feature XML.  Your best bet would be to have a "validate" phase or a manual step to configure the field.


Bluetooth: Mesh: Friend Clear Procedure Timeout

https://github.com/zephyrproject-rtos/zephyr/blob/99e935e2ba740c309d43a3608352e6bce494662e/subsys/bluetooth/mesh/friend.c#L647-L673
may be overflow:
// unsigned type
duration = k_uptime_get_32() - frnd->clear.start;
 	if (duration > 2 * frnd->poll_to) {

due to init as:
frnd->clear.start = k_uptime_get_32() + (2 * frnd->poll_to);

again, mesh spec define the  friend clear procedure timeout as 2*poll_timeout.

It's almost two years ago I wrote this code, but to me it looks like the frnd->clear.start should simply be initialised to k_uptime_get_32() rather than what the code now does. Do you agree?
Regarding integer overflow, as long as we don't go past the same value as the start timestamp after wrapping around to 0, the subtraction will still yield the correct result.

It's almost two years ago I wrote this code, but to me it looks like the frnd->clear.start should simply be initialised to k_uptime_get_32() rather than what the code now does. Do you agree?
Regarding integer overflow, as long as we don't go past the same value as the start timestamp after wrapping around to 0, the subtraction will still yield the correct result.

sure


Sending number zero as a value removes the corresponding field

Oops, yes, this is a bug. I will try and fix.
Would also be good to have some unit tests 😟


Potential breaking change: Upgrade to exclusively use Newtonsoft.Json >= 13.0.1

In the wake of a new vulnerability discovered in the Newtonsoft.Json library, it is recommended that consumers use version greater than or equal to 13.0.1. (https://github.com/JamesNK/Newtonsoft.Json/issues/2457)
In order to comply with the recommendation, I propose the community to consider upgrading the dependency on Netwonsoft.Json to the recommended version. It will help all the consumers of the Bond library to make their applications and services inherently more secure.
The Bond library states it's dependency as >= 9.0.1, and consumers are free to use any version that suits their needs. While this recommendation is not incorrect, it's not in the best interests of the consumers as we have the knowledge of the Newtonsoft issue. As a best practice, the latest version of the library should be using the most secure dependencies.
If there are enough community members that agree to this proposal, I'd be happy to work with the community and contribute to the proposed change.

I don't see a compelling reason to take this breaking change in Bond, so I'm going to leave Bond at the minimum version of Newtonsoft.Json that is requires.


asdf.tags.core.ndarray.validate_datatype not called for non-masked array

When validating a ndarray validate_datatype is not called unless the array is masked (and for a masked array, only on the mask). This can be observed by adding a breakpoint to validate_datatype and validating an appropriate tree or by running the following (which overwrites the validator to throw an exception).
import asdf
import numpy

class MyException(Exception):
    pass

def validate_datatype(validator, datatype, instance, schema):
    raise MyException("fail")

asdf.tags.core.ndarray.NDArrayType.validators['datatype'] = validate_datatype

a = numpy.arange(0, 10, dtype='f8')
try:
    asdf.AsdfFile({'a': a}).validate()
    print("!! array was incorrect in passing validation")
except MyException:
    print("Array was correct in failing to vaidate")

m = numpy.ma.array(a, mask=a>5)
try:
    asdf.AsdfFile({'m': m}).validate()
except MyException:
    print("Masked array was correct in failing to vaidate")

Output from running the above is as follows:
!! array was incorrect in passing validation
Masked array was correct in failing to vaidate

The above script was run against several combinations of versions of asdf (2.14.2, 2.13, 2.8) and json-schema (4.8, 4.10, 4.17) with identical results.
Is this expected behavior for a custom validator?

The mask triggers validate_datatype because the validation keyword datatype is used here in the ndarray schema. datatype for the array does not use the datatype validation keyword as the there is no one specific datatype.


Context passed as an arg to render_plugin()

Django 1.7.8
CMS 3.0.13
I'm getting an error from the render_plugin() call from one of my plugins which is sat within the columns plugin.
Just looking at the signature of render_plugin() makes me wonder if it's as glaringly obvious as it seems;
render_plugin(self, context=None, placeholder=None, admin=False, processors=None)
So my exception is as follows;
Exception Type:	AttributeError
Exception Value:	'NoneType' object has no attribute 'get'
Exception Location:	/var/www/django/myproj/lib/python2.7/site-packages/cms/models/pluginmodel.py in render_plugin, line 187

/var/www/django/myproj/lib/python2.7/site-packages/cms/templatetags/cms_tags.py in get_context

        return {
            'content': plugin.render_plugin(
                context,
                placeholder=placeholder,
                processors=processors ...
            )
        }

/var/www/django/myproj/lib/python2.7/site-packages/cms/models/pluginmodel.py in render_plugin

    def render_plugin(self, context=None, placeholder=None, admin=False, processors=None):
        instance, plugin = self.get_plugin_instance()
        if instance and not (admin and not plugin.admin_preview):
            if not placeholder or not isinstance(placeholder, Placeholder):
                placeholder = instance.placeholder
            placeholder_slot = placeholder.slot
            current_app = context.current_app if context else None
            context = PluginContext(context, instance, placeholder, current_app=current_app)
            context = plugin.render(context, instance, placeholder_slot)
            request = context.get('request', None)
            page = None
            if request:
                page = request.current_page

Now correct me if I'm wrong, but that's a call to render_plugin() with context as a standard positional arg while render_plugin expects context as a kwarg, right?

If args are given respecting the order the parameter, it doesn't really matter if they are passed as positional args or keywords args


UICONSET-166: Compare Authorization Role capabilities in full screen view

https://folio-org.atlassian.net/browse/UICONSET-166
Here we added a compare roles process for authorization roles.
Note: Authorization roles were added just for
https://github.com/folio-org/ui-consortia-settings/assets/72550466/e916bce4-1bba-4b01-8f65-10eeaa9372f4
testing and showing how it will work. Real authorization-role will be added in the scope of other tickets.
Highlight will be added after emerging https://github.com/folio-org/stripes-authorization-components/pull/5

@UladzislauKutarkin  if another role is not select, no diff should be displayed

@UladzislauKutarkin if another role is not select, no diff should be displayed

In ticket said that when When user has selected a role  Then its capability sets and capabilities are displayed in the corresponding accordions below. Same behavior we have in Permissions sets compare in users setting

Thanks @BogdanDenis , local hooks useRoleCapabilities and useRoleCapabilitySets replaced from @folio/stripes-authorization-components. useOkapiKy in that hooks updated with tenant in https://github.com/folio-org/stripes-authorization-components/pull/5/files.


Update y4m dependency to 0.4

Coverage decreased (-0.004%) to 71.1% when pulling 972d11b4bb40dee5c7ebc80de3f29db5199c738a on shssoichiro:update-y4m into 39b93e6a7ea4f99361fd88ab997a96ee9bc7a6f2 on xiph:master.


Error during the deployment of the template

Description
Error during the deployment of the next templates:
Name                            coreos-1745.7.0-5gb
Size                              4.43 GB
Direct Download          No
extractable                  Yes
Password Enabled      No
Dynamically Scalable  No
Public                          Yes
Featured                     Yes
OS Type                      CoreOS
Cross Zones               Yes
Steps to reproduce
Create VM with  "coreos-1745.7.0-5gb" template.
Actual result

Error during deployment
Expected result

No errors during deployment
Environment

Browser:
- [ ] Chrome version XX
- [ ] Firefox version XX
- [ ] Safari version XX
- [ ] Opera version XX
- [ ] IE version XX
- [ ] Edge version XX

Others:

Connected feature ID

vm_deploy
Screenshots

The problem occurs because the application passes the calculated rootize parameter to the virtual machine creation request.
Native UI request

CloudStack UI request

Test on
tamazlykar/1149-root-disk-size
Regression:
vm_deploy

Tested on tamazlykar/1149-root-disk-size


Allow Manual Closing of DAQ Context Manager Outside of the Context

We need a way to manually close an AsyncContextManager outside of the context manager. For example, in a parent process that spawns data acquisition threads.
We recommended that a user store the context in parent memory and call .cancel() at a later time. This doesn't work for several reasons. First, DPMContext returns a DPM object, not a Context Manager and DPM doesn't implement a close method. Second, even if we change the API to return the manager as well, calling .aclose() or dpm._shutdown() results in RuntimeError: aclose(): asynchronous generator is already running.

I like to try. first I need to repeat this situation.

Thanks, Zongwei. I'm experimenting with something at the moment. Hopefully it's a solution.


Fixes race condition in test

@mroloux just a FYI, this is a non-functional change that fixes an issue potentially caused by the time precision changes in the backend


Concern about labelstudio tracking our usage

Describe the bug
This is not a bug, but im not sure where else to ask this.
We installed label-studio in our private cloud and we see this message:

"get latest news..."
This means that a message with an email is sent to your servers and is tracked. How can I know nothing else is sent from our private usage to your servers?
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):

OS: [e.g. iOS]
Label Studio Version [e.g. 0.8.0]

Additional context
Add any other context about the problem here.

Checkout this issue https://github.com/HumanSignal/label-studio/issues/4612
"Get latest news" is just the start, a pretty insane amount of logging goes on

Hello,
If you set collect_analytics=0 that stops collecting everything. Does this help?

Thanks for the response @AbubakarSaad. Is there a documentation about what other "analytics" is collected? I know emails are collected and viewed by a human because after we installed label studio privately for a poc, we suddenly got approached by you guys.
Very concerning...

Label Studio collected API usage data, but that can be disabled using the environment variable mentioned by @AbubakarSaad.
collect_analytics=0

That will be sufficient to disable all reported analytics.


Add github link to menu

Please review, never worked with yaml. If I haven't missed anything, this PR should add a webhint.io github repo link to the menu as suggested in https://github.com/webhintio/webhint.io/issues/491.

As this is my first PR:

[x] Signed the Contributor License Agreement

Thanks @Malvoz !
I modified a bit the CSS to hide also the arrow so it looks the same as the scanner link.
Changes should hopefully be live soon!

All your changes are now live!


Off-season events not shown

Perhaps this is a known issue but off-season events are not being shown at https://www.statbotics.io/events when I select Offseason Only (or anything else I select) from the drop-down.

Bump

They were shown for 2023 and earlier but not from 2024 onwards


🐛 Bug Report: GitLab Auth Provider Doesnt Work

📜 Description
Hi,
I try to integrate gitlab auth method, but I give some errors when I try to login. I did everything succesfully. I checked the gitlab application scopes and secret tokens. Everything is fine, still the ui give me that error; Login failed, popup was closed and Login failed, rejected by user
These are my settings from app-config.yaml
  providers:
    gitlab:
      development:
        clientId: 8a50c1ad1a5025857****971759150b8dcf2e7d
        clientSecret: 913172083f3d3*****b5032e68d745183fb95cda564a
        callbackUrl: https://backstage.***.com.tr/api/auth/gitlab/handler/frame?env=development

Gitlab Scopes:
api,
read_api (Read Api),
read_user (Read the authenticated user's personal information)
Version: 1.8.3
👍 Expected behavior
It should login with gitlab.
👎 Actual Behavior with Screenshots

👟 Reproduction steps

Go To Backstage
Try to login with GitLab

📃 Provide the context for the Bug.

[X] I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

[X] I have read the Code of Conduct

Are you willing to submit PR?
No, I don't have time to work on this right now

We are also observing the same behaviour. We are running 1.8.3 version.

Sorry to hear that you're having problems. Some additional context to help debug would be fantastic. Did this happen after the upgrade meaning that if you downgrade to your previous version everything works again? Would be good to know which version it was in that case.
Could you also try skipping the callbackUrl in your config and see if that has any effect as it can also be computed by the instance. Do you get the login popup?

Hi @jhaals ,
I tried your opinion but results are same. Also gitlab needs callbackurl otherwise it doesn't give app tokens.

@iskifogl please provide more complete steps for how to reproduce this issue and then we'll be able to have a look.
I'll also note that using a clone of the project to deploy Backstage isn't something that we support, please set up a separate project with npx @backstage/create-app instead. Though if you're running unmodified code from the main repo and running into issues then that's something we can help out with.

Doesn't seem to work for us either:
app-config.yaml
app:
  title: Backstage
  baseUrl: http://localhost:3000

organization:
  name: Custom Organization

backend:
  # Used for enabling authentication, secret is shared by all backend plugins
  # See https://backstage.io/docs/tutorials/backend-to-backend-auth for
  # information on the format
  # auth:
  #   keys:
  #     - secret: ${BACKEND_SECRET}
  baseUrl: http://localhost:7007
  listen:
    port: 7007
    # Uncomment the following host directive to bind to specific interfaces
    # host: <IP_ADDRESS>
  csp:
    connect-src: ["'self'", 'http:', 'https:']
    # Content-Security-Policy directives follow the Helmet format: https://helmetjs.github.io/#reference
    # Default Helmet Content-Security-Policy values can be removed by setting the key to false
  cors:
    origin: http://localhost:3000
    methods: [GET, HEAD, PATCH, POST, PUT, DELETE]
    credentials: true
  # This is for local development only, it is not recommended to use this in production
  # The production database configuration is stored in app-config.production.yaml
  database:
    client: better-sqlite3
    connection: ':memory:'
  cache:
    store: memory
  # workingDirectory: /tmp # Use this to configure a working directory for the scaffolder, defaults to the OS temp-dir

integrations:
  github:
    - host: github.com
      # This is a Personal Access Token or PAT from GitHub. You can find out how to generate this token, and more information
      # about setting up the GitHub integration here: https://backstage.io/docs/getting-started/configuration#setting-up-a-github-integration
      token: ${GITHUB_TOKEN}
    ### Example for how to add your GitHub Enterprise instance using the API:
    # - host: ghe.example.net
    #   apiBaseUrl: https://ghe.example.net/api/v3
    #   token: ${GHE_TOKEN}

proxy:
  ### Example for how to add a proxy endpoint for the frontend.
  ### A typical reason to do this is to handle HTTPS and CORS for internal services.
  # '/test':
  #   target: 'https://example.com'
  #   changeOrigin: true

# Reference documentation http://backstage.io/docs/features/techdocs/configuration
# Note: After experimenting with basic setup, use CI/CD to generate docs
# and an external cloud storage when deploying TechDocs for production use-case.
# https://backstage.io/docs/features/techdocs/how-to-guides#how-to-migrate-from-techdocs-basic-to-recommended-deployment-approach
techdocs:
  builder: 'local' # Alternatives - 'external'
  generator:
    runIn: 'docker' # Alternatives - 'local'
  publisher:
    type: 'local' # Alternatives - 'googleGcs' or 'awsS3'. Read documentation for using alternatives.

auth:
  # see https://backstage.io/docs/auth/ to learn about auth providers
  providers: {}

scaffolder:
  # see https://backstage.io/docs/features/software-templates/configuration for software template options

catalog:
  import:
    entityFilename: catalog-info.yaml
    pullRequestBranchName: backstage-integration
  rules:
    - allow: [Component, System, API, Resource, Location]
  locations:
    # Local example data, file locations are relative to the backend process, typically `packages/backend`
    - type: file
      target: ../../examples/entities.yaml

    # Local example template
    - type: file
      target: ../../examples/template/template.yaml
      rules:
        - allow: [Template]

    # Local example organizational data
    - type: file
      target: ../../examples/org.yaml
      rules:
        - allow: [User, Group]

    ## Uncomment these lines to add more example data
    # - type: url
    #   target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/all.yaml

    ## Uncomment these lines to add an example org
    # - type: url
    #   target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/acme-corp.yaml
    #   rules:
    #     - allow: [User, Group]

app-config.local.yaml
auth:
  environment: development
  providers:
    gitlab:
      development:
        clientId: <redacted>
        clientSecret: <redacted>
        audience: https://git.<redacted>.com

yarn backstage-cli info:
OS:   Linux 5.15.0-53-generic - linux/x64
node: v16.19.0
yarn: 1.22.19
cli:  0.22.1 (installed)
backstage:  1.10.0

Dependencies:
  @backstage/app-defaults                          1.1.0
  @backstage/backend-app-api                       0.3.1
  @backstage/backend-common                        0.18.1
  @backstage/backend-plugin-api                    0.3.1
  @backstage/backend-tasks                         0.4.2
  @backstage/catalog-client                        1.3.0
  @backstage/catalog-model                         1.1.5
  @backstage/cli-common                            0.1.11
  @backstage/cli                                   0.22.1
  @backstage/config-loader                         1.1.8
  @backstage/config                                1.0.6
  @backstage/core-app-api                          1.4.0
  @backstage/core-components                       0.12.3
  @backstage/core-plugin-api                       1.3.0
  @backstage/errors                                1.1.4
  @backstage/integration-aws-node                  0.1.1
  @backstage/integration-react                     1.1.9
  @backstage/integration                           1.4.2
  @backstage/plugin-api-docs                       0.8.14
  @backstage/plugin-app-backend                    0.3.41
  @backstage/plugin-auth-backend                   0.17.4
  @backstage/plugin-auth-node                      0.2.10
  @backstage/plugin-catalog-backend                1.7.1
  @backstage/plugin-catalog-common                 1.0.10
  @backstage/plugin-catalog-graph                  0.2.26
  @backstage/plugin-catalog-import                 0.9.4
  @backstage/plugin-catalog-node                   1.3.2
  @backstage/plugin-catalog-react                  1.2.4
  @backstage/plugin-catalog                        1.7.2
  @backstage/plugin-github-actions                 0.5.14
  @backstage/plugin-org                            0.6.4
  @backstage/plugin-permission-common              0.7.3
  @backstage/plugin-permission-node                0.7.4
  @backstage/plugin-permission-react               0.4.9
  @backstage/plugin-proxy-backend                  0.2.35
  @backstage/plugin-scaffolder-backend             1.10.1
  @backstage/plugin-scaffolder-common              1.2.4
  @backstage/plugin-scaffolder-react               1.0.1
  @backstage/plugin-scaffolder                     1.10.1
  @backstage/plugin-search-backend-module-pg       0.5.2
  @backstage/plugin-search-backend-node            1.1.2
  @backstage/plugin-search-backend                 1.2.2
  @backstage/plugin-search-common                  1.2.1
  @backstage/plugin-search-react                   1.4.0
  @backstage/plugin-search                         1.0.7
  @backstage/plugin-tech-radar                     0.6.0
  @backstage/plugin-techdocs-backend               1.5.2
  @backstage/plugin-techdocs-module-addons-contrib 1.0.9
  @backstage/plugin-techdocs-node                  1.4.5
  @backstage/plugin-techdocs-react                 1.1.2
  @backstage/plugin-techdocs                       1.4.3
  @backstage/plugin-user-settings                  0.6.2
  @backstage/release-manifests                     0.0.8
  @backstage/test-utils                            1.2.4
  @backstage/theme                                 0.2.16
  @backstage/types                                 1.0.2
  @backstage/version-bridge                        1.0.3

GitLab Version: 15.7.5-ee
Backstage as an Application created as follows:

Redirect URI: http://localhost:7007/api/auth/gitlab/handler/frame
Confidential: false
Scopes: read_api & read_user

Error: Authentication failed, Failed to obtain access token

Let me know if I can provide any more information.

Hi @iskifogl and @adberger
I am facing same issue Error: Authentication failed, Failed to obtain access token
Can you guys please let me know here if you got it resolved or any workaround

Hi @jhaals facing same issue any updates on this is appreciated

@shahnawaz90 not resolved and no workaround found, sorry mate.

I don't see any issue with public gitlab. It is working as expected but with community edition Gitlab i am having this issue.
Just for your ref: @jhaals @adberger

For self-hosted gitlab version, ensure there is valid CA for gitlab instance or set NODE_TLS_REJECT_UNAUTHORIZED=0 to ignore self-cert error.

For self-hosted gitlab version, ensure there is valid CA for gitlab instance or set NODE_TLS_REJECT_UNAUTHORIZED=0 to ignore self-cert error.

FWIW NODE_EXTRA_CA_CERTS can be a slightly more secure alternative to the blanket NODE_TLS_REJECT_UNAUTHORIZED=0 option.

Hi @iskifogl  Where you able to fix this issue or find any workaround ?
I am also stuck with the same issue regarding Gitlab Authentication

@shahnawaz90 I am running into the same issue as you with Error: Authentication failed, Failed to obtain access token.
Anyone suggestions? My gitlab and backstage have valid certs.

It would be great if we can get some more information about why this is failing exactly. I think in the network tab there should be some more information in the response from gitlab itself possibly if you get a popup window?

I got the same issue, any details ?


Questions about linter quoting rules

I have a few questions about linting:

can we relax the | quote requirement to also allow | squote? The escape rules are different.
If we force an int to be placed in quotes, is it evaluated as a string? That could cause problems with arithmetic operations.

can we relax the | quote requirement to also allow | squote? The escape rules are different.

In theory this is supported but it is not listed in the test examples.

If we force an int to be placed in quotes, is it evaluated as a string? That could cause problems with arithmetic operations.

I have not looked into this yet which I think it is an important concern. Should I create another issue to look into it?