iOS 14: requestSubscription() issues

Version of react-native-iap
4.4.9
Version of react-native
.62.2
Platforms you faced the error (IOS or Android or both?)
iOS only.  Not testing on Android yet.

Actual behavior
call requestSubscription().  No purchase dialogs.  PurchaseUpdatedListener called immediately with no purchase dialogs or valid receipt.
Tested environment (Emulator? Real Device?)
Real iOS 14 device using TestFlight.
Steps to reproduce the behavior
call RNIap.requestSubscription().
Notes:

XCode 12
using Sign In with Apple Id.
testing in sandbox with real Apple id (not test user)
works fine on iPad running iOS 13.7.
have applied iOS 14 promise fix.

Same here. Any one can helps?

Not hanging, but returns []

For us the purchase dialog shows, and the purchase listener gets called correctly, but the requestSubscription call returns a promise that is never resolved.

Seems like this might have been resolved here https://github.com/dooboolab/react-native-iap/pull/1064

I applied that promise fix previously, so that's not the problem here.

Here's how I did this:
 // request purchase, listener registered above will receive notification when processing done         RNIap.requestSubscription(this.productIds[0]).catch(err => {             console.log(err.code, err.message);         });

I was calling the method with the boolean parameter false, which I think is the recommendation.  Either way I am still seeing very strange behavior on iOS14 no matter how I call the method, and it works fine on an iPad running iOS 13.7.  I tested requestSubscription with with and without the boolean and got two different behaviors, seemingly random.  But I am never getting the purchase dialogs.  Sometimes it never calls the RequestUpdatedListener.  Sometimes it calls it immediately with what looks like an invalid receipt.
Are there others who are having no problems with their subscriptions?

Hey, do you any of you use getPurchaseHistory() or getAvailablePurchases() just before requestSubscription() if so, this seem to be the problem on iOS14.

I do getAvailablePurchases() earlier in the flow - on app load, the screen right before my Subscription Screen.

yeah, unfortunately this seem to be a bug in iOS 14. No fault of this plugin. I've checked in native code without any RN or this plugin and it is the same issue.

That was the problem.  I removed the call to getAvailablePurchases() and the purchase popups occurred and purchase was successful. Regrettably, I need to make that call to getAvailablePurchases, since I check the status of their subscription on App Load.  So looks like we need to wait for a iOS 14 patch to fix this?  Nothing else anyone can do?  By the way, thank you so much theDev23!

You're welcome, the-dut. Your best bet is probably relying on a server-based check.

I was hoping to avoid having a server do this.  Is it as easy as copying the code from my react into something like a Firebase Cloud Function?  Or is this something that needs to be re-architectured?  Probably a dumb question, but hey.. we all started out asking dumb questions.  :)

@the-dut You can use iaphub to take care of the server receipt validation

Beverage of your choice if you're ever in the Midwest US.  :)

Same issue here affecting live users. @theDev23 do you happen to have a link any discussions about the issue in iOS? Are there any timelines for a fix?

@nicknjpconsultingllc no link, sorry. I did the digging myself.
@the-dut I am using Firebase Cloud Functions as well. Check out this repo.

I ended up saving the receipt to a database on successful purchase in the purchaseUpdatedListener, then on app load a pull the receipt from the database, validate it again and check to see if it's expired or not.  That allowed me to get rid of the call to getAvailablePurchases() is it works great in iOS14.  It's worked really well so far and is much faster.  Now a question on how far I need to take this.  If this is all I do, will the UpdatedPurchaseListener get called on all updates to the subscription (basically the auto-renewals) until I finish the transactions?  Or do I need to implement the server-to-server notifications to get informed of the auto-renewals?    It almost seems like using the UpdatedPurchaseListener is a better (and simpler) solution since I have been reading the server-to-server integration isn't perfect.

Still getting this issue after removing getPurchaseHistory and applying the promise fix. Can't get to make the purchase dialog pop up. Can anybody help?

I'd say take a look at any/all calls you are making to AIP before doing the request to purchase.  Are you calling getAvailablePurchases() as well?

I can't exclude the calls to getAvailablePurchases() and getPurchaseHistory(), is there anyway to make it requestSubscription() pop up the purchase menu with these calls?

Hi @HamzaIkram2727 , if I am not wrong, you can just create a new sandbox account to use and delete the old one. It should works.

@the-dut Even I am not using any getAvailablePurchases() and getPurchaseHistory() calls, still i am facing same problem. It does not show purchase dialog but purchased sucess automatically.
I am also calling finishTrsanactionIOS and finishTransation functions

I managed to get the iOS native purchase dialog by exiting the app. This looks to me like some promises finally resolved. I'm not sure if there's any promise dependencies in react-native-iap but the behavior seems weird. Calling requestSubscription resolves in the app but there's no dialog until I exit the app.

I'm having the same problem with getAvailblePruchases(), most of the times the promise does not resolve or reject. It started happening when testing on iOS 14 devices.

I'm having the same problem

I call getAvailablePurchases()
requestSubscription()

I removed getAvailablePurchase and apparently it works fine

Any updates?

Really weird behaviour and for what I'm noticing it is not only on the ios 14 that it happens... I'm contacting apple to try to understand what's going on coz I don't think it's a problem related with the package it self.

Here just to say that I'm facing the same problem. I already tested on iOS 12 and it's working nicely... iOS 14 is buggy.
My call of requestSubscription just last forever... And it's bugging in production!

Well, after a day of "having fun" this is what I've concluded so far... I have various devices on different OS versions and I have this problem on each one of them. I've contacted Apple in regards to this and we're still following up the case, however, without any solid conclusions. I do not believe it is a problem with the package, it must be something on the Apple's side. I've tried everything people are mentioning on this thread and in other threads that have reported a similar problem. The In App Purchase dialog is just inconsistent on showing up and until now, I cannot find a valid reason for it because the outcome are always different. Sometimes I get an E_UNKNOWN error, other times I get zero feedback, sometimes the pop up shows after 10 mins of waiting and other times it just works... I've tried with sandbox users, non sandbox users, I've had the same different outcomes for the same user and device and not even an error or a conclusive/insightful log is shown. This is just frustrating to debug because you can't find pretty much anything... Well, if you guys have any sort of advice or insight, I'm listening ... In the meantime, if I have some news from Apple, I'll keep you posted...

We have tested our app package with an App Store approved build and it looks like the inconsistency comes more prominently on iOS version 14+
Our conclusion is that the way Apple verifies the receipt on iOS 14+ is a bit different.

Create a new account in Sandbox
Make a purchase (it works fine because the restorePurchases has no purchases)
Wait until it expires.
Buy again. (Now the restore purchases responds an object and the popup to confirm the purchase no longer works)

It's weird. I was reviewing library code and I added some logs for test.  And i got the following

When restorePurchases is called. It run the listener "paymentQueueRestoreCompletedTransactionsFinished"

`-(void)paymentQueueRestoreCompletedTransactionsFinished:(SKPaymentQueue )queue {  ////////   RESTORE
NSLog(@"\n\n\n  paymentQueueRestoreCompletedTransactionsFinished  \n\n.");
NSMutableArray items = [NSMutableArray arrayWithCapacity:queue.transactions.count];
NSLog(@"Number of items in my array is: %d", [queue.transactions count]);//this will return (5)
for(SKPaymentTransaction *transaction in queue.transactions) {
    if(transaction.transactionState == SKPaymentTransactionStateRestored
       || transaction.transactionState == SKPaymentTransactionStatePurchased) {
        [self getPurchaseData:transaction withBlock:^(NSDictionary *restored) {
            [items addObject:restored];
            [[SKPaymentQueue defaultQueue] finishTransaction:transaction];
        }];
    }
}
NSLog(@"\n  finished paymentQueueRestoreCompletedTransactionsFinished ");

[self resolvePromisesForKey:@"availableItems" value:items];

}`
Here I print in log the transactions size and I got 23 items

The function paymentQueue is executed to finish the restore transactions

`case SKPaymentTransactionStateRestored: {
NSLog(@"Restored ");
     NSLog(@"\n  paymentQueue transaction : %@", transaction.transactionDate);
     [[SKPaymentQueue defaultQueue] finishTransaction:transaction];
       break;

}`

resolvePromisesForKey complete the restorePurchase

requestSubscription start calling the function buyProduct

Here I added logs for see the log

` NSLog(@"\n  produc purchase : %@", product);
SKMutablePayment *payment = [SKMutablePayment paymentWithProduct:product];
NSLog(@"\n  payment buy product : %@", payment.productIdentifier);
[[SKPaymentQueue defaultQueue] addPayment:payment];`
6.I got the following log:
a) RestorePurchases log
`2020-10-22 14:01:49.319677-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.319784-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 16:46:53 2020
2020-10-22 14:01:49.319930-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.319970-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 19:16:45 2020
2020-10-22 14:01:49.320044-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.320082-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 19:06:37 2020
2020-10-22 14:01:49.320155-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.320275-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 19:01:37 2020
2020-10-22 14:01:49.320421-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.320532-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 07:49:31 2020
2020-10-22 14:01:49.320671-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.320776-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 14:21:39 2020
2020-10-22 14:01:49.320915-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.321018-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 14:26:39 2020
2020-10-22 14:01:49.321180-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.321282-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 16:56:53 2020
2020-10-22 14:01:49.321414-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.321516-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 16:51:53 2020
2020-10-22 14:01:49.321648-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.321749-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 08:04:31 2020
2020-10-22 14:01:49.321879-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.321984-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 19:11:41 2020
2020-10-22 14:01:49.322113-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.322214-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 16:41:53 2020
2020-10-22 14:01:49.322346-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.322444-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 14:31:39 2020
2020-10-22 14:01:49.323955-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.324097-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 12:07:14 2020
2020-10-22 14:01:49.324240-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.324344-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 14:16:39 2020
2020-10-22 14:01:49.324474-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.324579-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 14:11:39 2020
2020-10-22 14:01:49.324709-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.324808-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 07:39:31 2020
2020-10-22 14:01:49.324934-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.325030-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 07:59:31 2020
2020-10-22 14:01:49.325163-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.325260-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 19:26:45 2020
2020-10-22 14:01:49.327079-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.327212-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 07:44:31 2020
2020-10-22 14:01:49.327360-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.327465-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 07:54:31 2020
2020-10-22 14:01:49.327594-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.327695-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 14:37:11 2020
2020-10-22 14:01:49.327823-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:49.327923-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 19:21:45 2020
2020-10-22 14:01:49.328099-0500 ApplicationTest[532:51039]
paymentQueueRestoreCompletedTransactionsFinished
2020-10-22 14:01:49.328204-0500 ApplicationTest[532:51039] Number of items in my array is: 23
2020-10-22 14:01:49.336873-0500 ApplicationTest[532:51039]
finished paymentQueueRestoreCompletedTransactionsFinished
2020-10-22 14:01:50.317721-0500 ApplicationTest[532:51234] `
b. buySubcription log:
2020-10-22 14:01:50.319607-0500 ApplicationTest[532:51234]
produc purchase : <SKProduct: 0x2839c0630>
2020-10-22 14:01:50.319694-0500 ApplicationTest[532:51234]
payment buy product : test.applicationtest
2020-10-22 14:01:50.319849-0500 ApplicationTest[532:51039] Restored
2020-10-22 14:01:50.319895-0500 ApplicationTest[532:51039]
paymentQueue transaction : Mon Oct  5 19:16:45 2020
So I don't understand why the paymentQueue method is executed with a SKPaymentTransactionStateRestored dated October 5

@andresordonezfm Have you called finishTransaction after the purchase?

Also regarding the pop-ups, here are great trials given in #863. Hope it helps.

@andresordonezfm Have you called finishTransaction after the purchase?

Sure! also I called clearTransactionIOS() anyway if it could be the problem but nothing.

Also regarding the pop-ups, here are great trials given in #863. Hope it helps.

Thank you. I gonna review-it

Any solutions/updates?

Nothing yet. I was reviewing and Apple added new features in StoreKit and Notification Types..
I'm not sure if that could be the problem
https://developer.apple.com/videos/play/wwdc2020/10661/

check if are calling
await RNIap.initConnection();
before requestSubscription()

check if you're calling
await RNIap.initConnection();
before RNIap.requestSubscription

Seems I solved following this issue:
https://github.com/dooboolab/react-native-iap/issues/1146
First of all, when I was validate the receipt I was passing true as isTest parameter (even when was a production build).
Then I added RNIap.clearTransactionIOS(); in the purchaseUpdatedListener before validating the receipt and  RNIap.finishTransaction(purchase, false) after RNIap.finishTransactionIOS(purchase.transactionId);
I tested in TestFlight and I could complete a purchase for the first time (with a sandbox account), now I going to test it in production 🤞

Seems I solved following this issue:
#1146
First of all, when I was validate the receipt I was passing true as isTest parameter (even when was a production build).
Then I added RNIap.clearTransactionIOS(); in the purchaseUpdatedListener before validating the receipt and RNIap.finishTransaction(purchase, false) after RNIap.finishTransactionIOS(purchase.transactionId);
I tested in TestFlight and I could complete a purchase for the first time (with a sandbox account), now I going to test it in production 🤞

Hi dude I tried this solution but not worked for me.

I'm running into the same issue. It happens when you try purchasing subscription after restoring purchase.
I can confirm logs from @andresordonezfm , where it seems that SKPaymentQueue receives transaction of the SKPaymentTransactionStateRestored type , instead of the SKPaymentTransactionStatePurchasing type, which is the reason for the "Restore" string being logged on purchase.

Hi Guys sorry for the delay.
I was reviewing and I found a solution for this problem. The problem is that before calling the restore purchases, the newly opened transactions of the same restore purchases do not close properly. Then I forcibly close them.
Then the clearTransactionIOS function of the library, the promise must be added to wait for the transactions to be closed using the function removedTransactions
Here is my code:
File: react-native-iap/ios/RNIapIos.m

Replace the function RCT_EXPORT_METHOD(clearTransaction) with the following code

RCT_EXPORT_METHOD(clearTransaction:(RCTPromiseResolveBlock)resolve
                  reject:(RCTPromiseRejectBlock)reject) {

    NSLog(@"\n\n\n  ***  clear remaining Transactions. Call this before make a new transaction   \n\n.");

    NSArray *pendingTrans = [[SKPaymentQueue defaultQueue] transactions];
    countPendingTransaction = (NSInteger)(pendingTrans.count);

    if (countPendingTransaction > 0) {
        [self addPromiseForKey:@"cleaningTransactions" resolve:resolve reject:reject];

        for (SKPaymentTransaction *transaction in pendingTrans) {
            [[SKPaymentQueue defaultQueue] finishTransaction:transaction];
        }

    } else {
        resolve(nil);
    }
}

Add the following new function (I added this function at the end of the RNIapIos.m file. Before the line "@end" )

-(void)paymentQueue:(SKPaymentQueue *)queue removedTransactions:(NSArray *)transactions {
    NSLog(@"removedTransactions");
    if (countPendingTransaction != nil && countPendingTransaction > 0) {
        countPendingTransaction--;
        if (countPendingTransaction == 0) {
            [self resolvePromisesForKey:@"cleaningTransactions" value:nil];
            countPendingTransaction = nil;
        }
    }
}

File: react-native-iap/ios/RNIapIos.h

Under the code "SKProduct * promotedProduct;" add the following line

  NSInteger countPendingTransaction;

Finally before calling the "RNIap.requestSubscription (productId, false)" function.
The clearTransactionIOS function must be called, for example:
    if (Platform.OS === 'ios') {
      await RNIap.clearTransactionIOS();
    }
    await RNIap.requestSubscription(productId, false);

I hope it helps. The strange behavior of restore purchases I have not been able to find the reason. But this solved for me testing in Sandbox.
If you have any suggestions please let me know.

I also have this problem，please author,help me ,thank your family

I also have this problem，please author,help me ,thank your family

Hi dude, In the previous comment, the steps I did to solve it are explained
If you have any comments, let me know

@andresordonezfm would you be willing to create a PR for these changes?

@andresordonezfm would you be willing to create a PR for these changes?

For the issue of purchases coming as SKPaymentTransactionStateRestored  that happens after restoring, I created a new account and made a few purchases and it's not happening for that account. I'm able to call RNIap.getAvailablePurchases() and RNIap.requestSubscription() afterwards.
It seems that it may be related with the amount of purchases a user has? I haven't run any tests on production yet.

Well, after a day of "having fun" this is what I've concluded so far... I have various devices on different OS versions and I have this problem on each one of them. I've contacted Apple in regards to this and we're still following up the case, however, without any solid conclusions. I do not believe it is a problem with the package, it must be something on the Apple's side. I've tried everything people are mentioning on this thread and in other threads that have reported a similar problem. The In App Purchase dialog is just inconsistent on showing up and until now, I cannot find a valid reason for it because the outcome are always different. Sometimes I get an E_UNKNOWN error, other times I get zero feedback, sometimes the pop up shows after 10 mins of waiting and other times it just works... I've tried with sandbox users, non sandbox users, I've had the same different outcomes for the same user and device and not even an error or a conclusive/insightful log is shown. This is just frustrating to debug because you can't find pretty much anything... Well, if you guys have any sort of advice or insight, I'm listening ... In the meantime, if I have some news from Apple, I'll keep you posted...
P.S EDIT. after reading the last comment, one thing is consistent and I can confirm it, on iOS 14 the thing seems more inconsistent even though it happens on every OS version that I've tested this (12, 13.7, 14). However, I'm still with the impression that the problem is not solely related the OS version and that's why I'm thinking that it must be something with the Sandbox server or any other thing in the Apple's end ..

Exactly what I'm experiencing on my end, even after the fix suggested by @andresordonezfm (https://github.com/dooboolab/react-native-iap/issues/1120#issuecomment-734805587). @106firestarter, did Apple ever reply?
I'll see what I can find in the developer forums meanwhile.

For the issue of purchases coming as SKPaymentTransactionStateRestored  that happens after restoring, I created a new account and made a few purchases and it's not happening for that account. I'm able to call RNIap.getAvailablePurchases() and RNIap.requestSubscription() afterwards.
It seems that it may be related with the amount of purchases a user has? I haven't run any tests on production yet.

I'll try that, but production is a concern as well, indeed.

Possibly related:

https://developer.apple.com/forums/thread/669435

I can confirm as well that with a brand new account the issue is no longer as @Paduado stated (https://github.com/dooboolab/react-native-iap/issues/1120#issuecomment-742685043).

@andresordonezfm would you be willing to create a PR for these changes?

Hi, dude sorry for the delay again. I created a pull request to react-native iap

~I can confirm as well that with a brand new account the issue is no longer as @Paduado stated (#1120 (comment)).~
Correction: using a brand new account together with the fix suggested by @andresordonezfm makes it work on my end. Without the fix, after the very first subscription, the same behavior I was experiencing before emerges again and I can no longer subscribe again. One more thing I noticed after applying the fix is that subscriptions appear to no longer auto-renew automatically, not even once, although using sandbox they are expected to renew 5 times.

Hi dude I think it is a sandbox problem. Do you keep giving that behavior?

@andresordonezfm yes, I suspect it is an issue with Apple's sandbox too. I didn't perform any more tests since my last comment and I'm yet check production.

The sandbox appears to be back on track on my end when using a brand new account, as previous reported by @Paduado, without any making any changes to the code. 🤷♂ 😕

The sandbox appears to be back on track on my end when using a brand new account, as previous reported by @Paduado, without any making any changes to the code. 🤷♂ 😕

@andresordonezfm I encountered the same thing and your patch seams to fix it. However, I think I see subtle bug in your code that could potentially cause cleartTransactionsIOS() to hang. removeTransactions may be called with 1 or more transactions to be removed. So in removeTransactions, I think we want:
        countPendingTransaction -= [transactions count];

In practice, I've only seen removedTransactions called with one transaction, but apple docs say it could be one or more. Nice find by the way! I was stuck on this for a while.

@andresordonezfm I encountered the same thing and your patch seams to fix it. However, I think I see subtle bug in your code that could potentially cause cleartTransactionsIOS() to hang. removeTransactions may be called with 1 or more transactions to be removed. So in removeTransactions, I think we want:
        countPendingTransaction -= [transactions count];

In practice, I've only seen removedTransactions called with one transaction, but apple docs say it could be one or more. Nice find by the way! I was stuck on this for a while.

@andresordonezfm I encountered the same thing and your patch seams to fix it. However, I think I see subtle bug in your code that could potentially cause cleartTransactionsIOS() to hang. removeTransactions may be called with 1 or more transactions to be removed. So in removeTransactions, I think we want:
        countPendingTransaction -= [transactions count];

In practice, I've only seen removedTransactions called with one transaction, but apple docs say it could be one or more. Nice find by the way! I was stuck on this for a while.

Hi dude, thank you for you recommendation. I tested your propose and it works well for me. I made a pull request with that change.
Note: Yes, I reviewed that documentation but since it only is fired from transaction to transaction I did not implement it. But it is a very good recommendation. Thank you!

@andresordonezfm I encountered the same thing and your patch seams to fix it. However, I think I see subtle bug in your code that could potentially cause cleartTransactionsIOS() to hang. removeTransactions may be called with 1 or more transactions to be removed. So in removeTransactions, I think we want:
        countPendingTransaction -= [transactions count];

In practice, I've only seen removedTransactions called with one transaction, but apple docs say it could be one or more. Nice find by the way! I was stuck on this for a while.

Hi dude, thank you for you recommendation. I tested your propose and it works well for me. I made a pull request with that change.
Note: Yes, I reviewed that documentation but since it only is fired from transaction to transaction I did not implement it. But it is a very good recommendation. Thank you!

Hey, do you any of you use getPurchaseHistory() or getAvailablePurchases() just before requestSubscription()? if so, this seem to be the problem on iOS 14.

I use getPurchaseHistory(). when i use this, no purchase dialogs in requestSubscription.

Hey, do you any of you use getPurchaseHistory() or getAvailablePurchases() just before requestSubscription()? if so, this seem to be the problem on iOS 14.

I use getPurchaseHistory(). when i use this, no purchase dialogs in requestSubscription.

I find that there's no problem, it's just extremely slow, like 5 - 10mins to init the iap, or get the history or clear transaction, the first time purchase is ok, after the first time purchase, dont know why it's so slow

Hi Guys sorry for the delay.
I was reviewing and I found a solution for this problem. The problem is that before calling the restore purchases, the newly opened transactions of the same restore purchases do not close properly. Then I forcibly close them.
Then the clearTransactionIOS function of the library, the promise must be added to wait for the transactions to be closed using the function removedTransactions
Here is my code:
File: react-native-iap/ios/RNIapIos.m

Replace the function RCT_EXPORT_METHOD(clearTransaction) with the following code

RCT_EXPORT_METHOD(clearTransaction:(RCTPromiseResolveBlock)resolve
                  reject:(RCTPromiseRejectBlock)reject) {

    NSLog(@"\n\n\n  ***  clear remaining Transactions. Call this before make a new transaction   \n\n.");

    NSArray *pendingTrans = [[SKPaymentQueue defaultQueue] transactions];
    countPendingTransaction = (NSInteger)(pendingTrans.count);

    if (countPendingTransaction > 0) {
        [self addPromiseForKey:@"cleaningTransactions" resolve:resolve reject:reject];

        for (SKPaymentTransaction *transaction in pendingTrans) {
            [[SKPaymentQueue defaultQueue] finishTransaction:transaction];
        }

    } else {
        resolve(nil);
    }
}

Add the following new function (I added this function at the end of the RNIapIos.m file. Before the line "@EnD" )

-(void)paymentQueue:(SKPaymentQueue *)queue removedTransactions:(NSArray *)transactions {
    NSLog(@"removedTransactions");
    if (countPendingTransaction != nil && countPendingTransaction > 0) {
        countPendingTransaction--;
        if (countPendingTransaction == 0) {
            [self resolvePromisesForKey:@"cleaningTransactions" value:nil];
            countPendingTransaction = nil;
        }
    }
}

File: react-native-iap/ios/RNIapIos.h

Under the code "SKProduct * promotedProduct;" add the following line

  NSInteger countPendingTransaction;

Finally before calling the "RNIap.requestSubscription (productId, false)" function.
The clearTransactionIOS function must be called, for example:
    if (Platform.OS === 'ios') {
      await RNIap.clearTransactionIOS();
    }
    await RNIap.requestSubscription(productId, false);

I hope it helps. The strange behavior of restore purchases I have not been able to find the reason. But this solved for me testing in Sandbox.
If you have any suggestions please let me know.

Working solution,First and Second steps code was already in my llibrary.
Only adding one line before requestSubscription -
if (Platform.OS === 'ios') {       await RNIap.clearTransactionIOS();  }
works for me. Thanks a ton

Fixed by adding 5 second delay before requestSubscription


Closure

Hello,
Thank you for publishing this interesting work. I can't wait to evaluate it on my use-cases. I have investigated main.py in this repository; yet, the closure doesn't calculate the loss again in there; as opposed to the following example from PyTorch website.
for input, target in dataset:
    def closure():
        optimizer.zero_grad()
        output = model(input)
        loss = loss_fn(output, target)
        loss.backward()
        return loss
    optimizer.step(closure)

My question is, if closure won't recalculate the loss, why it is necessary for AdamS? Which usage is proper for AdamS, the one from PyTorch website or the one in main.py? which is simply as follows:
        def closure(): return loss
        optimizer.zero_grad()
        loss.backward()
        optimizer.step(closure)

It also seems that I need to re-hypertune the learning rate for using AdamS, as the optimizer converges much slower when I experimented with that as a drop-in replacement for Adam.

Thank you for the question!
Unlike Conjugate Gradient/LGBFS, the AdamX (X=UCB/CB/S) optimizers do not re-evaluate the loss multiple times per update.
Only the current loss scalar value is necessary to estimate the running means and the variances. Therefore, closure is just used to pass the loss value for the current state of the network. So, it is one forward pass for one update.
Therefore, you can use either of the versions above.
We used the latter one as that'd keep loss and output tensors in the function scope for further use.

You can also try keeping the same learning rate you used to have in the original Adam but tune on the variance hyperparameter eta for AdamS for best convergence by quickly running it for ~3 epochs and picking the best one.

Thank you for your response. I have tried the values in the paper for the
eta value, will later try hypertuning it.
On Thu 6. Jun 2019 at 18:26, Vineeth S. Bhaskara<EMAIL_ADDRESS>wrote:

You can also try keeping the same learning rate you used to have in the
original Adam but tune on the variance hyperparameter eta for AdamS for
best convergence by quickly running it for ~3 epochs and picking the best
one.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/bsvineethiitg/adams/issues/1?email_source=notifications&email_token=ALF7SPZ5IXU2ZZNRMDUCM7LPZE3C5A5CNFSM4HVC736KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXDNBKQ#issuecomment-499568810,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALF7SP3ZZGDS6MELGXLHBFLPZE3C5ANCNFSM4HVC736A
.

FYI, I am basically trying to employ it in the following Deep RL challenge I have created.
https://github.com/kayuksel/forex-rl-challenge
The repository also includes a gradient-free Augmented Random Search implementation,
that seems to be much more successful than Adam version in converging to better policies.


Sidebar Body Displacement Logic

Tina's Sidebar uses some pretty simple code to displace the website, and it has a number of issues.

It assumes the user is using box-sizing: border-box;, but that's not always true.
It takes control of the body element's left padding and transition properties, which may be in use already.
It doesn't actually change the size of the viewport, so responsive elements might not work as expected.

Example of 1:

@jhuggett: "the sidebar is pushing my content off the screen."

I think this is now closed since the iframe doesn't displace the user's window at all.


extend the search bar to occupy 55% of the whole width

Expand this to have a width of 55%. Keep the max-width of 1087px


How to find the files without licenses/copyright using this tool? is there a flag for that?

Description

I want to find out the files without licenses/copyright content using scancode. How can I do that? Also if I want to put a filter exclusively for finding images inside the source code repo, is it possible?

System configuration

For bug reports, it really helps us to know:

Linux
What installation method was used to install/run scancode? source

@vvscloud this is an example screen shot from scancode.io

You can see how the files without licenses and without copyright are shown in the pie charts.
And clicking it leads to the list:

The same data is available through the API for integration in your workflow.

A new "Resources detected license expressions" panel was added in the project details view.
You click on the "no licenses" entry to view the list of files without licenses.

Alternatively, you can use the "Detected license expression" filter in the resources list view.


GridLayout broken

Noticed by @gsteckman:
The GridLayout isn't working with the latest version updates. I am getting the runtime error in the javascript console "Uncaught TypeError: elements.iterator_0_k$ is not a function" when going to the Grid Layout view in the showcase. I traced it down to the code in Styling.kt lines 36-38 (if you comment that out, the error stops, but of course then it doesn't apply all the styling). I tried making a small test to reproduce the error and isolate the issue, but have not been able to reproduce it yet. It's probably something with the compose or javascript compilers since the kotlin compiles ok. It looks like the components that use Styling.generate() are broken.

Thanks for noticing! I don't use GridLayout, so I didn't notice this in my production app...
I have to create tests for every component to automate the current manual test using showcase! But unfortunately I am currently busy too. I will take a short look later this day, but I can't promise anything.

It appears to be fixed in the latest update: id("org.jetbrains.compose") version "1.0.0-alpha4-build331". Possibly from the commit 49bfb89c117db78552c888d90a6b8461ecd6bd1e.

Yeah, it looks like it is fixed. Feel free to reopen if not.


Unable to download from ITV Hub

Checklist

[x] I've verified that I'm running youtube-dl version 2021.12.17
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar bug reports including closed ones
[x] I've read bugs section in FAQ

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--verbose', u'https://www.itv.com/hub/celebrity-juice/1a7045a0288']
[debug] Encodings: locale UTF-8, fs UTF-8, out None, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 2.7.18 (CPython) - Linux-5.4.0-96-generic-x86_64-with-Ubuntu-20.04-focal
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, phantomjs ., rtmpdump 2.4
[debug] Proxy map: {'no': 'localhost,<IP_ADDRESS>,::1'}
[debug] Using fake IP <IP_ADDRESS> (GB) as X-Forwarded-For.
[ITV] 1a7045a0288: Downloading webpage
ERROR: Unable to download webpage: ('The read operation timed out',) (caused by SSLError('The read operation timed out',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/local/bin/youtube-dl/youtube_dl/utils.py", line 2737, in https_open
    req, **kwargs)
  File "/usr/lib/python2.7/urllib2.py", line 1208, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1178, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 452, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 408, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 754, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 641, in read
    v = self._sslobj.read(len)

Description

Trying to download episodes of Celebrity Juice from the ITV Hub, but it's halting after 'Downloading webpage' and then several minutes later it errors out.

Tried several additional switches, but alas to no avail.
Including the verbose log from '--print-traffic' to hopefully shed a little more light;
[debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['--no-check-certificate', '--print-traffic', '-C', '--no-geo-bypass', '--proxy', '', '--verbose', 'https://www.itv.com/hub/celebrity-juice/1a7045a0288'] [debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8 [debug] youtube-dl version 2021.12.17 [debug] Python version 3.8.10 (CPython) - Linux-5.4.0-96-generic-x86_64-with-glibc2.29 [debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, phantomjs ., rtmpdump 2.4 [debug] Proxy map: {} send: b'GET /ip HTTP/1.1\r\nHost: yt-dl.org\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.73 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Date: Mon, 31 Jan 2022 10:02:15 GMT header: Content-Type: text/plain;charset=UTF-8 header: Transfer-Encoding: chunked header: Connection: close header: Vary: Accept-Encoding header: Server: nginx header: X-Xss-Protection: 1; mode=block header: X-Frame-Options: SAMEORIGIN header: Strict-Transport-Security: max-age=31536000 header: X-Content-Type-Options: nosniff header: Referrer-Policy: strict-origin-when-cross-origin header: Content-Encoding: gzip [debug] Public IP address: <IP_ADDRESS> send: b'GET /latest/version HTTP/1.1\r\nHost: yt-dl.org\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.73 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n' reply: 'HTTP/1.1 200 OK\r\n' header: Date: Mon, 31 Jan 2022 10:02:15 GMT header: Content-Type: text/plain; charset=UTF-8 header: Content-Length: 10 header: Connection: close header: Server: nginx header: Last-Modified: Thu, 16 Dec 2021 19:01:37 GMT header: ETag: "a-5d3480d45fe58" header: Accept-Ranges: bytes header: X-Xss-Protection: 1; mode=block header: X-Frame-Options: SAMEORIGIN header: Strict-Transport-Security: max-age=31536000 header: X-Content-Type-Options: nosniff header: Referrer-Policy: strict-origin-when-cross-origin [ITV] 1a7045a0288: Downloading webpage send: b'GET /hub/celebrity-juice/1a7045a0288 HTTP/1.1\r\nHost: www.itv.com\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.73 Safari/537.36\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'

The betting is that ITV has employed a bouncer that rejects or ignores your web requests if it doesn't like the look of them.
For the moment, --user-agent 'Mozilla/5.0' confuses the bouncer into allowing access, like some suede trainers.

Nice find, indeed! 🎉 (but ITVHub people are known to monitor downloading apps 😠 , so am wondering how long it'll be before it gets patched... ); additionally, OP is employing v2021.12.17 which, AFAICT, doesn't include #30266; perhaps the latter could be further "improved" by adding this latest "fix", what do you reckon? 😜

... OTOH, yt-dlp appears unaffected by this latest ITVHub shenanigan, so "they" must have dealt with it, inadvertentenly in this case, somehow differently:
yt-dlp -v -F "https://www.itv.com/hub/celebrity-juice/1a7045a0288" =>

[debug] Command-line config: ['--ffmpeg-location', '..', '--downloader-args', 'ffmpeg:-v 8 -stats', '-v', '-F', 'https://www.itv.com/hub/celebrity-juice/1a7045a0288']
[debug] Encodings: locale cp1253, fs utf-8, out utf-8 (No ANSI), err utf-8 (No ANSI), pref cp1253
[debug] yt-dlp version 2022.01.31 [7ba35c8] (win_exe)
[debug] ** This build is unofficial daily builds, provided for ease of use.
[debug] ** Please do not ask for any support.
[debug] Python version 3.7.9 (CPython 32bit) - Windows-Vista-6.0.6003-SP2
[debug] exe versions: ffmpeg 4.4.1 (setts), ffprobe 4.4.1
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
[debug] Using fake IP <IP_ADDRESS> (GB) as X-Forwarded-For
[debug] [ITV] Extracting URL: https://www.itv.com/hub/celebrity-juice/1a7045a0288
[ITV] 1a7045a0288: Downloading webpage
[ITV] 1a7045a0288: Downloading JSON metadata
[ITV] 1a7045a0288: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7),vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext,aext, hasaud, source, id
[info] Available formats for 1a7045a0288:
ID       EXT RESOLUTION FPS |   FILESIZE   TBR PROTO  | VCODEC        VBR ACODEC     ABR
----------------------------------------------------------------------------------------
hls-136  mp4 audio only     | ~ 33.84MiB  136k m3u8_n | audio only        mp4a.40.2 128k
hls-318  mp4 512x288     50 | ~ 79.13MiB  318k m3u8_n | avc1.4D401E  171k mp4a.40.2 128k
hls-601  mp4 512x288     50 | ~149.56MiB  601k m3u8_n | avc1.4D401F  438k mp4a.40.2 128k
hls-802  mp4 512x288     50 | ~199.58MiB  802k m3u8_n | avc1.4D401F  628k mp4a.40.2 128k
hls-1082 mp4 896x504     50 | ~269.25MiB 1082k m3u8_n | avc1.4D401F  892k mp4a.40.2 128k
hls-1670 mp4 896x504     50 | ~415.58MiB 1670k m3u8_n | avc1.4D401F 1447k mp4a.40.2 128k

yt-dlp is now wearing various shades of Chrome/90+ and sending some other slightly different HTTP headers. Otherwise, the yt_dlp extractor and PR #30266 should be doing the same thing.
Probably we should just use s/t like the Chrome/iOS UA which also satisfies TikTok (today).

various shades of Chrome/90+ and sending some other slightly different HTTP headers.

Well, ITVHub's "bouncer" doesn't appear to only check for palatable UA strings; I changed my browser's default UA to Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.73 Safari/537.36, which is the one rejected in OP's second log, and my browser has no issue loading that programme's page (cache and cookies fully wiped out before the page load):

I expect your current desktop browser is sending the slightly different HTTP headers.

I performed some additional tests with youtube-dl.exe and it looks like Chrome/89 is the minimum Chrome version (at this time 😉 ...) that www.itv.com will sanction when requested by yt-dl:
youtube-dl --user-agent "Chrome/88" -F "https://www.itv.com/hub/celebrity-juice/1a7045a0288"

exhibits the issue reported by OP, while
youtube-dl --user-agent "Chrome/89" -F "https://www.itv.com/hub/celebrity-juice/1a7045a0288"

successfully gets past the  [ITV] 1a7045a0288: Downloading webpage stage and on to [ITV] 1a7045a0288: Downloading JSON metadata; actually, setting the Chrome version to a big, future, value also does the trick (e.g. --user-agent "Chrome/200"); JFYI... 😃

Apologies but I don't fully understand the above - should this be working now using --user-agent "Chrome/89" ?
When using the example above, I get ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>);
I am running youtube-dl version 2021.12.17

Thank you for the troubleshooting, I have altered the user-agent switch and I'm now getting passed the [ITV] 1a7045a0288: Downloading webpage stage.
I'm now having the same issue as @boxgr0ve - it errors out while trying to download the JSON - ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>);
Full output here. Tried it with ITV credentials and without and neither work.
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '--print-traffic', '--user-agent', 'Chrome/90', '-u', 'PRIVATE', '-p', 'PRIVATE', 'https://www.itv.com/hub/celebrity-juice/1a7045a0288']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.8.10 (CPython) - Linux-5.4.0-99-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, phantomjs ., rtmpdump 2.4
[debug] Proxy map: {'no': 'localhost,<IP_ADDRESS>,::1'}
[debug] Using fake IP <IP_ADDRESS> (GB) as X-Forwarded-For.
[ITV] 1a7045a0288: Downloading webpage
send: b'GET /hub/celebrity-juice/1a7045a0288 HTTP/1.1\r\nHost: www.itv.com\r\nX-Forwarded-For: <IP_ADDRESS>\r\nUser-Agent: Chrome/90\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Encoding: gzip
header: Content-Type: text/html; charset=UTF-8
header: grace: none
header: Server: nginx
header: Vary: Accept-Encoding
header: X-Cache-Hits: 0
header: X-Frame-Options: DENY
header: X-Akamai-Transformed: 9 19358 0 pmb=mTOE,1
header: Cache-Control: public, max-age=28, s-maxage=60
header: Expires: Wed, 09 Feb 2022 10:24:20 GMT
header: Date: Wed, 09 Feb 2022 10:23:52 GMT
header: Content-Length: 15851
header: Connection: close
header: X-Device-BrandName:
header: X-Device-Os:
header: X-Device-MobileBrowser:
header: X-Device-ModelName:
header: X-Device-Os-Version:
[ITV] 1a7045a0288: Downloading JSON metadata
send: b'POST /playlist/itvonline/ITV2/1_7045_0288.001 HTTP/1.1\r\nContent-Length: 374\r\nHost: magni.itv.com\r\nAccept: application/vnd.itv.vod.playlist.v2+json\r\nContent-Type: application/json\r\nHmac: E8EDCDB18AE0E032C1A1C8BDE616100F270993CA\r\nX-Forwarded-For: <IP_ADDRESS>\r\nUser-Agent: Chrome/90\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: en-us,en;q=0.5\r\nConnection: close\r\n\r\n'
send: b'{"user": {"itvUserId": "", "entitlements": [], "token": ""}, "device": {"manufacturer": "Safari", "model": "5", "os": {"name": "Windows NT", "version": "6.1", "type": "desktop"}}, "client": {"version": "4.1", "id": "browser"}, "variantAvailability": {"featureset": {"min": ["hls", "aes", "outband-webvtt"], "max": ["hls", "aes", "outband-webvtt"]}, "platformTag": "dotcom"}}'
reply: 'HTTP/1.1 404 Not Found\r\n'
header: Content-Type: application/json
header: Vary: Origin
header: Content-Length: 123
header: Date: Wed, 09 Feb 2022 10:23:52 GMT
header: Connection: close
header: Access-Control-Allow-Credentials: true
header: Access-Control-Allow-Origin: *
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/lib/python3.8/dist-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

Today, using #30266:
$ python -m youtube_dl -F -v 'https://www.itv.com/hub/celebrity-juice/1a7045a0288' --user-agent 'Chrome/89'
[debug] System config: [u'--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-F', u'-v', u'https://www.itv.com/hub/celebrity-juice/1a7045a0288', u'--user-agent', u'Chrome/89']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Git HEAD: 0945174fa
[debug] Python version 2.7.17 (CPython) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[debug] Using fake IP <IP_ADDRESS> (GB) as X-Forwarded-For.
[ITV] 1a7045a0288: Downloading webpage
[ITV] 1a7045a0288: Downloading JSON metadata
[ITV] 1a7045a0288: Downloading m3u8 information
[info] Available formats for 1a7045a0288:
format code  extension  resolution note
hls-136      mp4        audio only  136k , mp4a.40.2@128k
hls-318      mp4        512x288     318k , avc1.4D401E@ 171k, 50.0fps, mp4a.40.2@128k
hls-601      mp4        512x288     601k , avc1.4D401F@ 438k, 50.0fps, mp4a.40.2@128k
hls-802      mp4        512x288     802k , avc1.4D401F@ 628k, 50.0fps, mp4a.40.2@128k
hls-1082     mp4        896x504    1082k , avc1.4D401F@ 892k, 50.0fps, mp4a.40.2@128k
hls-1670     mp4        896x504    1670k , avc1.4D401F@1447k, 50.0fps, mp4a.40.2@128k (best)
df@Spiridion:~/Documents/src/youtube-dl$ python -m youtube_dl --test 'https://www.itv.com/hub/celebrity-juice/1a7045a0288' --user-agent 'Chrome/89'
[ITV] 1a7045a0288: Downloading webpage
[ITV] 1a7045a0288: Downloading JSON metadata
[ITV] 1a7045a0288: Downloading m3u8 information
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 340
[download] Destination: Celebrity Juice - Series 25 - Episode 1-1a7045a0288.mp4
[download] 100% of 10.00KiB in 00:00
$

OT TV critic: see how I used the --test option to avoid getting a viewable version of the show itself! The selection of Plebs and Wossie as test shows in the extractor was more acceptable.

Perfect - installed that version of itv.py and it worked a treat - thank you

@jmoorey1 Apologies again - can you let me know how you downloaded itv.py and ran it? I've look all though https://github.com/ytdl-org/youtube-dl/pull/30266 and even downloaded yt-dlp but can't make sense of it.  I hope you can appreciate this my first time trying to install a different version.
For reference I originally installed YouTube-dl on a Mac via home-brew (if it makes a difference).
Thanks in anticipation.

The PR obviously needs an update to apply a different UA, either locally or for all extractors, before it can be merged. This isn't Microsoft or Google, you know.

This isn't Microsoft or Google, you know

... I never implied that 😜 (TBH, I've got a lot to blame both for... 😞 ) and your hard work is being highly appreciated 👍 ; to re-iterate: current flavour of https://github.com/ytdl-org/youtube-dl/pull/30266 ("working" itv.py from there) has to be manually applied at this time, the procedure depends on how you installed/invoke yt-dl on your system; this is a prerequisite to issuing the --user-agent switch(es) mentioned in this thread for a successful ITVHub fetch...

here's a link to the raw file; https://raw.githubusercontent.com/ytdl-org/youtube-dl/0945174fa4326d5bd4b04d2b6f5ed104569c5559/youtube_dl/extractor/itv.py

... The link, when clicked, loads https://github.com/ytdl-org/youtube-dl/issues/url :wink: ; no doubt a mishap :smile: , but it should be fixed...

Thanks all, I finally managed to download something from ITV hub!!  I probably made it a lot more complicated than it should have been but for anyone else reading this who's stupid like me, the file path where the new itv.py file needs to go (at least for me) is:
/usr/local/Cellar/youtube-dl/2021.12.17/libexec/lib/python3.10/site-packages/youtube_dl/extractor
"After replacing the file, I did start getting an error: ERROR: m3u8 download detected but ffmpeg or avconv could not be found. Please install one."
So I installed ffmpeg and it all seemed to work as per the attached screenshot.   When I've used YouTube-dl in the past I didn't normally get the yellow and purple text but hey it worked!! One thing I did notice was the quality wasn't particularly great, 896x504 however this is probably a limitation of ITV hub?

Previously you didn't have ffmpeg which is what is generating the diagnostics. Apparently the ffmpeg devs and whoever is responsible for ITV content playout disagree on the correct format of the stream.
You can use --external-downloader-args '-v 16' (or possibly, 8) to cut down the chatter from ffmpeg.

ERROR: m3u8 download detected but ffmpeg or avconv could not be found. Please install one.

Have you tried the  --hls-prefer-native switch? youtube-dl defaults to using ffmpeg for dumping AES-128 encrypted HLS streams (it's the type of mobile streams the ITV Info Extractor "harvests" in order to fetch the ITVHub programme :stuck_out_tongue_winking_eye: ), but said switch will dump the stream via a pure Python implementation, no external downloader needed...
However, since this is on a Mac, you may have to install an additional "decrypting" python module (hlsnative dependency), such as the deprecated PyCrypro or the active PyCryprodome equivalent; consult some Mac documentation on how to install one of these modules...
But, as your log shows, ffmpeg is still required to mux downloaded stream to the MP4 container, so it's a good thing you installed it anyway... :smiley:

... I did notice was the quality wasn't particularly great, 896x504
however this is probably a limitation of ITV hub?

Yes; look at the formats tables posted by me here and by dirkf here (resolution column) ...
Cheers :smile:

... The link points to https://github.com/ytdl-org/youtube-dl/issues/url 😉 ; no doubt a mishap 😄 , but it should be fixed...

@Vangelis66 - fixed - nice spot ;-)


expected FITSIO.TableHDU, got FITSIO.ImageHDU

@kbarbary Would take a look at this when you have chance? I'm running infer on tall and narrow slice of sky here, that intersects two different stripes.
jregier@cori11~/Celeste/bin $  ./celeste.jl infer-nersc 350 350.1 0 10 . --logging=DEBUG
...
gunzip --stdout /global/projecta/projectdirs/sdss/data/sdss/dr12/boss/photo/redux/301/7814
/objcs/4/fpM-007814-g4-0084.fit.gz > celeste/fpm/7814-4/fpM-007814-g4-0084.fit
gunzip --stdout /global/projecta/projectdirs/sdss/data/sdss/dr12/boss/photo/redux/301/7814
/objcs/4/fpM-007814-r4-0084.fit.gz > celeste/fpm/7814-4/fpM-007814-r4-0084.fit
gunzip --stdout /global/projecta/projectdirs/sdss/data/sdss/dr12/boss/photo/redux/301/7814
/objcs/4/fpM-007814-i4-0084.fit.gz > celeste/fpm/7814-4/fpM-007814-i4-0084.fit
gunzip --stdout /global/projecta/projectdirs/sdss/data/sdss/dr12/boss/photo/redux/301/7814
/objcs/4/fpM-007814-z4-0084.fit.gz > celeste/fpm/7814-4/fpM-007814-z4-0084.fit
ERROR: LoadError: TypeError: read_photoobj: in typeassert, expected FITSIO.TableHDU, got F
ITSIO.ImageHDU
 in read_photoobj at /global/homes/j/jregier/.julia/v0.4/Celeste/src/SDSSIO.jl:334
 in read_photoobj at /global/homes/j/jregier/.julia/v0.4/Celeste/src/SDSSIO.jl:331
 in read_photoobj_primary at /global/homes/j/jregier/.julia/v0.4/Celeste/src/api.jl:70
 in infer at /global/homes/j/jregier/.julia/v0.4/Celeste/src/api.jl:142
 in infer_nersc at /global/homes/j/jregier/.julia/v0.4/Celeste/src/api.jl:413
 in main at /global/u2/j/jregier/Celeste/bin/celeste.jl:37
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 in process_options at ./client.jl:280
 in _start at ./client.jl:378
while loading /global/u2/j/jregier/Celeste/bin/celeste.jl, in expression starting on line 54

I'm finding some other regions this issue affects too:
./celeste.jl --logging=DEBUG infer-nersc 200 200.01 -29 89 .
...
gunzip --stdout /global/projecta/projectdirs/sdss/data/sdss/dr12/boss/photo/redux/301/6751
/objcs/6/fpM-006751-z6-0027.fit.gz > /global/cscratch1/sd/jregier/celeste/fpm/6751-6/fpM-0
06751-z6-0027.fit
ERROR: LoadError: TypeError: read_photoobj: in typeassert, expected FITSIO.TableHDU, got F
ITSIO.ImageHDU
 in read_photoobj at /global/homes/j/jregier/.julia/v0.4/Celeste/src/SDSSIO.jl:334
 in read_photoobj at /global/homes/j/jregier/.julia/v0.4/Celeste/src/SDSSIO.jl:331
 in read_photoobj_primary at /global/homes/j/jregier/.julia/v0.4/Celeste/src/api.jl:70
 in infer at /global/homes/j/jregier/.julia/v0.4/Celeste/src/api.jl:142
 in infer_nersc at /global/homes/j/jregier/.julia/v0.4/Celeste/src/api.jl:413
 in main at /global/u2/j/jregier/Celeste/bin/celeste.jl:37
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:304
 in process_options at ./client.jl:280
 in _start at ./client.jl:378
while loading /global/u2/j/jregier/Celeste/bin/celeste.jl, in expression starting on line
54

Also
/global/projecta/projectdirs/sdss/data/sdss/dr12/boss/photoObj/301/6597/4/photoObj-006597-4-0025.fits

and
/global/cscratch1/sd/jregier/celeste/fpm/7265-5/fpM-007265-z5-0030.fit

According to Dustin, the problem is that, on these fields,  photo timed out and did not produce any objects. He suggests first checking the window_flist.fits file, for whether that field has a score=0, and cutting on that.

The window_flist.fits file to use is
/global/projecta/projectdirs/sdss/data/sdss/dr12/boss/resolve/2010-05-23

Why did you point to /global/cscratch1/sd/jregier/celeste/fpm/7265-5/fpM-007265-z5-0030.fit? I don't see anything wrong with that file or the associated photoObj file.

Sorry, I must have copied and pasted the wrong thing in that case.

Unfortunately changing the list of fields doesn't seem to have fixed them problem. Now that the second hdu's format is checked, we get errors like
ERROR: LoadError: error reading from FITS file
 in error at ./error.jl:21
 in fits_assert_ok at /global/homes/j/jregier/.julia/v0.4/FITSIO/src/libcfitsio.jl:198
 in fits_read_pix at /global/homes/j/jregier/.julia/v0.4/FITSIO/src/libcfitsio.jl:548
 in fits_read_pix at /global/homes/j/jregier/.julia/v0.4/FITSIO/src/libcfitsio.jl:553
 in read at /global/homes/j/jregier/.julia/v0.4/FITSIO/src/image.jl:62
 in read_frame at /global/homes/j/jregier/.julia/v0.4/Celeste/src/SDSSIO.jl:99
 in read_sdss_field at /global/homes/j/jregier/.julia/v0.4/Celeste/src/SkyImages.jl:244
 in infer at /global/homes/j/jregier/.julia/v0.4/Celeste/src/api.jl:174
 in infer_box_nersc at /global/homes/j/jregier/.julia/v0.4/Celeste/src/api.jl:378
 in main at /global/u2/j/jregier/Celeste/bin/celeste.jl:39
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:304
 in process_options at ./client.jl:280
 in _start at ./client.jl:378
while loading /global/u2/j/jregier/Celeste/bin/celeste.jl, in expression starting on line 52

Looks like this is coming from a different part of the code, but I'll have a more detailed look today.

I couldn't reproduce this. Could it be a corrupt unzipped frame file? The logic for unzipping files to scratch is pretty simple: If you stop the unzip in the middle, the unzipped file might be created but incomplete. Then on the next run, the code thinks the unzipped file already exists (and assumes it is complete!) and doesn't try to re-unzip it.
If you can figure out what file the error is coming from, just delete it from scratch and run again.

I can't recreate it either after deleting scratch


ableton/Link.hpp not found

Hi, I tried to use ofxAbletonLink but it cannot find the Link.hpp file.
Maybe the file no longer exist in the latest version of Ableton Link?
I downloaded LinkKit, and there are only 3 files in include folder that are ABLLink.h, ABLLinkSettingsViewController.h and ABLLinkUtils.h.
Am I missing something or is ofxAbletonLink no longer being maintained?
Thanks.

@cuinjune
you must read README.md and study git.

Sorry, I should have mentioned that I'm trying to use ofxAbletonLink with LinkKit which is an iOS SDK for Ableton Link. But it seems like it is not possible. I will find other ways. Thanks.

@cuinjune
don't worry :)
i don't try LinkKit, but maybe we can use this addon for ofxiOS, maybe.


Vectors of TString in a nested TTree

Summary
From the documentation, it seems that strings and vectors of strings are indeed supported.
However, vectors of TString cause problems:
Traceback (most recent call last):
...
  File "/software/cmsl1t/miniconda/envs/cms/lib/python2.7/site-packages/uproot/interp/auto.py", line 256, in interpret
    return asobjs(branch._vecstreamer.pyclass, branch._context)
AttributeError: 'TStreamerInfo' object has no attribute 'pyclass'

More details on the particular structure

https://github.com/cms-sw/cmssw/blob/master/L1Trigger/L1TNtuples/interface/L1AnalysisEventDataFormat.h#L43
https://github.com/cms-sw/cmssw/blob/master/L1Trigger/L1TNtuples/src/L1AnalysisEvent.cc#L66

Example code
Example file: issue_76.tar.gz
Code
import uproot

f = uproot.open('issue_76.root')
events = f['Events']
events['normalStrings'].show()
events['rootStrings'].show()

Output:
normalStrings              TStreamerSTL               asvar(ListStrings, skip_bytes=10)
Traceback (most recent call last):
  File "bin/dummy.py", line 8, in <module>
    events['rootStrings'].show()
  File "/software/cmsl1t/miniconda/envs/cms/lib/python2.7/site-packages/uproot/tree.py", line 1523, in show
    for line in self._format(foldnames):
  File "/software/cmsl1t/miniconda/envs/cms/lib/python2.7/site-packages/uproot/tree.py", line 1510, in _format
    out = [indent + "{0:26s} {1:26s} {2}".format(name, "(no streamer)" if self._streamer is None else self._streamer.__class__.__name__, interpret(self))]
  File "/software/cmsl1t/miniconda/envs/cms/lib/python2.7/site-packages/uproot/interp/auto.py", line 256, in interpret
    return asobjs(branch._vecstreamer.pyclass, branch._context)
AttributeError: 'TStreamerInfo' object has no attribute 'pyclass'

@reikdas: Could you look into this? Create a new branch and pull request as before.
It is true that TString has no pyclass. And I think the "handling of strings" went as far as std::string and char *, so there very well could be some special cases to handle with ROOT's TString.

Thanks! (It's good practice, I promise!)

Thanks!


Remove in-place add of eps

Remove in-place add of eps. Use in-place div for improved performance

I would be really cautious with this modification. It seems the in-place add helps performance. I would keep current version unless more extensive experiments show the in-place add is unnecessary.


Add parentFeature to SpatialResourceType

We need a way to reference the parent of a spatial resource.
For example, to be able to specify the parent Plot of a Plot Zone.
We propose to add parentFeature as a FeatureReferenceType to SpatialResourceType.
This will provide for the name, identifier and uri of the parent.

Yes, this makes sense. I had originally intended the links[] array in a ResourceType would support this. However, it requires that the target of the link is addressable via a URL, while the FeatureReferenceType supports a name and ID as well.
I could see this being used to identify the SiteResource (block) that a PlotResource (paddock or field) belonged to as well.


CFFL

CFFL

Linter check found the following problems:
The following problems have been found:
ERROR: /tmp/abs_a9xyh1v2bj/clone/recipe/meta.yaml:1: missing_python_build_tool: Python packages require a python build tool such as setuptools.
ERROR: /tmp/abs_a9xyh1v2bj/clone/recipe/meta.yaml:1: missing_python: For pypi packages, python should be present in the host and run sections. Missing in host
ERROR: /tmp/abs_a9xyh1v2bj/clone/recipe/meta.yaml:1: missing_wheel: For pypi packages, wheel should be present in the host section
ERROR: /tmp/abs_a9xyh1v2bj/clone/recipe/meta.yaml:14: pip_install_args: pip install should be run with --no-deps.
WARNING: /tmp/abs_a9xyh1v2bj/clone/recipe/meta.yaml:24: python_build_tool_in_run: The python build tool setuptools is in run depends
WARNING: /tmp/abs_a9xyh1v2bj/clone/recipe/meta.yaml:41: missing_description: The recipe is missing a description
ERROR: /tmp/abs_a9xyh1v2bj/clone/recipe/meta.yaml:41: missing_dev_url: The recipe is missing a dev_url
ERROR: /tmp/abs_a9xyh1v2bj/clone/recipe/meta.yaml:41: missing_license_family: The recipe is missing the about/license_family key.
ERROR: /tmp/abs_a9xyh1v2bj/clone/recipe/meta.yaml:41: missing_documentation: The recipe is missing a doc_url or doc_source_url
Errors were found


Fix caching scala test results

What changes were proposed in this pull request?
Fix caching scala test results added in #330
We weren't copying the circle test results file to the cache location.

scala flakes... merging anyway since this PR doesn't change spark code or behaviour, only caching


How to deal with angular module's config function when unit testing?

I have got the same question.
http://stackoverflow.com/questions/19287223/how-to-deal-with-angular-modules-config-function-when-unit-testing
In the config of my app module, I am creating the states and adding rules using $urlRouterProvider. So, when I am unit testing services which are using this states, I want to mock this config phase. How to do that?

https://gist.github.com/eddiemonge/f6a58169c2846731a1eb


chore(speed): use recommended arch flag

This PR resolves []
Description

While looking at the perf docs I saw this consequential little flag: https://reactnative.dev/docs/0.75/build-speed
When building your android app locally, by default you build all the 4 [Application Binary Interfaces (ABIs)](https://developer.android.com/ndk/guides/abis) : armeabi-v7a, arm64-v8a, x86 & x86_64.

However, you probably don't need to build all of them if you're building locally and testing your emulator or on a physical device.

This should reduce your native build time by a ~75% factor.

My very unscientific test of two clean builds didn't show quite that drastic an improvement but still significant:
With the flag:
BUILD SUCCESSFUL in 1m 26s

Without the flag:
BUILD SUCCESSFUL in 1m 55s

PR Checklist

[ ] I have tested my changes on iOS and Android.
[x] I hid my changes behind a feature flag, or they don't need one.
[x] I have included screenshots or videos, or I have not changed the UI.
[x] I have added tests, or my changes don't require any.
[x] I added an app state migration, or my changes do not require one.
[x] I have documented any follow-up work that this PR will require, or it does not require any.
[x] I have added a changelog entry below, or my changes do not require one.

To the reviewers 👀

[ ] I would like at least one of the reviewers to run this PR on the simulator or device.

Changelog updates
Changelog updates

Cross-platform user-facing changes

iOS user-facing changes

Android user-facing changes

Dev changes

only build android dev builds on active arch to speed up - brian

Need help with something? Have a look at our docs, or get in touch with us.

This PR contains the following changes:

Dev changes (only build android dev builds on active arch to speed up - brian - brainbicycle)

  Generated by :no_entry_sign: dangerJS against 16ececfd239ee21045385eae7da6d96e1fb4cbf6


Nit: Lalebot comments should not be counted in task list view

Only real user comments should be tracked as commentary in the task listing view. This is intended to show how much people go back and forth on a task, not how much the Lale bot chats...

Hi @dottorer I think I can look into this one. Also is there a list of issues that need to be done sorted by necessity?

@NolanChan Thanks for looking into it. I usually label the issues that could use some help with 'help wanted' so they can be identified. I will go through the list and check that they are correctly applied.


build: Add maven-compiler-plugin for java cross-build

Which issue does this PR close?

Closes #.
Rationale for this change

What changes are included in this PR?

How are these changes tested?

After @huaxingao tested it, this doesn't help on the issue. I will close this for now.

Wait for re-verifying
cc @huaxingao

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 34.01%. Comparing base (033fe6f) to head (0784bab).
Report is 10 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main     #911      +/-   ##
============================================
- Coverage     34.03%   34.01%   -0.03%
+ Complexity      883      875       -8
============================================
  Files           113      112       -1
  Lines         43170    43221      +51
  Branches       9516     9563      +47
============================================
+ Hits          14693    14701       +8
- Misses        25471    25491      +20
- Partials       3006     3029      +23

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Seems to be the right thing to do irrespective of whether it addresses the iceberg build issue.

Thanks @parthchandra .

Thanks @andygrove @huaxingao @parthchandra

Just realized that we already had
    <maven.compiler.source>${java.version}</maven.compiler.source>
    <maven.compiler.target>${java.version}</maven.compiler.target>

which set these properties for the compiler plugin. So it appears we are not adding anything new, which may explain why this  did not help.
There is a new property available since JDK 9 which we could use (and also remove the jdk 1.8 profile which is meaningless if the source and target version are set to 11 or greater)
<maven.compiler.release>${java.version}</maven.compiler.release>

https://maven.apache.org/plugins/maven-compiler-plugin/examples/set-compiler-release.html
@huaxingao


chore: bump to prettier 3

Prettier 3 made formatting asynchronous, the only thing I wasn't sure about is the reassigned PostgresMeta.format function, I couldn't find any usage
Given we expose this as library, PostgresMeta.format has a breaking change, going from non-async to async

There were problems when using Prettier 3 with Jest - we'll probably need to switch to Vitest

@soedirgo Jest does not support snapshots with prettier 2, thus the prettierPath change here


Support is:commander and commander:card-name in searching

This would return combos that
a) have a commander in them
b) have a color identity matching the commander
Down the line, once this data is exposed, we can use it

to link to the edhrec combo page for the commander
link to a search for other combos with the commander commander="card name"

Obsolete from next release (already available for preview)
commander="card name" now searches for combos that require card name as commander to work


re-add support for sourcing endpoint URLs from service-specific env config

The service env config work I did was flawed in that it didn't respect the precedence of programmatic config. This PR
adds a way of tracking the precedence up to the point of converting the SdkConfig into a service config. Now, env config will only be resolved when config was not set programmatically. I added tests to aws-config for the origin tracking to ensure it works.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Lastly, we probably need to a changelog entry that revises or corrects release notes that we put out in https://github.com/awslabs/aws-sdk-rust/releases/tag/release-2024-04-08 and the discussion topic, since the support is currently for endpoint url.

Lastly, we probably need to have a changelog entry that revises or corrects release notes that we put out in https://github.com/awslabs/aws-sdk-rust/releases/tag/release-2024-04-08 and the discussion topic, since the support is currently for endpoint url.

I updated the discussion. I think the Changelog entry is fine:

🎉 (https://github.com/smithy-lang/smithy-rs/issues/3493) Users may now set service-specific configuration in the environment. For more information, see https://github.com/smithy-lang/smithy-rs/discussions/3537.


Serialization issues with AWS Java SDK 1.10.62

After our build system automatically pulled the latest aws java jdk dynamodb 1.10.62, we are facing serious data consistency issues after creating some vertices and edges using the new version.
It looks like this sdk version introduced a new way of json marshalling which seems to cause problems at least in combination with Titan. I am pasting the exception details here hoping that anyone with deeper insight in the dynamodb - titan relation can make sense of it.
We are using the titan dynamodb storage backend from a Java EE application running on Wildfly application server, we faced the issues both with the local dynamo db and the real cloud service.
With aws-java-sdk 1.10.62, we get the following exception during initialization:
Caused by: java.lang.IllegalArgumentException: Encountered missing datatype registration for number:<PHONE_NUMBER>
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
at com.thinkaurelius.titan.graphdb.database.serialize.StandardSerializer.getDataType(StandardSerializer.java:187)
at com.thinkaurelius.titan.graphdb.database.serialize.StandardSerializer.readClassAndObject(StandardSerializer.java:267)
at com.thinkaurelius.titan.diskstorage.configuration.backend.KCVSConfiguration.staticBuffer2Object(KCVSConfiguration.java:250)
at com.thinkaurelius.titan.diskstorage.configuration.backend.KCVSConfiguration.toMap(KCVSConfiguration.java:183)
at com.thinkaurelius.titan.diskstorage.configuration.backend.KCVSConfiguration.asReadConfiguration(KCVSConfiguration.java:190)
at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1422)
at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:94)
at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:74)
The configuration passed to TitanFactory is basically the property file given in the examples.
After forcing the sdk version back to 1.10.61, the graph can be initialised again, however there are exceptions thrown by Titan with operations that involve loading the relations. Some data seems to be corrupted, trying to delete the vertices that can not be loaded also fails with the following exception:
java.lang.IllegalArgumentException: Invalid ASCII encoding offset: 654
at com.thinkaurelius.titan.graphdb.database.serialize.attribute.StringSerializer.read(StringSerializer.java:105)
at com.thinkaurelius.titan.graphdb.database.serialize.attribute.StringSerializer.read(StringSerializer.java:24)
at com.thinkaurelius.titan.graphdb.database.serialize.StandardSerializer.readObjectInternal(StandardSerializer.java:241)
at com.thinkaurelius.titan.graphdb.database.serialize.StandardSerializer.readObject(StandardSerializer.java:229)
at com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readPropertyValue(EdgeSerializer.java:191)
at com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readPropertyValue(EdgeSerializer.java:181)
at com.thinkaurelius.titan.graphdb.database.EdgeSerializer.parseRelation(EdgeSerializer.java:117)
at com.thinkaurelius.titan.graphdb.database.EdgeSerializer.readRelation(EdgeSerializer.java:60)
at com.thinkaurelius.titan.graphdb.transaction.RelationConstructor.readRelation(RelationConstructor.java:61)
at com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1$1.next(RelationConstructor.java:46)
at com.thinkaurelius.titan.graphdb.transaction.RelationConstructor$1$1.next(RelationConstructor.java:34)
at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.remove(AbstractVertex.java:87)
at com.thinkaurelius.titan.graphdb.vertices.StandardVertex.remove(StandardVertex.java:92)

I've a similar issue as well.

Version 1.10.63 of the SDK fixes this.

I tried updating to 1.10.63 but it doesnt fix the issue for me

The data written to DynamoDB with version 62 is corrupt. You will have to get rid of that from DynamoDB and use version 63. Let us know if this does not work for you. If it doesn't work, please provide steps to reproduce this.

@piyushaws thanks for the suggestion on clearing out the data. I saw that older version of sdk had created one table when I wiped it out and allowed the sdk to recreate fresh tables everything is back up.
Anyone to avoid such mishaps in future?

Thanks for the clarification, I will close the issue once I was able to test with SDK 1.10.63.
Was this an issue specific to the Dynamo Titan Storage Backend or was this a general SDK issue?

I believe it was the dynamodb sdk issue. As I didn't have to upgrade/downgrade Dynamo Titan Storage Backend sdk.

Works for me as well with the newer AWS SDK version, closing this issue. Thanks again.


How to obtain Dhrystone results?

In README.md, it is claimed that 8.333 DMIPS/MHz is reached. Meanwhile, I am able to simulate via the command obj_dir/VTop test_programs/dhry_1.s which executes 134733 cycles. How can I obtain the score of Dhrystone benchmark?
In the README.md file, it is mentioned that the system achieves 8.333 DMIPS/MHz. However, when I simulate using the command obj_dir/VTop test_programs/dhry_1.s, it executes 134733 cycles. Could you please guide me on how to obtain the Dhrystone benchmark score?
I meant, it is frustrating to extract relevant information from the output of the aforementioned command.
...
5core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
6
core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
core   0: 3 0x80000024 (0xa001)
134733 cycles

Sadly Spike generates a lot of messages when commit logging is enabled, commit logging is required for cosimulation though. However, all of this output is on stderr, which you can redirect to /dev/null.
You can also apply following hack to disable the log output entirely.
diff --git a/riscv/execute.cc b/riscv/execute.cc
index acf0e908..a3416adc 100644
--- a/riscv/execute.cc
+++ b/riscv/execute.cc
@@ -62,6 +62,7 @@ static void commit_log_print_value(FILE *log_file, int width, uint64_t val)

 static void commit_log_print_insn(processor_t *p, reg_t pc, insn_t insn)
 {
+  return;
   FILE *log_file = p->get_log_file();

   auto& reg = p->get_state()->log_reg_write;
diff --git a/riscv/mmu.h b/riscv/mmu.h

For a proper fix, I will have to fork Spike at some point.

Sadly Spike generates a lot of messages when commit logging is enabled, commit logging is required for cosimulation though. However, all of this output is on stderr, which you can redirect to /dev/null.

Can you add something like command obj_dir/VTop test_programs/dhry_1.s 2> /dev/null in README.md? So that, people can obtain the results.
My results on AMD Ryzen Threadripper 2990WX 32-Core Processor.
Runtime (cycles) 110994
Executed (instrs) 302040
mIPC 2721
mDMIPS/MHz 10256

Good point, done. The simulator is not using any measurements of real time, so the score should be identical everywhere. I've updated and clarified the score in README.md as well. Thanks again!


A query must be executed before performing export if ResultSet was closed

System information:

Operating system (distribution) and version Win10
DBeaver version 7.3.5 EA

Steps to reproduce, if exist:

Open new SQL Editor and execute a query
Close ResultSet
Select Export from Query in the context menu

Expected: the export wizard appears
Actual result: ResultSet with grey background appears
https://user-images.githubusercontent.com/51405061/109116417-86a5c800-7751-11eb-9b12-0e33037d8161.mp4
dbeaver-debug.log
.log

Fixed

verified


Scraper suddenly stopped working

Version of the custom_component
v6.7.0
Home Assistant 2023.5.3
Supervisor 2023.04.1
Operating System 10.1
Frontend 20230503.3 - latest
Configuration
multiscrape:
  - name: ACL Fuel Scraper
    resource: https://www.acl.lu/en-us/mobilite-et-tourisme/service-tourisme/prix-des-carburants
    scan_interval: 900
    log_response: false
    sensor:
      - unique_id: acl_95_fuel_price
        name: 95 Fuel Price
        icon: mdi:gas-station
        select: "#form > div.main-wrapper > div.section-pt.section-pb-70.noPaddingBottom > div > div > div > table:nth-child(9) > tbody > tr:nth-child(2) > td:nth-child(4)"
        unit_of_measurement: '€'
      - unique_id: acl_98_fuel_price
        name: 98 Fuel Price
        icon: mdi:gas-station
        select: "#form > div.main-wrapper > div.section-pt.section-pb-70.noPaddingBottom > div > div > div > table:nth-child(9) > tbody > tr:nth-child(2) > td:nth-child(5)"
        unit_of_measurement: '€'
      - unique_id: acl_diesel_fuel_price
        name: Diesel Fuel Price
        icon: mdi:gas-station
        select: "#form > div.main-wrapper > div.section-pt.section-pb-70.noPaddingBottom > div > div > div > table:nth-child(9) > tbody > tr:nth-child(2) > td:nth-child(6)"
        unit_of_measurement: '€'
      - unique_id: acl_last_updated
        name: Last Updated
        icon: mdi:calendar
        select: "#form > div.main-wrapper > div.section-pt.section-pb-70.noPaddingBottom > div > div > div > table:nth-child(9) > tbody > tr:nth-child(2) > td:nth-child(7)"

2023-05-16 11:36:43.638 DEBUG (MainThread) [custom_components.multiscrape] # Start loading multiscrape
2023-05-16 11:36:43.638 DEBUG (MainThread) [custom_components.multiscrape] # Reload service registered
2023-05-16 11:36:43.638 DEBUG (MainThread) [custom_components.multiscrape] # Start processing config from configuration.yaml
2023-05-16 11:36:43.639 DEBUG (MainThread) [custom_components.multiscrape] ACL Fuel Scraper # Setting up multiscrape with config:
{'name': 'ACL Fuel Scraper', 'resource': 'https://www.acl.lu/en-us/mobilite-et-tourisme/service-tourisme/prix-des-carburants', 'scan_interval': datetime.timedelta(seconds=900), 'log_response': True, 'sensor': [{'unique_id': 'acl_95_fuel_price', 'name': '95 Fuel Price', 'icon': Template<template=(mdi:gas-station) renders=0>, 'select': Template<template=(#form > div.main-wrapper > div.section-pt.section-pb-70.noPaddingBottom > div > div > div > table:nth-child(9) > tbody > tr:nth-child(2) > td:nth-child(4)) renders=0>, 'unit_of_measurement': '€', 'force_update': False}, {'unique_id': 'acl_98_fuel_price', 'name': '98 Fuel Price', 'icon': Template<template=(mdi:gas-station) renders=0>, 'select': Template<template=(#form > div.main-wrapper > div.section-pt.section-pb-70.noPaddingBottom > div > div > div > table:nth-child(9) > tbody > tr:nth-child(2) > td:nth-child(5)) renders=0>, 'unit_of_measurement': '€', 'force_update': False}, {'unique_id': 'acl_diesel_fuel_price', 'name': 'Diesel Fuel Price', 'icon': Template<template=(mdi:gas-station) renders=0>, 'select': Template<template=(#form > div.main-wrapper > div.section-pt.section-pb-70.noPaddingBottom > div > div > div > table:nth-child(9) > tbody > tr:nth-child(2) > td:nth-child(6)) renders=0>, 'unit_of_measurement': '€', 'force_update': False}, {'unique_id': 'acl_last_updated', 'name': 'Last Updated', 'icon': Template<template=(mdi:calendar) renders=0>, 'select': Template<template=(#form > div.main-wrapper > div.section-pt.section-pb-70.noPaddingBottom > div > div > div > table:nth-child(9) > tbody > tr:nth-child(2) > td:nth-child(7)) renders=0>, 'force_update': False}], 'method': 'GET', 'timeout': 10, 'list_separator': ',', 'parser': 'lxml', 'verify_ssl': True}
2023-05-16 11:36:43.639 DEBUG (MainThread) [custom_components.multiscrape] ACL Fuel Scraper # Log responses enabled, creating logging folder: /config/multiscrape/acl_fuel_scraper/
2023-05-16 11:36:43.649 DEBUG (MainThread) [custom_components.multiscrape] ACL Fuel Scraper # Initializing scraper
2023-05-16 11:36:43.649 DEBUG (MainThread) [custom_components.multiscrape.scraper] ACL Fuel Scraper # Initializing scraper
2023-05-16 11:36:43.678 DEBUG (MainThread) [custom_components.multiscrape.http] ACL Fuel Scraper # Initializing http wrapper
2023-05-16 11:36:43.678 DEBUG (MainThread) [custom_components.multiscrape] ACL Fuel Scraper # Initializing coordinator
2023-05-16 11:36:43.678 DEBUG (MainThread) [custom_components.multiscrape.coordinator] ACL Fuel Scraper # Scan interval is 0:15:00
2023-05-16 11:36:43.682 DEBUG (MainThread) [custom_components.multiscrape.coordinator] ACL Fuel Scraper # New run: start (re)loading data from resource
2023-05-16 11:36:43.682 DEBUG (MainThread) [custom_components.multiscrape.coordinator] ACL Fuel Scraper # Deleting logging files from previous run
2023-05-16 11:36:43.904 DEBUG (MainThread) [custom_components.multiscrape.coordinator] ACL Fuel Scraper # Rendered resource template into: https://www.acl.lu/en-us/mobilite-et-tourisme/service-tourisme/prix-des-carburants
2023-05-16 11:36:43.905 DEBUG (MainThread) [custom_components.multiscrape.coordinator] ACL Fuel Scraper # Request data from https://www.acl.lu/en-us/mobilite-et-tourisme/service-tourisme/prix-des-carburants
2023-05-16 11:36:43.905 DEBUG (MainThread) [custom_components.multiscrape.http] ACL Fuel Scraper # Executing page-request with a get to url: https://www.acl.lu/en-us/mobilite-et-tourisme/service-tourisme/prix-des-carburants.
2023-05-16 11:36:44.122 DEBUG (MainThread) [custom_components.multiscrape.http] ACL Fuel Scraper # request_headers written to file: page_request_headers.txt
2023-05-16 11:36:44.140 DEBUG (MainThread) [custom_components.multiscrape.http] ACL Fuel Scraper # request_body written to file: page_request_body.txt
2023-05-16 11:36:46.994 DEBUG (MainThread) [custom_components.multiscrape.http] ACL Fuel Scraper # Response status code received: 200
2023-05-16 11:36:47.012 DEBUG (MainThread) [custom_components.multiscrape.http] ACL Fuel Scraper # response_headers written to file: page_response_headers.txt
2023-05-16 11:36:47.028 DEBUG (MainThread) [custom_components.multiscrape.http] ACL Fuel Scraper # response_body written to file: page_response_body.txt
2023-05-16 11:36:47.028 DEBUG (MainThread) [custom_components.multiscrape.scraper] ACL Fuel Scraper # Loading the content in BeautifulSoup.
2023-05-16 11:36:47.111 DEBUG (MainThread) [custom_components.multiscrape.scraper] ACL Fuel Scraper # page_soup written to file: page_soup.txt
2023-05-16 11:36:47.111 DEBUG (MainThread) [custom_components.multiscrape.coordinator] ACL Fuel Scraper # Data succesfully refreshed. Sensors will now start scraping to update.
2023-05-16 11:36:47.111 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Finished fetching multiscrape data in 3.430 seconds (success: True)
2023-05-16 11:36:47.137 DEBUG (MainThread) [custom_components.multiscrape.sensor] ACL Fuel Scraper # 95 Fuel Price # Setting up sensor
2023-05-16 11:36:47.139 DEBUG (MainThread) [custom_components.multiscrape.sensor] ACL Fuel Scraper # 98 Fuel Price # Setting up sensor
2023-05-16 11:36:47.143 DEBUG (MainThread) [custom_components.multiscrape.sensor] ACL Fuel Scraper # Diesel Fuel Price # Setting up sensor
2023-05-16 11:36:47.144 DEBUG (MainThread) [custom_components.multiscrape.sensor] ACL Fuel Scraper # 95 Fuel Price # Start scraping to update sensor
2023-05-16 11:36:47.162 DEBUG (MainThread) [custom_components.multiscrape.scraper] ACL Fuel Scraper # 95 Fuel Price # Tag selected: None
2023-05-16 11:36:47.162 ERROR (MainThread) [custom_components.multiscrape.sensor] ACL Fuel Scraper # 95 Fuel Price # Unable to scrape data: Could not find a tag for given selector
Consider using debug logging and log_response for further investigation.
2023-05-16 11:36:47.164 DEBUG (MainThread) [custom_components.multiscrape.sensor] ACL Fuel Scraper # 95 Fuel Price # On-error, set value to None
2023-05-16 11:36:47.164 DEBUG (MainThread) [custom_components.multiscrape.entity] ACL Fuel Scraper # 95 Fuel Price # Icon template rendered and set to: mdi:gas-station
2023-05-16 11:36:47.164 DEBUG (MainThread) [custom_components.multiscrape.entity] ACL Fuel Scraper # 95 Fuel Price # Updated sensor and attributes, now adding to HA
2023-05-16 11:36:47.165 DEBUG (MainThread) [custom_components.multiscrape.sensor] ACL Fuel Scraper # Last Updated # Setting up sensor
2023-05-16 11:36:47.166 DEBUG (MainThread) [custom_components.multiscrape.sensor] ACL Fuel Scraper # 98 Fuel Price # Start scraping to update sensor
2023-05-16 11:36:47.185 DEBUG (MainThread) [custom_components.multiscrape.scraper] ACL Fuel Scraper # 98 Fuel Price # Tag selected: None
2023-05-16 11:36:47.185 ERROR (MainThread) [custom_components.multiscrape.sensor] ACL Fuel Scraper # 98 Fuel Price # Unable to scrape data: Could not find a tag for given selector
Consider using debug logging and log_response for further investigation.
2023-05-16 11:36:47.185 DEBUG (MainThread) [custom_components.multiscrape.sensor] ACL Fuel Scraper # 98 Fuel Price # On-error, set value to None
2023-05-16 11:36:47.186 DEBUG (MainThread) [custom_components.multiscrape.entity] ACL Fuel Scraper # 98 Fuel Price # Icon template rendered and set to: mdi:gas-station
2023-05-16 11:36:47.186 DEBUG (MainThread) [custom_components.multiscrape.entity] ACL Fuel Scraper # 98 Fuel Price # Updated sensor and attributes, now adding to HA
2023-05-16 11:36:47.188 DEBUG (MainThread) [custom_components.multiscrape.sensor] ACL Fuel Scraper # Diesel Fuel Price # Start scraping to update sensor
2023-05-16 11:36:47.209 DEBUG (MainThread) [custom_components.multiscrape.scraper] ACL Fuel Scraper # Diesel Fuel Price # Tag selected: None
2023-05-16 11:36:47.209 ERROR (MainThread) [custom_components.multiscrape.sensor] ACL Fuel Scraper # Diesel Fuel Price # Unable to scrape data: Could not find a tag for given selector
Consider using debug logging and log_response for further investigation.
2023-05-16 11:36:47.209 DEBUG (MainThread) [custom_components.multiscrape.sensor] ACL Fuel Scraper # Diesel Fuel Price # On-error, set value to None
2023-05-16 11:36:47.209 DEBUG (MainThread) [custom_components.multiscrape.entity] ACL Fuel Scraper # Diesel Fuel Price # Icon template rendered and set to: mdi:gas-station
2023-05-16 11:36:47.209 DEBUG (MainThread) [custom_components.multiscrape.entity] ACL Fuel Scraper # Diesel Fuel Price # Updated sensor and attributes, now adding to HA
2023-05-16 11:36:47.215 DEBUG (MainThread) [custom_components.multiscrape.sensor] ACL Fuel Scraper # Last Updated # Start scraping to update sensor
2023-05-16 11:36:47.232 DEBUG (MainThread) [custom_components.multiscrape.scraper] ACL Fuel Scraper # Last Updated # Tag selected: None
2023-05-16 11:36:47.232 ERROR (MainThread) [custom_components.multiscrape.sensor] ACL Fuel Scraper # Last Updated # Unable to scrape data: Could not find a tag for given selector
Consider using debug logging and log_response for further investigation.
2023-05-16 11:36:47.233 DEBUG (MainThread) [custom_components.multiscrape.sensor] ACL Fuel Scraper # Last Updated # On-error, set value to None
2023-05-16 11:36:47.233 DEBUG (MainThread) [custom_components.multiscrape.entity] ACL Fuel Scraper # Last Updated # Icon template rendered and set to: mdi:calendar
2023-05-16 11:36:47.233 DEBUG (MainThread) [custom_components.multiscrape.entity] ACL Fuel Scraper # Last Updated # Updated sensor and attributes, now adding to HA

Describe the bug
Scraper has been working fine until last few days, and even after restarting HA the same - I did notice by using Chrome to inspect the fields I scrape (I also tried Firefox), that they had changed (that I forgot to note down), but even with the new check they always fail
Debug log
page_response_body.txt
page_soup.txt
page_request_headers.txt
page_response_headers.txt
page_request_body.txt

Sorry, think I just spotted the issue - was not looking closely enough.


Apply Docker logs size limit fix

When you apply this to production, can you merge rather than cherry-pick?


Add ability to print from another thread

My usecase:
By default I'm waiting for some input from user. But sometimes my program's background thread needs to print some output. If I do this — this outputs is being inserted as answer into console and, of course, it's not correct behaviour.
Minimalistic example:
        fun main(args: Array<String>) {
            thread {
                Thread.sleep(3000)
                textio.textTerminal.println("Some text here")
                Thread.sleep(1000)
                textio.textTerminal.println("Some text here")
            }
            textio
                    .newStringInputReader()
                    .withMaxLength(2)
                    .read("some input")
        }

Is it possible to make terminal redraw question if another thread is trying to print something?

I think this would be a nice feature.
I'm currently working on another project, which will keep me busy for several months.
Therefore, I put the label "up-for-grabs" on this issue, to indicate that this feature has been specifically chosen to be implemented by contributors.
Some guidelines for contributors:

Implementing this feature is not a trivial task, because Text-IO has not been designed to be thread-safe.

In order to avoid intermingling the messages printed by different threads, TextTerminal's print methods should not be allowed to run in parallel.

When a print method runs in parallel with a read method, an ideal implementation will allow specifying the desired behavior:

(as requested by @asm0dey) Print the message and redraw the question.
Take care to restore the properties that were in use by the initial call of the read method.
Also take care of the partial input available at the moment when the read method has been interrupted.
Postpone the print operation until after the end of the read operation.

(It is OK to send a PR implementing only one of these two approaches, leaving the other one open for a future contribution.)

In order to prevent that two people work on the same thring, leave a comment here to claim this issue before starting to work on it.

Any plans on introducing it? This library is pretty useless without this feature.

Sorry, currently I have no time to implement this feature and no one stepped in to do this.
Would you be interested perhaps in sending a pull request for implementing this feature?


Feature/markdown pattern references

Allow pattern authors to reference other patterns within the description of a pattern via hyperlinks:

when creating a new pattern  (via a new button that was added to the the markdown editor)
when editing a pattern (via a new select field in the editor-dialog)

This fixes #81

Looks good!
I would just do some minor change regarding the look.

Add some spacing in between the dropdown and the add button
Test if it would look better, to put the dropdown to the right side of the title to save some space

@mbeisel could you recheck the changes? besides the css changes, it's mostly whitespace :)