Add possibility to define custom fasthttp server MaxRequestBodySize parameter to handle uploading of big files.

In what area(s)?

/area runtime

Describe the feature
During the writing and deployment of the service which uploads files (up to 2GB), I've encountered an issue that files bigger than 4 megabytes are not uploading. I've gone on an investigation where exactly the issue lies and eventually find out that dapr using for HTTP server fasthttp library, which by default got a 4 MB limit on max body size and in dapr code it seems that default fasthttp server is defined.
So, my question is - is it possible to add the possibility of providing MaxRequestBodySize parameter of fasthttp via annotation for example. I think I can lead that effort and provide a pull request if you guys would be okay with that kind of feature.
Release Note
Add the possibility of providing MaxRequestBodySize parameter of fasthttp server via annotation to handle uploading of big files.

Thanks for raising this, and we would gladly appreciate a PR.
This would need to propagate to Dapr in the form of command line flag, I suggest using -dapr-http-max-request-size.
The CLI flags are located here: https://github.com/dapr/dapr/blob/master/pkg/runtime/cli.go

Thanks for raising this, and we would gladly appreciate a PR.
This would need to propagate to Dapr in the form of command line flag, I suggest using -dapr-http-max-request-size.
The CLI flags are located here: https://github.com/dapr/dapr/blob/master/pkg/runtime/cli.go

@yaron2 Thank you for the fast reply. I've figured out about the CLI flag, although it's still not clear to me how can I pass it via annotation in yaml. Is that located here or I looking in not the right way? https://github.com/dapr/dapr/blob/master/pkg/injector/pod_patch.go

@yaron2 Thank you for the fast reply. I've figured out about the CLI flag, although it's still not clear to me how can I pass it via annotation in yaml. Is that located here or I looking in not the right way? https://github.com/dapr/dapr/blob/master/pkg/injector/pod_patch.go

@yaron2 Thank you for the fast reply. I've figured out about the CLI flag, although it's still not clear to me how can I pass it via annotation in yaml. Is that located here or I looking in not the right way? https://github.com/dapr/dapr/blob/master/pkg/injector/pod_patch.go

Yes, the args are set here: https://github.com/dapr/dapr/blob/4678e477562e7e35a1d6ba04a9b17b6c9c00a025/pkg/injector/pod_patch.go#L524

@yaron2 Thank you for the fast reply. I've figured out about the CLI flag, although it's still not clear to me how can I pass it via annotation in yaml. Is that located here or I looking in not the right way? https://github.com/dapr/dapr/blob/master/pkg/injector/pod_patch.go

Yes, the args are set here: https://github.com/dapr/dapr/blob/4678e477562e7e35a1d6ba04a9b17b6c9c00a025/pkg/injector/pod_patch.go#L524

Hi @yaron2
I've made appropriate changes and during testing encountered on the two new issues.

When I trying pass to my upload service via darp file more than 4 MB and less than 16 MB (I'll explain it in the next item) receiving the following error:
{  "errorCode": "ERR_DIRECT_INVOKE",  "message": "rpc error: code = ResourceExhausted desc = grpc: received message larger than max (10730238 vs. 4194304)" }
Do you have any ideas on how can I fix that as well?
When I pass to my upload service via darp file more than 16 MB (seems that it is a default value for the maximum file size in memory https://github.com/valyala/fasthttp/blob/70e00dc4f3b8c1ac01a2e82410c9f4d59248c099/http.go#L969) in echoapp daprd logs I found this message:
2021/01/14 16:12:50 error when serving connection "<IP_ADDRESS>:3500"<->"<IP_ADDRESS>:51830": cannot read multipart/form-data body: open /tmp/multipart-222081088: no such file or directory it can't tell me much, but I think it is something related with permissions on daprd sidecar, or something related to fs.
I would appreciate any help, thank you.

Hi @yaron2
I've made appropriate changes and during testing encountered on the two new issues.

When I trying pass to my upload service via darp file more than 4 MB and less than 16 MB (I'll explain it in the next item) receiving the following error:
{  "errorCode": "ERR_DIRECT_INVOKE",  "message": "rpc error: code = ResourceExhausted desc = grpc: received message larger than max (10730238 vs. 4194304)" }
Do you have any ideas on how can I fix that as well?
When I pass to my upload service via darp file more than 16 MB (seems that it is a default value for the maximum file size in memory https://github.com/valyala/fasthttp/blob/70e00dc4f3b8c1ac01a2e82410c9f4d59248c099/http.go#L969) in echoapp daprd logs I found this message:
2021/01/14 16:12:50 error when serving connection "<IP_ADDRESS>:3500"<->"<IP_ADDRESS>:51830": cannot read multipart/form-data body: open /tmp/multipart-222081088: no such file or directory it can't tell me much, but I think it is something related with permissions on daprd sidecar, or something related to fs.
I would appreciate any help, thank you.

Hi @yaron2
I've made appropriate changes and during testing encountered on the two new issues.

When I trying pass to my upload service via darp file more than 4 MB and less than 16 MB (I'll explain it in the next item) receiving the following error:
{ "errorCode": "ERR_DIRECT_INVOKE", "message": "rpc error: code = ResourceExhausted desc = grpc: received message larger than max (10730238 vs. 4194304)" }
Do you have any ideas on how can I fix that as well?
When I pass to my upload service via darp file more than 16 MB (seems that it is a default value for the maximum file size in memory https://github.com/valyala/fasthttp/blob/70e00dc4f3b8c1ac01a2e82410c9f4d59248c099/http.go#L969) in echoapp daprd logs I found this message:
2021/01/14 16:12:50 error when serving connection "<IP_ADDRESS>:3500"<->"<IP_ADDRESS>:51830": cannot read multipart/form-data body: open /tmp/multipart-222081088: no such file or directory it can't tell me much besides that fasthttp tries to save file more than 16 MB on tmp storage, but I think it is something related with permissions on daprd sidecar, or something related to fs.
I would appreciate any help, thank you.

Regarding 1, since Dapr uses gRPC to send messages in service invocation, the max size needs to propagate to the the code where Dapr starts its gRPC server and makes the gRPC call. In both cases it's a simple addition, you can see this issue that explains how to do it (the first comment shows the server side, and the second the client: https://github.com/grpc-ecosystem/grpc-gateway/issues/943)
The place where Dapr starts the gRPC server is here: https://github.com/dapr/dapr/blob/4678e477562e7e35a1d6ba04a9b17b6c9c00a025/pkg/grpc/server.go#L196
And here is the call to a remote Dapr sidecar that needs to include the client option: https://github.com/dapr/dapr/blob/944bcaeec10617021317e2123b129162689d19f9/pkg/messaging/direct_messaging.go#L160

Hi @yaron2
I've made appropriate changes and during testing encountered on the two new issues.

When I trying pass to my upload service via darp file more than 4 MB and less than 16 MB (I'll explain it in the next item) receiving the following error:
{ "errorCode": "ERR_DIRECT_INVOKE", "message": "rpc error: code = ResourceExhausted desc = grpc: received message larger than max (10730238 vs. 4194304)" }
Do you have any ideas on how can I fix that as well?
When I pass to my upload service via darp file more than 16 MB (seems that it is a default value for the maximum file size in memory https://github.com/valyala/fasthttp/blob/70e00dc4f3b8c1ac01a2e82410c9f4d59248c099/http.go#L969) in echoapp daprd logs I found this message:
2021/01/14 16:12:50 error when serving connection "<IP_ADDRESS>:3500"<->"<IP_ADDRESS>:51830": cannot read multipart/form-data body: open /tmp/multipart-222081088: no such file or directory it can't tell me much besides that fasthttp tries to save file more than 16 MB on tmp storage, but I think it is something related with permissions on daprd sidecar, or something related to fs.
I would appreciate any help, thank you.

Regarding 1, since Dapr uses gRPC to send messages in service invocation, the max size needs to propagate to the the code where Dapr starts its gRPC server and makes the gRPC call. In both cases it's a simple addition, you can see this issue that explains how to do it (the first comment shows the server side, and the second the client: https://github.com/grpc-ecosystem/grpc-gateway/issues/943)
The place where Dapr starts the gRPC server is here: https://github.com/dapr/dapr/blob/4678e477562e7e35a1d6ba04a9b17b6c9c00a025/pkg/grpc/server.go#L196
And here is the call to a remote Dapr sidecar that needs to include the client option: https://github.com/dapr/dapr/blob/944bcaeec10617021317e2123b129162689d19f9/pkg/messaging/direct_messaging.go#L160

Hi, @yaron2 I figured out how to pass an argument to the dapr/pkg/grpc/server.go, and I did a hardcode for testing purposes in dapr/pkg/messaging/direct_messaging.go, it seems that resolved the issue,  but I am struggling to pass this property as a config argument. Is there any example that I can use as a reference?
And if you have any thoughts regarding the second issue that I wrote, please share them.

Hi, @yaron2 I figured out how to pass an argument to the dapr/pkg/grpc/server.go, and I did a hardcode for testing purposes in dapr/pkg/messaging/direct_messaging.go, it seems that resolved the issue,  but I am struggling to pass this property as a config argument. Is there any example that I can use as a reference?
And if you have any thoughts regarding the second issue that I wrote, please share them.

Hi, @yaron2 I figured out how to pass an argument to the dapr/pkg/grpc/server.go, and I did a hardcode for testing purposes in dapr/pkg/messaging/direct_messaging.go, it seems that resolved the issue, but I am struggling to pass this property as a config argument. Is there any example that I can use as a reference?
And if you have any thoughts regarding the second issue that I wrote, please share them.

This is where you need to pass the new argument to the direct messaging package: https://github.com/dapr/dapr/blob/2849ede55102ff4c6a5da74de727b486f652f449/pkg/runtime/runtime.go#L456
Regarding the log error issue, it seems like the HTTP server is trying to write to a file and is denied because there's no filesystem mounted on the container. We (maintainers) will fix that in parallel to your PR.

Hi, @yaron2 I figured out how to pass an argument to the dapr/pkg/grpc/server.go, and I did a hardcode for testing purposes in dapr/pkg/messaging/direct_messaging.go, it seems that resolved the issue, but I am struggling to pass this property as a config argument. Is there any example that I can use as a reference?
And if you have any thoughts regarding the second issue that I wrote, please share them.

This is where you need to pass the new argument to the direct messaging package: https://github.com/dapr/dapr/blob/2849ede55102ff4c6a5da74de727b486f652f449/pkg/runtime/runtime.go#L456
Regarding the log error issue, it seems like the HTTP server is trying to write to a file and is denied because there's no filesystem mounted on the container. We (maintainers) will fix that in parallel to your PR.

Hi, @yaron2 I made a PR in the WIP stage, feel free to comment on it, thank you.

Hi, @yaron2 I made a PR in the WIP stage, feel free to comment on it, thank you.

@yaron2 Could you, please notify me (for example it that issue) when you'll provide some changes from your side, so I can test the whole solution itself.

@yaron2 Could you, please notify me (for example it that issue) when you'll provide some changes from your side, so I can test the whole solution itself.

Just wanted to drop a thank you, @apuzyrevsky and @yaron2. I came here to search because I ran into this exact problem, and... thank you for adding this code. 🙏

Just wanted to drop a thank you, @apuzyrevsky and @yaron2. I came here to search because I ran into this exact problem, and... thank you for adding this code. 🙏

@apuzyrevsky @yaron2 Anything else pending in this Issue? Was it part of a previous release?

@apuzyrevsky @yaron2 Anything else pending in this Issue? Was it part of a previous release?

I am waiting for a reply from @yaron2 regarding "it seems like the HTTP server is trying to write to a file and is denied because there's no filesystem mounted on the container. We (maintainers) will fix that in parallel to your PR." if it is done or not.

@apuzyrevsky it seems that this issue has not been finished yet, while docs released.

Does Dapr buffer the whole request? If so than it's not right behaviour for uploading large files, it must act like a proxy then sending the stream of bytes to the destination service.
I'm also interested to be able to pass huge files without size restriction through Dapr.

@Betula-L Please, where is the docs you are referring to?
@denis111 I agree that ideally we should stream requests and not buffer. Can you create an issue for that?
@apuzyrevsky Please, send a PR to add this configuration setting. Even if not ideal for a large file upload, it can unblock users that need this for a slightly larger payload than the default. This should be configurable for K8s too, not only standalone more, please.
Thanks.

https://docs.dapr.io/operations/configuration/increase-request-size/

@artursouza will do that

@Betula-L @apuzyrevsky  The problem is that the flag was added to daprd but not CLI. I am moving this issue to CLI repo.


Contents of microprofile-config.properties should be filtered

based on selected specifications.

Merged :umbrella:


Improve code editor UI for multiple tabs

When the number files gets to be more than 4 or 5 it might better to just show the ones that need to be worked on to pass the exercise.
Author: Emmanuel Geanrd
Slide: Local,Public

Maybe just keep them in file tree? Or related to/could be addressed by #299?

Looks like we're ending up reimplementing whatever every other IDE Is doing :)

This is in


Toyota: 2021 Highlander Platinum AWD Fingerprint (car not recognized) and Pre-Collision Malfunction

Describe the bug
Car Not Recognized (was able to manually edit values.py using the following to get car recognized), but please add the car as it is in the supported list.
Fingerprint for 2021 Highlander Platinum AWD:
36: 8, 37: 8, 114: 5, 119: 6, 120: 4, 170: 8, 180: 8, 186: 4, 355: 5, 401: 8, 426: 6, 452: 8, 464: 8, 466: 8, 467: 8, 544: 4, 550: 8, 552: 4, 562: 6, 565: 8, 608: 8, 610: 8, 643: 7, 658: 8, 705: 8, 728: 8, 740: 5, 761: 8, 764: 8, 765: 8, 791: 8, 800: 8, 810: 2, 812: 8, 814: 8, 818: 8, 822: 8, 824: 8, 830: 7, 835: 8, 836: 8, 865: 8, 869: 7, 870: 7, 871: 2, 877: 8, 881: 8, 885: 8, 889: 8, 891: 8, 896: 8, 898: 8, 900: 6, 902: 6, 905: 8, 913: 8, 918: 8, 921: 8, 933: 8, 934: 8, 935: 8, 942: 8, 944: 8, 945: 8, 951: 8, 955: 8, 956: 8, 976: 1, 987: 8, 998: 5, 999: 7, 1000: 8, 1001: 8, 1002: 8, 1014: 8, 1017: 8, 1020: 8, 1041: 8, 1042: 8, 1044: 8, 1056: 8, 1059: 1, 1063: 8, 1076: 8, 1077: 8, 1082: 8, 1114: 8, 1161: 8, 1162: 8, 1163: 8, 1164: 8, 1165: 8, 1166: 8, 1167: 8, 1172: 8, 1228: 8, 1235: 8, 1237: 8, 1263: 8, 1264: 8, 1279: 8, 1541: 8, 1552: 8, 1553: 8, 1556: 8, 1557: 8, 1568: 8, 1570: 8, 1571: 8, 1572: 8, 1592: 8, 1593: 8, 1594: 8, 1595: 8, 1649: 8, 1653: 8, 1654: 8, 1655: 8, 1696: 8, 1745: 8, 1775: 8, 1779: 8, 1786: 8, 1787: 8, 1788: 8, 1789: 8, 1904: 8, 1912: 8
carFw = [
( ecu = eps,
fwVersion = "8965B48310\000\000\000\000\000\000",
address = 1953,
subAddress = 0 ),
( ecu = esp,
fwVersion = "\001F15260E051\000\000\000\000\000\000",
address = 1968,
subAddress = 0 ),
( ecu = engine,
fwVersion = "\001896630EB2200\000\000\000\000",
address = 1792,
subAddress = 0 ),
( ecu = fwdRadar,
fwVersion = "\0018821F6201200\000\000\000\000",
address = 1872,
subAddress = 15 ),
( ecu = fwdCamera,
fwVersion = "\0028646F4803000\000\000\000\0008646G5301200\000\000\000\000",
address = 1872,
subAddress = 109 ) ],
Once you manually add the information above in the specified format (converting addresses to hex and remembering commas), a Pre-Collision Malfunction error on the dash once you enable openpilot.
Device/Version information (please complete the following information):

Device: comma two

Dongle ID: 73484e3ac5b1b0a7

Route: [e.g. 77611a1fac303767|2020-05-11--16-37-07, can be found in my.comma.ai/useradmin]

Timestamp: [When in the route the bug occurs (e.g. 4min 30s into the drive)]

Version: .8.3 release2

Car make/model: 2021 Toyota Highlander Platinum AWD

A fix was shipped in the subsequent release (0.8.4) for the PCS fault. I also checked your logs, and your car's fingerprint is in release. Let us know if you have any more issues.


tools: cleanup eyrie usage

This pulls eyrie directly into our app so we can control it better and not incur the cost of a react + deps bundle.

Makes sense to bundle. It's ours, we can count it as a dev feature.


Update README + Fix examples formatting

Doesn't really change anything in the code. Just fixes some formatting and adds typing for more readablity.


Unsafe macro in cfe_version.h

Is your feature request related to a problem? Please describe.
Unsafe macro (violates coding standard)
https://github.com/nasa/cFE/blob/06c08268f2d7a7fe72aceec8c33201ce6ccba2f4/fsw/cfe-core/src/inc/cfe_version.h#L48
Describe the solution you'd like
Add parentheses
Describe alternatives you've considered
None
Additional context
None
Requester Info
Jacob Hageman - NASA/GSFC

Nevermind.  Read up on it, since it quotes it's safe.

Nevermind. Read up on it, since it quotes it's safe.

Can you add the stuff you read so we can reference it if it comes up again? I was actually discussing this earlier and wondering if we need to have the integer macros at all. If they're all strings then we wouldn't need the helper functions.

My preference is to remove any unused macros!  I just brushed up on the preprocessor macro at https://www.cprogramming.com/reference/preprocessor/stringizing-operator.html, and wrote a quick test program to try and break it and I couldn't (just to confirm it behaved as expected on random strings).


Upgrade jawn-fs2, keypool, log4cats, vault to last CE2 versions

Looks like not.  It died on a PGP password from an archaic build setup.  I just did it manually, but won't be awake to see the results.  Anyone west or east of me, feel free to restart it.


Add secret token management commands

Regenerate the deployment token for a SWA (needs a big warning here)
Retrieve the deployment token for a SWA

Related to #231 which needs more discussion.


Fix missing trace in CheckResponse

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 1 (0.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 61.223%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

cache/check.go
0
1
0.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
2793

Relevant Lines:
4562

💛  - Coveralls


Urgent: App Crashing in Production after adding customer.io SDK

I just released a new version of my app which ships with the customer.io React Native SDK.
I'm getting many different crashes all related to the SDK all of them seem to have the common pattern of attempting to dereference a garbage pointer:
EXC_BAD_ACCESS
Exception 1, Code 8, Subcode 8 > Attempted to dereference garbage pointer 0x8000000000000008.

Here are some of the crashes:

Occur after some call to the Dictionary Dictionary._Variant.setValue which is called within DIGraph.sharedSdkConfigStore.getter
When calling DIGraph.sharedSdkConfigStore.getter
Calling Dictionary.filter inside CioHooksManager.deviceAttributesHooks.getter
Calling DIGraph.sharedQueueRequestManager.getter

This seems to be happening to users in both iOS 15 and 16 so it's not isolated to the new iOS 16 release. My current Xcode version is 14.0.1 and I am using Objective-C.

Another crash report came up:
SIGABRT
> (4655,0x16dcf3000) malloc: Incorrect checksum for freed object 0x103e34ef8: probably modified after being freed.
Corrupt value: 0x3ffffffe00000000

Which occurred at: CustomerIO.postInitialize (CustomerIO.swift:250)

Hi @Shahroz16, thank you for following up on this issue. Here is some code related to the setup of the SDK. Please let me know if there's anything else that I can share that would be helpful. In the meanwhile I will take a look at the logs to see if anything comes up.
I have created a handler file like in the docs:
import Foundation
import CioMessagingPushAPN
import UserNotifications

@objc
public class CioPushNotificationsHandler : NSObject {

  public override init() {}

  @objc(registerPushNotification:)
  public func registerPushNotification(withNotificationDelegate notificationDelegate: UNUserNotificationCenterDelegate) {

    UNUserNotificationCenter.current().getNotificationSettings(completionHandler: { settings in
      guard settings.authorizationStatus == .authorized else { return }
      DispatchQueue.main.async {
        UIApplication.shared.registerForRemoteNotifications()
      }
    })
  }

  @objc(application:deviceToken:)
  public func application(_ application: UIApplication, didRegisterForRemoteNotificationsWithDeviceToken deviceToken: Data) {
    MessagingPush.shared.application(application, didRegisterForRemoteNotificationsWithDeviceToken: deviceToken)
  }

  @objc(application:error:)
  public func application(_ application: UIApplication, didFailToRegisterForRemoteNotificationsWithError error: Error) {
    MessagingPush.shared.application(application, didFailToRegisterForRemoteNotificationsWithError: error)
  }

  @objc(userNotificationCenter:didReceiveNotificationResponse:withCompletionHandler:)
  public func userNotificationCenter(_ center: UNUserNotificationCenter, didReceive response: UNNotificationResponse, withCompletionHandler completionHandler: @escaping () -> Void) {
    let handled = MessagingPush.shared.userNotificationCenter(center, didReceive: response,
  withCompletionHandler: completionHandler)

    // If the Customer.io SDK does not handle the push, it's up to you to handle it and call the
    // completion handler. If the SDK did handle it, it called the completion handler for you.
    if !handled {
      completionHandler()
    }
  }
}

And the relevant parts in the AppDelegate.mm file:
CioPushNotificationsHandler* pnHandlerObj = [[CioPushNotificationsHandler alloc] init];

@implementation AppDelegate

- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions
{
  RCTAppSetupPrepareApp(application);

  RCTBridge *bridge = [[RCTBridge alloc] initWithDelegate:self launchOptions:launchOptions];

  // Define UNUserNotificationCenter
  UNUserNotificationCenter *center = [UNUserNotificationCenter currentNotificationCenter];
  center.delegate = self;

  [RNBranch.branch delayInitToCheckForSearchAds];

  // When in DEBUG mode, iAd framework will return fake Apple Search Ads payloads to simulate the install
  // being claimed by Apple Search Ads. Call this so that Branch won't callback with Apple's test data, this is still sent to the server.
  [RNBranch.branch ignoreAppleSearchAdsTestData]; // **IMPORTANT**: Comment this when you want to test search ads in DEBUG environment
#ifdef DEBUG
  [RNBranch.branch enableLogging];
#endif
  [RNBranch initSessionWithLaunchOptions:launchOptions isReferrable:YES];

  [ReactNativeNavigation bootstrapWithBridge:bridge];

  if ([FIRApp defaultApp] == nil) {
    [FIRApp configure];
  }

  // Initialize Facebook SDK
  [[FBSDKApplicationDelegate sharedInstance] application:application
                           didFinishLaunchingWithOptions:launchOptions];

  // FIX white modal background
  // @ref: https://github.com/wix/react-native-navigation/issues/5618
  self.window.backgroundColor = [UIColor blackColor];

  // Register for remote push notifications w/ cio
  [pnHandlerObj registerPushNotification:self];

  return YES;
}

// Required for the register event.
- (void)application:(UIApplication *)application didRegisterForRemoteNotificationsWithDeviceToken:(NSData *)deviceToken
{
  [RNCPushNotificationIOS didRegisterForRemoteNotificationsWithDeviceToken:deviceToken];

  // notify AppsFlyerLib
  [[AppsFlyerLib shared] registerUninstall:deviceToken];

  // Register device to receive push notifications with device token
  [pnHandlerObj application:application deviceToken:deviceToken];
}

// Required for the notification event. You must call the completion handler after handling the remote notification.
- (void)application:(UIApplication *)application didReceiveRemoteNotification:(NSDictionary *)userInfo fetchCompletionHandler:(void (^)(UIBackgroundFetchResult))completionHandler
{
  [RNCPushNotificationIOS didReceiveRemoteNotification:userInfo fetchCompletionHandler:completionHandler];
}

// Required for the registrationError event.
- (void)application:(UIApplication *)application didFailToRegisterForRemoteNotificationsWithError:(NSError *)error
{
  [RNCPushNotificationIOS didFailToRegisterForRemoteNotificationsWithError:error];
  [pnHandlerObj application:application error:error];
}

- (void)userNotificationCenter:(UNUserNotificationCenter *)center
didReceiveNotificationResponse:(UNNotificationResponse *)response
         withCompletionHandler:(void (^)(void))completionHandler
{
  [RNCPushNotificationIOS didReceiveNotificationResponse:response];

  if ([response.notification.request.trigger isKindOfClass:[UNPushNotificationTrigger class]]) {
    // User did tap at remote notification
    [pnHandlerObj userNotificationCenter:center didReceiveNotificationResponse:response withCompletionHandler:completionHandler];
    return;
  }

  // Write data in async storage
  RCTAsyncLocalStorage *storage = [[RCTAsyncLocalStorage alloc] init];

  dispatch_async(storage.methodQueue, ^{
      NSMutableDictionary *formattedNotification = [NSMutableDictionary dictionary];

      formattedNotification[@"actionIdentifier"] = response.actionIdentifier;

      NSDateFormatter *formatter = [NSDateFormatter new];
      [formatter setDateFormat:@"yyyy-MM-dd'T'HH:mm:ss.SSSZZZZZ"];
      NSString *dateString = [formatter stringFromDate: [NSDate date]];
      formattedNotification[@"date"] = dateString;

      NSError *error;
      NSData *jsonData = [NSJSONSerialization dataWithJSONObject:formattedNotification
                                                         options: 0 // Pass 0 if you don't care about the readability of the generated string
                                                           error: &error];
      if (!jsonData) {
          NSLog(@"Got an error: %@", error);
      } else {
          NSString *jsonString = [[NSString alloc] initWithData:jsonData encoding:NSUTF8StringEncoding];

          if ([storage respondsToSelector:@selector(multiSet:callback:)]) {
              [storage performSelector:@selector(multiSet:callback:)
                withObject:@[@[@"@NOTIFICATION_ACTION_RESPONSE", jsonString]]
                withObject:[^(NSArray* res) {
                  NSLog(@"Contents: %@", res);
              } copy]];
          } else {
             NSLog(@"storage does not respond!!");
          }
      }
  });

  // Delay 5 seconds
  dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(30 * NSEC_PER_SEC)), dispatch_get_main_queue(), ^{
      completionHandler();
  });
}

-(void)userNotificationCenter:(UNUserNotificationCenter *)center
      willPresentNotification:(UNNotification *)notification
        withCompletionHandler:(void (^)(UNNotificationPresentationOptions options))completionHandler
{
  // show push when the app is in foreground
  completionHandler(UNNotificationPresentationOptionAlert + UNNotificationPresentationOptionSound);
}

Just another note, I did find out that my project and target SWIFT_VERSION is at 4.0. I've changed it to 5.0 and submitted a release to see if that has any effect. It's still awaiting review from Apple.
Would you have any idea if this could somehow be related?

It could be because EXC_BAD_ACCESS was very common in SWIFT_VERSION = 4, you can check out this github issue as well for Swift.
In the production build, when you upgrade the swift version can you reproduce this issue?

Unfortunately, I've been unable to reproduce the issue on my end and have been only able to observe this in the wild through reports in Sentry from my users. Doesn't seem to be affecting everyone, but it is a fairly large percentage.
Hopefully I can get the release out soon to see the effect, but if that would be the cause of the EXC_BAD_ACCESS... any ideas on the SIGABRT issue?

I feel like they might be both related as well.
can you please explain why is there a 5 secs delay?
// Delay 5 seconds
  dispatch_after(dispatch_time(DISPATCH_TIME_NOW, (int64_t)(30 * NSEC_PER_SEC)), dispatch_get_main_queue(), ^{
      completionHandler();
  });

In the method you referenced, we do some cleanup as well. Could it be the case, where you try to access something after it was cleaned up?
private func postInitialize(siteId: String) {
        let diGraph = DIGraph.getInstance(siteId: siteId)

        // Register Tracking module hooks now that the module is being initialized.
        let hooksManager = diGraph.hooksManager
        hooksManager.add(key: .tracking, provider: TrackingModuleHookProvider(siteId: siteId))

        cleanupRepository = diGraph.cleanupRepository

        // run cleanup in background to prevent locking the UI thread
        threadUtil?.runBackground { [weak self] in
            self?.cleanupRepository?.cleanup()
            self?.cleanupRepository = nil
        }
    }

I am on the latest RN SDK: 1.0.0-beta.1
The 5 second delay is just some workaround that had to be implemented to make notification actions work on the local notifications when the device is in the background/suspended. I believe due to the if statement here:
  if ([response.notification.request.trigger isKindOfClass:[UNPushNotificationTrigger class]]) {
    // User did tap at remote notification
    [pnHandlerObj userNotificationCenter:center didReceiveNotificationResponse:response withCompletionHandler:completionHandler];
    return;
  }

That the delay would never be reached when receiving a remote notification anyway.
Also, it looks to be that this is always happening on the first app launch (initial download of the app) so I don't think this would be executed anyway because there would be no notifications being received at that point and I think didReceiveNotificationResponse only gets called upon interacting with a notification (via opening or pressing on an action; though I could be wrong).

@ninjz we are truly sorry for this, but as you cant replicate we can't replicate this on our end as well. But we are trying to debug and see where the issue lies and how we can fix it asap.

Thank you for your efforts. Please do let me know if there's anything I can do to

@Shahroz16 I saw this response on stack overflow:
https://stackoverflow.com/a/28910283/1749580
To quote:

MUTABLE dictionaries and arrays (as well as NSMutableDictionary and NSMutableArray) aren't thread safe, and problems arise when using them within Singletons that are accessed from multiple threads

I see that the DIGraph class is in fact a singleton and uses mutable dictionaries:
https://github.com/customerio/customerio-ios/blob/develop/Sources/Common/DIGraph.swift
This to me looks to be the problem, but seeing as the iOS SDK has been out and stable for quite some time now could this just be because of how the RN SDK is accessing this singleton (maybe via multiple threads?)

I have the exact same problem : I just tracked screens + a dummy event every time I change screen and after a few changes in screen it crashes. Hope it helps to reproduce on your side

@matthieuMay So you're receiving a similar stack trace? I don't see anything related to screen tracking in my stack trace and am not able to reproduce it on my end. Are you testing on a device or simulator?

Hi @ninjz @matthieuMay
Jumping in to provide more support on the issue.
@matthieuMay Would you be comfortable sharing code snippet for screen tracking and the dummy event ? This might help us reproduce the issue on our end and come up with a solution.

@Shahroz16 I saw this response on stack overflow: https://stackoverflow.com/a/28910283/1749580
To quote:

MUTABLE dictionaries and arrays (as well as NSMutableDictionary and NSMutableArray) aren't thread safe, and problems arise when using them within Singletons that are accessed from multiple threads

I see that the DIGraph class is in fact a singleton and uses mutable dictionaries: https://github.com/customerio/customerio-ios/blob/develop/Sources/Common/DIGraph.swift
This to me looks to be the problem, but seeing as the iOS SDK has been out and stable for quite some time now could this just be because of how the RN SDK is accessing this singleton (maybe via multiple threads?)

Hi @ami-aman, thanks for jumping in. Would you be able to comment on this?
I've gone through the code on the Customer.IO iOS SDK and it does seem like there are measurements in place to make the dictionary thread-safe. However, I just discovered in my app because I have multiple notification libraries for handling local notifications I ended up making multiple calls to request remote notifications which does result in multiple invocations of the didRegisterForRemoteNotificationsWithDeviceToken delegate. I suspect that this would then result in multiple calls to the DIGraph singleton class.
I'm pushing out a new release now that removes that extra call which I had running from the JS side to see what happens, but I have no idea if this will be successful.

Thanks for the reply @ami-aman, so my latest release has gone live and it looks to me like the frequency of the crashes has lessened, though I don't want to speak too soon as there hasn't been a high level of adoption of this new release yet. If this turns out to be the solution it would be good to update the docs for future users of the SDK to warn about this case (yes I am using RNPushNotificationIOS for handling local notifications.
Anyhow, I still received 3 crashes related to the SDK in my latest release and I've noticed 2 things:

One crash occurs when initializing In-App Messaging, and I noticed that there are two threads (inclusive of the thread that the crash occurred) which is executing code from the CustomerIO SDK, which I believe must be the suspect. The other thread seems to be executing CustomerioReactnative.setDeviceAttributes and I see that initializeInApp is in fact being run on another thread: https://github.com/customerio/customerio-reactnative/blob/b7257fe07408702fba5af2ed64305948c243a49a/ios/CustomerioReactnative.swift#L109-L111
The other crash occurs just after calling CustomerioReactnative.setDeviceAttributes and I do not observe another thread executing anything related to CustomerIO SDK. So it seems there must be something going on when setting device attributes... In my code, here is how the SDK is initialized:

export default class CioService {
  static async init() {
    const data = new CustomerioConfig()
    data.logLevel = CioLogLevel.debug

    const env = new CustomerIOEnv()
    env.siteId = config.CIO_SITE_ID
    env.apiKey = config.CIO_API_KEY
    env.organizationId = config.CIO_ORGANIZATION_ID
    env.region = Region.US

    CustomerIO.initialize(env, data)

    // Set device attributes to be tied to this specific device
    const deviceAttributes = {
 	timezone: RNLocalize.getTimeZone(),
    };
    CustomerIO.setDeviceAttributes(deviceAttributes)
  }
  ...
}

Sry ended up closing this by accident in a private commit reference.
Just another update, I ended up having the crash occur on my end when testing another build on Testflight. Again, it's not clear how to reproduce because it's just happening sporadically and I've tried dozens of times and still unable to.
A look at the crash logs, this one seems to be different from the previous 2 cases I mentioned in my last comment. Again it's an EXC_BAD_ACCESS exception occurring when accessing the dictionary, but this one is coming from a thread calling the application:didRegisterForRemoteNotificationsWithDeviceToken delegate method again and I've already removed the call on the JS side which means the delegate is not called twice anymore...
Prior to this call there is a call to -[GULAppDelegateSwizzler application:donor_didRegisterForRemoteNotificationsWithDeviceToken:] which I believe is something related to a service required by Firebase SDK.
Not sure if that is of any interest, but I did look up the active threads again and saw another thread which is also accessing DIGraph.sharedLockManager which originated from a call to  CustomerioReactnative.initialize.
Could it be that there is a race condition between the initialization of the CustomerIO SDK on the JS side and when the callback to the application:didRegisterForRemoteNotificationsWithDeviceToken delegate is invoked?

Update: My latest release seems to be crash-free. I removed the initialization of the SDK on the JS side and initialized the SDK + set the device attributes natively on iOS. So far, so good.

Hi @ninjz
Thanks for your incredibly detailed comments. This definitely helps us a lot.
We have been investigating on the issue at our end as well and we suspect the same i.e a possible race condition which eventually leads to a crash. I will make sure to update you on this as we fix the issue. Meanwhile, I would appreciate you sharing your finding with us as it helps a lot as a possible lead to the issue.

Hi @ninjz
Thanks for your patience here.
We have deployed a new package release - version 1.0.0-beta.3. With this change, we're no longer see crash in our test app. Can you confirm if this update works fine in your app ?

Hi thanks for the update @ami-aman
Honestly, right now the way I have it set up by initializing the SDK on the native side on iOS I haven't seen any crashes yet so I'm a bit hesitant to update to the new version.
Were you able to reproduce the issue on your end?
Also the commit diff seem to just show an upgrade to the iOS SDK dependency. Was there something that was fixed on that side that contributed to the fix? Would be great to know more about what actually went on to fix this.

I totally understand your hesitation, @ninjz !
We were able to reproduce with the steps mentioned by matthieuMay i.e. change screens / call screen tracking function with other events and we did observe the crash which was happening in iOS SDK, which has been fixed !

@ami-aman Would you be able to point to me where in the changes to the iOS SDK that the fix was applied to so I can also have a look myself? Thanks

@ninjz In one line, SDK was creating new instances every time in a scenarios which was causing the crash. You may check out the latest released version of our iOS SDK here.

Closing due to inactivity. Feel free to re-open if required !


Deselecting an element should select root (vs. hide tree). Blank slate moves to properties

I think this is a regression on develop right now, the tree view should ideally never be blank, as I might want to select or search the tree for the element (vs. select in the UI)

I do like the blank slate for the selection though — ideally that that blank slate belongs in the properties tab.

I pushed quick fix to develop. It should unblock using develop.


Remove ${JAVA_VERSION} from native lib path for jlink test

We do not require ${JAVA_VERSION} to be provided as part of the path to native library for the jlink test. Moreover, removing this will mean openjdk.test.modularity/build.xml do not have any dependency on ${JAVA_VERSION}.

Fixed via https://github.com/AdoptOpenJDK/openjdk-systemtest/pull/121


Fix escapes in tools and scripts

Related command

Description
Related to #28829.

[x] The PR title and description has followed the guideline in Submitting Pull Requests.

[x] I adhere to the Command Guidelines.

[x] I adhere to the Error Handling Guidelines.

Thank you for your contribution! We will review the pull request and get back to you soon.


STL file provided as is needs supports(if its possible to print cleanly at all) due to 2 items in one STL

never mind

Reasonable feedback - we are going to update the STL with a new, more discrete design shortly.


[LS] Update add contract API change

Description
Update flowkit API change for adding contracts.

[x] Targeted PR against master branch
[ ] Linked to Github issue with discussion and accepted design OR link to spec that describes this work
[x] Code follows the standards mentioned here
[x] Updated relevant documentation
[x] Re-reviewed Files changed in the Github PR explorer
[x] Added appropriate labels

Sorry yeah, this PR should be draft until I update the CLI version. Will update.


Feat: round trip time metrics for request/response cycles

This PR introduces some internal RTT metrics for certain request/response cycles. It also updates the grafana dashboard to display these. Unfortunately there is currently no perfectly clean way of introducing them with feature flags so the occasional Option is necessary.
The Pong message isn't technically a cycle but as it does some operations based on the block locators, I thought it would be interesting to have a sense of how expensive that is.

The Grafana dashboard can be seen here: https://github.com/AleoHQ/snarkOS/pull/1720#issuecomment-1088637071

Rebased.

More broadly as a question, is there really no better way to integrate metrics into the structs other than to just inject commands into the codebase? I would be surprised if this was the best way to design a solution for metrics

Rebased.


[Question] Is project dead?

The project hasn't been updated for about a month now..
I wonder whether it is still alive..

It surely is, but it is vacation time ;-)

Good! 😀


Derive more traits for events::ModifiersState

Thanks! Please add an entry in the changelog as well!

Okay, I added an entry to the changelog. Thanks for creating winit; it's making my window-management code simple!


Primitive scraper for SH

echo SH; d=$(curl --silent 'https://sh.ch/CMS/content.jsp?contentid=3209198&language=DE&_=1584807070095' | grep data_post_content | sed -E -e 's/\\n/\n/g'); echo "Scraped at $(date --iso-8601=seconds)"; echo -n "Date and time: "; echo "$d" | grep "Im Kanton Schaffhausen gibt es" | sed -E -e 's/^.*\(([0-9.]+)\).*$/\1/'; echo -n "Confirmed cases: "; echo "$d" | grep "bestätige" | sed -E -e 's/^.*strong>([0-9]+)[^0-9]*$/\1/'

SH
Scraped at 2020-03-21T16:19:46+00:00
Date and time: 20.03.2020
Confirmed cases: 14

This is really hacky, because sh.ch is absolutely abhorrent with amount of JavaScript and content that is loaded dynamically. But it works.
The URL that generates json content, was reverse engineered from looking at network traffic when loading this site: https://sh.ch/CMS/Webseite/Kanton-Schaffhausen/Beh-rde/Verwaltung/Departement-des-Innern/Gesundheitsamt-3209198-DE.html
I have no idea what the _ parameter is, some kind of timestamp I think, but I am not sure. Could be caching, and might not work. To be seen.

;) thanks for hacking!

Please do UR too :)

@baryluk great hacking Sir, great Hacking!

Please do UR too :)

Done. https://github.com/openZH/covid_19/issues/72


Image Corruption

I'm getting what looks like an image corruption. I was able to reproduce this in the Cube demo by adding VK_IMAGE_USAGE_SAMPLED_BIT to the depth image, using VK_IMAGE_TILING_OPTIMAL for depth and texture_object images, and allocating the depth and texture_object images with VulkanMemoryAllocator (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator). This could be a bug in VMA, but I thought I'd bring it here first.
This does not occur when running the Cube demo on iOS. Only Mac. My MacOS is 10.15.3 with Radeon Pro 570 4 GB.
I'm testing with the latest commit bfde618. I've attached the modified cube.c. I attached vk_mem_alloc.h and vma.cpp which I add to the Cube demo project. I also attached a screenshot of the corruption I'm seeing when I run the Cube demo.
In my own similar demo it seems like the corruption happens when I begin/end the related renderpass. If I submit the command buffer with just the begin and end of the renderpass, with nothing in between, and afterwards transfer the texture_object image to a file I can view, then it will be corrupted.
Archive.zip

Thanks for submitting the sample app demonstrating the issue. The error appears in my environment too.
I've done a fair bit of digging into it. The error is triggered by a long combination of things:

VMA uses a single large underlying VkDeviceMemory to bind to both the depth image and the samplable texture being loaded from file (via a buffer copy). The native Cube demo uses separate device memory instances. Changing back fixes this issue.

MoltenVK uses a placement MTLHeap within the device memory to create the underlying MTLTextures for each image. Configuring MoltenVK to avoid using MTLHeaps also fixes this issue.

The issue only appears if the the depth image includes samplable usage. Changing this back also fixes the issue.

There doesn't seem to be a clear reason why this is happening. I've verified that VMA is picking byte offsets and alignments for the MTLTextures that should avoid conflicts within the placement MTLHeap memory space. And it's hard to see why modifying the usage property should naturally have any effect on where in the heap the memory is allocated from.
At this point, I'm suspecting a possible bug within the Metal driver that triggers a layout conflict inside the placement MTLHeap when using samplable depth images.
I don't know if we can do much of anything about this within MoltenVK, other than to possibly provide an environment variable to optionally allow an application to disable MTLHeaps if they encounter this issue.

@DiegoAce PR #845 fixes this issue. Please retest and close this issue if it fixes the problem you encountered.

I'm not getting the corruption anymore relating to the conditions above. Thanks! I'll close the issue. Although, I'm still getting an identical looking corruption in another part of my project that I thought was related to this, but perhaps it's not. It's not a critical part of my project and I fear tracking it down will be difficult. So I may not do so, but if I do I'll re-open this issue.


The game shouldn't keep running while backgrounded

Causes battery death / excessive heat production on mobile devices, even though nothing is being displayed.

May be specific to xamarin debug scenarios, but should investigate regardless that we are doing suspension correctly.

I'm exploring this now.
I tested in iOS land that upon locking the device, or dismissing the app, all of the appropriate UIApplicationDelegate callbacks (eg, "device will suspend") are being called properly without issue.
That being said, I'm using Activity Monitor in Instruments to detect how much CPU usage osu!framework tests takes up.
At any point when open, osu!framework consumes 70% of the CPU load of my iPad Pro. If I lock the device, usage drops to 1%.
But. If I dismiss the app and return to the Home Screen, without actually closing the app in the background, the CPU usage doesn't drop and continues at 70% while the app is completely not in the foreground.

This screenshot was taken while the iPad was unlocked, but osu!framework was in the background.
FWIW, this is a Debug build of osu!framework.

Did a stack trace in Instruments Time Profiler.

Closed via #3367 (to some extent). There are still some background threads we aren't yielding, but at very least it should be greatly improved to the point of not affecting battery life.


add new achievemens - new branch for achievements 2024

Please review my new achievements.

Fun fact: Windows and Unix are different in that one is case-sensitive (Unix) and the other is not (Windows). So I'm guessing the capital-letters name doesn't make a difference on Windows in terms of ref, but does on Unix/MacOS.


Estimate gas

Feature: Estimate gas for contract deployment using ethgasnow API.
make contract=Greeter estimate
Questions:

Should this be part of the dapp native toolchain? So, perhaps PR it into there as well
Should we change deploy as well, so that we can invoke it with make contract=Greeter deploy ? Currently, contract name is hardcoded into deploy.sh
jq, tput, column are implicitly dependencies. I assume that they exist in most POSIX shells, but just in case, should we put them as dependencies in the nix package manager?

I think installing dapp installs jq as well, so shouldn't be a problem. The Greeter contract is intentionally hardcoded, when you write a deploy script you write all the contracts you wanna deploy and let it run for all of them. You can ask in their repo, it can prob be useful yeah.

btw it might me better to use another gas API coz the gasnow API will be ended
https://twitter.com/gasnow_org/status/1443511830253498374?s=20

yeah @pyk,
This feature will move upstream to dapp itself (there is an issue discussing it) and here we will change the code to use that.


DefectDojo SonarQube Integration

First of all, thank you for this amazing tool, I'm just starting to use it.
I would like to use this recent Integration, however I am unable to generate HTML report in SonarQube.
Sample HTML file can be obtained at https://github.com/DefectDojo/sample-scan-files/blob/master/sonarqube/sonarqube_v6.7.5.html and it works like a charm.
I am also using SonarQube 6.7.5 version, on its Community Edition.
Having a look to HTML sample code, it contains some banner about @AdrienGuillerme employer, which leads me to think, that provided HTML report is an internal report generated by them. If so, would you mind to share how you generate it? Maybe some template / SonarQube API calls.
Thanks in advance.

I will tak a look at it

Hi @AdrienGuillerme just wanted to checkin to see how the file is generated. Thanks!

Hi @AdrienGuillerme, any update on this?

also interested, for what it's worth :)

Hello,
I'm working for SonarSource the company making SonarQube. We are looking at the options to provide an official integration with DefectDojo that will be supported by SonarSource.
Do you know if DefectDojo is supporting a generic import format? Or shall we define our own export format and provide the importer in the DefectDojo repo?
Thanks

@agigleux That's fantastic news, we do have a generic csv or if it's easier you can define a json export and we can create it. Thanks!

@aaronweaver I tried to follow what is documented here and generate a file corresponding to it but no luck so far. Can you provide a sample CSV file so I compare with what I generated?
I used https://defectdojo.herokuapp.com/ and selected "Generic Findings Import". Do you confirm this instance allows upload of files, maybe that's the reason why I can't import it?

I'll take a look and get back to you shortly. Thanks!

Hi @agigleux, I updated the csv field here. If you need any additional fields let me know and I can add that to this importer. Thx

I'm able to use the "Generic Findings Import" thanks to your sample.
I have a couple of remarks:

the generic CSV format doesn't allow to upload Findings with Active=True and Verified=True: once the import is done, even by setting these columns to "True", the findings are always Inactive and Verified is unticked. Is that expected? In that case, what's the point to have these columns in the CSV format?
the CSV format is import Findings as "Dynamic finding" by default and it's not possible to specify that our Findings are generated by our SAST feature - so I can't set by default "Static finding" on Findings.
is there any field in DefectDojo where we can feed the Vendor URL where the user can look at the findings in the Vendor's UI? SonarQube is providing the full execution flow steps, and I believe it would be valuable to provide a direct access to such information from DefectDojo.
this is not related to this CSV import, but I'm very surprised it's possible to edit the Hash of a Finding - I believe this is what is used to guess that a Finding has been already imported and avoid too many duplicated entries. If I'm correct, this Hash data should be hidden.
Findings imported as visible as "Found by Generic Findings Import" : it means SonarQube name is not visible and that's a problem from a product promotion point of view, so I'm thinking more and more that we will have to define our own JSON format and have a dedicated importer developed.

Thanks

Hi @agigleux, Thanks for the comments. I'll file a bug report and work on those. You are correct hash shouldn't be editable. It would probably be a better integration with a custom importer via json. I agree that deep linking to the issue would be important as well.

@agigleux, regarding your last poing, I've been working on some improvements under the PR https://github.com/DefectDojo/django-DefectDojo/pull/843, many of them were basically extending the API functionalities to enable CRUD for some models, but the one of the most important features implemented on the API was to be able import findings specifying differents test_type and scan_type, for example, one of the problem we handled was we wanted to import findings using the Generic Findings Import, but we wanted to create a SonarQube test type instead of Generic Findings Import. I guess this feature can be also extended for any other kind of test_type, regardless of the scan_type you want to use. These feature was implemented on the API only, i do not recall if this is available through the GUI.

I'm able to use the "Generic Findings Import" thanks to your sample.
I have a couple of remarks:

the generic CSV format doesn't allow to upload Findings with Active=True and Verified=True: once the import is done, even by setting these columns to "True", the findings are always Inactive and Verified is unticked. Is that expected? In that case, what's the point to have these columns in the CSV format?
the CSV format is import Findings as "Dynamic finding" by default and it's not possible to specify that our Findings are generated by our SAST feature - so I can't set by default "Static finding" on Findings.
is there any field in DefectDojo where we can feed the Vendor URL where the user can look at the findings in the Vendor's UI? SonarQube is providing the full execution flow steps, and I believe it would be valuable to provide a direct access to such information from DefectDojo.
this is not related to this CSV import, but I'm very surprised it's possible to edit the Hash of a Finding - I believe this is what is used to guess that a Finding has been already imported and avoid too many duplicated entries. If I'm correct, this Hash data should be hidden.
Findings imported as visible as "Found by Generic Findings Import" : it means SonarQube name is not visible and that's a problem from a product promotion point of view, so I'm thinking more and more that we will have to define our own JSON format and have a dedicated importer developed.

Thanks

Hello @aaronweaver,
Finally SonarSource won't have the possibility to add a dedicated API to extract the data for DefectDojo.
Instead, we would like to try to reuse the issues/search API that is used to display the issues in SonarQube and also to extract issues in JSON format.
I wrote the specification of the mapping between SonarQube and DefectDojo here: https://jira.sonarsource.com/browse/MMF-1672
Can you take the lead on it and implement a parser.py dedicated to the SonarQube output?
I know that some data are missing like the CWE-ID but we can think about adding them later.
Thanks

@agigleux @aaronweaver @AdrienGuillerme sorry it didn't come earlier, I was able to get the agreement from our hierarchy to disclose the sonar export source code.
The code is in javascript; it's depoyed as an npm package
It's using sonar API to get the required information, notably

/api/rules/search : get the rules
/api/issues/search : get the vulnerabilities

It exports the list of vulnerabilities and rules to html, which can be parsed by the sonarQube parser that was previously commited to DefectDojo.
Not all the fields from https://jira.sonarsource.com/browse/MMF-1672 are implemented yet but it can be a starting point. Currently we have for each vuln:

Rule
Severity
Description
Message

Are you still interested?
The main difference with what Alexandre proposed is that we export html instead of json. It might be cleaner to use json, however it's handy to have a human readable format too. I don't think there's any free plugin to export to html from sonar. Last time I did that was with sonar preview mode - which may still work although it's been deprecated for some time.
Here's what the export looks like:

Hi,
I've released our sonar html export code here: https://github.com/soprasteria/sonar-report
Feedback is welcome

Hi,
I've released our sonar html export code here: https://github.com/soprasteria/sonar-report
Feedback is welcome

Hi @ptrovatelli  i can't find SOnarQube in my supported scan type, how to get it.

i am using following defectdojo code:
908118a: Merge pull request #1412 from Maffooch/dev [2019-07-30 08:47:59 -0500]

@praveendvd good question :) I need to re-activate it. Right now it's just commented out. I was waiting for some reactions on this thread, if it's fine by everyone I will reactivate it in dev branch and throw in a bit of documentation

@praveendvd good question :) I need to re-activate it. Right now it's just commented out. I was waiting for some reactions on this thread, if it's fine by everyone I will reactivate it in dev branch and throw in a bit of documentation

Hi @ptrovatelli thanks for your huge effort, the report looks great but i am facing one problem. I am getting only those bugs which have a vulnerability tag.
SonarQube with filter:

SonarQube without filter:

Sonar-html Report:

The html report shows only the 4 vulnerabilities , is it the expected behavior ?. Could you please let me know about any work around to make full report ?.
Thank you so much @ptrovatelli

@praveendvd thank you; I didn't code it myself though :)
about your question, that project was done specifically to export vulnerabilities, not code smells and other things you'll find in sonar. If you want to export everything, you can tune the filters and remove "types=VULNERABILITY" (see index.js).

@praveendvd thank you; I didn't code it myself though :)
about your question, that project was done specifically to export vulnerabilities, not code smells and other things you'll find in sonar. If you want to export everything, you can tune the filters and remove "types=VULNERABILITY" (see index.js).

@ptrovatelli !!! Thank you so much, it worked perfectly. Does defectdojo sonarqube integration supports uploading the entire result like the one i have provided below?
For any one who wants to do the same , below are the steps :

search for the sonar-report installation :

sudo find / -name sonar-report

navigate to the lib directory from the search result

cd <path>/node-v10.16.0-linux-x64/lib/node_modules/sonar-report

Edit the index.js

vi index.js

search for 'types=VULNERABILITY &' and remove it

Now run the command :

@praveendvd yes it should work as well. right now it's broken on the dev branch due to python3; i'm looking into it. If you're using a python2 branch (master or legacy-python2.7) it should work after uncommenting ("SonarQube Scan", "SonarQube Scan"), in dojo/forms.py (and rebuilding the images if using docker)

@praveendvd yes it should work as well. right now it's broken on the dev branch due to python3; i'm looking into it. If you're using a python2 branch (master or legacy-python2.7) it should work after uncommenting ("SonarQube Scan", "SonarQube Scan"), in dojo/forms.py (and rebuilding the images if using docker)

I am getting below error for all the uploads in master branch through importscan API. It seems like master branch upload api is broken.

In dev branch everything is working fine
.
Will wait for the dev branch fix  :) thank you @ptrovatelli

@praveendvd  I have just made a PR with the fix for python3 and other improvements.
See https://github.com/DefectDojo/django-DefectDojo/pull/1445
You'll be glad to see that I have also added a parameter to export all sonar bugs, not only vulnerabilities in the sonar-report export code :)
Last version can be found here https://github.com/soprasteria/sonar-report (the npm registry is not up to date yet)

@praveendvd feel free to try this PR too that brings a different approach, with a 2-way integration with snoarqube:
https://github.com/DefectDojo/django-DefectDojo/pull/1444

@praveendvd feel free to try this PR too that brings a different approach, with a 2-way integration with snoarqube:
#1444

sure @ptrovatelli will give it a look and try to give you the feed back at the earliest as i can. Thank you so much.

@praveendvd feel free to try this PR too that brings a different approach, with a 2-way integration with snoarqube:
#1444

sure @ptrovatelli will give it a look and try to give you the feed back at the earliest as i can. Thank you so much.

Hi @ptrovatelli ,
I cloned and build the below repository:

https://github.com/twsagarcia/django-DefectDojo.git

and installed new sonar-report using below command:

npm install -g https://github.com/soprasteria/sonar-report.git

I was able to upload default report which takes in only vulnerabilities:
File used :
Output:

But for full report , it throws parsing error:
File used:
Output:

@praveendvd the parser wasn't meant to import non-vuln bugs. there may be a field missing on those. i reproduced with the tomcat report; i'll have a look.

@praveendvd the parser wasn't meant to import non-vuln bugs. there may be a field missing on those. i reproduced with the tomcat report; i'll have a look.

Thank you so much @ptrovatelli  , i also checked the second solution you suggested.
but i am getting below error message

my sonar server is at localhost:9000 and my DefectDojo is at localhost:8000
i also tried giving url field in the tool configuration as 'http://localhost:9000/api' but still gets same error.
I am able to access the http://localhost:9000/api/projects/search?q=Juice , through browser

@praveendvd I haven't tried this one yet. have you tried the last version? (last commit 3 days ago). If it's still not working i advise to continue the discussion here https://github.com/DefectDojo/django-DefectDojo/pull/1444 or on slack if @twsagarcia  is there

@ptrovatelli
I tested your fix , it works perfectly. I was able to upload both the vulnerability and full report. I will try with API on monday.  Thank you so much for the help

sonarqube was reactivated in dev branch: see https://github.com/DefectDojo/django-DefectDojo/pull/1445
There is also another solution in development with 2-way integration with sonar: see https://github.com/DefectDojo/django-DefectDojo/pull/1444


Fix and_wrap_original failing with ruby 3.2

This spec only fails in ruby 3.2, it succeeds with ruby 3.1. The failure in 3.2:
     ArgumentError:
       wrong number of arguments (given 1, expected 0; required keyword: my_object)
     Shared Example Group: "using and_wrap_original" called from ./spec/rspec/mocks/and_wrap_original_spec.rb:121
     # ./spec/rspec/mocks/and_wrap_original_spec.rb:27:in `initialize'

Closing as my spec was incorrect, it should have been:
allow(MyService).to receive(:call_now).and_wrap_original do |blk, **args|


[Feature] Show daily/weekly totals

For the today / last week / all time section, I would find it helpful if it showed the total count of how many challenges I completed. I'm usually aiming to do 5-10 a day and currently have to remember how many I've done.

Honestly, I agree as well, unfortunately the developers aren't focusing on this project anymore


Replace share extension with action extension

After reading a discussion on Slack I realised that the current extension for subscribing to a feed using the share sheet (in Safari or another app) is wrong. There are two types of extensions in the share sheet: share extensions and action extensions (see the iOS HIG). Share extensions are meant for sharing content, and action extensions for performing tasks.
Currently the subscribe to feed extension is a share extension. Since subscribing to a feed is a task it should instead be a action extension. The different styling applied to action extension should also hopefully help with discoverability (which was the topic in the discussion on Slack).

(The placement in the list of available action can be configured by the user, and by default seems to be done by some sort of frequent use algorithm)

That was an intentional abuse of the HIG and a pretty common abuse by other apps.
My thinking was that the Share extensions are more noticeable and thus more discoverable.  As you point out, that might not be the case in iOS 13 since iOS 13 seems to surface Action extensions without having to directly edit their availability.


fix(query/influxql): normalize selectors when an interval is present

The selector will be normalized to the start time interval when GROUP BY time(X) is present in the query.
Fixes #10747.

Those are skip test declarations.


Updates for latest compiler

Thanks for making this! :)
This PR updates the code so it will work with the latest PureScript compiler. Also, I made a note about how to install mustache and compiling for the front-end in the README (you'll have to use pulp browserify instead of pulp build).

@dgendill hello Dominick, thanks for taking a look and for updating this project to the latest compiler!
I haven't used PureScript for some time but I've updated everything locally and your changes work fine with the latest compiler.
The pull request didn't appear to branch off the master branch:

so to keep the project history clean I have combined your two commits and applied them to the master branch (https://github.com/svisser/purescript-mustache/commit/f49b300b061cb1123cf5e278d6415f366ca396bf).
I've also tagged the project as v3.0.1. If everything works for you I'll close this pull request.

Oops!  Sorry about that, I did the forking process a little differently than I normally do.
I pulled the tag you made into my project and it looks like everything is working.  Thanks!


cordova add plugin error

I'm using cordova building ionic project, when i add my custom plugin by
<framework src="SUPRAcceptSDK/All" type="podspec" spec="2.2.3" />
It should auto pod install SUPRAcceptSDK/All in my project. But i got this error:
[!] <PBXCopyFilesBuildPhase name=Embed Frameworks UUID=A4750ADD1CE641F29F38C7DC> attempted to initialize an object with an unknown UUID. 826B38BCAF324BA8BBAE5F52 for attribute: files. This can be the result of a merge and  the unknown UUID is being discarded.
(Success when i add my other plugin the same way)

Hello yujido,
It seems that Cordova does not support podspecs with embedded frameworks.
So I can suggest you two possible options to try:

Install Cordova plugin which allows you to automatically install pods:
https://github.com/blakgeek/cordova-plugin-cocoapods-support

Create your own custom Podspec file where you can specify submodules you want to use (eg. Core and Spire).

If no one from provided solutions will work please provide more specific steps how to reproduce your issue. Thank you.

It seems a cocoapods problem. Already solved.


Group of Persons, Agent, Person

Is every Person an Agent? Regardless of cognitive development? If not, why is Group of Persons a subclass of Group of Agents? Shouldn't there be groups containing also Persons who are not Agents?
Person has alternative label "human". From which point on is an Organism a Person/human? Are there Persons/humans before birth or maybe even before nidation?

We don't make any commitment as to when in the developmental process a human comes into existence. I understand there is a sense of 'person' that invokes the notions of agency and legal status, and perhaps the deeper existential question as well, but our definition makes it clear (I hope) that CCO Person does touch on those things. In CCO a Person, aka human, is just that, a member of the species, so defined, as current understanding dictates.
We use the term Agent simply to classify things which are intentionally active and causal participants in processes. Agents are things that perform intentional acts. We don't offer a formal treatment of intentionality and thus it can be regarded as primitive.
I agree that Group of Persons shouldn't be explicitly asserted as a sub of Group of Agents. Will update that for next release.

Thanks for the helpful reply.
I would submit that Organization shouldn't be subclassed under Group of Agents either. There seem to be examples of members of organizations where it's not clear that they must be agents. For example, a newborn could become shareholder (and thus member of a public corporation?) by inheriting shares.

I suggest that in addition to the reasons given above, the term 'person' could follow BFO design principles more closely.
'person' is defined as "an animal that is a member of homo sapiens". But 'corporate person' (a legal status, as Mark mentioned) would intuitively be a subclass of 'person' and it would not be a subclass of 'animal'. I realize corporate personhood is contentious, but the term is codified in this way in various laws (US and perhaps elsewhere). It may matter in the industrial ontology foundry, for example, and so we should be sensitive to future-proofing. I don't have an interest in debating the possibility of sentient AI, immaterial angels, God, etc. that might also be counter-examples.
If 'person' just is an animal that is a member of homo sapiens and if 'person' often evokes such connotations of agency or legal status, why isn't the term just 'Human' or 'Human Person'? After all, whatever debates about when a human comes into existence (embryologists could tell us), if the class is to pick out humans and only humans irrespective of 'personhood debates', why not just avoid this minefield and go with 'human'?

Addressed in 1.5.


Add REST API GET, POST, PATCH tests for cloud storage

Motivation and context

PR contains REST API tests for cloud storage functionality
Resolves #3996
How has this been tested?

Checklist

[x] I submit my changes into the develop branch
[ ] I have added a description of my changes into CHANGELOG file
- [ ] I have updated the documentation accordingly
[x] I have added tests to cover my changes
- [ ] I have linked related issues (read github docs)
- [ ] I have increased versions of npm packages if it is necessary (cvat-canvas,
cvat-core, cvat-data and cvat-ui)

License

[x] I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
[x] I have updated the license header for each file (see an example below)

# Copyright (C) 2022 Intel Corporation
#
# SPDX-License-Identifier: MIT

@Marishka17 , could you please merge develop and share your experience about the process? I see that you have conflicts with cvat_data.tar.bz2, cvat_db.sql.

@kirill-sizov , could you please review the patch and provide your suggestions.

@Marishka17 When I run these tests locally they fails, for example this test failed due to status code is 400
FAILED test_cloud_storages.py::TestPostCloudStorage::test_sandbox_user_create_cloud_storage[user-True]

Am I doing something wrong? Maybe I need to change something in my setup?

You need to export TESTING environment variable and run docker containers with -f docker-compose.testing.yml option.

@kirill-sizov, Could you please help and resolve conflicts with assets?

@nmanovic, I've changed approach with test cloud storage functionality. Main idea is use Multi-Cloud Object
Storage. Now, before running REST API tests MinIO server is started and MinIO client creates several buckets locally that are used in tests.
I'm removing WIP status now because I've mostly done all (maybe I will do only some cosmetics edits) and want to be sure that tests will be passed on CI as locally after resolving conflicts.

@kirill-sizov, Could you please help and resolve conflicts with assets?

Yes sure, I will resolve conflicts.


Separate driver-related stuff into its own file

I'm new to ooc and Rock; this is my first real ooc programming I've done, so please bear with any mistakes.
I saw the PR for a CMake driver and freaked out with all the code duplication. It's scary. So, I decided to factor out the common stuff into GenericDriver.ooc, which contains some base classes for MakeDriver (and the not-yet-merged CMakeDriver) to remove all the code duplication.
Cons? This is likely going to 100% break #847, which (ironically) is the very issue that started this whole thing to begin with.

Any chance of this (and maybe #862) being reviewed now that you seem to be better from your cold? ;)

@kirbyfan64 But if I merge this everyone will jump me with their pull requests in hand!
Ahhhh open source. Just let me at least see this to completion!

(Also yes, it's been merged. "Generic" didn't seem descriptive enough to me so I changed it, but otherwise fine! @zhaihj, want to adapt your CMake driver to it? Shouldn't be too much hassle afaict, although I haven't read your code in details yet.)

@fasterthanlime Thanks!!!

Finished adapting CMake driver to this commit

However github tells me there are unresolved conflicts. I will rebase my fork and open a new issue.

@zhaihj Let's continue the discussion in #847 directly


fix(ibc-core): send_packet_validate checks for timeouts

Closes: #1198
Description

PR author checklist:

[ ] Added changelog entry, using unclog.
[ ] Added tests.
[ ] Linked to GitHub issue.
[ ] Updated code comments and documentation (e.g., docs/).
[ ] Tagged one reviewer who will be the one responsible for shepherding this PR.

Reviewer checklist:

[ ] Reviewed Files changed in the GitHub PR explorer.
[ ] Manually tested (in case integration/unit/mock tests are absent).

Looks like these two tests (without any timeout) are expected to pass.
https://github.com/cosmos/ibc-rs/blob/16666c2c62d0f075b573aa6b94d34c5b6bceadfd/ibc-testkit/tests/core/router.rs#L347-L359
Judging by the impl TryFrom<RawMsgTransfer> for MsgTransfer, this should have failed
https://github.com/cosmos/ibc-rs/blob/16666c2c62d0f075b573aa6b94d34c5b6bceadfd/ibc-apps/ics20-transfer/types/src/msgs/transfer.rs#L65-L67


Mobile app - Documents- Dark mode , ui color issue

Android= Folder/ image remove buttons is not visible  in dark mode

IOS - Folder/ image remove buttons is not appear in dark mode

The dark mode is not available at this time


Helm (presumably any /* */) comments aren't supported

Doesn't work:
{{-     $password = index $split 1 }}{{- /* pragma: allowlist secret */}}
Does work:
{{-     $password = index $split 1 }}{{- /* # pragma: allowlist secret */}}
This gives me teh sadz, but I figured I'd at lest document the workaround in an issue for future people. 🤷

@dannysauer Hello, can you please let me know which file type this is in and some more context around the line in question.

I'm not sure what else to say, @jpdakran :)
This was in a helm template file (so, Go template format).  That templating language allows for basically C-style delimited comments.  I have a variable called password which gets populated by an element from an array.  The detect-secrets parser does not recognize the delimited comments; specifically it does not recognize the /* marking the beginning of a comment.  It inaccurately expects comments in the helm template files to start with #.
So, the workaround above is to make a proper comment in the template file, and then to precede the pragma with the expected comment character.
In a helm chart, only the yaml files (so basically just the values file and the chart definition) work with # comment style comments; the template files use {{- /* comment */}}.  So, I'm guessing that the delimited comments aren't supported.  This would be similar to """ multi-line python comment """ or ` support, for another couple of potential examples.  Supporting delimited comments is probably something moderately useful, as they can be used on one line as in the above example, or could possibly precede the line like this:
{{- /*
 pragma: allowlist secret
 */}}
{{- $password = index $split 1 }}

I don't know if that works in, say, C - but the general "comment start marker" and "comment end marker" style might be something handy for quite a few languages.
Here's the Helm comment documentation, for reference: https://helm.sh/docs/chart_best_practices/templates/#comments-yaml-comments-vs-template-comments

Granted, the "line above" mechanism could work with # pragma ....  However, in this case, what would actually happen is that the variable being defined would not show up in the emitted YAML  It's used later; that line doesn't print anything.  But the YAML comment line would show up in the template.  So the generated YAML in the k8s manifest would have a weird comment which does not apply to the actual YAML content, potentially confusing someone in the future.  Using the native templae language's delimited comments means the pragma is omitted from the generated YAML, making everyone happy. :)


Feature/content order

fixed the order in which guidance and example displays

Closing this PR as the fix will be merged into the feature/typography-refactor branch and from there merged into master.


data: image cause CSP header issues

The EmojiOne Area plugin uses a data: image in the CSS for the default button.
In my extension this default icon is hidden and another is used with the core icon() helper.
But the image is still in the CSS and still triggers CSP error if data: is not allowed by the img-src rule. It does not prevent the use of the plugin, but it will spam the CSP report URI if enabled.
It might be a good idea to extract the useful parts of the CSS into extension.less and not import all the original rules.

The data-image is also used for the triangle under the popup. This one is still used and disapears if CSP blocks it.


New GNU Fortran intrinsics.

Added missing intrinsic functions from newest GFortran documentation at Intrinsic-Procedures.
Also Fix issue #113 by adding ELSE and STOP as keywords

[x] Add intrinsics on js file
[x] Add intrinsic json files

We should add the new intrinsics here as well https://github.com/krvajal/vscode-fortran-support/blob/master/src/docs/

I'll add those soon, thank you.

Done that tedious work of copying the documentation to json files. :+1:

@krvajal Is that a reason why our intrinsic documentation files have 3 or 4 copies of the same structure?
Like the first one ABORT.json and all others I've oppended, the structure of the file should resemble the below:
{
"keyword":"",
"docstr":""
}

But they are in one line and repeated 4 times or more
{"keyword":"","docstr":""}{"keyword":"","docstr":""}{"keyword":"","docstr":""}{"keyword":"","docstr":""}

If you look for the closing bracket and enter return after each one this is what you'll get

Should we consider this a bug?

Should we consider this a bug?

Yes, it is indeed a bug as it is not valid JSON.

Ok done. Used VSCode smart search and replace to remove duplicates and then indent the files better.

Maybe we should also add functions for double precision float number like DABS, DSIN, and etc.?
A list of them can be found at
https://github.com/vim/vim/blob/master/runtime/syntax/fortran.vim#L146


[WIP] Add the bulk-import backend

Description
Adds the bulk-import backend with the initial github bulk import feature.
Currently still work in progress, as only the repository listing functionality works
Corresponding Epic
https://issues.redhat.com/browse/RHIDP-955
What issue(s) does this fix?
https://issues.redhat.com/browse/RHIDP-1208
https://issues.redhat.com/browse/RHIDP-1207
PR acceptance criteria
Please make sure that the following steps are complete:

[ ] GitHub Actions are completed and successful
[ ] Unit Tests are updated and passing
[x] Documentation is updated if necessary (requirement for new features)
[ ] Add a screenshot if the change is UX/UI related

How to test changes / Special notes to the reviewer
Currently still work in progress, as only the repository listing functionality works
Need to finish writing the initial unit tests

Work continuing in https://github.com/janus-idp/backstage-plugins/pull/1471
/close


Add support for LeakyReLU and Split in ModelSpeedup?

What would you like to be added:
Could you add support for LeakyReLU and aten::split in ModelSpeedup.
Why is this needed:
I tried to prune yolov4 but failed with an error RuntimeError: Has not supported infering output shape from input shape for.
I think supporting these node is useful since they are ubiquitous.
Without this feature, how does current nni work：
Show error as above.
Components that may involve changes:
ModelSpeedup
Brief description of your proposal if any:

@YuyaKobayashi thanks for reporting your issue. We will support these ops soon. Also we will refactor the implementation to make users easily add the support of new ops. We highly encourage users to contribute new ops :)

Thank you

@YuyaKobayashi thanks for reporting your issue. We will support these ops soon. Also we will refactor the implementation to make users easily add the support of new ops. We highly encourage users to contribute new ops :)

Hi there!
Can I work on this?

@hxri
If you are using nn.module's leaky_relu, we have already supported it, but we have not yet supported nn.functional's leaky_relu, you are welcome to contribute code

@hxri @zheng-ningxin - has this issue been supported in previous releases?

https://github.com/microsoft/nni/pull/4996
It should work when this PR is merged. Thanks

hi, @hxri  You could use nni v2.9 to test it. It will be released soon. Please reopen or new an issue if you have any problem.

hi, @hxri You could use nni v2.9 to test it. It will be released soon. Please reopen or new an issue if you have any problem.

thanks for the follow up, I'm closing as fixed downstream.


ui-ace need ace to work... (o rly?)

what that means? can any one explain me?

I think you need ace-builds to get it work. I had this issue and used ace-builds and it worked.
include this in your project: https://github.com/ajaxorg/ace-builds/tree/master/demo

I had this issue and it appears to be a dependency issue.  I required the following files and it worked for me

bower_components/ace-builds/src-min-noconflict/ace.js
bower_components/ace-builds/src-min-noconflict/ui-ace.js.js


SIRF-exercise jupyter issue in VM 1.0.0 rc1 and rc3

jupyter is not installed? "command not found" . And there is no browser to download from the web.

This is true. You could try to install firefox and pip install jupyter but this is likely to fail due to changes in packages in the mean time.
Better to get a new VM really. sorry.


Unable to install (64-bit)

Installation of bootstrap zipball fails on the first stage, during downloading. According to the traffic data, it downloads around 5 KB, then stops with the brief pop-up message about error.
My tablet with MT8735A definitely 64-bit and fit to launch Nix - I had a fully working installation with nix-in-termux script.
How do I fix this problem or at least collect more information about it?

No activity here since a long time. Feel free to reopen if still relevant.

It keeps crashing, I have installed it and it still continues with the same error.
I tried downloading it from the url, from the file manager and change url and everything.

My cell phone information


ACK controller have duplicate CRD causing issues when using kustomize

Is your feature request related to a problem?
Yes.
To understand our issue, we are using argocd and kustomize, and we have defined an argocd application that is regrouping all ACK controller we want to use.
Basically, we use IAM, RDS, S3.
We have an issue due to kustomize, when kustomize is building the yaml, it throws an error due to CRDs duplication...
The CRDs concerned are the following:
services.k8s.aws_adoptedresources.yaml
services.k8s.aws_fieldexports.yaml
Theses CRDs are present in each helm charts, and they are exactly the same..
Step to reproduce our problem
Create your kustomize file:
helmCharts:
- repo: oci://public.ecr.aws/aws-controllers-k8s
  name: s3-chart
  includeCRDs: true
  releaseName: s3
  version: 1.0.9
- repo: oci://public.ecr.aws/aws-controllers-k8s
  name: rds-chart
  includeCRDs: true
  releaseName: rds
  version: 1.1.11

Then apply:
kustomize build --enable-helm .
At this step, kustomize will fail saying that some CRDs are the same, and need to be merge or patch.
Describe the solution you'd like
Define this CRDs as helm dependencies could allow us to activate the CRD creation once.
Describe alternatives you've considered
Giving more flexibility to choose which CRDs are deployed from the helm values.

Hi!
Interesting thing. I really wonder how I did not notice it, as I am using FluxCD and did not run into this issue. For the more context - I am installing every chart as a separate release.

/remove-lifecycle stale


MM-14417: Add messaging for channel mentions of users not in constrai…

…ned groups.
Summary
Updates the display of the ephemeral message notifying of users mentioned in a channel when the channel is group-constrained and some or all of the mentioned users are not permitted in the channel.
Requires this server change: https://github.com/mattermost/mattermost-server/pull/10594
The equivalent web app PR is: https://github.com/mattermost/mattermost-webapp/pull/2647
Ticket Link
https://mattermost.atlassian.net/browse/MM-14417
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[ ] Added or updated unit tests (required for all new features)
[ ] All new/modified APIs include changes to mattermost-redux (please link)
[x] Has UI changes
[x] Includes text changes and localization file updates

Device Information
This PR was tested on: iPhone X, iOS 12.2 (emulated)
Screenshots

@mkraft can you post an example where the groups options is not enabled?

@enahum Here's are two notifications, one with a single user and the other with multiple users.

@mkraft I mean for a server that does not have the option or that the option is NOT enabled, or even a channel that is not controlled by any groups.

@enahum The screenshot is for a group that is not group-constrained. Am I misunderstanding?

@mkraft so basically what I want to see if how the app behaves with backwards compatibility

@enahum Here's the master branch of mattermost-mobile pointing to the server-side changes associated to MM-14417 using a non-group-constrained private channel:

So this means that this PR pointing to a server that has the  MM-14417 changes will always show that they do not belong to a group even though LDAP groups is not enabled?

So this means that this PR pointing to a server that has the MM-14417 changes will always show that they do not belong to a group even though LDAP groups is not enabled?

I don't understand what you mean. Where does it show that they don't belong to a group? Do you mean belong to a channel? The users in that example don't belong to the channel.

@enahum Thanks for reminding me to also test an upgraded mobile app with an old server. Here's a screenshot of it working:

Now I have covered the following:

[x] Upgraded mobile app, upgraded server
[x] Existing mobile app, upgraded server
[x] Upgraded mobile app, existing server


Remove remaining release tasks

The remaining release tasks don't work well and for the last couple of
releases I've updated the AUR package manually.
Maintaining distribution specific packages outside of the repo creates a
clear separation of concerns.

This follows up on #135.

Right on.
Do you want to add a note (to yourself, or for other devs) on how to make an Arch or Homebrew release?
In rcm I keep a list of the package maintainers so that I can contact them when a new release comes out (bottom of https://raw.githubusercontent.com/thoughtbot/rcm/master/DEVELOPERS.md ).

That's a great idea! I'll compile a list.

Hey @xuchunyang @eavgerinos @fornwall @neelchauhan,
I've gone through the commit history of Pick to figure out who's maintaining the different Pick packages in order to compile a list of package maintainers to email about new versions.
Please let me know if you do not want to be listed or no longer want to maintain these packages.

I've emailed the above package maintainers regarding this PR and am waiting for a replies either here or by email.

I'm fine being listed as maintainer for the homebrew package.

I'm fine being listed as a maintainer for the FreeBSD port/package.

Pick on MacPorts has been updated to latest version v1.4.  Sorry for the delay.
I'm fine being listed as maintainer for the MacPorts port.

Thanks everyone for replying! I've also heard back from @eavgerinos via email so I'll merge this PR now.


Fix test_helper.py for NumPy 1.18

Error message was changed in 1.18.
I think error message can be simply ignored.

Jenkins, test this please

Jenkins CI test (for commit f29e6df0a6fb31cdd22ee011d9b752498567c448, target branch master) succeeded!

@emcastillo This pull-request is marked as to-be-backported, but the corresponding backport PR could not be found. Could you check?


some examples

Hi, could u add any examples of mongoose schema use?

Some examples would really be helpful.
I define my server and register fastify-mongoose in server.js, my route in route.js, my controller in another file, and my model in its own file. I don't quite understand how to pass the database connection between them all.

@tomclark I think this can help you:
const fastify = require('fastify')()
const { Schema } = require('mongoose')

// Declare a route
fastify.get('/', function (request, reply) {
  reply.send({ hello: 'world' })
})

fastify.register(require('fastify-mongoose'), {
  uri: 'mongodb://test:test@localhost/test'
}, err => {
  if (err) throw err
}).after(() => {
  var schema = new Schema({
    foo1: {
      type: String,
      required: false
    },
    bar1: {
      type: String,
      required: false
    }
  })
  let Model = fastify.mongo.db.model('model', schema)
  let m1 = new Model({
    foo: 'value_for_foo',
    bar: 'value_for_bar'
  })
  m1.save(function (err) {
    if (err) {
      console.log(err.stack)
    }
    console.log('saving done...')
  })
})

// Run the server!
fastify.listen(3000, function (err) {
  if (err) throw err
  console.log(`server listening on ${fastify.server.address().port}`)
})

Legend, thank you!
I actually managed it in the end in a slightly different way where I connect to the database in my db module, import it to the server, and then wrap my server with the db connection. Like this:
// db.js
mongoose.connect(config.db.uri, opts)
const db = mongoose.connection
db.on('error', err => {
  if (err.message.code === 'ETIMEDOUT') {
    mongoose.connect(config.db.uri, opts)
  }
})

module.exports = db

Then in the server:
// server.js
import db from './db'

db.once('open', () => {
  /**
   * Bind routes
   */
  server.register(BaseRoute)
  server.register(BaseRoute, { prefix: '/api/v1' })

  // Run the server
  const start = async () => {
    try {
      await server.listen(config.port, '<IP_ADDRESS>')
    } catch (err) {
      server.log.error(err)
      process.exit(1)
    }
  }

  start()
})

Then later in the routes I can just do something like this:
  async detail (itemId) {
    return ItemModel.findById(itemId).lean()
  }

I can then just use models etc in my routes and controllers by importing mongoose into them.
Not sure it's quite the right way to do it, but it works just fine.
Thanks for coming back to me.

Start server only if the connection to the database is open seems to me a problem and it won't help with microservices.
This module import mongoose through fastify decorator, so, mongoose will be avalible for all application through fastify.mongo.db (connection).
Maybe I'll fork this plugin to improve some things like schema and mongoose plugins.
About your code, fastify is architected for microservices. So, reusing code through decorators, hooks and registers will improve your performance and development speed.

But if the database connection is required for any API call at all, am I not better waiting until it's open before bringing up the server? i.e. there are no API calls in my app where you don't hit the database.
Am obviously a newbie, so I appreciate the tips, thank you. I'll have a look at refactoring as you suggest. This is my first project with Fastify, so I'm grateful for your advice.

It's my first too 😃
When you use process.exit(1) your application will close if db is unreachable.
Your application may be reachable even if the database is not.
You can send a 501 error informing you that the server is unavailable.

Do you understand me?

@derevandal So I think I have a slightly more appropriate way of doing this. Would appreciate your feedback.
I have a models module that I register with the Fastify server object like this:
server.register(mongoose, {
  uri: config.db.uri
})
server.register(models)
server.register(route)

models.js contains:
import MyModel from '../Models/MyModel.model'

export default async function models (server) {
  server.register(MyModel)
}

Where MyModel.js looks something like this:
import { Schema } from 'mongoose'

export default async function (server) {
  const myModel = new Schema(
    {
      username: { type: String, trim: true },
      someNumber: { type: Number, unique: false, required: true },
    },
    { collection: 'MyModel' }
  )

  return server.mongo.db.model('MyModel', myModel)
}

Then in my route.js, I do:
server.get('/mymodel/:id', async (req, res) => {
  try {
    const models = await new GetMyModelController(server.mongo.db.models.MyModel).detail(req.params.id)
    SuccessResponse(
      res,
      `${res.length}`,
      models
    )
  } catch (exception) {
    if (!parseInt(req.params.id, 10)) {
      BadRequestResponse(res, 'Invalid ID') // in case someone gets /mymodel/foo
    } else {
      InternalServerErrorResponse(res, exception, 500)
    }
  }
})

And finally, in the controller, I do:
export class GetMyModelController {
  constructor (MyModel) {
    this._myModel = MyModel
  }

  async detail (ModelId) {
    return this._myModel.findById(ModelId).lean()
    .then(model => {
      if (model !== null) {
        return model
      } else {
        throw new NotFoundException('modelId not found')
      }
    })
  }

Some code/edited for brevity/relevance, so there may be errors, but hopefully it gets across the approach. The various response objects just return some json and an HTTP code.
I realise I could simplify it by getting the model directly in the route rather than via a controller, but there's other reasons for having the controller, though they're not that important, so I guess I could drop the controller in favour of a leaner approach.
Does that look a little more like how one should do it in Fastify? It works, but as I say, I'm grateful for your feedback as to whether that's correct, as I don't have anyone else to do code review.

Sorry, posted that before I'd seen your reply about the database - yep, gotcha, that makes sense.

@tomclark I realise that this is super late, and perhaps a bit irrelevant, but then again this is an issue to describe examples (which could hopefully make it to the readme?).
Anyways, I found your example above to be super useful (and simple to understand) man, thanks for that.
Just a simple high-five.

Glad it helped! :-)

Okay okay this should be on the README.md


Missing .h file in source distribution, causing pip install no-binary to fail.

Was getting below error on mac M1 laptop which needed to compile the source distribution.
src/cpp/astar.cpp:7:10: error: 'experimental_heuristics.h' file not found with <angled> include; use "quotes" instead
      #include <experimental_heuristics.h>
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~
               "experimental_heuristics.h"
      1 warning and 1 error generated.
      error: command 'clang' failed with exit status 1
      [end of output]

You should be able to reproduce the issue everywhere with below commands:
pip uninstall pyastar2d
pip install pyastar2d --no-binary :all:

The new single MANIFEST.in entry, adds the .h to source dist and allows it to be referenced correctly when installing without binary.
Apologies not testing this scenario and only finding out the issue now.

Thanks for fixing! I also completely forgot to mention that this would be necessary so don't worry about it at all.

Great, thanks for getting it merged so quickly.
Looks good.
Skipping wheel build for pyastar2d, due to binaries being disabled for it.
Installing collected packages: pyastar2d
  Running setup.py install for pyastar2d ... done
Successfully installed pyastar2d-1.0.6


Bad Link on Shows Page

Link to Scott's webpage is bad. Works by adding "www." or by forcing "http" over "https" and relying on the redirect to correctly reroute.

Fixed in https://github.com/wesbos/Syntax/commit/42df32697990bb70431308335d12d77a542be646
If you clear your cache, leveluptutorials.com will correctly redirect to www. The issue is browser cached to be https://leveluptutorials.com which isn't throwing a redirect due to these restrictions: https://support.dnsimple.com/articles/redirector-https
The solution would be to have a custom redirect server running on https. Either way, I've updated the links.

@stolinski Ah thanks for taking the fix, I guess I'll close the pull request I made then. Interesting I was having the problem even though I had never been to either site before in the past. Not sure exactly how you are doing redirects but in my case I would rewrite the URL in apache mod_rewrite.

Oh, I see now that the links were using https.
Unfortunately, I'm hosting on a platform that doesn't give me that kind of control without firing up another container. I could point the domain to an apache server or node server that handles the redirect... maybe a lambda function would do the trick.

Just to clarify, so the expected behavior of leveluptutorials.com is for the connection to be refused(die) in the event "www" is not included in the URL? Even with my cache cleared I am still not being redirected with the URL of (https://leveluptotorials.com)

It's because of the https which isn't supported. If you were to have http://leveluptutorials it would properly redirect. That is currently the expected behavior although I should have a better solution working today.

FYI, I setup an nginx redirect server for https://leveluptutorials.com all seems to be working now.


Align int to main

Due to ongoing merge conflicts

This got merged though https://github.com/transcom/mymove/pull/13485, this PR itself was not merged but the commits were via another PR