add sbt-spark-package plugin

After adding credentials to build.sbt, a new version can be released using sbt spPublish
https://github.com/databricks/sbt-spark-package

Thanks!


Gen2: DynamoDB Streams - CloudFormation fails due to Circular dependency between resources

Before opening, please confirm:

[X] I have searched for duplicate or closed issues and discussions.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework
Not applicable
Amplify APIs
GraphQL API
Amplify Version
v6
Amplify Categories
auth, function, api
Backend
Amplify Gen 2 (Preview)
Environment information

# Put output below this line
  System:
    OS: macOS 14.4.1
    CPU: (10) arm64 Apple M1 Max
    Memory: 4.63 GB / 32.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 22.1.0 - /opt/homebrew/bin/node
    Yarn: 1.22.22 - /opt/homebrew/bin/yarn
    npm: 10.7.0 - /opt/homebrew/bin/npm
    Watchman: 2<IP_ADDRESS> - /opt/homebrew/bin/watchman
  Browsers:
    Chrome: 124.0.6367.119
    Safari: 17.4.1
  npmPackages:
    %name%:  0.1.0
    @aws-amplify/backend: ^1.0.0 => 1.0.1
    @aws-amplify/backend-cli: ^1.0.1 => 1.0.2
    @aws-amplify/react-native: ^1.1.0 => 1.1.0
    @aws-amplify/ui-react-native: ^2.1.6 => 2.2.0
    @aws-lambda-powertools/logger: ^2.1.0 => 2.1.0
    @babel/core: ^7.20.0 => 7.24.5
    @expo/vector-icons: ^14.0.0 => 14.0.1
    @react-native-async-storage/async-storage: ^1.23.1 => 1.23.1
    @react-native-community/netinfo: ^11.3.1 => 11.3.1
    @react-navigation/native: ^6.0.2 => 6.1.17
    @types/aws-lambda: ^8.10.137 => 8.10.137
    @types/jest: ^29.5.12 => 29.5.12
    @types/react: ~18.2.45 => 18.2.79 (18.3.1)
    @types/react-test-renderer: ^18.0.7 => 18.3.0
    HelloWorld:  0.0.1
    aws-amplify: ^6.3.0 => 6.3.0
    aws-amplify/adapter-core:  undefined ()
    aws-amplify/analytics:  undefined ()
    aws-amplify/analytics/kinesis:  undefined ()
    aws-amplify/analytics/kinesis-firehose:  undefined ()
    aws-amplify/analytics/personalize:  undefined ()
    aws-amplify/analytics/pinpoint:  undefined ()
    aws-amplify/api:  undefined ()
    aws-amplify/api/server:  undefined ()
    aws-amplify/auth:  undefined ()
    aws-amplify/auth/cognito:  undefined ()
    aws-amplify/auth/cognito/server:  undefined ()
    aws-amplify/auth/enable-oauth-listener:  undefined ()
    aws-amplify/auth/server:  undefined ()
    aws-amplify/data:  undefined ()
    aws-amplify/data/server:  undefined ()
    aws-amplify/datastore:  undefined ()
    aws-amplify/in-app-messaging:  undefined ()
    aws-amplify/in-app-messaging/pinpoint:  undefined ()
    aws-amplify/push-notifications:  undefined ()
    aws-amplify/push-notifications/pinpoint:  undefined ()
    aws-amplify/storage:  undefined ()
    aws-amplify/storage/s3:  undefined ()
    aws-amplify/storage/s3/server:  undefined ()
    aws-amplify/storage/server:  undefined ()
    aws-amplify/utils:  undefined ()
    aws-cdk: ^2.140.0 => 2.141.0
    aws-cdk-lib: ^2.140.0 => 2.141.0
    constructs: ^10.3.0 => 10.3.0
    esbuild: ^0.21.1 => 0.21.1 (0.20.2)
    expo: ~51.0.0 => 51.0.2
    expo-constants: ~16.0.1 => 16.0.1
    expo-font: ~12.0.4 => 12.0.4
    expo-linking: ~6.3.1 => 6.3.1
    expo-router: ~3.5.10 => 3.5.11
    expo-splash-screen: ~0.27.4 => 0.27.4
    expo-status-bar: ~1.12.1 => 1.12.1
    expo-system-ui: ~3.0.4 => 3.0.4
    expo-web-browser: ~13.0.3 => 13.0.3
    jest: ^29.2.1 => 29.7.0
    jest-expo: ~51.0.1 => 51.0.1
    react: 18.2.0 => 18.2.0
    react-dom: 18.2.0 => 18.2.0
    react-native: 0.74.1 => 0.74.1
    react-native-gesture-handler: ~2.16.1 => 2.16.2
    react-native-get-random-values: ^1.11.0 => 1.11.0
    react-native-reanimated: 3.10.0 => 3.10.0
    react-native-safe-area-context: ^4.10.1 => 4.10.1
    react-native-screens: 3.31.1 => 3.31.1
    react-native-url-polyfill: ^2.0.0 => 2.0.0
    react-native-web: ~0.19.10 => 0.19.11
    react-test-renderer: 18.2.0 => 18.2.0
    tsx: ^4.9.3 => 4.9.3
    typescript: ^5.4.5 => 5.4.5 (4.4.4, 4.9.5)
  npmGlobalPackages:
    eas-cli: 7.4.0
    npm: 10.7.0
    yarn: 1.22.22

Describe the bug
Follwing the official Gen 2 Guide to configure a Lambda function with an Amazon DynamoDB stream as an event source I get the following error during deployment/sandbox:
The CloudFormation deployment has failed.
Caused By: ❌ Deployment failed: Error [ValidationError]: Circular dependency between resources: [auth179371D7, data7552DF31, function1351588B]

Resolution: Find more information in the CloudFormation AWS Console for this stack.

The error happens when I add the Event Source according to the official Docs:
import { defineBackend } from '@aws-amplify/backend';
import { StartingPosition } from 'aws-cdk-lib/aws-lambda';
import { DynamoEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';
import { auth } from './auth/resource';
import { data } from './data/resource';
import { myDynamoDBFunction } from './functions/dynamoDB-function/resource';

const backend = defineBackend({
  auth,
  data,
  myDynamoDBFunction,
});

const eventSource = new DynamoEventSource(
  backend.data.resources.tables['Todo'],
  {
    startingPosition: StartingPosition.LATEST,
  },
);

backend.myDynamoDBFunction.resources.lambda.addEventSource(eventSource); // <-- this causes the Circular dependency

Edit: I was concerned my Auth preTokenGeneration handler caused this, but it dit not. When I remove preTokenGeneration from triggers, I still get the error:
The CloudFormation deployment has failed.
Caused By: ❌ Deployment failed: Error [ValidationError]: Circular dependency between resources: [data7552DF31, function1351588B]

So the circular dependency definitely results from the fact that the backend resource table is referenced in the eventSource and then being injected back into the backend object. Design flaw?
const eventSource = new DynamoEventSource(
  backend.data.resources.tables['Todo'], // <- reference from backend
  {
    startingPosition: StartingPosition.LATEST,
  },
);

backend.myDynamoDBFunction.resources.lambda.addEventSource(eventSource); // <- inject back into backend

Expected behavior
It should be possible to add the event source for the lambda according to the docs to invoke the lambda by DynamoDB Stream.
Reproduction steps
Just follow your own official guide: https://docs.amplify.aws/react/build-a-backend/functions/examples/dynamo-db-stream/
Code Snippet
// Put your code below this line.

Log output

// Put your logs below this line

aws-exports.js
No response
Manual configuration
No response
Additional configuration
No response
Mobile Device
No response
Mobile Operating System
No response
Mobile Browser
No response
Mobile Browser Version
No response
Additional information and screenshots
No response

Hey guys, I get the same error:
The CloudFormation deployment has failed.
Caused By: ❌ Deployment failed: Error [ValidationError]: Circular dependency between resources: [auth179371D7, data7552DF31, function1351588B]
Here is the code I added to backend.ts:
const SMSResponseEventSource = new DynamoEventSource(backend.data.resources.tables['SMSResponse'], {
  startingPosition: StartingPosition.LATEST,
});
const EmailResponseEventSource = new DynamoEventSource(
  backend.data.resources.tables['EmailResponse'],
  {
    startingPosition: StartingPosition.LATEST,
  },
);
const VoiceCallResponseEventSource = new DynamoEventSource(
  backend.data.resources.tables['VoiceCallResponse'],
  {
    startingPosition: StartingPosition.LATEST,
  },
);

backend.OnAppointmentReminderDeliveryHandler.resources.lambda.addEventSource(
  SMSResponseEventSource,
);
backend.OnAppointmentReminderDeliveryHandler.resources.lambda.addEventSource(
  EmailResponseEventSource,
);
backend.OnAppointmentReminderDeliveryHandler.resources.lambda.addEventSource(
  VoiceCallResponseEventSource,
);

I'm hoping this gets fixed soon so I don't have to set this all up manually.

I also get the same error when trying to set up environment variables. The weird things with this is that the error won't always happen immediately. Instead, I'll start getting the error after making other unrelated changes to my sandbox. Then I have to comment out the environment variable code in my backend.ts file to get rid of the circular dependency.
The CloudFormation deployment has failed.
Caused By: ❌ Deployment failed: Error [ValidationError]: Circular dependency between resources: [auth179371D7, data7552DF31, function1351588B]
Commenting out these lines in backend.ts resolves the circular dependency:
const copyDynamoDBTableLambdaFunction = backend.CopyDynamoDBTableHandler.resources
  .lambda as Function;
copyDynamoDBTableLambdaFunction.addEnvironment('APP_SYNC_API_ID', backend.data.apiId);

Assigning to @AnilMaktala to try & repro -- I'm not able to get this error to occur.

Just wanted to pop in to say that I also am getting the same error when trying to add in event source for dynamodb streams. Same issue circular dependency between auth, data, and function which doesn't make any sense to me since my lambda needs the table but my table doesn't need my lambda?

The issue should be marked as bug rather than as question.

@jgo80 do you have a workaround by any chance for now?

@jgo80 do you have a workaround by any chance for now?

I don't 😞 Just waiting for an answer as a desperate customer moving from Gen1 to Gen2...

I found a similar issue with a resolution: https://github.com/aws-amplify/amplify-backend/issues/1552
I'm not sure if it exactly applies to our situation because I didn't quite follow the recommended solution. Let me know if it helps you guys.

Thanks @MattWlodarski I will take a look at this today and give it a try and then update this thread

Yeah the code provided in the document operates normally when only event mapping is used, but a circular dependency error occurs as soon as data custom query is used.
So if you are using a custom query (both query and mutation), you have to manually make a separate cdk stack for the streaming target table and set the event mapping and policy on that stack.
As @MattWlodarski mentioned, i wrote down that issue and found the solution. the well-worked repository is on my repository (you can find it on that mentioned issue), so you might be able to refer it.

I had some triggers in the auth resource. If I remove them, it removes the circular dependency.
triggers: {
    customMessage,
    postConfirmation,
    preSignUp,
    defineAuthChallenge,
    createAuthChallenge,
    verifyAuthChallengeResponse: verifyAuthChallenge,
  },

@ggj0418 @MattWlodarski do you either of you have any auth triggers?

import { defineAuth, secret } from "@aws-amplify/backend";
import { customMessage } from "./custom-message/resource";
import { postConfirmation } from "./post-confirmation/resource";
import { preSignUp } from "./pre-signup/resource";
import { defineAuthChallenge } from "./define-auth-challenge/resource";
import { createAuthChallenge } from "./create-auth-challenge/resource";
import { verifyAuthChallenge } from "./verify-auth-challenge/resource";

/**
 * Define and configure your auth resource
 * @see https://docs.amplify.aws/gen2/build-a-backend/auth
 */
export const auth = defineAuth({
  loginWith: {
    email: true,
    phone: true,
  },
  triggers: {
    customMessage,
    postConfirmation,
    preSignUp,
    defineAuthChallenge,
    createAuthChallenge,
    verifyAuthChallengeResponse: verifyAuthChallenge,
  },
});

@LukaASoban hm... no i do not have any auth trigger. did you add that trigger with custom query or something else component of amplify gen2 backend?

@LukaASoban on another project, im using that auth trigger with data custom queries, but it does not have a circular error

const backend = defineBackend({
  auth,
  //preSignUp,
  //storage,
  data,
  //getUsersByEmailOrPhoneFunction,
  //createAuthChallenge,
  //checkIfCognitoUserExistsFunction,
  //onUpload,
  dynamodbStreamPlaces,
});

const eventSource = new DynamoEventSource(
  backend.data.resources.tables["Place"],
  {
    startingPosition: StartingPosition.LATEST,
  }
);

backend.dynamodbStreamPlaces.resources.lambda.addEventSource(eventSource);

and then my auth resource.ts
export const auth = defineAuth({
  loginWith: {
    email: true,
    phone: true,
    // externalProviders: {
    //   signInWithApple: {
    //     clientId: secret("SIWA_CLIENT_ID"),
    //     keyId: secret("SIWA_KEY_ID"),
    //     privateKey: secret("SIWA_PRIVATE_KEY"),
    //     teamId: secret("SIWA_TEAM_ID"),
    //     scopes: ["email"],
    //     attributeMapping: {
    //       email: "email",
    //       custom: {
    //         email_verified: "email_verified",
    //       },
    //     },
    //   },
    //   google: {
    //     clientId: secret("GOOGLE_CLIENT_ID"),
    //     clientSecret: secret("GOOGLE_CLIENT_SECRET"),
    //     scopes: ["email"],
    //     attributeMapping: {
    //       email: "email",
    //       custom: {
    //         email_verified: "email_verified",
    //       },
    //     },
    //   },
    //   callbackUrls: ["http://localhost:5173/", "flaire://"],
    //   logoutUrls: ["http://localhost:5173/", "flaire://"],
    // },
  },
  //accountRecovery: "EMAIL_AND_PHONE_WITHOUT_MFA",
  triggers: {
    customMessage,
    // postConfirmation,
    // preSignUp,
    // defineAuthChallenge,
    // createAuthChallenge,
    // verifyAuthChallengeResponse: verifyAuthChallenge,
  },
});

and then in my customMessage trigger function, there is no custom query or dependency on any other component. If I comment out the customMesssage function here it works.

@LukaASoban I think that dynamoDB event mapping is the point. here is my suggestion.
const backend = defineBackend({
  auth,
  //preSignUp,
  //storage,
  data,
  //getUsersByEmailOrPhoneFunction,
  //createAuthChallenge,
  //checkIfCognitoUserExistsFunction,
  //onUpload,
  dynamodbStreamPlaces,
});

const placeTable = backend.data.resources.tables["Place"]
backend.dynamodbStreamPlaces.resources.lambda.role?.attachInlinePolicy(
    new Policy(
        Stack.of(placeTable),
        "DynamoDBPolicy",
        {
            statements: [
                new PolicyStatement({
                    effect: Effect.ALLOW,
                    actions: [
                        "dynamodb:DescribeStream",
                        "dynamodb:GetRecords",
                        "dynamodb:GetShardIterator",
                        "dynamodb:ListStreams",
                    ],
                    resources: ["*"],
                }),
            ],
        }
    )
);

new EventSourceMapping(
    Stack.of(placeTable),
    "test",
    {
        target: backend.dynamodbStreamPlaces.resources.lambda,
        eventSourceArn: placeTable.tableStreamArn,
        startingPosition: StartingPosition.LATEST,
    }
);

Oh i forgot to tell the resolved code's branch. here is the branch. it is not a main. i'm sorry for the missing description.

@ggj0418 Thanks for your input here. I seem to be getting this error when I try that:
CREATE_FAILED        | AWS::Lambda::EventSourceMapping | data/amplifyData/Place/DynamodbStreamPlacesEventSourceMapping (DynamodbStreamPlacesEventSourceMappingE2426243) Resource handler returned message: "Invalid request provided: Stream arn:aws:dynamodb:us-east-2:*********:table/Place-2xgy7oir4fa5tndcygcxeusazu-NONE/stream/2024-05-30T20:36:57.557 is Disabled. You cannot create a lambda mapping on a stream that is Disabled. (Service: Lambda, Status Code: 400, Request ID: 1578ca62-c2ba-4039-ab59-15f48b13ba0e)" (RequestToken: ea51563f-a008-1cb5-0938-f9869001ab6f, HandlerErrorCode: InvalidRequest)
Not sure why my lambda stream is disabled

@LukaASoban hm... could i see your dynamdbStreamPlaces' handler?

Looks exactly like in the docs:
import type { DynamoDBStreamHandler } from "aws-lambda";
import { Logger } from "@aws-lambda-powertools/logger";

const logger = new Logger({
  logLevel: "INFO",
  serviceName: "dynamodb-stream-handler",
});

export const handler: DynamoDBStreamHandler = async (event) => {
  for (const record of event.Records) {
    logger.info(`Processing record: ${record.eventID}`);
    logger.info(`Event Type: ${record.eventName}`);

    if (record.eventName === "INSERT") {
      // business logic to process new records
      logger.info(`New Image: ${JSON.stringify(record.dynamodb?.NewImage)}`);
    }
  }
  logger.info(`Successfully processed ${event.Records.length} records.`);

  return {
    batchItemFailures: [],
  };
};

@LukaASoban oh yeah sandbox's hotswap is not perfectly worked...
do you talk about the amplify env vars?

How come the official docs  publishes a procedure that clearly causes this issue?
Does no one checks what is being published?
This should be marked as an issue.

@heclon are you also getting this same error?

Hey @LukaASoban I also had an auth trigger as well. I appreciate you guys explaining the workaround but I agree with @heclon. This should be marked as an issue and fixed asap. I can't believe they haven't caught this already. They must just test their code in the simplest possible ways. It doesn't take much to trigger either of these circular dependencies...

HI @jgo80, We are able to replicate this issue and marking this as bug for the team to evaluate further.

I get the exact same error as everyone else here from just following the official docs. Please fix your broken code.

@leonanderson88 Does the same error occur even if you comment that ses part?

@leonanderson88 The below works - here is my code to get this to work to break the circular dependency (you can reduce the IAM role scopes as needed I just gave it all permissions to see if it works)
You basically need to create a custom resource that is a lambda that runs, uses the AWS SDK to get the streamARN for the table and then returns it. You can modify it to include as many tables as you want.
backend.ts
const userProfileTable = backend.data.resources.tables["UserProfile"];

const dynamodbStreamingLambda =
  backend.dbToOpensearchStreamingFunction.resources.lambda;
dynamodbStreamingLambda.addToRolePolicy(
  new iam.PolicyStatement({
    sid: "AllowStreaming",
    actions: ["dynamodb:*"],
    resources: ["*"],
  }),
);

// getStreamARN Lambda IAM Policy
const getStreamArnsLambdaPolicy = new iam.PolicyStatement({
    actions: ["dynamodb:*"],
    resources: ["*"],
  });

// Create a stack for the custom resources
const customResourceStack = backend.createStack("CustomResourceStack");

// Create a custom resource to init the event mappings (we need to get the stream ARNs)
const getStreamArnsLambda = new lambda.Function(
  customResourceStack,
  "GetStreamArnsLambda",
  {
    runtime: lambda.Runtime.NODEJS_LATEST,
    code: lambda.Code.fromAsset(
      "./amplify/custom/trigger-functions/get-dynamodb-stream-arns",
    ),
    handler: "getStreamArns.handler",
    timeout: Duration.seconds(30),
    initialPolicy: [getStreamArnsLambdaPolicy],
  },
);

// Create a custom resource to init the event mappings (we need to get the stream ARNs for each table)
const streamArnsProvider = new customResources.Provider(
  customResourceStack,
  "StreamArnsProvider",
  {
    onEventHandler: getStreamArnsLambda,
  },
);
const streamArns = new CustomResource(customResourceStack, "StreamArns", {
  serviceToken: streamArnsProvider.serviceToken,
  properties: {
    TableNames: [
      userProfileTable.tableName,
    ], // Add more tables here as we need
  },
});

// Ensure the custom resource depends on the DynamoDB tables creation
streamArns.node.addDependency(userProfileTable);

// Event source mapping for the User Profile table
const userProfileStreamingEventSourceMapping = new lambda.EventSourceMapping(
  customResourceStack,
  "UserProfileStreamingEventSourceMapping",
  {
    target: backend.dbToOpensearchStreamingFunction.resources.lambda,
    eventSourceArn: streamArns.getAttString("UserProfile"),
    startingPosition: StartingPosition.LATEST,
  },
);
userProfileStreamingEventSourceMapping.node.addDependency(streamArns);

and then in my getStreamArns lambda
/* GetStreamArns lambda function */

const {
  DynamoDBClient,
  DescribeTableCommand,
} = require("@aws-sdk/client-dynamodb");

const client = new DynamoDBClient({ region: process.env.AWS_REGION });

exports.handler = async (event) => {
  const tableNames = event.ResourceProperties.TableNames;

  try {
    const streamArns = {};
    for (const tableName of tableNames) {
      const command = new DescribeTableCommand({ TableName: tableName });
      const data = await client.send(command);
      const baseTableName = tableName.split("-")[0]; // Extract the base name
      streamArns[baseTableName] = data.Table.LatestStreamArn || "";
    }

    console.log("Stream ARNs:", JSON.stringify(streamArns)); // Log the returned ARNs

    return {
      PhysicalResourceId: tableNames.join(","),
      Data: streamArns,
    };
  } catch (error) {
    console.error("Error:", error);

    return {
      PhysicalResourceId: tableNames.join(","),
      Data: {
        Error: error.message,
      },
    };
  }
};

I've run into the circular dependency issue several times as well in a fairly simple Gen2 project. In the latest case my setup is as follows:
Setup

const backend = defineBackend({
  auth,
  data,
  storage
});

I've used data category to define the AppSync API from schema. This creates the only AppSync API with the tables (no resolvers). I've defined a function to create all resolvers manually by using backend.data.addResolver(...). I have one resolver which calls a lambda function which needs to access Storage to create a presigned URL. So far so good, I'm at:
Dependency: AppSync >> Storage
Next I used the LambdaToSqsToLambda L3 Construct to implement:

Storage needs to trigger a Lambda Function to send a message to the SQS queue of on file creation. Dependency: Storage >> [L3Construct: Lambda Function (producer) >> SQS]
SQS queue needs to notify a Lambda Function (consumer), do processing on the object in Storage, then send the updates to AppsSync. Dependency: [L3Construct: SQS >> Lambda Function] >> Storage & AppSync

// Definition of LambdaSQSLambda L3 Construct
// See: https://constructs.dev/packages/@aws-solutions-constructs/aws-lambda-sqs-lambda/v/2.61.0?lang=typescript
//
//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

const lambdaSQSLambdaStack = backend.createStack('LambdaSQSLambdaStack');

const lambdaToSQSToLambdaConstruct = new LambdaToSqsToLambda(lambdaSQSLambdaStack, 'LambdaToSqsToLambdaPattern', {
  producerLambdaFunctionProps: {           // << -- this way doesn't work because it creates the lambda in the same stack
                                           //         as the SQS queue
    runtime: LambdaRuntime.NODEJS_20_X,
    handler: 'index.handler',
    code: fromAssetHelper(path.resolve('amplify/data/features/my-feature/upload/s3-to-sqs-handler.lambda.cjs'))
  },

  consumerLambdaFunctionProps: {           // << -- this way doesn't work because it creates the lambda in the same stack
                                           //         as the SQS queue
    runtime: LambdaRuntime.NODEJS_20_X,
    handler: 'index.handler',
    code: fromAssetHelper(path.resolve('amplify/data/features/my-feature/upload/sqs-to-appsync-handler.lambda.cjs'))
  },

});

// PRODUCER gets notified of uploads and writes the event to the SQS queue
backend.storage.resources.bucket.addObjectCreatedNotification(new LambdaDestination(lambdaToSQSToLambdaConstruct.producerLambdaFunction));

lambdaToSQSToLambdaConstruct.sqsQueue.grantSendMessages(lambdaToSQSToLambdaConstruct.producerLambdaFunction);

// CONSUMER listens to the SQS queue, does processing and sends response to AppSync (data)
backend.data.resources.graphqlApi.grantMutation(
  lambdaToSQSToLambdaConstruct.consumerLambdaFunction,
  '*'
);

backend.storage.resources.bucket.grantRead(
  lambdaToSQSToLambdaConstruct.consumerLambdaFunction,
  '*'
);

lambdaToSQSToLambdaConstruct.consumerLambdaFunction.addToRolePolicy(new PolicyStatement({
  actions: [
    "appsync:GraphQL",
  ],
  resources: [
    `${backend.data.resources.graphqlApi.arn}/*`,
  ],
}));

lambdaToSQSToLambdaConstruct.consumerLambdaFunction.addEnvironment('API_ENDPOINT', backend.data.graphqlUrl);

Problem
So the circular dependency comes because now I have:
❌ Dependency: AppSync >> Storage >> [L3Construct: SQS >> Lambda Function] >> Storage & AppSync
Solution
The solution I found was to control the stack that the resources were created in, placing related resources in a common stack. The L3 construct above allows defining and providing a reference to lambda functions. This allows us to reference resources in a different stack. Concretely, this allowed me to define the producer lambda function (which sends data from Storage to SQS) in the Stack.of(backend.storage.resources.bucket) and the consumer lambda function (which receives the data from SQS and then uses Storage and AppSync in the Stack.of(backend.data). This leaves me with:
✅ Dependency: [L3Construct: SQS] >> [Stack: AppSync, Consumer Function] >> [Stack: Storage, Producer Function]
✅ Dependency: [L3Construct: SQS] >> [Stack: Storage, Producer Function]

const lambdaSQSLambdaStack = backend.createStack('LambdaSQSLambdaStack');

const lambdaToSQSToLambdaConstruct = new LambdaToSqsToLambda(lambdaSQSLambdaStack, 'LambdaToSqsToLambdaPattern', {
  // producerLambdaFunctionProps: {           // << -- this way doesn't work because it creates the lambda in the same stack
                                              //       as the SQS queue
  //   runtime: LambdaRuntime.NODEJS_20_X,
  //   handler: 'index.handler',
  //   code: fromAssetHelper(path.resolve('amplify/data/features/my-feature/upload/s3-to-sqs-handler.lambda.cjs'))
  // },
  existingProducerLambdaObj: new LambdaFunction(Stack.of(backend.storage.resources.bucket), 'LambdaToSqsProducerLambda', {
    handler: 'index.handler',
    code: fromAssetHelper(path.resolve('amplify/data/features/my-feature/upload/s3-to-sqs-handler.lambda.cjs')),
    runtime: LambdaRuntime.NODEJS_20_X,
  }),
  // consumerLambdaFunctionProps: {           // << -- this way doesn't work because it creates the lambda in the same stack
                                              //       as the SQS queue
  //   runtime: LambdaRuntime.NODEJS_20_X,
  //   handler: 'index.handler',
  //   code: fromAssetHelper(path.resolve('amplify/data/features/my-feature/upload/sqs-to-appsync-handler.lambda.cjs'))
  // },
  existingConsumerLambdaObj: new LambdaFunction(Stack.of(backend.data), 'LambdaToSqsConsumerLambda', {
    handler: 'index.handler',
    code: fromAssetHelper(path.resolve('amplify/data/features/my-feature/upload/sqs-to-appsync-handler.lambda.cjs')),
    runtime: LambdaRuntime.NODEJS_20_X,
    // environment: {
    // }
  })
});

This broke the dependency for me.

TLDR: as a workaround you could try defining your lambda function manually (instead of defineFunction()) and placing it in Stack.of(data) where your tables are. That way, there aren't an inter stack dependencies.

I also bumped into this issue when trying to grant dynamodb table permissions to my lambda handler or pass table name into lambda as an env variable. And for me, it is quite difficult to break this circular reference, so I'm thinking about giving custom resource a try just as what @LukaASoban shared( thank you for sharing this workaround!)
Personally, I think this is a bug that has a big impact when working with gen2. Hopefully it could be fixed soon

@MyNameIsTakenOMG , I also found that ampx doesn't automatically take care of our .ts files when not created using defineFunction().
I've created a helper function to copy the .cjs file that is tranpiled from my .ts file to the correct asset folder that ampx has designated for my function. It then gets packaged and uploaded. It could be modified to include the transpile step directly.

@MyNameIsTakenOMG , I also found that ampx doesn't automatically take care of our .ts files when not created using defineFunction().
I've created a helper function to copy the .cjs file that is tranpiled from my .ts file to the correct asset folder that ampx has designated for my function. It then gets packaged and uploaded. It could be modified to include the transpile step directly.

Oh, many thanks 🙏  I was looking for a way to transpile the ts code and put it into the correct place so that ampx could pick up and upload. I will definitely check it out. Thank you, really appreciate it 👍

Is this going to be fixed? Or is this the solution? https://github.com/ggj0418/amplify-next-pages-template/compare/main...error/circular-dependency
Trying understand where this issue is, because it's a blocker for what we're trying to solve.

bump, my team and I aren't able to progress a production application due to the issue!

Same issue here. I'm using a custom mutation (not a function)
deleteTickets: a
    .mutation()
    .arguments({
      ids: a.id().array(),
    })
    .returns(a.json())
    .authorization((allow) => [allow.authenticated()])
    .handler(
      a.handler.custom({
        dataSource: a.ref('Ticket'),
        entry: './handlers/tickets/deleteTickets.js',
      })
    ),

bump, my team and I aren't able to progress a production application due to the issue!

me too

I think it is not a good idea to use scheduled Job or data stream feature with amplify gen2.
Our team are in a situation where scheduled jobs are essential, so we had no choice but to use aws sam & aws cdk to implement the feature. I hope this function will be officially supported by amplify gen2 ASAP.

I submit to you a way to reproduce the error from the provided starter template:

Follow instructions for "DynamoDB Streams" ✅ example
Follow instructions for "Add user to group" ❌ example

The CloudFormation deployment has failed.
Caused By: ❌ Deployment failed: Error [ValidationError]: Circular dependency between resources: [auth179371D7, data7552DF31, function1351588B]

Resolution: Find more information in the CloudFormation AWS Console for this stack.

PS. The resolution part of the message above is misleading, there is no more information in the CloudFormation console. DS.

The docs have been updated. https://docs.amplify.aws/react/build-a-backend/functions/examples/dynamo-db-stream/
The root cause is that all functions are placed in a single CFN stack. There will be a circular dependency between the data and function stack if:

There is a dependency from data to any functions in the app.
There is dependency from any function to any data resource.

I'll be updating https://github.com/aws-amplify/amplify-category-api/issues/2375 soon with a recommendation on how to move a function to a different stack.


Support changing colormap with string like visualize(cmap='RdBu')

[x] Tests added / passed
[x] Passes flake8 dask
[x] Fully documented, including docs/source/changelog.rst for all changes
and one of the docs/source/*-api.rst files for new API

I plan to merge this soon if there are no comments.


 : Upgrading clang to 3.8

 : config

https://github.com/swiftdocker/docker-swift/pull/58

How can the "3.1" and "latest" dockerfiles still be on Clang 3.6 after this was merged?


Heterogeneous (partially archived) setup

Hi! Is it possible to create a setup where one can have once serialized tree that is loaded as a writeable resource. So that when write occurs, a serialized node is replaced by an in-memory representation? Something like a read-only filesystem with writable overlay that can later be serialized too to merge serialized and not-yet-serialized stuff into a single archive.

You might be able to do this by defining an enum (i.e Heterogeneous<T>) with an Archived variant containing an Archived<T> and an Unarchived variant containing a T. The serialization implementation would just check the variant and either deserialize and re-serialize (if Archived) or just serialize (if Unarchived). There are a few more things to figure out, but I think that general shape will work.
As always, make sure your types are archive-safe (enums should have a defined repr) and you should be good to go.

Closing this for now, feel free to reopen if you have more questions 🙂


The App Crashes On Launch

The app (latest 1.6.0) when launched crashes with the following message:

OS X: 10.11.4

Thanks @insha. I am working on it.

Fixed in the version 1.6.1. Could you please check if it works for you?

Here is the error for 1.6.1 on windows:

That worked! Thanks 👍


Upgrade to latest galois-field and elliptic-curve deps

@Acentelles This looks good to merge. Just fix the benchmarks merge conflicts.

Also update the Changelog.

Fixed the merge conflicts. I think.

Force push is evil.


[Bug] Single Block producer creates multiple blocks in one slot

While we were testing some stuff our single node created 3 blocks at the same slot which is 1185. The duplicated blocks got rejected but the out of sync blocks just can go and with a single node without any extra my node pushed 3 blocks in a slot. This might create an issue with parties flooding invalid blocks in a single slot effecting other nodes and more. I feel like it should be investigated a bit further.

This has been replicated by others on Discord and I also just experienced this. Here's my metrics for block produced showing multiple blocks produced in a single slot (epoch 1 slot 864)

Many more reports of this on Github, plus if you win multiple slots within a short time frame this can produce a lot of additional blocks.

@EmrePiconbello - Could you send us the logs from this time period when these were evaluated?  Also are you passing in multiple block producer keys and if so, using what method?

@aneesharaines - I didn't have it right now but I can replicate the same thing. Simply you can replicate it by stoping the node and wiping the node's data just before a block production. I was keeping a node in the old state and run it before the block production slot.
I am not passing multiple block producer keys it's just a normal node I am just manipulating the node by stopping it down and starting back up. I didn't have much time to go into details and test various situations so I didn't follow it up.

Just to add to @EmrePiconbello description this will also happen on a node that persists the state. The only thing you need to do is to be in catchup when it is your turn for block production and multiple blocks will be produced in the slot - it seems as fast as your machine can complete a proof.
There are many instances of this already reported on Discord or visible by looking at blocks produced in a slot. These are just normal nodes in catchup producing multiple blocks per slot.

Adding one more example to this. Unfortunately I let the logs rollover and just noticed it now. Contributing anecdote anyway in case helpful.
I have a scenario where a single BP produced a block that made it in as the canonical block, and then produced again. Canonical: https://minaexplorer.com/block/3NKXYYvB5yH7BD9aDrwW3kHmDdnpDZGEK6mDEztXkgY9MghmMguw
and duplicate about a minute later: https://minaexplorer.com/block/3NKbJcUrrCbKKHFUYmWxnDzWSMUyWCgwYLgPsqWBSKBuLmk8yGGG
(My redundant BPs are now identified by unique coinbase receiver, so I know this was the same node.)

Throwing in my hat into this ring, had this occur to me today:

None of my blocks ended up being gossiped, missed the slot as a result. Bummer!

Our thought is this bug is not reproducible in the latest releases. Will reopen if its encountered again.


modify the route entry file name, resulting in an incomplete path

https://github.com/angular-examples/toh-6
modify the file name web/index.html to web/main.html
http://localhost:8080/main.html to http://localhost:8080/#dashboard
Refresh
wrong address http://localhost:8080/#dashboard Not Found
correct address http://localhost:8080/main.html#dashboard
click Heroes to http://localhost:8080/#heroes
Refresh
wrong address http://localhost:8080/#heroes Not Found
correct address http://localhost:8080/main.html#heroes

Yes, that is expected. If you rename the index.html file then that will impact the URL at which you access your app. That has nothing to do with Angular, it has to do with how index.html files are treated. Why are you renaming index.html? (I can't think of any use cases where you would need to do that.)

I was personally also surprise that "index.html" is treated differently than others html:
https://github.com/dart-lang/angular/blob/master/angular_router/lib/src/location/location.dart#L193
I spend hours tracking a bug because of this.
I have a web folder with plenty of entry points named other than index.html. And I actually advice to not use "index.html" because of this difference.

Yes, you are right that some web servers serve the index.html when requesting a directory. But not the other way: it doesn't redirect you to the directory you when you are requesting index.html.
The bug hit me when I was when trying to port my app to the new router. Lot of pieces were moving and I was switching between HashStrategy and PathStrategy to understand the new behavior. In the end the bug was on my end, but it was painful to understand that my 2 tests pages were behaving differently because of their name.

But not the other way: it doesn't redirect you to the directory you when you are requesting index.html.

You may have a point there, but I vaguely recall this behavior from the AngularTS router too (but I could be mistaken).
@leonsenft - any comment about https://github.com/dart-lang/angular/blob/master/angular_router/lib/src/location/location.dart#L193 and why it is there?

@chalin
Unfortunately no idea, we inherited it that way.


Regular Expression

Can I add awesome-regexp in this list?

Collection of anything about Regular Expression (RegExp)

That's not what an awesome list should be. It should be a curated list of the best things. Not a "collection of everything".

@arthurvr Thanks for your suggestion.
I will add those later.
How about

A curated list of awesome Regular Expression resources, tools,  and shiny things

Maybe, but you need to be a lot more strict on what you're including.
Including w3schools doesn't really give a high confidence in its quality.
http://www.w3fools.com/

You just created the list. Give it some time to mature and improve. Try submitting again in a month or so.

Ok @sindresorhus Thanks.

One question, why w3schools still always show on the top on the search page

@huei90 Because they use sleazy and spammy SEO optimizations to get there.

I have moved the project to awesome-regex, see
https://github.com/aloisdg/awesome-regex/issues/5


FAPI-RW-5.2.3-3: claims_parameter_supported must be: true

The test module CheckDiscEndpointClaimsParameterSupported expects the discovery endpoint returns claims_parameter_supported = true. But keycloak returns false.
FAPI-RW-5.2.3-3 describes the client shall request user authentication by requesting the acr claim as an essential claim. It means OP needs to support use of the claims parameter.
When OP supports the parameter, the discovery endpoint should return claims_parameter_supported = true.

related keycloak-jira ticket created : https://issues.jboss.org/browse/KEYCLOAK-14380

https://github.com/keycloak/keycloak/pull/7232


Typo in :help tag-priority

Hi,
I think I've found a little typo in the documentation displayed by :help tag-priority; it currently says:
- when 'tagcase' is "followic" and 'ignorecase' is off

but it should say:
- when 'tagcase' is "followic" and 'ignorecase' is on

If that's the case, then the documentation for :help 'smartcase' might be updated as well, from:
":g" and ":s".  Not used for "*", "#", "gd", tag search, etc.  After

to something like:
":g" and ":s"; used for tag search only when 'tagcase' is "followscs".  Not used for "*", "#", "gd", etc.  After

Thanks.

Here is another. It should say
This is a very good one. It works for C, C++, Java,

instead of
This a very good one. It works for C, C++, Java,

Hmm, these two lists don't look correct.  How about this:
The ignore-case matches are not found for a ":tag" command when:

'tagcase' is "followic" and the 'ignorecase' option is off
'tagcase' is "followscs" and the 'ignorecase' option is off and the
'smartcase' option is off or the pattern contains an upper case character.
'tagcase' is "match"
'tagcase' is "smart" and the pattern contains an upper case character.

The ignore-case matches are found when:

a pattern is used (starting with a "/")
for ":tselect"
when 'tagcase' is "followic" and 'ignorecase' is on
when 'tagcase' is "followscs" and 'ignorecase' is on or the 'smartcase'
option is on and the pattern does not contain an upper case character
when 'tagcase' is "ignore"
when 'tagcase' is "smart" and the patter does not contain an upper case
character

That looks good, but :help 'smartcase' already says it'll override the 'ignorecase' option only when the search pattern is typed and 'ignorecase' option is on, so I think the previous wording of:

'tagcase' is "followscs" and the 'ignorecase' option is off and the
'smartcase' option is off or the pattern contains an upper case character.

and

when 'tagcase' is "followscs" and 'ignorecase' is on or the 'smartcase'
option is on and the pattern does not contain an upper case character

are just okay.

fixed by 12ee7ff00b91d852e060bb24951d1c94239


Fix error when viewing timelines in Optimize-Bottlenecks in IE

https://bugzilla.redhat.com/show_bug.cgi?id=1234588
@dclarizio, @h-kataria please review.

@dclarizio Fix verified in IE9

@bmclaughlin Can we try to consolidate the duplicated code into a common method?  Thx, Dan

@Fryguy, @tenderlove, do you have any recommendations on DRY'ing up the getTLdata method between an application controller concern and two other controllers?
https://github.com/ManageIQ/manageiq/blob/master/app/controllers/application_controller/timelines.rb#L130
https://github.com/ManageIQ/manageiq/blob/master/app/controllers/dashboard_controller.rb#L597
https://github.com/ManageIQ/manageiq/pull/3529/files#diff-8c3427b37f1658d1cc09ffb1b7d975e5R67

Yes, we should definitely dry these up.  Are they exactly the same?  Just at a glance they look very similar.  We should extract a function with the similar things, then extract the different bits.  Sorry, that's a little vague, but essentially we should use the Extract Method technique a couple times.  I can show you a patch to get you started if that helps!

Not exactly the same, I found a one line difference that will be easily solved with one parameter.  I wasn't sure where the best home for the extracted method would be (application helper or application controller?)?

@bmclaughlin well it looks like this method creates XML that is only to do with VM instances.  I would either move it to a VM related model, or a VM related presenter ("presenter" being code word for "plain old ruby object").  If we don't have a VM related presenter, I don't see why we shouldn't add one! :)

@bmclaughlin need to correct the failing spec.

@chessbyte @Fryguy @dclarizio @matthewd, recommended changes implemented, please review.

if session[:tl_xml_blob_id] != nil should probably be spelled if session[:tl_xml_blob_id]
... but is it even useful? AFAICS, if it's not true, we'll fail to render, and (given there's no corresponding view) still end up with an error either way. So I think we can just lose it.

Err, we seem to have lost a destroy... was that deliberate?

@bmclaughlin Travis failure, seems a spec still has the CamelCased route in it.

@matthewd, I agree on the nil check, and I lost the destroy on purpose as BinaryBlob is no longer being instantiated to a variable.
@dclarizio, fixed that pesky spec.

@bmclaughlin the destroy removes the row from the database.  Not sure I am following your explanation of why it was removed. (cc @matthewd)

@chessbyte, apologies, thought that was destroying the object in memory.  Need more coffee.  Patch coming momentarily!

Patch coming momentarily!

bump @bmclaughlin

@chessbyte, 'Momentarily' became slightly longer due to a double rendering issue between rendering the view and rendering the xml for IE.

@chessbyte, @dclarizio this should be ready when travis finishes.

@bmclaughlin there still seems to be code duplication between the new multiple timeline_data methods.  Will that be refactored at some point via a new PR?

@tenderlove @Fryguy @matthewd what is the best way to refactor the duplicate timeline_data methods?  Is it via mixin, concern or some other technique/pattern?

... but is it even useful? AFAICS, if it's not true, we'll fail to render, and (given there's no corresponding view) still end up with an error either way. So I think we can just lose it.

@chessbyte I don't think there's enough code there to duplicate, to warrant extraction. Especially if my above allegation is true.

@chessbyte, @matthewd  Originally the miq_capacity_controller was throwing errors when trying to access the copy of timeline_data in application controller, however that seems to have been due to an intervening bug and the duplicate code is now properly DRY'd out as well as the suggested optimizations implemented.  And I squashed the commits.

@bmclaughlin Nice and DRY!  Waiting for Travis to go green before merging.

@bmclaughlin I'm still missing an answer to https://github.com/ManageIQ/manageiq/pull/3529#issuecomment-127423356... I don't understand what the two return lines are gaining us: won't we still just get an error?

@matthewd I agree, we will still just get an error.  Would you like me to remove those lines in a quick follow-up PR?

Yeah, let's kill those two, and switch to find instead of find_by_id. 3 lines is better than 5 :smile:
(and while I'd generally favour find over find_by_id anyway, in any situation where we're not explicitly planning to handle a nil return... I think using find should convert both those error conditions to 404s, which is a nice bonus)


Fix hasChildNodes method.

Welcome to Codecov :tada:
Once merged to your default branch, Codecov will compare your coverage reports and display the results in this comment.
Thanks for integrating Codecov - We've got you covered :open_umbrella:


datepicker fails 2.0.0-rc.4/1.0.24

I've seen tickets for this from a few weeks ago, but I still get the same result:
platform-browser.umd.js:1900 EXCEPTION: Error: Uncaught (in promise): Template parse errors:
No provider for NgModel ("-bootstrap {{ date.toDateString() }}</alert>
<div style="display:inline-block; min-height:290px">
  [ERROR ->]<datepicker [(ngModel)]="date" showWeeks="true"></datepicker>
</div>
"): Ng2BootstrapComponent@2:2

It doesn't seem to recognise datepicker at all, and I'm lost as to what to twiddle to fix it..

Same here, I've been looking for answer for 2 hours.

check out https://angular.io/docs/ts/latest/guide/forms.html if you haven't already. I had the same problem and almost made it working by adding the following bootsrapping code:
import { bootstrap }    from '@angular/platform-browser-dynamic';
import { disableDeprecatedForms, provideForms } from '@angular/forms';
import { AppComponent } from './app.component';
bootstrap(AppComponent, [
  disableDeprecatedForms(),
  provideForms()
 ])
 .catch((err: any) => console.error(err));

also, I am now importing FORM_DIRECTIVES from angular/forms instead of common and also adding
[ngModelOptions]="{standalone: true}"
to all inputs I use ngModel..
Hope that helps..

That's true, starting from v1.0.17 ng2-bootstrap doesn't work with deprecated forms

I'm having the same issue with pagination component,
@koraybalci what do you mean about [ngModelOptions]="{standalone: true}" to all inputs?
such as:
<pagination  [totalItems]="bigTotalItems" [(ngModel)]="bigCurrentPage" [ngModelOptions]="{standalone: true}"></pagination>
but I tried it, error still exist

@whileLooper yes, that's what I did, along with the other stuff I mentioned. Are you getting the same error as OP? Did you do all the pieces I mentioned?

@koraybalci
finally I got it fixed after 2 hours research in this repo, the problem is
import { disableDeprecatedForms, provideForms } from '@angular/forms'
not
import { disableDeprecatedForms, provideForms } from '@angular/forms/index'
can anyone tell me whats that different? import from folder and import from index file


Improve the general readme

Is your feature request related to a problem? Please describe.
Add the components to general readme and improve it wherever it is not linked.

Assign it to me !! @siddhi-244

Is your feature request related to a problem? Please describe. Add the components to general readme and improve it wherever it is not linked.

Kindly assign this issue to me.

Is your feature request related to a problem? Please describe. Add the components to general readme and improve it wherever it is not linked.

Kindly assign this issue to me.

Assigning it to you !


Question: adding new resolvers/services?

Hey there!
Thanks for this sample, really nice and simple, one question though, how can new services be added to the NewResolver method? https://github.com/alexzimmer96/gqlgen-example/blob/e63752d46e2a7d37e9bd10058e6c3178aed12679/graphql/resolvers.go#L13
it looks like only a single service, in this case articleService passed through, would changing this block to the following do the trick? I'm still new to go but quite familiar with building gql backends with apollo.
type Resolver struct {
	articleService service.IArticleService,
        someServiceA service.ISomeServiceA,
        someServiceB service.ISomeServiceB,
}

func NewResolver() *Resolver {
	return &Resolver{
		articleService: service.IArticleService,
                someServiceA: service.ISomeServiceA,
                someServiceB: service.ISomeServiceB,
	}
}

Anyway thanks for providing this example

Hi @jay-manday
Wow.. I took me more than one year to realize there's actually a Issue in one of my projects. I apol(l)ogize for the late feedback. At least with the version i've used, this should do the trick.


Imported obj-files (3D-models) not usable when Godot is compiled with double precision floats

Godot version
v4.0.alpha.custom_build [15db72c53]
System information
Windows 10, Vulkan
Issue description
Obj-files (3D-models) added into the project directory can not be used as MeshInstance3D's mesh if godot is compiled with double-precision floats (option float=64). Trying to use imported obj-file (in this case MeshLabCube.obj) leads to following errors (First line and "Dict1" and "Array"-strings added because I tried to find the cause):

core\io\resource_format_binary.cpp:602 - type: 0
core\io\resource_format_binary.cpp:603 - Method/function failed. Returning: ERR_FILE_CORRUPT
Error when trying to parse Variant. Dict1
Error when trying to parse Variant. Array
Failed loading resource: res://.godot/imported/MeshLabCube.obj-4ecfb58fb3f397871e450daffa9d1ea1.mesh. Make sure resources have been imported by opening the project in the editor at least once.
Failed loading resource: res://MeshLabCube.obj. Make sure resources have been imported by opening the project in the editor at least once.
Cannot load resource from path 'res://MeshLabCube.obj'.

It is possible to work around this problem by first starting single-precision-version of Godot (compiled from same sources) to generate ".godot"-directory and "*.obj.import"-files. After that the project can be opened with double-precision version as well. If ".godot"-directory and *.obj.import-files are deleted after using the erroneously imported obj-file in the project, the project fails to load in double-precision version:

This also can be fixed by deleting those import-files/dirs and regenerating the files again by loading the project with single-precision-version, although it gives a warning when doing so (which is ok, of course):

While trying to find the cause I made these modification into "core/io/resource_format_binary.cpp" (only added the "Dict x" and "Array"-texts to error strings in function parse_variant(Variant& r_v)):
case VARIANT_DICTIONARY: {
	uint32_t len = f->get_32();
	Dictionary d; //last bit means shared
	len &= 0x7FFFFFFF;
	for (uint32_t i = 0; i < len; i++) {
		Variant key;
		Error err = parse_variant(key);
		ERR_FAIL_COND_V_MSG(err, ERR_FILE_CORRUPT, "Error when trying to parse Variant. Dict1");
		Variant value;
		err = parse_variant(value);
		ERR_FAIL_COND_V_MSG(err, ERR_FILE_CORRUPT, "Error when trying to parse Variant. Dict2");
		d[key] = value;
	}
	r_v = d;
} break;
case VARIANT_ARRAY: {
	uint32_t len = f->get_32();
	Array a; //last bit means shared
	len &= 0x7FFFFFFF;
	a.resize(len);
	for (uint32_t i = 0; i < len; i++) {
		Variant val;
		Error err = parse_variant(val);
		ERR_FAIL_COND_V_MSG(err, ERR_FILE_CORRUPT, "Error when trying to parse Variant. Array");
		a[i] = val;
	}
	r_v = a;

} break;

Code around line 603 with some debug-output added:
default: {
	char errStrBuffer[100];
	sprintf(errStrBuffer, "type: %d", type);
	_err_print_error(FUNCTION_STR, __FILE__, __LINE__, errStrBuffer);
	ERR_FAIL_V(ERR_FILE_CORRUPT);
} break;

This is the default-branch of the same switch as the cases above.
I tried to figure out what's going on here, but got quite lost. Looking at the files generated in ".godot"-directory it looked like this could be related to AABB encoded erroneously there (using doubles in the file output from double-version of Godot with the same "variant-id"?). But this is really a guess as I still couldn't find a way to fix this...
Steps to reproduce

Create a 3D-scene with Godot compiled with double-precision floats (option float=64)
Add .obj-file to be imported into project directory
Try to use the imported file as MeshInstance3D's mesh. This will output the aforementioned errors.

Minimal reproduction project
obj_import_bug_double.zip
Loading this fails in Godot compiled with double-precision floats.

cc @aaronfranke


Enable the use of "now()" and "today()" in presentation requests

Some very typical things in Presentation Requests are requesting the revocation status of a credential "as of now", and requesting a person prove they are older than a certain age "today".  It would be nice to be able to have Traction replace certain magic values in a presentation request before sending it.  See the alternate ideas below.
Notably:

replace any instance of now() in a presentation request with the UnixTime equivalent of the second the presentation request is submitted.  This allows the user to "now()" instead of determining the UnixTime equivalent.
replace any instance of today(<num>) in a presentation request with the "date_int" version of today minus  years. So if asking to prove a person is older than 19, the user would put "today(19)" into the predicate, and Traction would replace it with 20041124 (the integer -- no double-quotes).

I think it would be safe enough to do a naive implementation, where the text is searched on and replaced.  A regex would work for the first, although not the second...
To be totally safe, you may want to limit the substitutions for now() into the revoke_inteval from and to values, and today() to predicates, but I think that would be overkill in keeping the presentation request writer safe...
I think this should only apply to a presentation request submitted via the Traction UI.  This is a general feature that is needed, but I think we can safely assume any app doing this can handle the pre-processing itself.
Alternate ideas:

Have a button in the UI (e.g., above the "Submit") that can be clicked to trigger the replacement, so that the person sees what gets replaced before submitting.
Have the UnixTime of now() displayed with a "copy" button on the form, and an "Over [ ] years old today" that the person could fill in and copy to get the data, and then paste them into their presentation request.

Another hacky alternative -- put 99999999 and<PHONE_NUMBER> into the request, so there is no messing with "now()" around the text versions and leaving them off when replaced by numbers.  The latter is a pain -- getting the exact number of 9s right. And of course, the minor chance that those numbers are needed in a presentation request :-).


createReactSpec type too restrictive.

In my experience, having to supply the initial state without access to any effects while calling createReactSpec is a bit to restrictive. Given that the ReactSpec has a getInitialState field which allows effects, should't we allow the same opportunity here?

Sounds useful. Would you like to make a PR and we can compare?
Thanks!

sure, should be easy

Here's a pr with breaking changes, but maybe we can start here
https://github.com/paf31/purescript-thermite/compare/master...blinky3713:effectful-initial-state

Thanks! This would be a breaking change, but maybe it's better this way. Could you please open a PR?


Whats the shame with pinouts?

Where's the truth?

Sorry I mixed up with T-Display

Hi, do you still have a question or should you close the issue?
Cheers, Terry
Cheers !
Terry Stainton
@.***

On Aug 1, 2023, at 5:26 AM, Vladimir Yershov @.***> wrote:
Sorry I mixed up with T-Display
—
Reply to this email directly, view it on GitHub https://github.com/Xinyuan-LilyGO/T-Display-S3/issues/179#issuecomment-1659926741, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATCNKOGTZU3XDDZEYRXMAI3XTDDVDANCNFSM6AAAAAA27NT7PA.
You are receiving this because you are subscribed to this thread.

Sure. close it

yes


LSP Range: allow empty range overlap predicate.

codeAction requests often just include the cursor position in the
request which is an empty range. If the diagnostic in question is
a one-character range, make sure we recognize that as overlap.
Signed-off-by: Henner Zeller<EMAIL_ADDRESS>

Codecov Report

Merging #1266 (5e2ec2a) into master (30d3d89) will increase coverage by 0.00%.
The diff coverage is 100.00%.

:exclamation: Current head 5e2ec2a differs from pull request most recent head d683b5f. Consider uploading reports for the commit d683b5f to get more accurate results

@@           Coverage Diff           @@
##           master    #1266   +/-   ##
=======================================
  Coverage   92.83%   92.83%
=======================================
  Files         339      339
  Lines       23498    23500    +2
=======================================
+ Hits        21814    21816    +2
  Misses       1684     1684

Impacted Files
Coverage Δ

common/lsp/lsp-protocol-operators.h
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 30d3d89...d683b5f. Read the comment docs.


[SPARK-30188][SQL][WIP] Enable adaptive query execution by default

What changes were proposed in this pull request?
Enable adaptive query execution default

Why are the changes needed?
To expand the usage of AQE.

Does this PR introduce any user-facing change?
No

How was this patch tested?
Existing unit tests

Test build #115034 has started for PR 26813 at commit 9a819b7.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19854/
Test PASSed.

Test build #115034 has finished for PR 26813 at commit 9a819b7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115034/
Test FAILed.

Could you take a look at the UT failures, @JkSelf ?

fyi @maryannxue, @cloud-fan

@dongjoon-hyun sure. I will take a look at all the failure uts. Thanks

@JkSelf are we confident that AQE would be beneficial in most cases while not lead to regression ?

@manuzhang Based on the benchmark data in databricks, heard from @cloud-fan, after enable AE, there is no regression. So we decide enable AQE default. Next I will also test in my local environment and post the benchmark data later. Thanks.

@JkSelf, this is major feature and could have big (unexpected) user impact. We'd better firstly discuss it on dev list. @HyukjinKwon @cloud-fan @maryannxue, what do you think ?

I agree, would be good to bring up on dev list, I thought different companies have been running off the branch of this in 2.x, is anyone doing a lot of testing work on the version with 3.0?  Would be curious to see what all workloads/testing has been done on it so far and what kind of results you are seeing (ie generally performance increase with no slow downs, sometimes you see slow downs, etc.)

I've seen a regression with tpcds q87 on 1TB data with AQE enabled. The regression was due to executor GC issue brought by ReduceNumShufflePartitions and setting spark.sql.adaptive.shuffle.targetPostShuffleInputSize=16777216 could alleviate it.

Thank you for sharing the information, @manuzhang .

@dongjoon-hyun
I tried same query with latest master branch (shuffle services are still 2.3) and saw the same regression and executor GC issue with AQE enabled.

Another thing is the default enabled dynamic partition pruning isn't supported in AQE yet which would effectively disable AQE in some cases.

Given the above advices, I'll close this PR for now.
@JkSelf , you can reopen this after 3.0.0 release.
Thank you all!

It's arguable that if we should turn on AQE by default in 3.0 or not. But I think it's worthwhile to try turn it on and fix tests. We can merge this PR with tests fixed and still keep AQE off.
f we end up with deciding turn on AQE by default, then it will be a one-line change if we fix all the failed tests here.
@JkSelf can you reopen and fix tests first?

@cloud-fan Ok, I will fix the failed tests firstly. Thanks.

Test build #115379 has started for PR 26813 at commit 9a819b7.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20185/
Test PASSed.

Test build #115379 has finished for PR 26813 at commit 9a819b7.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115379/
Test FAILed.

Test build #115785 has started for PR 26813 at commit c3f8391.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20578/
Test PASSed.

Test build #115785 has finished for PR 26813 at commit c3f8391.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115785/
Test FAILed.

Test build #115811 has started for PR 26813 at commit e5791c1.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20603/
Test PASSed.

Test build #115811 has finished for PR 26813 at commit e5791c1.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115811/
Test FAILed.

We can merge this PR with tests fixed and still keep AQE off.

Sounds good

If we end up with deciding turn on AQE by default, then it will be a one-line change if we fix all the failed tests here.

Since AQE is a moving piece and the execution result could be indeterministic with AQE enabled , I'm not sure it would be as simple as one-line change after we merging this PR.

@manuzhang even if it's not a one-line change, it will be a small change, as we've fixed so many tests already here.

Test build #115838 has started for PR 26813 at commit a3d1720.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20631/
Test PASSed.

Test build #115838 has finished for PR 26813 at commit a3d1720.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115838/
Test FAILed.

Test build #115897 has started for PR 26813 at commit b84253a.

Test build #115897 has finished for PR 26813 at commit b84253a.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test PASSed.

Build finished. Test FAILed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20687/
Test PASSed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115897/
Test FAILed.

Test build #115899 has started for PR 26813 at commit 98c3c20.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20689/
Test PASSed.

Test build #115899 has finished for PR 26813 at commit 98c3c20.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115899/
Test FAILed.

Test build #115900 has started for PR 26813 at commit 0cac79d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20690/
Test PASSed.

Test build #115900 has finished for PR 26813 at commit 0cac79d.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115900/
Test FAILed.

Test build #115901 has started for PR 26813 at commit e06cd15.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20691/
Test PASSed.

Test build #115901 has finished for PR 26813 at commit e06cd15.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115901/
Test FAILed.

retest this please

Test build #115903 has started for PR 26813 at commit e06cd15.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20693/
Test PASSed.

Test build #115903 has finished for PR 26813 at commit e06cd15.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115903/
Test FAILed.

Test build #115907 has started for PR 26813 at commit 2a309b4.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20697/
Test PASSed.

Test build #115907 has finished for PR 26813 at commit 2a309b4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115907/
Test FAILed.

Test build #115909 has started for PR 26813 at commit 0dea683.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20699/
Test PASSed.

Test build #115909 has finished for PR 26813 at commit 0dea683.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115909/
Test FAILed.

retest this please

Test build #115914 has started for PR 26813 at commit 0dea683.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20704/
Test PASSed.

Test build #115914 has finished for PR 26813 at commit 0dea683.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115914/
Test FAILed.

Test build #115915 has started for PR 26813 at commit c0b58af.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20705/
Test PASSed.

Test build #115915 has finished for PR 26813 at commit c0b58af.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115915/
Test FAILed.

Test build #115925 has started for PR 26813 at commit acfaa7a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20716/
Test PASSed.

Test build #115925 has finished for PR 26813 at commit acfaa7a.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115925/
Test FAILed.

retest this please

Test build #115933 has started for PR 26813 at commit acfaa7a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20725/
Test PASSed.

Test build #115933 has finished for PR 26813 at commit acfaa7a.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115933/
Test FAILed.

Test build #115956 has started for PR 26813 at commit 9562ce9.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20748/
Test PASSed.

Test build #115956 has finished for PR 26813 at commit 9562ce9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115956/
Test FAILed.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20752/
Test PASSed.

Test build #115959 has started for PR 26813 at commit 9562ce9.

Test build #115959 has finished for PR 26813 at commit 9562ce9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115959/
Test FAILed.

can we update the description of this jira to fix tests with adaptive execution

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20778/
Test PASSed.

Test build #115986 has started for PR 26813 at commit 15f58ac.

Test build #115986 has finished for PR 26813 at commit 15f58ac.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115986/
Test FAILed.

Test build #115989 has started for PR 26813 at commit 2f9307e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20781/
Test PASSed.

Test build #115989 has finished for PR 26813 at commit 2f9307e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/115989/
Test FAILed.

Test build #116009 has started for PR 26813 at commit 32e8bea.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20801/
Test PASSed.

Test build #116009 has finished for PR 26813 at commit 32e8bea.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116009/
Test FAILed.

Test build #116041 has started for PR 26813 at commit 79d4dff.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20832/
Test PASSed.

Test build #116041 has finished for PR 26813 at commit 79d4dff.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116041/
Test FAILed.

Test build #116096 has started for PR 26813 at commit ea1d810.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20889/
Test PASSed.

Test build #116096 has finished for PR 26813 at commit ea1d810.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116096/
Test FAILed.

Test build #116110 has started for PR 26813 at commit 1ac455a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/20902/
Test PASSed.

Test build #116110 has finished for PR 26813 at commit 1ac455a.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116110/
Test FAILed.

@cloud-fan @dongjoon-hyun @HyukjinKwon  updated all your comments. Thanks.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21005/
Test PASSed.

Test build #116215 has started for PR 26813 at commit 0982708.

Test build #116215 has finished for PR 26813 at commit 0982708.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116215/
Test FAILed.

Test build #116216 has started for PR 26813 at commit 5652338.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21008/
Test PASSed.

Test build #116216 has finished for PR 26813 at commit 5652338.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116216/
Test FAILed.

Test build #116219 has started for PR 26813 at commit d9e98fb.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21011/
Test PASSed.

Test build #116219 has finished for PR 26813 at commit d9e98fb.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116219/
Test FAILed.

retest this please

Test build #116222 has started for PR 26813 at commit d9e98fb.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21014/
Test PASSed.

Test build #116222 has finished for PR 26813 at commit d9e98fb.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116222/
Test FAILed.

Test build #116239 has started for PR 26813 at commit 2daeed9.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21031/
Test PASSed.

Test build #116239 has finished for PR 26813 at commit 2daeed9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116239/
Test FAILed.

can you please update the title on this PR to not be enable by default. Change it to fixing tests or something.

+1 for @tgravescs 's comment.

@travishegner @dongjoon-hyun Thanks for your suggestions. Updated the title.

Test build #116308 has started for PR 26813 at commit 9c986bf.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21100/
Test PASSed.

Test build #116308 has finished for PR 26813 at commit 9c986bf.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116308/
Test FAILed.

Test build #116311 has started for PR 26813 at commit 0f3991e.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21103/
Test PASSed.

Test build #116311 has finished for PR 26813 at commit 0f3991e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116311/
Test FAILed.

Test build #116340 has started for PR 26813 at commit 4eacd0b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21131/
Test PASSed.

Test build #116340 has finished for PR 26813 at commit 4eacd0b.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116340/
Test FAILed.

Test build #116344 has started for PR 26813 at commit 2ec1914.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21135/
Test PASSed.

Test build #116344 has finished for PR 26813 at commit 2ec1914.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116344/
Test FAILed.

Test build #116346 has started for PR 26813 at commit 752728d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21137/
Test PASSed.

Test build #116346 has finished for PR 26813 at commit 752728d.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116346/
Test FAILed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21142/
Test PASSed.

Test build #116352 has started for PR 26813 at commit bb46c1d.

Test build #116352 has finished for PR 26813 at commit bb46c1d.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116352/
Test FAILed.

Test build #116358 has started for PR 26813 at commit c72c20d.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21149/
Test PASSed.

Test build #116358 has finished for PR 26813 at commit c72c20d.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116358/
Test FAILed.

Test build #116361 has started for PR 26813 at commit a8f5e03.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/21152/
Test PASSed.

Test build #116361 has finished for PR 26813 at commit a8f5e03.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/116361/
Test FAILed.

Test build #116364 has started for PR 26813 at commit e3ef524.


Change api to just get plist

Closed? Changed your mind?

hmmm... It does seem useful. I'll reopen


Record some more of the thinking for parsing

@KrzysztofCwalina, thoughts?

@terrajobst can you also update the parsing.md to include the relative priorities we discussed over email:
Parsing

(pri 0) performance tests and improvement
(pri 0) support fast lower case and upper case hexadecimal parsing
(pri 1) remove old APIs
(pri 1) remove EncodingData(byte[][] symbols, TextEncoding encoding, Tuple<byte, int>[] parsingTrie)
(pri 1) add all primitive parser to PrimitiveParser (e.g. GUID, URI, date time, etc.)
(pri 1) add support for multi-span parsing
(pri 2) support ASCII
(pri 2) clean up code
(pri 2) full test coverage
(pri 2) support all formats
(pri 2) provide EncodingData.InvariantUtf16BE
(pri 2) write custom EncodingData generation tool

While we're still fine tuning the spec, let's merge what we have so far.


🐛 Zipped Files: Missing Identifier - BOM issue

Summary
The client would import CSVs that had identifier columns, but the importer would still fail with "Missing at least one required element, missing element(s) are: identifier"
original/spike ticket:

https://github.com/scientist-softserv/adventist_knapsack/issues/680

The Spike determined the following:
Debugging from console -
Comparing "identifier" with "identifier"
Bytes do not match: "identifier" != "identifier"
Expected bytes: [105, 100, 101, 110, 116, 105, 102, 105, 101, 114]
Actual bytes: [239, 187, 191, 105, 100, 101, 110, 116, 105, 102, 105, 101, 114]

The difference in bytes indicates that the identifier key in the data contains a Byte Order Mark (BOM) character (represented by the byte sequence [239, 187, 191]) at the beginning. This can happen if the CSV file was saved with a BOM, which is common for UTF-8 encoded files.
Acceptance Criteria

[ ] A user should be able to import the following sample file: DOWNLOAD

Screenshots or Video
BEFORE

AFTER

Testing Instructions

Create a Bulkrax Importer with the sample file. This CSV is big - I recommend setting a limit of 3
Verify the completion and success of the importer
Verify the created works have files attached

Notes

ssl cert expired

https://assaydepot.slack.com/archives/C0311DN2YCA/p1720805659060129

Blocked until valkyrie staging bugs are resolved

TODO: retest in staging

cc @KatharineV We are revisiting this to make sure things work with valkyrie. The file link has since expired. Would it be possible to refresh it or provide a new link to the same file, for importer:

BOX-010-corrected-e_percent.csv
BOX-010-corrected-c_plus.csv
@ShanaLMoore here are the two CSVs I see referenced in importers 380 (file name ends with c_plus) and 381 (ends with e_percent). Both importers and CSVs rely on related_url to import files. The file name endings indicate what characters were inserted into the related URLs in place of spaces, so you can see that the one with plus signs was probably being affected by that. The other file has %20 in place of the spaces.
Also, at some point I hit refresh on one of the related importers that was attempting to bring in the works in these spreadsheets. I did finally get them to import. Here's the CSV that finally worked:
BOX-010-corrected-f_success.csv

@ShanaLMoore I'm so sorry, but I was able to replicate the error! I hope it's still human error at work, but I've seen this message before and I just don't know. Thanks for putting this back in the important milestones for this final push.

@KatharineV This issue is seen on production which we won't be making a fix to.
Let me test it again within the staging instance and that'll determine how we move forward. Thanks for brining it up again.

@KatharineV I just tested this in staging, substituting out the actual pdfs with dummy pdfs but with the same csv and it was successful.  I'm hoping this means we're in the clear once this goes to production.
https://adl.s2.adventistdigitallibrary.org/importers/320?locale=en

I'm moving this to the next column, since today I was able to take an importer that failed in prod (for missing identifier) and ran it successfully in two staging tenants.


In settings window tab in common should be frame

In settings window tab in common should be frame

Fixed in Master.

Fixed in Prod.


CDATA Fix

Fix the URI fields to use CDATA as per the spec.
Added a CDATA struct for the cases where the URI field isn't
an unnamed field within another with attributes.

The cdata attribute for xml was introduced in 1.6 I think, so this PR breaks backwards compatibility :P

@kalbasit please review

on it

@jeffreypicard please address my comment and rebase over master.

@jeffreypicard can you please address my comments so we can merge this PR? Thank you.


Adds support to localized components files

Related: #894
I just combined the "localized" information from component files (my_component.es.html.erb) into variant attribute to avoid adding new parameters or functions and worked well :smile:
It's not done yet, needs some work, more tests, benchmarks... but I wanted to see some opinions about this approach, if there's something else I'm not seeing.

Benchmark:
localized components:

Warming up --------------------------------------
component:    13.862k i/100ms
partial:     2.296k i/100ms
Calculating -------------------------------------
component:    139.812k (± 0.9%) i/s -      1.400M in  10.014746s
partial:     22.876k (± 3.2%) i/s -    229.600k in  10.047667s

main branch:

Warming up --------------------------------------
component:    13.007k i/100ms
partial:     2.440k i/100ms
Calculating -------------------------------------
component:    129.883k (± 1.9%) i/s -      1.301M in  10.018044s
partial:     24.145k (± 3.6%) i/s -    241.560k in  10.021579s

I'm doing some real world tests in a dummy app, with the following configuration:
I18n.available_locales = [:en, :es, :fr]
I18n.default_locale = :en

I'm adding a FooComponent with 3 partials:
app/components/foo_component.en.html.erb
# no partial for `es`
app/components/foo_component.fr.html.erb
app/components/foo_component.html.erb

If I set I18n.locale = :en in my request, I get the foo_component.html.erb :arrow_right: I would expect foo_component.en.html.erb
If I set I18n.locale = :es in my request, I get the foo_component.html.erb :heavy_check_mark:
If I set I18n.locale = :fr in my request, I get the foo_component.fr.html.erb :heavy_check_mark:
If I set no locale in my request, I get the foo_component.html.erb :arrow_right: I would expect the I18n.default_locale to be used, so foo_component.en.html.erb

Now If I'm removing this partial :
app/components/foo_component.en.html.erb
app/components/foo_component.fr.html.erb
- app/components/foo_component.html.erb

I get an error in all cases: No such file or directory @ rb_sysopen - /home/lfalcao/review/app/components/foo_component.html.erb
I guess it should work without a "default" partial, what do you think?

Thanks for reviewing it @Spone
I think every case is expected as I used the I18n.default_locale as the default without locale information in the filename. I mean, if you set I18n.default_locale there's no need to have all your files with '.en'.... :thinking:
There's a (test) for this.

If I set I18n.locale = :es in my request, I get an error undefined local variable or method 'call' for #FooComponent:0x00007f9720c56610

This happened because there's no foo_component.es.html.erb. You think we should to fallback to the foo_component.html.erb ?

If I set no locale in my request, I get an error undefined local variable or method 'call' for #FooComponent:0x00007f9720c56610

Also expected, as the file foo_component.html.erb was deleted.

If I set no locale in my request, I get the show.en.html.erb
Actually, I never get the show.html.erb and I can remove it without getting errors

Without locale, but with I18n.default_locale = :en right?

This happened because there's no foo_component.es.html.erb. You think we should to fallback to the foo_component.html.erb ?

In this case, Rails localized views fall back to the default locale (so foo_component.en.html.erb in this case).

Also expected, as the file foo_component.html.erb was deleted.

Yes but in this case Rails localized views fall back to the default locale as well.

Without locale, but with I18n.default_locale = :en right?

Yes. Actually you can also remove it from the config since :en is the default value.
To sum up, I think the fall back behavior should be the same as Rails standard behavior with localized views, as described in the EDIT of my previous comment. What do you think?

Thanks @Spone and @jonspalmer for the comments. I'll have a look into it carefully.

@lfalcao is there any news on this PR?

I can take some time to revisit this @lfalcao if you're still interested. Please let me know!

I think it's worth to get this. Sometimes this is better than the current translations support

Closing as stale. Please reopen if/when you pick this up, @Spone!


macOS packaged applications not backwards compatible

STATUS:

@ntoll just packaged the Mu Editor 1.1.0b6 for release.
He used a mac running 10.15.
The packaged bundle does not launch on my system, running 10.14.

MORE:

Running otool -l ./Mu\ Editor/Contents/MacOS/Mu Editor on the official release vs on my system shows different binary data/linker info.
Compiling laucher.c with clang -mmacosx-version-min=10.9 ... adds a "thingie" to the binary:

Load command 9
      cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.9
      sdk 10.14

WONDERING:

Could this be enough?

LET's TRY!

Closing after having mergeed #178.


Bump the codec version to 10.0.

Lucene100Codec is the exact same file format as Lucene912Codec. This codec dance just makes things slightly easier to reason about since our backward compatibility guarantees are aligned with major version: once we drop support for 9.x indices, we can remove all Lucene9XXCodecs.

Codec 💯 ! 😂

I was in a meeting yesterday and someone told me "oh we are using a very recent Lucene release, 9.1".  And I asked them to double check ... and it was 9.10 they were using.  This conflict between floating point numbers and release versioning sure is tricky!

Thanks for merging @ChrisHegarty, I missed that PR.