nextToken not accepted and gives error "Invalid pagination token given"

Describe the bug
Query is working in development and build environments but when deployed in the Test environment the nextToken is never accepted.
The same error occurs in the code as well as the AppSync console where we do a query with a limit and get a nextToken. Then we do a query straight away with the same token and it gives the following error:
com.amazonaws.deepdish.common.pagination.InvalidPaginationTokenException: Invalid pagination token given.
To Reproduce
Do following query in the console (same as in the code) FYI There are 15 items in the table.
{
  listUsers(limit:3){
    items{
      name
    }
    nextToken
  }
}

The response is:
{
  "data": {
    "listUsers": {
      "items": [
        {
          "name": "Joergen Von Klopp"
        },
        {
          "name": "Yolanda McAdams"
        },
        {
          "name": "Info Test"
        }
      ],
      "nextToken": "eyJ2ZX...shortened="
    }
  }
}

Then we do it again with the nextToken:
{
  listUsers(limit:3,nextToken:"eyJ2ZX...shortened="){
    items{
      name
    }
    nextToken
  }
}

The error is:
{
  "data": {
    "listUsers": null
  },
  "errors": [
    {
      "path": [
        "listUsers"
      ],
      "data": null,
      "errorType": "DynamoDB:UserIllegalArgumentException",
      "errorInfo": null,
      "locations": [
        {
          "line": 2,
          "column": 3,
          "sourceName": null
        }
      ],
      "message": "com.amazonaws.deepdish.common.pagination.InvalidPaginationTokenException: Invalid pagination token given."
    }
  ]
}

We experience the same behavior with all the tables in this environment. If we increase the limit we of course get all the elements but for some reason nextToken does not work.
Expected behavior
Should get more data returned like on the other environments.
Screenshots
If applicable, add screenshots to help explain your problem.

Environment
npx envinfo --system --binaries --browsers --npmPackages --npmGlobalPackages
npx: installed 1 in 1.301s

  System:
    OS: Windows 10 10.0.18362
    CPU: (12) x64 Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
    Memory: 8.41 GB / 23.94 GB
  Binaries:
    Node: 12.13.1 - C:\Tools\nodejs\node.EXE
    npm: 6.13.2 - C:\Tools\nodejs\npm.CMD
  Browsers:
    Edge: 44.18362.449.0
    Internet Explorer: 11.0.18362.1

Additional context
Amplify --version 4.5.0
"aws-amplify": "^2.2.2",
"aws-amplify-react": "^3.1.3",

We are having a similar issue. We have a react.js / amplify app with a "master - detail" view where we paginate through records from an aws dynamo db using the 'next token' property returned from a graphQl 'list all' query. The app was working great until we tried to set up a team dev environment with continuous deploys pulling from a git repo. All queries seem to be working well from the new environments with the exception of the 'next token' returned from the list all query ...

Thanks @AaronHarris , we have 6 environments,

2 DEV environments,
a nightly BUILD env,
a TEST env
and a PROD env.

We merged from DEV -> BUILD -> TEST and this broke in TEST,
It works in DEV and BUILD but not TEST with the token error. FYI TEST worked before the merge and now ALL tables in TEST have the same issue.
Here is Query.listUsers.req.vtl  for both DEV and TEST. They are the same.
#set( $limit = $util.defaultIfNull($context.args.limit, 10) )
#set( $ListRequest = {
  "version": "2017-02-28",
  "limit": $limit
} )
#if( $context.args.nextToken )
  #set( $ListRequest.nextToken = "$context.args.nextToken" )
#end
#if( $context.args.filter )
  #set( $ListRequest.filter = $util.parseJson("$util.transform.toDynamoDBFilterExpression($ctx.args.filter)") )
#end
#if( !$util.isNull($modelQueryExpression)
                        && !$util.isNullOrEmpty($modelQueryExpression.expression) )
  $util.qr($ListRequest.put("operation", "Query"))
  $util.qr($ListRequest.put("query", $modelQueryExpression))
  #if( !$util.isNull($ctx.args.sortDirection) && $ctx.args.sortDirection == "DESC" )
    #set( $ListRequest.scanIndexForward = false )
  #else
    #set( $ListRequest.scanIndexForward = true )
  #end
#else
  $util.qr($ListRequest.put("operation", "Scan"))
#end
$util.toJson($ListRequest)

I am also coming across this error when using next token for pagination. This seems to have began after I pushed to my development environment the same day this ticket was raised (The error wasn't spotted until today). This push included a two field change to a table in my graphql schema, where said fields had their type changed from '[String]!' to 'AWSJSON!'. Interestingly, the now failing query that uses next token for pagination reads from a different table than that which was changed.
I am using Cognito user groups for authentication, and I can provide full schema if necessary.
Here is the error that I have captured in the AppSync console:
{
  "data": {
    "articlesByDate": null
  },
  "errors": [
    {
      "path": [
        "articlesByDate"
      ],
      "data": null,
      "errorType": "DynamoDB:UserIllegalArgumentException",
      "errorInfo": null,
      "locations": [
        {
          "line": 2,
          "column": 3,
          "sourceName": null
        }
      ],
      "message": "com.amazonaws.deepdish.common.pagination.InvalidPaginationTokenException: Invalid pagination token given."
    }
  ]

And here is the query req vtl in question (not quite as tidy as the gentleman above):
## [Start] Set query expression for @key **
#set( $modelQueryExpression = {} )
## [Start] Validate key arguments. **
#if( !$util.isNull($ctx.args.datePublished) && $util.isNull($ctx.args.dataType) )
  $util.error("When providing argument 'datePublished' you must also provide arguments dataType", "InvalidArgumentsError")
#end
## [End] Validate key arguments. **
#if( !$util.isNull($ctx.args.dataType) )
  #set( $modelQueryExpression.expression = "#dataType = :dataType" )
  #set( $modelQueryExpression.expressionNames = {
  "#dataType": "dataType"
} )
  #set( $modelQueryExpression.expressionValues = {
  ":dataType": {
      "S": "$ctx.args.dataType"
  }
} )
#end
## [Start] Applying Key Condition **
#if( !$util.isNull($ctx.args.datePublished) && !$util.isNull($ctx.args.datePublished.beginsWith) )
  #set( $modelQueryExpression.expression = "$modelQueryExpression.expression AND begins_with(#sortKey, :sortKey)" )
  $util.qr($modelQueryExpression.expressionNames.put("#sortKey", "datePublished"))
  $util.qr($modelQueryExpression.expressionValues.put(":sortKey", { "S": "$ctx.args.datePublished.beginsWith" }))
#end
#if( !$util.isNull($ctx.args.datePublished) && !$util.isNull($ctx.args.datePublished.between) )
  #set( $modelQueryExpression.expression = "$modelQueryExpression.expression AND #sortKey BETWEEN :sortKey0 AND :sortKey1" )
  $util.qr($modelQueryExpression.expressionNames.put("#sortKey", "datePublished"))
  $util.qr($modelQueryExpression.expressionValues.put(":sortKey0", { "S": "$ctx.args.datePublished.between[0]" }))
  $util.qr($modelQueryExpression.expressionValues.put(":sortKey1", { "S": "$ctx.args.datePublished.between[1]" }))
#end
#if( !$util.isNull($ctx.args.datePublished) && !$util.isNull($ctx.args.datePublished.eq) )
  #set( $modelQueryExpression.expression = "$modelQueryExpression.expression AND #sortKey = :sortKey" )
  $util.qr($modelQueryExpression.expressionNames.put("#sortKey", "datePublished"))
  $util.qr($modelQueryExpression.expressionValues.put(":sortKey", { "S": "$ctx.args.datePublished.eq" }))
#end
#if( !$util.isNull($ctx.args.datePublished) && !$util.isNull($ctx.args.datePublished.lt) )
  #set( $modelQueryExpression.expression = "$modelQueryExpression.expression AND #sortKey < :sortKey" )
  $util.qr($modelQueryExpression.expressionNames.put("#sortKey", "datePublished"))
  $util.qr($modelQueryExpression.expressionValues.put(":sortKey", { "S": "$ctx.args.datePublished.lt" }))
#end
#if( !$util.isNull($ctx.args.datePublished) && !$util.isNull($ctx.args.datePublished.le) )
  #set( $modelQueryExpression.expression = "$modelQueryExpression.expression AND #sortKey <= :sortKey" )
  $util.qr($modelQueryExpression.expressionNames.put("#sortKey", "datePublished"))
  $util.qr($modelQueryExpression.expressionValues.put(":sortKey", { "S": "$ctx.args.datePublished.le" }))
#end
#if( !$util.isNull($ctx.args.datePublished) && !$util.isNull($ctx.args.datePublished.gt) )
  #set( $modelQueryExpression.expression = "$modelQueryExpression.expression AND #sortKey > :sortKey" )
  $util.qr($modelQueryExpression.expressionNames.put("#sortKey", "datePublished"))
  $util.qr($modelQueryExpression.expressionValues.put(":sortKey", { "S": "$ctx.args.datePublished.gt" }))
#end
#if( !$util.isNull($ctx.args.datePublished) && !$util.isNull($ctx.args.datePublished.ge) )
  #set( $modelQueryExpression.expression = "$modelQueryExpression.expression AND #sortKey >= :sortKey" )
  $util.qr($modelQueryExpression.expressionNames.put("#sortKey", "datePublished"))
  $util.qr($modelQueryExpression.expressionValues.put(":sortKey", { "S": "$ctx.args.datePublished.ge" }))
#end
## [End] Applying Key Condition **
## [End] Set query expression for @key **
#set( $limit = $util.defaultIfNull($context.args.limit, 10) )
#set( $QueryRequest = {
  "version": "2017-02-28",
  "operation": "Query",
  "limit": $limit,
  "query": $modelQueryExpression,
  "index": "byDate"
} )
#if( !$util.isNull($ctx.args.sortDirection)
                    && $ctx.args.sortDirection == "DESC" )
  #set( $QueryRequest.scanIndexForward = false )
#else
  #set( $QueryRequest.scanIndexForward = true )
#end
#if( $context.args.nextToken ) #set( $QueryRequest.nextToken = "$context.args.nextToken" ) #end
#if( $context.args.filter ) #set( $QueryRequest.filter = $util.parseJson("$util.transform.toDynamoDBFilterExpression($ctx.args.filter)") ) #end
$util.toJson($QueryRequest)

And the res vtl:
$util.toJson($ctx.result)
Has anyone rebuilt their environment from scratch to see if this fixes it? I would prefer not to have to do this, but when in doubt...

I am waiting a week or so before I blow my test away and rebuild. I am hoping we can figure out a resolution because go live is coming quick and I cannot be rebuilding a production environment. It has taken a while to stabilize the Amplify environments and make sure everything is done in a certain order so env don't get blown up.

Just a bit of info, I was running into this as well after I upgraded from amplify-cli 4.13.1 to 4.13.2. Once I downgraded to 4.13.1 and did another amplify push life was better.

Thank you @malcomm

Having the same issue with the latest CLI version (4.13.3) on any @model list query. Forcing version 4.13.1 of the CLI in build settings -> Live package updates does not seem to solve it for me, at least for existing @models.
It happens with any @model, and is reproducible directly from the AppSync Console.
Generate a simple @model
type Todo @model @auth(rules: [{ allow: owner }]) {
  id: ID!
  owner: ID!
  text: String
  checked: Boolean
}

Create a few objects, and query them, copying the value of nextToken verbatim from a previous query directly in AppSync Console
query ListTodos {
  listTodos(limit: 2, nextToken: "ey...SRiJ9") {
    items {
      id
      text
      checked
    }
    nextToken
  }
}

The error shows up consistently
{
  "data": {
    "listTodos": null
  },
  "errors": [
    {
      "path": [
        "listTodos"
      ],
      "data": null,
      "errorType": "DynamoDB:UserIllegalArgumentException",
      "errorInfo": null,
      "locations": [
        {
          "line": 2,
          "column": 3,
          "sourceName": null
        }
      ],
      "message": "com.amazonaws.deepdish.common.pagination.InvalidPaginationTokenException: Invalid pagination token given."
    }
  ]
}

@malcomm For me, this issue was occurring on CLI version 4.13.1, however once I downgraded to 4.13.0 it cleared up. I did try and upgrade to 4.13.2 to see if that might be a fix, but I could still reproduce the error.
@olivoil I'd recommend taking the CLI down to 4.13.0 and then doing a amplify push and see if that works.
Hopefully the Amplify team can get to the bottom of this, but I have a work around for now.

@a-j-olly I am on Amplify 4.5.0 and I haven't upgraded for a while, so there were no environment changes. Though saying that the problem I am having is with an environment that is built by "AWS Amplify" from GIT. I assume they would be using the latest version of Amplify when building the backend. I will add an echo to the amplify.yml to see what version they are using.
BUT saying that my build environment is also built by "AWS Amplify" so I would expect the same issue if it was a version issue. But BUILD works (and the DEV environments I built by hand work) but TEST does not.

+1 same problem with @key and custom queryField.

Hi @apanagos
I'm trying to reproduce this though I'm not able to
this is the schema I am trying on version 4.5.0
type Todo @model @auth(rules: [{ allow: owner }]) {
  id: ID!
  owner: ID!
  text: String
  checked: Boolean
}

I'm running this in us-west-2.
If possible could you provide the schema you are using, here or email<EMAIL_ADDRESS>and provide the region this api exists.

Hi @SwaySway, problem is that I am experiencing the problem with all the tables in my TEST environment.
Here is a simple one that is causing issues:
type Tags @model @aws_cognito_user_pools @aws_iam{
  id:ID!
  tag:String!
}

I am running this in us-east-1
Let me go through the process of how I promote in case the problem is there.
git checkout test
git merge build
amplify env checkout testedge
amplify env pull
amplify push

once its is finished i do the following:
git add *
git commit -m "upgrade"
git push

Then the build automatically starts in "AWS Amplify" as I have set it up to build from updating the branch.
Some other things to take into account.
My command shell has amplify at version 4.5.0
BUT when it builds in "AWS Amplify" I guess it uses the latest version of Amplify CLI?
Just a little more info.  The follwoing is the build file I have checked into GIT and is used to build in "AWS Amplify"
file: amplify.yml
version: 0.5
backend:
  phases:
    build:
      commands:
        - '# Execute Amplify CLI with the helper script'
        - echo "User $USER_BRANCH"
        - echo "AWS User $AWS_BRANCH"
        - amplifyPush --environment $USER_BRANCH --simple
frontend:
  phases:
    preBuild:
      commands:
        - echo "Front End Prebuild"
        - npm install --global gulp-cli
        - npm ci
    build:
      commands:
        - gulp
        - npm run build
  artifacts:
    baseDirectory: build
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*

If you need anything else, please just reach out and I will help as much as i can.
Thanks

I have tried deploying a test api through both  Amplify Version(4.13.3 and 4.13.2) with the following schema :
type TODO @model {
id: ID!
amount : Int!
}
The resolver generated using amplify CLI
Query.listTODOs.req.vtl
#set( $limit = $util.defaultIfNull($context.args.limit, 10) )
#set( $ListRequest = {
"version": "2017-02-28",
"limit": $limit
} )
#if( $context.args.nextToken )
#set( $ListRequest.nextToken = $util.toJson($context.args.nextToken) )
#end
#if( $context.args.filter )
#set( $ListRequest.filter = $util.parseJson("$util.transform.toDynamoDBFilterExpression($ctx.args.filter)") )
#end
#if( !$util.isNull($modelQueryExpression)
&& !$util.isNullOrEmpty($modelQueryExpression.expression) )
$util.qr($ListRequest.put("operation", "Query"))
$util.qr($ListRequest.put("query", $modelQueryExpression))
#if( !$util.isNull($ctx.args.sortDirection) && $ctx.args.sortDirection == "DESC" )
#set( $ListRequest.scanIndexForward = false )
#else
#set( $ListRequest.scanIndexForward = true )
#end
#else
$util.qr($ListRequest.put("operation", "Scan"))
#end
$util.toJson($ListRequest)
We keep getting the same error even when running queries  with Appsync console directly . This issue needs to be addressed urgently . Proper testing must be ensured before releasing new
versions.

As per @a-j-olly's comment above, can confirm it works when I downgraded the cli from 4.13.2 to 4.13.0 and do amplify push.

Just an update.
Having downgraded to 4.13.0 and finding success, I was annoyed to find out this morning that once again the problem has cropped up. I have not made any alterations whatsoever since I did the downgrade and amplify push.
I'm getting the same error:
{
  "data": {
    "articlesByDate": null
  },
  "errors": [
    {
      "path": [
        "articlesByDate"
      ],
      "data": null,
      "errorType": "DynamoDB:UserIllegalArgumentException",
      "errorInfo": null,
      "locations": [
        {
          "line": 2,
          "column": 3,
          "sourceName": null
        }
      ],
      "message": "com.amazonaws.deepdish.common.pagination.InvalidPaginationTokenException: Invalid pagination token given."
    }
  ]
}

Pagination is a critical feature of GraphQL. Fixing this issue is of vital importance.
If you need anymore info see my above posts or PM me.

I ran into the same problem when I went to 4.13.3, downgraded to 4.13.1 which still works for me.

So I just did another 'empty' push, and it has cleared up again. I think I'm just going to take a break...

We've root-caused the issue with the latest release and working on a fix for this which will be released soon.
cc: @attilah

Have the same problem
com.amazonaws.deepdish.common.pagination.InvalidPaginationTokenException: Invalid pagination token given.

I attempted to build the application again from scratch, and still getting the pagination error for Amplify version 4.13.3
The resolver generated 'Query.listSecurSamples.req.vtl' file:
#set( $limit = $util.defaultIfNull($context.args.limit, 10) )
#set( $ListRequest = {
"version": "2017-02-28",
"limit": $limit
} )
#if( $context.args.nextToken )
#set( $ListRequest.nextToken = "$context.args.nextToken" )
#end
#if( $context.args.filter )
#set( $ListRequest.filter = $util.parseJson("$util.transform.toDynamoDBFilterExpression($ctx.args.filter)") )
#end
#if( !$util.isNull($modelQueryExpression)
&& !$util.isNullOrEmpty($modelQueryExpression.expression) )
$util.qr($ListRequest.put("operation", "Query"))
$util.qr($ListRequest.put("query", $modelQueryExpression))
#if( !$util.isNull($ctx.args.sortDirection) && $ctx.args.sortDirection == "DESC" )
#set( $ListRequest.scanIndexForward = false )
#else
#set( $ListRequest.scanIndexForward = true )
#end
#else
$util.qr($ListRequest.put("operation", "Scan"))
#end
$util.toJson($ListRequest)
Worked correctly after downgrading to Amplify version 4.13.0

Could you please try this out with 4.13.4? We fixed this in our latest version

@kaustavghosh06 Please note we  (i.e., myself and @AshwiniRaysecur )have tested publishing our app with version 4.13.4 and the issue seems to be resolved. Smoke tests show our app once again behaves as expected.

I just upgraded to version 4.13.4 and things appear to be working for me as well. I'll keep folks posted if otherwise. Thanks!

We tested it today and I can confirm that the fix does work. Thank you @kaustavghosh06 , @SwaySway , @AaronHarris , @sammartinez , @yuth , @attilah for addressing the issue so quickly.
Now I have made some changes to our process. I changed "AWS Amplify" to not use the latest version of Amplify but to use a versioned (4.13.4) so it is in-sync with what we are developing with. As we upgrade and test our version of Amplify then we will change the "AWS Amplify" version so we are not surprised.
Thanks Again.

Upgraded to 4.13.4 . The issue is resolved .Thanks
On Fri, 21 Feb 2020, 03:03 Andrew Panagos<EMAIL_ADDRESS>wrote:

We tested it today and I can confirm that the fix does work. Thank you
@kaustavghosh06 https://github.com/kaustavghosh06 , @SwaySway
https://github.com/SwaySway , @AaronHarris
https://github.com/AaronHarris , @sammartinez
https://github.com/sammartinez , @yuth https://github.com/yuth ,
@attilah https://github.com/attilah for addressing the issue so quickly.
Now I have made some changes to our process. I changed "AWS Amplify" to
not use the latest version of Amplify but to use a versioned (4.13.4) so it
is in-sync with what we are developing with. As we upgrade and test our
version of Amplify then we will change the "AWS Amplify" version so we are
not surprised.
Thanks Again.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/aws-amplify/amplify-cli/issues/3411?email_source=notifications&email_token=AE5UPKLAYER57S6PXF6AYT3RD3ZMJA5CNFSM4KT7CL52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMQHIIY#issuecomment-589329443,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AE5UPKMMSR7TTIIIRJG3L4TRD3ZMJANCNFSM4KT7CL5Q
.

This is still happening for me on  4.13.4

@oliverandersencox Are you using @key with your query?

@SwaySway yes I am

@oliverandersencox Could you try the workaround mentioned in #3493?
If it is related to that please follow that issue for updates.

This is still happening for me on 4.13.4

same here, downgraded to 4.13.1 and it works

I just wanted to confirm that I had this issue with 4.13.3 but upgrading to 4.13.4 resolve it for me.  The steps I took were:

npm install -g @aws-amplify/cli@latest
make a simple change to the schema (I added a field)
amplify api gql-compile
amplify push
That was it.. works fine now.. thanks guys.

@WerewolFox
Same problem. same fix.
Except that i didnt need amplify api gql-compile

I just wanted to confirm that I had this issue with 4.13.3 but upgrading to 4.13.4 resolve it for me. The steps I took were:

npm install -g @aws-amplify/cli@latest
make a simple change to the schema (I added a field)
amplify api gql-compile
amplify push
That was it.. works fine now.. thanks guys.

I had the same issue. However I was on Amplify --version 4.13.4. Bumped it to the latest --version 4.17.1.
• amplify api gql-compile
• amplify push
That fixed it for me!

Hello @SwaySway
I'm having this issue even when I try the query from the Appsync console.
My type is NOT using the @key directive.
I'm using the latest Amplify CLI 4.17.2
Could you please address my issue?
Thank you!

Hello,
This issue is back with the latest (4.18.1) Amplify CLI Version. Overriding the version with 4.16.1 (which is the one I happen to know is working) as described at https://docs.aws.amazon.com/amplify/latest/userguide/custom-build-image.html#setup-live-updates no longer throws this error.

Hello,
I'm having this issue with amplify 4.25.0.

I'm having this issue too with 4.18.1

I have the same issue. I'm using version 4.13.4. Base on some comments above I upgraded to 4.27.3, it works fine for me.

I'm still having this issue, v: 4.29.7.
It has been months, impossible to to use Amplify in production 👎

Hello @iShavgula-TacTill
Could you provide a schema which can reproduce this scenario?

Thank you for having a look at this.
This happens for every DynamoDB Query (2018-05-29) which goes via Pipeline Resolvers and uses GSI.
I tried to delete and recreate the these queries many times, but that does not help, it can be reproduced every time with every query following these patterns. Here is a (pseudo) code :
#set( $QueryRequest = {
  "version": "2018-05-29",
  "operation": "Query",
  "limit": $limit,
  "query": some expression,
  "index": "index name"
} )
#if( !$util.isNull($ctx.args.sortDirection) && $ctx.args.sortDirection == "DESC" )
  #set( $QueryRequest.scanIndexForward = false )
#else
  #set( $QueryRequest.scanIndexForward = true )
#end
#if( $context.args.nextToken )
  #set( $QueryRequest.nextToken = $util.toJson($context.args.nextToken) )
#end
#if( $context.args.filter )
  #set( $QueryRequest.filter = $util.parseJson("$util.transform.toDynamoDBFilterExpression($ctx.args.filter)") )
#end
$util.toJson($QueryRequest)

Please let me know if you need more information.
P.S. I did check that nextToken is correctly used in my pipelines, it was working fine until it just stopped working after one of the Amplify cli updates.

@iShavgula-TacTill
Currently the CLI does not support pipeline resolvers. Do you have pipeline resolvers setup by overriding the generated resolvers? If so the fix you could apply here is to change the query resolver to remove $util.toJson(). This is what the transformer currently generates for query resolvers that use nextToken.
#if( $context.args.nextToken )
  #set( $QueryRequest.nextToken = $context.args.nextToken )
#end

That fix did work, but I find that very strange because of couple of reasons :

I had some auto-generated queries which used toJSON() for nextToken before commenting on this issue.
I have not changed DynamoDB Query version

Could that be linked to AppSync utilities' updates ?
Any tips how to avoid this kind of issues ?
Thank you

In an earlier version of the Amplify CLI a bug was introduced which double encoded the payload toJSON. What I suspect is you might have overridden the template when the CLI had the bug, which got you the buggy resolver. This was fixed in the CLI later, but since the template was overriden, that fix not propagate to that template.


[OLS-4 Cohort call week 2]

Weekly emails
[ ] Send email to mentees
[ ] Send email to speakers
Cohort call video preparation
[ ] Download from Zoom cloud recordings
[ ] Upload to Otter.ai (will need a few minutes to process)
[ ] Upload to the OLS YouTube account
[ ] Create a front page slide for the video - templates are available in Google Drive
[ ] Make sure the YouTube video is added to the current cohort playlist
[ ] Correct Otter.ai transcripts (make sure to use uploaded video, not the original transcript, as the timings won't match the video)
[ ] Download corrected transcript as a .srt file from Otter
[ ] Upload it to the YouTube video
[ ] Post a link to the recording in the OLS Slack! 🙌
Website (in schedule in _data folder)
[ ] Add link to recording
Add talks

[ ] Add link to slide deck
[ ] Add person detail to people.yaml if needed

Just wanna say the "Cohort call video preparation" is done except for posting the link in the OLS slack, which I think it's better to be left from the organiser team!
I created a file in Otter.ai for OLS4 transcripts but feel free to remove it. I also created a new playlist on YouTube for OLS4. If I did anything incorrectly, please let me know - so I don't repeat it!


[BUG]: restrict_to_aoi not implemented

restrict_by_aoi is defined as a boolean argument in the raster tiler but doesn't appear to be implemented. I was thinking of implementing this but instead of making it a boolean, the portion of the raster that falls within the aoi would remain and the portion of the tile that falls outside the aoi would be filled with the mean of the training dataset. I think this type of filling would work since labels are not present outside of the aoi_boundary (by definition of how aoi_boundary should be used). Does this sound good @nrweir ?

Hey @rbavery,
That sounds great! My long-term vision for this is to have a few different options - including reflecting values across the boundary (similar to what's often implemented for image scaling algorithms for deep learning) and a couple of others. Maybe we can start with two options:

Use the mean;
Use a custom user-provided value (e.g. so they can use 0 or something if they really want to)

...while leaving flexibility to add additional options in the future. What do you think?

Sounds good to me! The easiest thing to do would be to use the mean of the image used for tiling. But we could also use the mean of all images that will be tiled by supplying a directory. The former is simpler but the latter means a consistent fill value will be used across all tiles in cases where multiple images are tiled, which would make visualization of the tiles easier. I'm leaning toward the latter option as the default, what do you think @nrweir ?

Ideally, we enable both: allow users to specify a fill value (which they could derive from the full-dataset mean) or solaris could derive the mean within the image itself. Would essentially need to provide an argument specifying either a value to fill with, or allow users to specifiy 'mean', in which case solaris will get the image's mean value and use that. e.g. a new argument nodata_fill=None where the user can provide, for example, nodata_fill=50 (or whatever other specific value they want) or nodata_fill='mean' in which case solaris will get it from the image for them.

I just wanted to say filling in with custom values (0 in particular) would be great to have as one of the options at least. Computing the mean can be time-consuming over a large area, and being able to mask the label raster with 0 where the imagery tile has no data would be useful (i.e. to not have these contribute to the loss during training).

This is solved by https://github.com/CosmiQ/solaris/pull/331 @yangsiyu007 . there's now an option to fill with either the mean of the source image or a chosen fill_value. See the fill_all_nodata method of the RasterTiler class: https://github.com/CosmiQ/solaris/blob/dev/solaris/tile/raster_tile.py#L455
And https://github.com/CosmiQ/solaris/blob/dev/solaris/vector/mask.py#L978

Good job @rbavery ! Looking forward to trying it out


Android Screenshots for Version: 2.0.0

Android Screenshots for Version: 2.0.0

Open-Source Project Corona-Warn-App – Screenshots Android.pdf

@dsarkar
Wouldn't it make sense to add a new section "Event-Registration" to the screenshots page?
Should I open another issue for this?

@Ein-Tim we have a ticket already! Thanks!

Some screenshots have a dark overlay (second screenshot in "Contact Journal " section, first four screenshots of "Statistics" section, last screenshot in "Test registration & Key sharing" & third screenshot in "Test result home" section)

@Ein-Tim Will check, thanks.

we have a ticket already! Thanks!
EXPOSUREAPP-6420

@dsarkar Is this ticket publicly available? I couldn't find it on Github.

@Art4
This is an internal tracking ID, it's not public.


Distinguish direct calls from indirect calls

//- @f defines/binding FnF
struct A { virtual void f() {} };
//- @"A::f()" ref/call/direct FnF
struct B : public A { void f() override { A::f(); }};

This helps various tools that use the callgraph by excluding overrides where appropriate.

The edge name isn't finalized.


Resin enclosure

This is out of scope for the class, and may be completed at a future date as a side project.


auto pwn issue

Enabling autopwn fucks up any possibility of using the whid other than the one code you selected if im even able to connect to the wifi I can't stop the code from running I can't reset the device I can't do anything but plug it in.

plug in, connect to the wifi from the plugged devices, and flash it back with the settings

Doesn't work
On Fri., Mar. 4, 2022, 11:11 a.m. Hugo, @.***> wrote:

plug in, connect to the wifi from the plugged devices, and flash it back
with the settings
—
Reply to this email directly, view it on GitHub
https://github.com/whid-injector/WHID/issues/86#issuecomment-1059400944,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AWWVPXGW6E4UW42EPLYWCFLU6JG3HANCNFSM5J2XSAQQ
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you authored the thread.Message ID:
@.***>

wdym doesnt work ?
provide somes steps, else you can flash it back to default via the wiki


Refactor RPC call get_all_stake_info_for_coldkey to use Bytes

closing in favour of https://github.com/opentensor/subtensor/commit/4dbe0e35d4f21b56dca872eb7cae094f37131503 . In addition to this , the test runtime crate has incompatiblities with v1.0.0


Logo

Summary
What: Add logo and reference it in the readme

Hi ebebbington, can you describe me what you need for your logo, i can make one for you. Kind regards

Hey @xroll, we'd be happy for you to try :) @crookse is the genius behind the logos and he uses Sketch for them, i was hoping  a logo along the lines of:

(ROUGH design lol, the idea is it's the visual representation of dawn)
I think it's possible, and use we use polygons to design the end result, see https://drash.land/#modules as a reference on how we design them :) So for example, the polygons in my rough design would follow the same colour scheme, and act as the horizon, and the partial circle and lines act as the sun and it's 'rays'
If you also want to think of your own designs then feel free to propose them :) but i guess they must fit the current colour/design scheme

I have a logo ready . I am new on git hub . How do I attach it ?

Can you assing it to me

Hey @LySergicaciD125, you can save it as dawn/logo.png on your fork and make a pull request, though it would be best to mention this before @xroll so multiple people aren't trying to tackle the same issue
@xroll feel free to still make one 👍

@lakshya125 / @xroll,
Just an FYI, you can upload the logos to these comments. Just drag and drop the logo into the comment box.

Due to the age of this issue and the comments, we have made a logo and thus i am locking conversation on this issue and a PR will close this


NPM fails to start using Pihole dns in docker

Describe the bug
When starting docker app (2.7.2) using PiHole dns running in a separate container on the same host, the following is logged during startup:
app_1  | [2/18/2021] [2:03:37 PM] [Migrate  ] › ℹ  info      Current database version:<PHONE_NUMBER>3841
app_1  | [2/18/2021] [2:04:20 PM] [Global   ] › ✖  error     Command failed: pip3 install certbot-dns-cloudflare==1.8.0 cloudflare
app_1  | WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f9c8898ae20>: Failed to establish a new connection: [Errno -3] Try again')': /simple/certbot-dns-cloudflare/
app_1  | WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f9c8898ad60>: Failed to establish a new connection: [Errno -3] Try again')': /simple/certbot-dns-cloudflare/
app_1  | WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f9c8896a490>: Failed to establish a new connection: [Errno -3] Try again')': /simple/certbot-dns-cloudflare/
app_1  | WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f9c8896a340>: Failed to establish a new connection: [Errno -3] Try again')': /simple/certbot-dns-cloudflare/
app_1  | WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f9c8896a4c0>: Failed to establish a new connection: [Errno -3] Try again')': /simple/certbot-dns-cloudflare/
app_1  | ERROR: Could not find a version that satisfies the requirement certbot-dns-cloudflare==1.8.0 (from versions: none)
app_1  | ERROR: No matching distribution found for certbot-dns-cloudflare==1.8.0

If I add the 'dns' option to my docker-compose file and point it to <IP_ADDRESS> NPM loads without issues.
I have tried setting the same option to the various docker and physical ips, but they yield the same error.
I have several other containers, servers and clients that successfully use the Pihole container as their dns.
Any advise on this strange issue?

Both PiHole and Nginx use the port 80 and 443, how did you work around that ? I don't think you can have the same ports in use for 2 containers on 1 host.

@Zerwin, I use "expose" rather than "ports" in my pihole container config.  Then let NPM proxy to pihole using the docker network. The only thing Im unsure of is that this is not blocking SSL based adverts which I'm looking at right now.
Here's my docker-compose.yml for pihole (but NPM uses the same external "proxy" network).
`version: "3.2"
More info at https://github.com/pi-hole/docker-pi-hole/ and https://docs.pi-hole.net/
services:
pihole:
container_name: pihole
image: pihole/pihole:latest
ports:
- 53:53/tcp
- 53:53/udp
expose:
- 67/udp
- 80/tcp
- 443/tcp
environment:
TZ: 'Europe/London'
WEBPASSWORD: '***********'
# Volumes store your data between container upgrades
volumes:
- etc-pihole:/etc/pihole/
- etc-dnsmasq.d:/etc/dnsmasq.d/
# Recommended but not required (DHCP needs NET_ADMIN)
#   https://github.com/pi-hole/docker-pi-hole#note-on-capabilities
#cap_add:
#  - NET_ADMIN
restart: unless-stopped
volumes:
etc-pihole:
etc-dnsmasq.d:
networks:
default:
external:
name: proxy`

From the PiHole docker hub site:
Port 443 is to provide a sinkhole for ads that use SSL. If only port 80 is used, then blocked HTTPS queries will fail to connect to port 443 and may cause long loading times. Rejecting 443 on your firewall can also serve this same purpose. Ubuntu firewall example: sudo ufw reject https

As you are only exposing port 443 it's never available to anything outside NPM and PiHole. You need to properly publish port 443 for this to work.
Overall I would suggest moving away from exposing ports and putting PiHole on a different VM/Server. You can still use docker there if you like. Exposing ports like you are now is probably also what is causing the original problem.


app metadata logo key error

I got the following error but can not anchor which app cause it. It would be better if the error message is more detailed to tell the user where the app metadata structure incompatibility happened.
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/opt/aiidalab-home/home/start_page.py in render(self)
    106                 # Create app widget if it has not been created yet.
    107                 if name not in self._app_widgets:
--> 108                     self._app_widgets[name] = self._create_app_widget(name)
    109
    110                 display(self._app_widgets[name])

/opt/aiidalab-home/home/start_page.py in _create_app_widget(self, name)
     66         """Create the widget representing the app on the home screen."""
     67         config = self.read_config()
---> 68         app = AiidaLabApp(name, None, AIIDALAB_APPS)
     69
     70         if name == "home":

/opt/conda/lib/python3.7/site-packages/aiidalab/app.py in __init__(self, name, app_data, aiidalab_apps_path, watch)
    563
    564         self.name = self._app.name
--> 565         self.logo = self._app.metadata["logo"]
    566         self.categories = self._app.metadata["categories"]
    567         self.is_installed = self._app.is_installed

KeyError: 'logo'

This issue occurs if the version of aiidalab is <= 21.11.2. I make a PR to this package to fix the lower dependency.


[Site Request] Asciinema

Checklist

[x] I'm reporting a new site support request
[x] I've verified that I'm running youtube-dlcc version 2020.10.31 (2020.11.11-3)
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that none of provided URLs violate any copyrights
[x] I've searched the bugtracker for similar site support requests including closed ones

Example URLs

Single video: https://asciinema.org/a/332708

Description
I opened this feature request on the legacy bug tracker, but the owner closed it as invalid because, as he said, the website does not host videos, that is just ascii art. I disagree with that, so let me convice you that there are videos and that it makes sense to add asciinema video downloader to yt-dlc.
The website provides a tool for recording console output and uses a special format to represent the console output in time. You can upload recorded "videos" (which are basically text files) to their website and they can play the files as videos by animating the text in the browser itself.
So, if you think about it, what the definition of a video actually is? The video is a sequence of images that are supposed to be played at some framerate. And each image is a 2D grid of pixels (colors). But a console that supports colors can actually be viewed as a video player. You can change the width/height of the console windows and you can customize the color palette, so that you can play any video by literally rendering ascii output of a program in the console window itself.
My suggestion is that we add support for asciinema to the yt-dlc by retrieving the "video" file (that is just a plaintext with special color codes and escape sequences) and then use python (on nodejs or some other program) to actually render each frame as a pixel matrix and then pass to ffmpeg that can read raw pixel data and save as mp4 file or some other format.
What do you think about this idea?

Asciinema does not provide any video content , and asciinema doesn't turn the recorded output into some magical format it's just json that asciinema player reads and print like your regular terminal emulator.
if you like to, here's a small script to download asciinema recordings for you then use some tool to convert it like asciicast2gif then use ffmpeg to convert gif to mp4, unfortunately yt-dlc does only care about media content and asciinema is not included.
import sys
import requests as req

if __name__ == "__main__":
    if len(sys.argv) != 3:
        print("./asciinema_download <URL> <OUTPUT_NAME>")
        exit(0)
    r = req.get(sys.argv[1]+'.cast?dl=1')
    open(sys.argv[2], 'w').write(r.text)
    print(f"Download Complete {sys.argv[2]}")


Avoid method that assume default platform encoding is suitable.

Dm: Reliance on default encoding (DM_DEFAULT_ENCODING)
Found a call to a method which will perform a byte to String (or String to byte) conversion, and will assume that the default platform encoding is suitable. This will cause the application behavior to vary between platforms. Use an alternative API and specify a charset name or Charset object explicitly.

Hi @juanpablo-santos,
I've made changes to avoid reliance on default platform encoding,  I think this would be a nice addition to the 2.12.2 release. Please take a look.
Best,
Arturo


Hyperparameters and accuracy for AtrialFibrillation dataset.

Hello,
I am attempting to perform pre-training and fine-tuning on the AtrialFibrillation dataset, but I am unable to locate the hyperparameters and corresponding performance metrics in the relevant paper. Can you please provide me with this information, if available?

Is it possible to get some help with this? @gzerveas

Hi, this dataset was not included in the evaluation of the paper. If it is in the time series regression/classification archive, you can download it and put it in a directory as the README indicates, and the class TSRegressionArchive(BaseData): in the data.py should be able to handle it.
However, I can't really tell what would be good hyperparameters without having worked with this dataset. I suggest looking at the paper's table with default hyperparamters as a starting point, and experiment a bit (have a look at other datasets' hyperparameters to see which ones are the ones that most often affect performance). You may also consider using a sub-sample factor if the time series are very long or sampled with a high temporal resolution.


jgenesis-config.toml not taken into account when running from cli

Hello,
Is there a way to use the settings defined within the "jgenesis-config.toml" when running the -cli executable ?

This is not currently possible. As you've noticed, input configuration in the CLI is currently extremely limited (to be honest, I'm a little surprised to see someone other than me trying to use it).
It would be pretty easy to make the CLI able to read the GUI's input configuration from jgenesis-config.toml, although that might be somewhat confusing if it's not reading all configuration from there (which would be a little more complex). It would also be easy to add a command line arg to the GUI to automatically open a specific game file at startup, as you suggested.

Hi,
Thanks for the answer, from my point of view both would work, as I can manage either to update settings in the toml file through my launcher build in c#.
Both would be a nice addition and would enable integration with our front-end.

I implemented the GUI arg first because that was simpler. Dev builds are available here:
https://github.com/jsgroth/jgenesis/actions/runs/8901751196
There's a new optional -f/--file-path arg that, if set, will immediately launch the specified file after the GUI is initialized and then close the GUI when the emulator window is closed. If there is an initialization error (e.g. file doesn't exist) then the GUI will log an error and immediately exit.
I would still like to fix this for the CLI as well, because there's not really any reason to have the GUI window running in the background if it's not going to be used.

I'll test tomorrow and report back if I notice any issue.

Works fine thanks a lot

Hello,
I still have an issue with this : when running the gui executable with a game starting automatically, the game windows does never get the focus, making it mandatory to click with the mouse to be able to use the controller.
Windows 11, tested on windows 10 also.
It seems that the emulator gui window appears and steels the focus.
@jsgroth

Ah, I only tested on Linux before - I see the same behavior with this flag on Windows 10. I'll see if I can fix it by waiting to launch the emulator until after the first GUI paint.

I don't have a Windows 11 machine to test on, but waiting until after the first full GUI paint fixed this on Windows 10 for me. Builds here:
https://github.com/jsgroth/jgenesis/actions/runs/8931723133

Thanks, I'll test later today when coming home from work.
Win11 has some strange behavior with focus since they implemented multi-tabs in explorer, however there is no reason it would act differently than win10.

Hello: working fine on windows 11 also

I just pushed a change that has the CLI read all configuration from the GUI's config file, including input configuration:
https://github.com/jsgroth/jgenesis/actions/runs/9022312465
By default the CLI uses the same config file path as the GUI, and there's also an optional --config arg to override the config path.
Most of the previously-existing args are still present but they now function as temporary config overrides rather than being the only way to configure the CLI.

Awesome.
So the arguments have the priority over the config from what I read, right ?
That's great, will test this when I am back from holiday.

That is correct, the CLI args have priority over the config file if they are set.


Restore animation behaviour for Solar Beam in Sun

In #4974 I tried to fix Solar Beam in Sun but unfortunately I hadn't realised that this meant that the move animated before it got prepared. This reverts the changes dealing with animating fully charged two-turn moves including moving the animation back to the onTry handler, so it only happens if the fully charged move has a target. Razor Wind is a bit hacky though.

I fixed the Power Herb animation in https://github.com/Zarel/Pokemon-Showdown/commit/37a5cf25ccd4feff222c2bfa9044f269d15440f0

If I'm reading this correctly this is fixing the no-target case?

What do no-target moves usually do? Don't they usually tag [still]? How do they usually animate?

This is fixing a) the Power Herb with no target case and b) the Solar Beam in Sun case (with or without target).
No-target moves tag [notarget] rather than [still]; I don't know what the difference is supposed to be as they seem to have very similar effects at least as far back as 2013. I guess it's possible that [notarget] moves used to animate as a miss or something...

It seems more sense to move the no-target check before the Power Herb check.

And then also change SolarBeam to work like Power Herb.

No, that's what caused the original bug. The move needs to check for Power Herb before checking for no target. If there's no Power Herb then it doesn't matter about the target, because the move is charging.

Okay, I think I understand now.
I've pushed c752b327ca0ee and aba474869d20, which should fix it, but with significantly less code. Can you review?

aba4748 also needs to be ported to Solar Blade (which didn't exist in Gen 6 Triples so I hadn't specifically mentioned it before).

Ah, so what's happening here is that the [notarget] ends up on the -anim minor, thus disabling the animation again. Nice!

Feel free to port it yourself.


How to switch BuildType Debug/Release quickly?

Currently, I  can switch CMake build type by setting g:cmake_default_config='Debug'/'Release'. Is there any better way to do this?
another question is how to use :CMakeSwitch <config>, I don't understand what does <config> mean.
Thanks!

Hi, I think :CMakeSwitch is indeed what you're looking for. For instance, to switch to the build configuration Release, just run :CMakeSwitch Release. You can also trigger autocompletion for the available configurations by pressing <Tab> after typing :CMakeSwitch in the command line.
By the way, all these commands are documented, try :help cmake-commands.
Let me know if this does not solve your problem.

Hi, I think :CMakeSwitch is indeed what you're looking for. For instance, to switch to the build configuration Release, just run :CMakeSwitch Release. You can also trigger autocompletion for the available configurations by pressing <Tab> after typing :CMakeSwitch in the command line.
By the way, all these commands are documented, try :help cmake-commands.
Let me know if this does not solve your problem.

Thank you for your reply. I read the manual and know to use these commands now.
But I get a new issue when use CMakeGenerate:

the pwd is: /home/rave/tmp/test_project,  but the project root path is pwd followed by a while space, which does not exits.

Hmm that's strange. Could you please open a separate issue for that? Please also

pull the latest master branch
enable logging by setting g:cmake_log_file to something like /tmp/vim-cmake.log before running any command
fill in the complete issue template for bugs

Thanks a lot!

Sorry for this late reply, I opened a new issue for this.


Adding config file to Ethercat.

@magni-mar why was this closed?
To populate interfaces to json schema during runtime you could define to_json_schema for the given type.
Example of implementation:
namespace tfc::ethernet {
struct interface{
std::string value{};
static std::vector<interface> const& get() {
// This function should be implemented in source file and would return a reference to a global symbol.
// That will allow making string_view in to_json_schema call to this refernce.
// grab interfaces from system and populate vector to global variable within the source file for each call
return {};
}
std::string operator() const {
return value;
}
};
} // namespace tfc::ethernet
namespace tfc::json::detail {

template <>
struct to_json_schema<tfc::ethernet::interface> {
  template <auto opts>
  static void op(auto& schema, auto& defs) {
    // std::span schema.enumeration to view of tfc::ethernet::interface::get()
  }
};
}


Add Copyright symbol to CopyrightInfo component's storybook.

Describe the bug
It seems that the copyright symbol is not displaying as it should.
To Reproduce
Steps to reproduce the behavior:

Go to (https://restaurant-app-components.vercel.app/?path=/docs/atoms-copyrightinfo--docs)

Hey @bruno-keiko can you assign this to me.
Hello, I have assigned this task to you.


[SPARK-3541][MLLIB] New ALS implementation with improved storage

This PR adds a new ALS implementation to spark.ml using the pipeline API, which should be able to scale to billions of ratings. Compared with the ALS under spark.mllib, the new implementation

uses the same algorithm,
uses float type for ratings,
uses primitive arrays to avoid GC,
sorts and compresses ratings on each block so that we can solve least squares subproblems one by one using only one normal equation instance.

The following figure shows performance comparison on copies of the Amazon Reviews dataset using a 16-node (m3.2xlarge) EC2 cluster (the same setup as in http://databricks.com/blog/2014/07/23/scalable-collaborative-filtering-with-spark-mllib.html):

I keep the spark.mllib's ALS untouched for easy comparison. If the new implementation works well, I'm going to match the features of the ALS under spark.mllib and then make it a wrapper of the new implementation, in a separate PR.
TODO:

[X] Add unit tests for implicit preferences.

Test build #25195 has finished for   PR 3720 at commit 2a8deb3.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long)
  case class Movie(movieId: Int, title: String, genres: Seq[String])
  case class Params(
class ALS extends Estimator[ALSModel] with ALSParams
 case class RatingBlock(srcIds: Array[Int], dstIds: Array[Int], ratings: Array[Float])

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25195/
Test PASSed.

Test build #25207 has started for   PR 3720 at commit a76da7b.

This patch merges cleanly.

@srowen @coderxiang This PR is almost ready, pending few unit tests. Would you be interested in reviewing the code?

Test build #25207 has finished for   PR 3720 at commit a76da7b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long)
  case class Movie(movieId: Int, title: String, genres: Seq[String])
  case class Params(
class ALS extends Estimator[ALSModel] with ALSParams
 case class RatingBlock(srcIds: Array[Int], dstIds: Array[Int], ratings: Array[Float])

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25207/
Test FAILed.

Test build #25337 has started for   PR 3720 at commit b84f41c.

This patch merges cleanly.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25335/
Test FAILed.

Test build #25337 has finished for   PR 3720 at commit b84f41c.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long)
  case class Movie(movieId: Int, title: String, genres: Seq[String])
  case class Params(
class ALS extends Estimator[ALSModel] with ALSParams
 case class RatingBlock(srcIds: Array[Int], dstIds: Array[Int], ratings: Array[Float])

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25337/
Test PASSed.

Test build #25353 has started for   PR 3720 at commit dd0d0e8.

This patch merges cleanly.

Test build #25353 has finished for   PR 3720 at commit dd0d0e8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long)
  case class Movie(movieId: Int, title: String, genres: Seq[String])
  case class Params(
class ALS extends Estimator[ALSModel] with ALSParams
 case class RatingBlock(srcIds: Array[Int], dstIds: Array[Int], ratings: Array[Float])

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25353/
Test FAILed.

test this please

Test build #25485 has started for   PR 3720 at commit dd0d0e8.

This patch merges cleanly.

Test build #25485 has finished for   PR 3720 at commit dd0d0e8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long)
  case class Movie(movieId: Int, title: String, genres: Seq[String])
  case class Params(
class ALS extends Estimator[ALSModel] with ALSParams
 case class RatingBlock(srcIds: Array[Int], dstIds: Array[Int], ratings: Array[Float])

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25485/
Test FAILed.

test this please

@srowen @coderxiang T on the implementation?

Test build #25765 has started for   PR 3720 at commit dd0d0e8.

This patch merges cleanly.

Test build #25765 has finished for   PR 3720 at commit dd0d0e8.

This patch fails to build.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long)
  case class Movie(movieId: Int, title: String, genres: Seq[String])
  case class Params(
class ALS extends Estimator[ALSModel] with ALSParams
 case class RatingBlock(srcIds: Array[Int], dstIds: Array[Int], ratings: Array[Float])

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25765/
Test FAILed.

Test build #25767 has started for   PR 3720 at commit 1b9e852.

This patch merges cleanly.

Test build #25767 has finished for   PR 3720 at commit 1b9e852.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
  case class Rating(userId: Int, movieId: Int, rating: Float, timestamp: Long)
  case class Movie(movieId: Int, title: String, genres: Seq[String])
  case class Params(
class ALS extends Estimator[ALSModel] with ALSParams
 case class RatingBlock(srcIds: Array[Int], dstIds: Array[Int], ratings: Array[Float])

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25767/
Test PASSed.

@srowen @coderxiang Do you have more comments? I'm thinking about merging this and then port nonnegative support. After that, we can replace the ALS implementation under "spark.mllib".

@mengxr Given how familiar you are with this implementation, and the tests, I can only be pretty sure it works. I didn't see any style issues, and thought through some of the loops for speed/correctness but every spot check was fine. It is looking good to me.

@mengxr the logic and style also look good to me.

Thanks! I've merged this into master.

hi, we use als example, but we found same data, same input arguments, but output different result, spark 1.2.0 can result data we want, but spark 1.3.0 cannot return right data, it all return zero matrix in userFeature.


Support clamp and clamp!

Closes #207.

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
Attention: Patch coverage is 0% with 24 lines in your changes missing coverage. Please review.

Project coverage is 57.62%. Comparing base (6e404ca) to head (4031bab).

Files with missing lines
Patch %
Lines

src/overloads/utils.jl
0.00%
12 Missing :warning:

src/overloads/arrays.jl
0.00%
7 Missing :warning:

src/overloads/three_arg.jl
0.00%
5 Missing :warning:

:exclamation:  There is a different number of reports uploaded between BASE (6e404ca) and HEAD (4031bab). Click for more details.
HEAD has 8 uploads less than BASE

Flag
BASE (6e404ca)
HEAD (4031bab)

9
1

Additional details and impacted files
@@             Coverage Diff             @@
##             main     #208       +/-   ##
===========================================
- Coverage   93.86%   57.62%   -36.24%
===========================================
  Files          29       30        +1
  Lines        1190     1147       -43
===========================================
- Hits         1117      661      -456
- Misses         73      486      +413

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Feature Pin to Dashboard

Hi all, right now its hard to know updates from specific region, im implementing feature to pin specific region to dashboard so user can easily get update about his/her interested region

Hi all, right now its hard to know updates from specific region, im implementing feature to pin specific region to dashboard so user can easily get update about his/her interested region

Niceee feature! 👍 🎉

nice feature btw! 🎉

Maybe we can add feature to unpin too @fajarnuha? what do u think?

that would be cool, but that's another feature i guess, we can make it as backlog in Issues, lets focus on this one first @pandubaraja

can any one of you improve if this PR looks good already? so I can merge this and make it available on master branch faster @pandubaraja @isfaaghyth @rizmaulana


Mapped a bunch of random stuff

Mapped a little bit of everything.

Amazing work, but we broke it I'm afraid!

I'll fix it up myself.


Should there be a Planet Rust?

A number of programming organizations host a "planet" blog that aggregates content from a variety of other blogs in the organization's community. Would such a thing be desirable for Rust?
This would provide a single easy-to-follow feed for people looking for a variety of quality content from throughout the Rust community, and could also be a decent way to discover new Rust blogs. Best of all, it would involve minimal maintenance on the part of the Rust team, beyond keeping the server up, keeping the aggregation application running, and fielding requests to be added to the list of aggregated blogs.
It looks like the Planet Feed Reader is the standard software for creating this sort of site.
Any thoughts?

We've historically talked about it a few times, but many many people replied "what is a planet" even.

(Also, I would argue that this is probably better as an internals thread than a bug on the website, visibility-wise)

Fair point. I've posted it there: https://internals.rust-lang.org/t/should-there-be-a-rust-planet/3434


how to get messageId when use sparkstreaming

dependency:pulsar-spark
thanks!

@zhouxin-116 Do you have a concrete example of your question? It is a bit hard for me to understand what you are looking for.

val pulsarReceiver = new SparkStreamingPulsarReceiver(
serviceUrl,
pulsarConf,
new AuthenticationDisabled()
)
//  receiver DStream
val pulsarDStream = ssc.receiverStream(pulsarReceiver)
//  process
pulsarDStream.foreachRDD(rdd => {
rdd.foreachPartition(partition => {
partition.foreach (row=>{
})
})
})

@sijie
val pulsarReceiver = new SparkStreamingPulsarReceiver(
serviceUrl,
pulsarConf,
new AuthenticationDisabled()
)
// receiver DStream
val pulsarDStream = ssc.receiverStream(pulsarReceiver)
// process
pulsarDStream.foreachRDD(rdd => {
rdd.foreachPartition(partition => {
partition.foreach (row=>{
})
})
})

Closed as stale.
Feel free to open a Q&A discussion at https://github.com/apache/pulsar/discussions if it's still relevant for maintained versions.


1.4 GUID Issue

Although the mod seems to work on the surface it fails to register any of the GUIDs after you close a save leaving  you with a bunch of empty pouches. Beyond that functionally it seems to work.

I know the update broke stuff. Please be patient until I fixed it.

No worries, was more of just giving a play test and stating the issue. Didn't mean to rush.


Modify path

If I'm interpreting https://github.com/amwmedia/node-plop/blob/master/src/actions/modify.js#L25-L28 correctly, the modify action does not allow to change the path. Is there a recommendation for moving/renaming files while modifying them? I'm currently relying on a custom action with Gulp for this since I'm using it in other places anyway, but I'm not sure whether there would be a better solution:
return [
  new Promise((resolve) => {
    gulp.src(/* Read files */)
      .pipe(/* Manipulate paths */)
      .pipe(gulp.dest(/* Write to new location */))
      .on('end', () => {
        del(/* Delete original files */).then(() => resolve());
      });
  }),
];

Currently a custom action would be the way to go for this. Maybe if you could explain your use case a little more, another solution might come to mind? I'm having trouble understanding why you would want to "modify and move" some files around your codebase.

It’s mostly about renaming components. For this they need to be moved as well as slightly refactored (renaming of class names etc.).

The class renaming could be taken care of by a capable IDE though.

Are you building a one time script to mass refactor a codebase? Like something you might use jscodeshift for?

It’s only about occasional renaming of single components. But thanks for pointing out jscodeshift, I’ll have a look!

What I have personally done for renaming a component, is just plop a new component with the name I want and copy the file contents from the old to the new. Reversing what the template does so you can rename component-a to component-b can be brittle because it joins those 2 generators at the hip so that a change to one requires a change to the other.
All that said, if you wanted to build a rename action, it would be pretty straight forward to make your own actionType or custom action function that did what you want.


IOError running downloadcalaccessrawdata

Cleaning data files
 Cleaning CVR2_SO_CD.TSV
 Cleaning LOBBYIST_EMP_LOBBYIST2_CD.TSV
 Cleaning FILER_ADDRESS_CD.TSV
 Cleaning LOBBYIST_FIRM3_CD.TSV
 Cleaning CVR_REGISTRATION_CD.TSV
  Bad parse of line 30381 (71 headers, 72 values)
CVR_REGISTRATION_CD.TSV had 1 errors
Traceback (most recent call last):
  File "example/manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/home/aaron/.envs/calaccess/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/home/aaron/.envs/calaccess/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/aaron/.envs/calaccess/local/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/aaron/.envs/calaccess/local/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "calaccess_raw/management/commands/downloadcalaccessrawdata.py", line 154, in handle
    self.clean()
  File "calaccess_raw/management/commands/downloadcalaccessrawdata.py", line 276, in clean
    call_command("cleancalaccessrawfile", name)
  File "/home/aaron/.envs/calaccess/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 115, in call_command
    return klass.execute(*args, **defaults)
  File "/home/aaron/.envs/calaccess/local/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/home/aaron/.envs/calaccess/local/lib/python2.7/site-packages/django/core/management/base.py", line 503, in handle
    label_output = self.handle_label(label, **options)
  File "calaccess_raw/management/commands/cleancalaccessrawfile.py", line 22, in handle_label
    self.clean(label)
  File "calaccess_raw/management/commands/cleancalaccessrawfile.py", line 122, in clean
    self.log_errors(name, log_rows)
  File "calaccess_raw/management/commands/cleancalaccessrawfile.py", line 136, in log_errors
    log_file = open(log_path, 'wb')
IOError: [Errno 2] No such file or directory: '/home/aaron/Code/california-civic-data-coalition/django-calaccess-raw-data/example/data/log/cvr_registration_cd.errors.csv'

Not sure why there's an error since it's writing a file now reading

Code should create log_dir if it doesn't exist:
if not os.path.exists(self.log_dir):
    os.makedirs(self.log_dir)

Will send in shortly

I think that patch is in the latest trunk. Right hurr Maybe a git pull will fix it?

You right. WE're good to go


Convention to respect to create the labels and component name

Enhancement
I propose that we adopt the following convention to create :

Component metadata name

For a Maven project, we use as Component metadata name the artifactId
For a Gradle project we use the Gradle description where we convert the title in lower cases and replace the space with "-" [1]

Labels

For a Maven/Gradle project, we will include the following labels

    labels:
      app: <artifactID> OR <gradle_description_lowercase_with_hyphen>
      version: <project_version>
      group: <???>

Question: What should we do for the group field ? My idea is to specify there the project or application to which the component or module belong to ? Where can we get such info ?
WDYT: @iocanel @geoand @aureamunoz @metacosm
[1] https://github.com/spring-projects/spring-framework/blob/master/spring-beans/spring-beans.gradle#L1

ap4k uses the current user as the group, but in this case I don't think we can do that.

Regarding the component name: I don't agree with what you're proposing. The component name should be independent of the build system. Whatever convention we choose, it needs to be independent of whatever build tool is using.
For the labels, maybe some context is needed to provide an answer. What are these labels used for? What semantics do we want them to carry? @cmoulliard

What are these labels used for?

Labels could be used by a k8s client to fetch resources using one or all the labels specified. Idem also at the server side

The component name should be independent of the build system

My proposition has nothing to do with the build system. Can you elaborate your concern please with an example ?

@cmoulliard since your proposition has nothing to do with the build system (and it shouldn't), why do you propose that the component names depend on the build system (i.e. a component will have a different name depending on whether it's built with maven or gradle)?

depend on the build system (i.e. a component will have a different name depending on whether it's built with maven or gradle)?

Got it. What I was saying is that, when we will create a Component CRD using ap4k, kreate, then the tools can determine what will be the component name as they can locally discover if this is a maven, gradle, nodejs, ... project

@cmoulliard that still depends on the build system. The component name shouldn't be derived from the build system. If it's a Spring Boot project, for example, its name should be independent of whether it is built using maven or gradle. It also happens that projects provide both build systems (though, rare, granted). The component name should be based on the artifact being build and the runtime (or some combination thereof that is repeatable).

@metacosm .. not my place, but.. if I'm a maven user, the name of my artifact is in the pom.xml file. If I use gradle, it's in the build.gradle file. If I use node, it's in the package.json file. I think @cmoulliard is suggesting we go look for it in the most language-normal place to find it.

This is the idea/plan proposed @ebullient ;-) Many thanks for your comment

@ebullient I agree but I worry about a strategy that would rely on the build system to name a component because that seems brittle to me… For example, what would happen if you were to switch from gradle to maven to build your project? Presumably, you would adjust your build parameters accordingly but…

from gradle to maven to build your project?

I don't think that many projects will change from one build strategy to another. Even if this is the case ap4k or kreate tools when they are used should be able to discover that we switched from maven to gradle or gradle to maven and then able to update the config of the component.
As this use case is border line, I would prefer than we document it in this case and tell to the user what should be done

For me, this enhancement is outside of the scope of the project. Why do we care how components are named (as long as they are valid k8s names)? What people call their components is up to them and we shouldn't try to infer any kind of semantics from the name. If we need more information, that information should be explicit and not encoded in the component name.
What benefit(s) would such an "enhancement" bring?

Why do we care how components are named (as long as they are valid k8s names)?

I think that you don't understand the purpose of this ticket. See the proposition of the enhancement which is already partially implemented for maven as we use artifactId as Component.Metadata.Name

@metacosm Is this still needed/wanted?

No.


Spectator view should defer loading large assets until needed.

Spectator view loads asset caches initially, so if a lot of meshes/large memory assets are included in asset caches, you can hit a point where the app crashes based on a lack of memory. We should assess whether loading these assets can be deferred until they are needed.

Any update on this? It is preventing me from progressing using Spectator View

We are looking into this now.

After I run "Update All Asset Caches" in my project, the project build size has increased by 1G. And when I open it on Hololens,it crashed.I have tested several situations.Without "Update All Asset Caches" the build can run normally on Hololens.
Is the crash caused by the large assets problem?Is there a solution now?

@RealFrogee the issue is based on how asset caches currently work. Basically all assets in your unity project get declared in an asset cache. Then when the hololens starts up it tries to load all asset caches. This is okay for small unity projects/targetted demos, but is really bad for real world applications whose projects contain a lot of content. The review we have out in #295 may be able to unblock you. It still has the same amount of asset references in a resource folder, which i would imagine will still increase your project build size. But compared to attempting to load all assets on start, a list of asset names is loaded. Then when spectator view observes assets declared in your scene, this asset name list is used to load individual asset references. There will still be some caveats, if you have any assets with the same name and type, all of them will be loaded at the same time even if you are only using one of them.


ADR for exposing MCAD observability metrics

Issue link

What changes have been made
Added an ADR for exposing MCAD metrics. Relates to PR 651
Verification steps

Checks

[ ] I've made sure the tests are passing.
Testing Strategy

[ ] Unit tests
[ ] Manual tests
[ ] Testing is not required for this change

Since the CFO is taking responsibility of serving the metrics then it seems we won't be needing an ADR.


Creating a bowl-shaped patch from a path

I have previous experience in using vanilla OpenSCAD to create very basic parts, but I am new to BOSL2. I read the docs on the wiki; the library is very impressive.
I am restoring a Japy Message typewriter. It does not have much value except a sentimental one. It is a 1963 typewriter, used by my mum in her first job up to the early 70s when I was born. I try to model a key cap to create replacement parts, as many were damaged by decades of storage.

I managed to create the overall shape of the key cap using smooth paths around trapezoids.  However, I have no idea how to model the top of the caps. The frame is planar, but the inside is bowl-shaped. I imagine creating a Bezier patch from the path and shifting down the inner control points. But I can't see how to achieve that. Unless there is a different approach to the problem?

include <BOSL2/std.scad>
include <BOSL2/rounding.scad>

top_width = 13;
bottom_width = 15;
keycap_height = 9;

top_face = smooth_path(
    trapezoid(w1=top_width, w2=top_width-2, h=15.5),
    closed=true);

bottom_face = smooth_path(
    trapezoid(w1=bottom_width, w2=bottom_width-2, h=15.5),
    closed=true);

skin(slices=0,
    profiles=[bottom_face, top_face],
    z=[0, keycap_height]);

I don't have an easy answer to this.  It looks like you don't actually want a bowl.  You want a flat recess with gently sloping sides.  You could do a recess maybe by taking your border shape, offsetting it in, and using offset_sweep to make a version that is flat on the bottom, with a rounded profile.  You then subtract that from your original.

Thanks, Adrian, for the answer. I didn't manage to use offset_sweep() for my purpose. My best solution for now was to create the "finger rest" shape as the hull() around the top face and a sphere (whose radius was empirically estimated). When I flatten the shape using zscale(), it doesn't look that bad. Maybe it is a bit too "conical" compared with the original. I wonder if this will be noticeable for a 1mm-height recess. I will probably print a prototype to validate the design.
module finger_rest() {
    r = 7;
    up(keycap_height+1/r/2) {
        zscale(1/r) hull() {
            linear_extrude(height=1, center=true)
                polygon(top_face);
            sphere(r);
        }
    }
}

module key() {
    skin(slices=0,
        profiles=[bottom_face, top_face],
        z=[0, keycap_height]);
}

difference() {
    key();
    finger_rest();
}

You probably need to use offset_sweep() to chamfer or round that shape.

Does this issue call for any action, or can we close it?

Thanks a lot for your help with that issue.


Fuzz Agora endpoints

Before we go in open beta

I am currently looking into https://github.com/microsoft/restler-fuzzer and https://github.com/dcarr178/har2openapi to make this happen with minimal modifications to agora and code written by us.

It's too bad that https://gitlab.com/gitlab-org/security-products/har-recorder does not like HTTPS connections we recently enabled on our nodes.

I have finally done what I should've done way before and decided to ditch the auto-generated spec. I am currently writing one myself for the Validator.
This will make it easier to write apps that interact with Agora nodes for everyone.


Inputs duplicated in summary

Some calculations, such as geomopt, save the structure in the inputs, which can output weird binaries as it's writing out the Atoms object.
As we already handle the structure separately, we should just be able to delete the object from the inputs dict.

This only occurs for trajectories, otherwise inputs["struct"] is overwritten, which is why we don't normally see this, as most calculations aren't designed to run on trajectories anyway - see #241


Handle Miri sysroot entirely outside the Miri driver

(Extracted from https://github.com/rust-lang/miri/pull/3409)
This entirely moves the responsibility of setting miri-sysroot to whatever invokes the Miri driver.  cargo-miri knows whether it is inside rustdoc or not and can adjust accordingly. I previously avoided doing that because there are a bunch of places that are invoking the driver (cargo-miri, the ui test suite, ./miri run, ./x.py run miri) and they all need to be adjusted now. But it is also somewhat less fragile as we usually have more information there -- and we can just decide that ./miri run file.rs --sysroot path is not supported. The advantage of this is that the driver is reasonably clean and doesn't need magic environment variables like MIRI_SYSROOT, and we don't have to fight rustc_driver to use a different default sysroot. Everything is done in cargo-miri (and the other much simpler driver wrappers) where it can hopefully be debugged much better.

:umbrella: The latest upstream changes (presumably #3414) made this pull request unmergeable. Please resolve the merge conflicts.

autocfg got patched. :) So this should pass CI now.
It's still a bit fragile though due to https://github.com/rust-lang/cargo/issues/10885. Also if we land this now cross-builds will break for anyone using autocfg older than 1.2. (But host builds should work, autocfg probes will just use the wrong sysroot.)

Ah the eyre build probe also still fails to apply RUSTC_WRAPPER. So landing this would break everything that uses eyre...

:umbrella: The latest upstream changes (presumably #3418) made this pull request unmergeable. Please resolve the merge conflicts.

Also this should only break with old autocfg when using --target; plain cargo miri test should be unaffected.

how long do you think we should wait before rolling this out?

Based on the downloads graph on crates.io, 1.2 seems like just over a third of downloads. I'm concerned we have a surprising amount of --target usage due to our much better shim support on x86_64-unknown-linux-gnu, so I would prefer to wait a week from now.

:umbrella: The latest upstream changes (presumably #3453) made this pull request unmergeable. Please resolve the merge conflicts.

I would prefer to wait a week from now.

We have waited 2 weeks since then. Download stats for autocfg haven't really changed.
Let's ship this; we can always revert if there are too many problems.
@bors r+

:pushpin: Commit<PHONE_NUMBER>f3b7df3460af8b678cadf77cf16a1c has been approved by RalfJung
It is now in the queue for this repository.

:hourglass: Testing commit<PHONE_NUMBER>f3b7df3460af8b678cadf77cf16a1c with merge f42ede2f19693d2b23e94536250b0f4b9416e607...

The build has long finished, but somehow bors didn't notice...?
@bors retry

:hourglass: Testing commit<PHONE_NUMBER>f3b7df3460af8b678cadf77cf16a1c with merge f250781a1397517aa19d87ffad07ac2234b6f7dd...

:sunny: Test successful - checks-actions
Approved by: RalfJung
Pushing f250781a1397517aa19d87ffad07ac2234b6f7dd to master...


Update Sinope_TH1124ZB.md

Adding information for Ouellet thermostat oth4000-zb that is manufactured by Sinopé

Nope, that's a new device so it needs a new file

I already submitted it last week … ;-)
Le 28 oct. 2022 à 18:35, blakadder @.***> a écrit :

Nope, that's a new device so it needs a new file
—
Reply to this email directly, view it on GitHubhttps://github.com/blakadder/zigbee/pull/788#issuecomment-1295571791, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUKZIKKGQBQ6HAIC5OBQQMDWFRIMDANCNFSM6AAAAAARRKSG3Q.
You are receiving this because you authored the thread.Message ID: @.***>

Completly agree. Already submitted it as a new device using website form last week ;-)
Without answer I thought that it was rejected.
May I have to submit it again ?
Le 28 oct. 2022 à 19:33, Baba Is You @.***> a écrit :

I already submitted it last week … ;-)
Le 28 oct. 2022 à 18:35, blakadder @.***> a écrit :

Nope, that's a new device so it needs a new file
—
Reply to this email directly, view it on GitHubhttps://github.com/blakadder/zigbee/pull/788#issuecomment-1295571791, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUKZIKKGQBQ6HAIC5OBQQMDWFRIMDANCNFSM6AAAAAARRKSG3Q.
You are receiving this because you authored the thread.Message ID: @.***>

its added from the submitted form


Install: wrong version in docker-compose.yml

Using the following docker and docker-compose versions,
$ docker-compose --version
docker-compose version 1.17.1, build unknown
$ docker --version
Docker version 18.09.4, build d14af54266
which as I understand should support a docker-compose.yml 3.7 version, I get errors when trying to install ShinyStudio. First, a few errors that seem to originate from the fact that there is no running containers
bash ./control.sh setup
*** Stopping
"docker stop" requires at least 1 argument.
See 'docker stop --help'.
Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
Stop one or more running containers
"docker rm" requires at least 1 argument.
See 'docker rm --help'.
Usage:  docker rm [OPTIONS] CONTAINER [CONTAINER...]
Remove one or more containers
ERROR: Version in "./../docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the services key, or omit the version key and place your service definitions at the root of the file to use version 1.
For more on the Compose file format versions, see https://docs.docker.com/compose/compose-file/
*** Building
ERROR: Version in "./../docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the services key, or omit the version key and place your service definitions at the root of the file to use version 1.
For more on the Compose file format versions, see https://docs.docker.com/compose/compose-file/
*** Starting
ERROR: Version in "./../docker-compose.yml" is unsupported. You might be seeing this error because you're using the wrong Compose file version. Either specify a supported version (e.g "2.2" or "3.3") and place your service definitions under the services key, or omit the version key and place your service definitions at the root of the file to use version 1.
For more on the Compose file format versions, see https://docs.docker.com/compose/compose-file/

For starters, you can ignore the error about there being no containers to stop. The ./control.sh setup operation begins by stopping all pertinent containers before rebuilding and restarting the site(s). If there is nothing to stop (as is the case in the initial setup), the error will be thrown, but it can be ignored. The setup script could be made to be more robust in this way; I will address this if I get time.
Regarding the version incompatibility, is there anything preventing you from upgrading to newer versions of Docker / Docker Compose? This was designed and tested with Docker version 18.09.2, Docker compose version 1.23.2. If you can, I suggest upgrading Docker to newer builds.
I did some testing and will be able to drop the docker-compose file version down from 3.7 to 3.5 to ensure compatibility with older Docker builds, but I cannot go lower than 3.5 without getting errors.

docker-compose's version was to be blamed, all clear now, thanks for your
work.
On Sun, Apr 7, 2019 at 6:19 PM Donald Mellenbruch<EMAIL_ADDRESS>wrote:

For starters, you can ignore the error about there being no containers to
stop. The ./control.sh setup operation begins by stopping all pertinent
containers before rebuilding and restarting the site(s). If there is
nothing to stop (as is the case in the initial setup), the error will be
thrown, but it can be ignored. The setup script could be made to be more
robust in this way; I will address this if I get time.
Regarding the version incompatibility, is there anything preventing you
from upgrading to newer versions of Docker / Docker Compose? This was
designed and tested with Docker version 18.09.2, Docker compose version
1.23.2. If you can, I suggest upgrading Docker to newer builds.
I did some testing and will be able to drop the docker-compose file
version down from 3.7 to 3.5 to ensure compatibility with older Docker
builds, but I cannot go lower than 3.5 without getting errors.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dm3ll3n/ShinyStudio/issues/3#issuecomment-480605957,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAcVPknx0imxKRJD3PHalK_94GkERRVpks5vehp3gaJpZM4cg1fM
.

Glad you got it worked out. I'll be sure to mention version requirements in the setup doc.


C++ new line brace positioning is affected

Normally when I type:
if (a > b){}  //the closing brace is inserted automatically

and hit enter, I end up with this:
if (a > b) {
    //the caret is here
}

With this extension enabled I end up with this:
if (a > b) {
} //the caret is before the closing brace

The same thing probably happens with functions etc. also.
Settings at Tools > Options > Text Editor > C/C++ > Formatting > New Lines don't seem to have much effect.

I guess VS behaviour is also to blame here, perhaps even more so.
If you move the carrret and move it back (this is actually done by the extension)  VS won't format the block after pressing enter.


ESAS: ObservationDistance

ObservationDistance
Distance at which the birds or marine mammals were observed.
In case of the standard 300 m transect used for ship-based sampling of marine mammals and birds in contact with the water:
A In contact with the water, 0 - 50 m
B In contact with the water, 50 - 100 m
C In contact with the water, 100 - 200 m
D In contact with the water, 200 - 300 m
E In contact with the water, >300 m, beyond transect
F Flying, no contact with water
W In contact with the water, but distance not recorded
For any other type of distance bin classification, the distance is an integer indicating the distance (m) to middle point of one the distance bins specified in the DistanceBins. Observations outside this transect are indicated by a > sign (e.g. >1000).

Anna will check with Carlos regarding the codes

codes are created. numeric values will be handled separately


Darktable appimage not working on EL7

Hello, when trying to run Darktable appimage on Enterprise Linux 7 I get these errors:
'/home/enkidu/Downloads/Darktable-2.2.5.glibc2.14-x86_64.AppImage'
/tmp/.mount_DarktazvJZXN/usr/bin/darktable: /lib64/libstdc++.so.6: version CXXABI_1.3.8' not found (required by /tmp/.mount_DarktazvJZXN/usr/bin/../lib/x86_64-linux-gnu/darktable/libdarktable.so) /tmp/.mount_DarktazvJZXN/usr/bin/darktable: /lib64/libstdc++.so.6: version GLIBCXX_3.4.21' not found (required by /tmp/.mount_DarktazvJZXN/usr/bin/../lib/x86_64-linux-gnu/darktable/libdarktable.so)
/tmp/.mount_DarktazvJZXN/usr/bin/darktable: /lib64/libstdc++.so.6: version GLIBCXX_3.4.20' not found (required by /tmp/.mount_DarktazvJZXN/usr/bin/../lib/x86_64-linux-gnu/darktable/libdarktable.so) /tmp/.mount_DarktazvJZXN/usr/bin/darktable: /lib64/libstdc++.so.6: version GLIBCXX_3.4.20' not found (required by /tmp/.mount_DarktazvJZXN/usr/lib/x86_64-linux-gnu/libIlmImf-2_2.so.22)
/tmp/.mount_DarktazvJZXN/usr/bin/darktable: /lib64/libstdc++.so.6: version GLIBCXX_3.4.21' not found (required by /tmp/.mount_DarktazvJZXN/usr/lib/x86_64-linux-gnu/libIlmImf-2_2.so.22) /tmp/.mount_DarktazvJZXN/usr/bin/darktable: /lib64/libstdc++.so.6: version GLIBCXX_3.4.20' not found (required by /tmp/.mount_DarktazvJZXN/usr/lib/x86_64-linux-gnu/libexiv2.so.14)
/tmp/.mount_DarktazvJZXN/usr/bin/darktable: /lib64/libstdc++.so.6: version GLIBCXX_3.4.21' not found (required by /tmp/.mount_DarktazvJZXN/usr/lib/x86_64-linux-gnu/libexiv2.so.14) /tmp/.mount_DarktazvJZXN/usr/bin/darktable: /lib64/libstdc++.so.6: version GLIBCXX_3.4.21' not found (required by /tmp/.mount_DarktazvJZXN/usr/lib/x86_64-linux-gnu/libpugixml.so.1)
/tmp/.mount_DarktazvJZXN/usr/bin/darktable: /lib64/libstdc++.so.6: version CXXABI_1.3.8' not found (required by /tmp/.mount_DarktazvJZXN/usr/lib/x86_64-linux-gnu/libicuuc.so.55) /tmp/.mount_DarktazvJZXN/usr/bin/darktable: /lib64/libstdc++.so.6: version GLIBCXX_3.4.21' not found (required by /tmp/.mount_DarktazvJZXN/usr/lib/x86_64-linux-gnu/libIex-2_2.so.12)
/tmp/.mount_DarktazvJZXN/usr/bin/darktable: /lib64/libstdc++.so.6: version GLIBCXX_3.4.20' not found (required by /tmp/.mount_DarktazvJZXN/usr/lib/x86_64-linux-gnu/libIex-2_2.so.12) /tmp/.mount_DarktazvJZXN/usr/bin/darktable: /lib64/libstdc++.so.6: version GLIBCXX_3.4.21' not found (required by /tmp/.mount_DarktazvJZXN/usr/lib/x86_64-linux-gnu/libIlmThread-2_2.so.12)
/tmp/.mount_DarktazvJZXN/usr/bin/darktable: /lib64/libgpg-error.so.0: no version information available (required by /tmp/.mount_DarktazvJZXN/lib/x86_64-linux-gnu/libgcrypt.so.20)
Please could you tell me what could be the problem?
Thank you very much and have a nice day.

Please ask the developers of Darktable to provide an AppImage that runs on CentOS 7.
http://www.darktable.org/contact/


Gh/deleted entities

@geneh,
Thanks for your contribution as a Microsoft full-time employee or intern. You do not need to sign a CLA.
Thanks,
Microsoft Pull Request Bot

Sigh, I accidentally hit the sync button in this branch and pushed another set of changes.  We should revert commit b46222f.  I'm not 100% confident in doing that so want to either sit with someone while I do it or have someone else do it. Sorry.

b46222f changes seem identical to a73704a. Not sure if anything should be done here.

I should have compared.  There are differences.  https://github.com/Microsoft/ghcrawler/compare/a73704a...b46222f
Having said that, the differences are good so we should leave the new commit.


Pip using python2 even if running build script with python3

If I start my build script like this
python3 build.py --build outdated

It will still use pip2 to install the conan packages, see packager.py
https://github.com/conan-io/conan-package-tools/blob/60ed0b215e704092beb2bead7d0fe3933a30a6c4/conan/packager.py#L406
Assuming pip == pip2, which it normally is

I assume that it is hard to know which one to execute, I could add a parameter to specify the "pip" command to run. WDYT?

Yes, a parameter/environment variable would be helpful. Even better would be automatic detection, but that might be a bit tricky if the naming of python/pip binaries are not standardized across platforms.
To detect the running environment you can use sys.version_info, and if that is version 3, then try using pip3, otherwise pip2. Maybe you need something like this to check if the those binares are available. If not, fall back to pip.

Since we started moving the build tools slowly to Python 3 this might become obsolete.

What about...?
import pip

pip.main(['install', 'whatever'])

or pip._internal.main depending on python version.
or running <path/to/python> -m pip ... where <path/to/python> = sys.executable.


Add a firebase function that listen to billing alert and hard stop the server at max budget

Implemented and tested as of b3c8ba697eaf85ecfee3466530d3391923863ac4


refactor code-based reply in responsive agent

Utilize the config in register_auto_reply to sanitize the implementation of code-based reply.
### Tasks
- [ ] In `execute_code_blocks`: Use a passed dict as the config instead of `self._code_execution_config`.
- [ ] When registering `generate_code_execution_reply`, set `config=code_execution_config` parameter.
- [ ] Remove `_code_execution_config` as a variable from ResponsiveAgent.

Already supported.


APS3 collaborative development from PS36 - test 1:  "copy suite au-aa279"  from rosie go: create a new trunk/suite id

|    by<EMAIL_ADDRESS>
This is a test to create a new trunk/suite id from the "trunk au-aa279" which is the metoffice PS36 suite (mi-ae510).

Issue migrated from trac:168 at 2024-01-31 18:10:42 +1100

<EMAIL_ADDRESS>commented

This ticket is associated with the suite id au-aa280:
URL: svn+ssh://accessdev.nci.org.au/home/access-svn/roses_au_svn/a/a/2/8/0/trunk
local copy: /home/548/ffb548/roses/au-aa280
browser: https://accessdev.nci.org.au/trac/browser/roses/a/a/2/8/0

<EMAIL_ADDRESS>commented

Exploring the possibility of having BOM_SUITE (configurable option in rose-suite.conf) in order to keep unchanged the logic of the metoffice suite, but diversify the path metoffice/bom only where necessary (e.g. pbs/archive/install_cold etc.)

<EMAIL_ADDRESS>changed _comment0 which not transferred by tractive

<EMAIL_ADDRESS>commented

To quick visualize difference between 2 repositories (e.g. reference development):
svn diff --diff-cmd='meld' URL_reference URL_development
svn diff --diff-cmd='meld' !^/a/a/2/7/9/trunk/ !^/a/a/2/8/0/trunk/


DownloadFile() overload to a Stream

Hi Harsh,
Any chance of DownloadFile() overload with a Stream instead of to disk? Recipricol of #32.

I thought of that too, but I didn't know how to implement it. Would it create a new MemoryStream and return that? The byte[] version is obvious and can easily be implemented. However since its a different return type, an overload will be impossible. So the API will get a bit messed up:

UploadFile() -- 3 overloads
DownloadFile() -- download and save to disk
DownloadFileAsBytes() -- new
DownloadFileAsStream() -- new

How about out params that are either a stream or a byte array?

Okay so something like:

DownloadFile(string, string)
DownloadFile(string, out byte[])
DownloadFile(string, out Stream)

Would that be discoverable? Anyways I think as an overload that should be pretty easy to discover. What do you think?

That looks good to me. It would come up nicely in code completion so be discoverable IMHO.

Done. Published to nuget as https://www.nuget.org/packages/FluentFTP/16.0.19
Sadly the byte[] and other versions appear in VS autocompletion before the string version. Do you know how to ask VS to prefer a certain overload?

Can you also test those methods to see if they work in your use case? I was going to use out Stream but decided to go with Stream since it allows the user to write data into any stream, whereas with out Stream I'm forced to create a new MemoryStream and set it into the var for C# to accept it.

Hi Harsh,
The stream version works for me in my use case. Only thing that may need consideration is that there is still an OverwriteFile parameter. This is redundant as the file is being written to a stream so I don't think that should be there anymore as a parameter.

Good points. Yes, I already discovered the overwrite parameter and removed it in the latest version.

I've made improvements to this in the latest version. You can fetch it from nuget. Can you see if everything works ok?

@hgupta9 Was this removed? I just updated to 17.x from 16.5.0 and I'm not seeing this in the 17.4.1 release. Was this moved to a different function or was it completely removed? If it was completely removed, can we get this back in? I'll be staying on 16.x until this works itself out.

@jaxidian Its now called Download() and Upload() for the stream/byte versions to make it more discoverable.

@hgupta9 Ahh, that'd be what I'm looking for. I could have sworn that I looked for renamed versions and couldn't find something - I guess I just didn't look hard enough. Thanks!