Broken CLI TypeScript support

Environment
Knex version: 0.20.15
Database + version: sqlite3
OS: macOS 10.14.6 (18G3020)
@lorefnon
Example repository with bugs
https://github.com/htunnicliff/knex-cli-ts-errors
Bug (1)
Explain what kind of behaviour you are getting and how you think it should do
Running knex migrate:make some_migration_name with a knexfile.ts in the current working directory should produce a TypeScript migration file with no errors.
Error message
Failed to load external module ts-node/register
Failed to load external module typescript-node/register
Failed to load external module typescript-register
Failed to load external module typescript-require
Failed to load external module @babel/register
sqlite does not support inserting default values. Set the `useNullAsDefault` flag to hide this warning. (see docs http://knexjs.org/#Builder-insert).
Created Migration: /knex-cli-ts-errors/migrations/20200416104822_create_example_table.ts

Bug (2)
Explain what kind of behaviour you are getting and how you think it should do
Running knex migrate:up should migrate with no errors.
Error message
Failed to load external module ts-node/register
Failed to load external module typescript-node/register
Failed to load external module typescript-register
Failed to load external module typescript-require
Failed to load external module @babel/register
sqlite does not support inserting default values. Set the `useNullAsDefault` flag to hide this warning. (see docs http://knexjs.org/#Builder-insert).
/knex-cli-ts-errors/migrations/20200416104822_create_example_table.ts:1
import * as Knex from "knex";
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at wrapSafe (internal/modules/cjs/loader.js:1072:16)
    at Module._compile (internal/modules/cjs/loader.js:1122:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1178:10)
    at Module.load (internal/modules/cjs/loader.js:1002:32)
    at Function.Module._load (internal/modules/cjs/loader.js:901:14)
    at Module.require (internal/modules/cjs/loader.js:1044:19)
    at require (internal/modules/cjs/helpers.js:77:18)
    at FsMigrations.getMigration (/knex-cli-ts-errors/node_modules/knex/lib/migrate/sources/fs-migrations.js:81:12)
    at /knex-cli-ts-errors/node_modules/knex/lib/migrate/Migrator.js:146:69
    at arrayFilter (/knex-cli-ts-errors/node_modules/lodash/lodash.js:582:11)

@htunnicliff What was the last version that this worked for?

@kibertoad I've had issues with this since I started using knex, around 0.16

@htunnicliff Have you tried using ts-node? This seems like you are trying to import TS file in JS context. After @briandamaged 's fixes liftoff should handle at least some of the TS cases out-of-the-box, but apparently not all of them.

I think we support knexfile to be TS, but not migration files per se. ts-node should help.

I have used this with ts-node in the past, but I wanted to file an issue because the expected behavior here just doesn't work unless you jump through hoops like require("ts-node/register"), etc. in your knexfile.

If this is truly the desired behavior, I think it is imperative that the docs elaborate on exactly how to get TypeScript working with ts-node. The docs don't make a single mention of ts-node.

I think we support knexfile to be TS, but not migration files per se. ts-node should help.

From the docs (Migrations > CLI > knexfile.js):
Generated migration extension:
You can control extension of generated migrations.
module.exports = {
  client: 'pg',
  migrations: {
    extension: 'ts'
  }
};

Yeah, I guess we need to look into what is supported and what is not and update docs.
Failed to load external module ts-node/register looks fishy, though.
For curiousity sake, which Node version are you using?

@htunnicliff But the migration creation case says Created Migration: /knex-cli-ts-errors/migrations/20200416104822_create_example_table.ts. Are you sure it doesn't actually get created?

I'm using Node 12.16.1.

It does create the migration, however, when running knex migrate:up you get another error and it is unable to actually do the migrating.

If there is a need for ts-node to enable CLI TypeScript support, would it make sense to have it as  a dependency in knex?

This problem also exists for CLI seed generation and execution

I think there are two things to be done here:

We need to document what is needed to interpret knexfile (or other files which knex imports) written in typescript  - which is basically one of these modules.

Currently the utilities for inferring extension for migrations/seeds to be generated are not aware of knexfile extension where as IMHO they should default to knexfile extension in absense of explicit extension specified somewhere in config.

What is the recommended workflow for running Typescript migrations? The documentation still suggests that running knex migrate:latest is sufficient:
Once you have a knexfile.js, you can use the migration tool to create migration files to the specified directory (default migrations). Creating new migration files can be achieved by running:

$ knex migrate:make migration_name

# or for .ts

$ knex migrate:make migration_name -x ts

...

Once you have finished writing the migrations, you can update the database matching your NODE_ENV by running:

$ knex migrate:latest

See:
root@1e4d151bd246:/srv/app# node --version
v12.18.2
root@1e4d151bd246:/srv/app# npm run knex -- --version

><EMAIL_ADDRESS>knex /srv/app
> knex "--version"

Failed to load external module ts-node/register
Failed to load external module typescript-node/register
Failed to load external module typescript-register
Failed to load external module typescript-require
Failed to load external module sucrase/register/ts
Failed to load external module @babel/register
Knex CLI version: 0.19.5
Knex Local version: 0.19.5
root@1e4d151bd246:/srv/app#
root@1e4d151bd246:/srv/app#
root@1e4d151bd246:/srv/app#
root@1e4d151bd246:/srv/app# npm run knex -- migrate:make initial_migration -x ts

><EMAIL_ADDRESS>knex /srv/app
> knex "migrate:make" "initial_migration" "-x" "ts"

Failed to load external module ts-node/register
Failed to load external module typescript-node/register
Failed to load external module typescript-register
Failed to load external module typescript-require
Failed to load external module sucrase/register/ts
Failed to load external module @babel/register
Using environment: development
Using environment: development
Using environment: development
Created Migration: /srv/app/migrations/20200722212137_initial_migration.ts
root@1e4d151bd246:/srv/app# npm run knex -- migrate:latest

><EMAIL_ADDRESS>knex /srv/app
> knex "migrate:latest"

Failed to load external module ts-node/register
Failed to load external module typescript-node/register
Failed to load external module typescript-register
Failed to load external module typescript-require
Failed to load external module sucrase/register/ts
Failed to load external module @babel/register
Using environment: development
/srv/app/migrations/20200722212137_initial_migration.ts:1
import * as Knex from "knex";
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at wrapSafe (internal/modules/cjs/loader.js:1054:16)
    at Module._compile (internal/modules/cjs/loader.js:1102:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1158:10)
    at Module.load (internal/modules/cjs/loader.js:986:32)
    at Function.Module._load (internal/modules/cjs/loader.js:879:14)
    at Module.require (internal/modules/cjs/loader.js:1026:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at FsMigrations.getMigration (/srv/app/node_modules/knex/lib/migrate/sources/fs-migrations.js:84:12)
    at /srv/app/node_modules/knex/lib/migrate/Migrator.js:82:69
    at arrayFilter (/srv/app/node_modules/lodash/lodash.js:582:11)
    at filter (/srv/app/node_modules/lodash/lodash.js:9173:14)
    at /srv/app/node_modules/knex/lib/migrate/Migrator.js:81:13
    at tryCatcher (/srv/app/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/srv/app/node_modules/bluebird/js/release/promise.js:547:31)
    at Promise._settlePromise (/srv/app/node_modules/bluebird/js/release/promise.js:604:18)
    at Promise._settlePromise0 (/srv/app/node_modules/bluebird/js/release/promise.js:649:10)

Workaround that worked for me using ts-node:
ts-node ./node_modules/.bin/knex migrate:make --knexfile knexfile.ts -x ts <migration_name>

This successfully creates the migration file with a .ts extension.
As for running the migration, I still run them using the transpiled .js files via npm run scripts:
"db:migrate:latest": "cd build && env DOTENV_CONFIG_PATH=../.env node --require dotenv/config ../node_modules/.bin/knex migrate:latest",
"db:migrate:rollback": "cd build && env DOTENV_CONFIG_PATH=../.env node --require dotenv/config ../node_modules/.bin/knex migrate:rollback"

I tried running the migrations as .ts files before but I encountered weird runtime transpilation issues when running on deployed containers (not enough memory?).

https://stackoverflow.com/questions/40910864/cannot-find-module-ts-node-register
I think this solves the issue

https://stackoverflow.com/questions/40910864/cannot-find-module-ts-node-register
I think this solves the issue

No, this does not solve issue. I have ts-node installed locally and the issue still persists.
same error with ts-node:
╰─ npx ts-node ../../../../node_modules/knex/bin/cli  migrate:latest
Using environment: development
/Users/jowparks/Projects/reflect/apps/api/src/app/migrations/20200816085511_add_questions.ts:1
import { __awaiter } from "tslib";
^^^^^^

SyntaxError: Cannot use import statement outside a module
...

Found a work around here:
https://github.com/knex/knex/issues/3849#issuecomment-643411244
in my package.json added this line, then executed whatever command I want:
scripts: {
  ...
  "api-knex": "TS_NODE_COMPILER_OPTIONS='{ \"module\": \"commonjs\" }' knex --knexfile=./path/to/knexfile.ts"
...

and run a command in command-line like:
npm run api-knex migrate:up

Note: My problem was that esnext is not supported for the knex cli

@jowparks Thanks, this workaround does work, but for some reason it doesn't allow me to pass the --all argument to the migrate:rollback command. Adding some console.log statements to node_modules/.bin/knex shows that commander.js receives cmd.all as undefined. It receives the argument correctly when not using the workaround, but then of course it breaks after it encounters the knexfile written in typescript.

Yea I am not a whiz with npm, guessing the argument is somehow not getting passed. Maybe you can try hardcoding the command in the package.json to see if it works.

as we were calling the following in our app init:
knex.migrate.latest(config);
our solution was to add ts-node to package.json:
"ts-node": "^8.10.2"
and the following near the top of our server.ts file.
require('ts-node/register');

I use env-cmd to load env vars from script  "migrate": "env-cmd knex migrate:make -x ts"

remove "include" from "compilerOptions" in tsconfig.ts

npm install ts-node worked for me


@auth rules not working

Before opening, please confirm:

[X] I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
[X] I have searched for duplicate or closed issues.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
[X] I have removed any sensitive information from my code snippets and submission.

How did you install the Amplify CLI?
npm
If applicable, what version of Node.js are you using?
v14.17.2
Amplify CLI Version
9.2.1
What operating system are you using?
Mac
Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.
I changed the mapping templates for two mutations, a query, and a subscription in the console. The subscription is the same subscription that i am trying to add the custom resolver to. I also added a lambda resolver for a query. The Function was created in the cli. But it was set as the resolver in the console. The function was also modified in the console. None of these are on any of the types that are the subject of this issue.
Amplify Categories
auth
Amplify Commands
push
Describe the bug
When I create a type and add auth rules. The auth rules do not work. I have put an example type below but the auth rules don't work on any of the types that i have pushed recently. Auth rules on older types work. When I examine the mapping templates for the newer types, whose auth rules don't work, there is no static group authorization. And in the example below, users were able to perform all operations regardless of their group.
type UserChannelPermission
    @model
    @key(fields: ["userId"])
    @auth(
        rules: [
            { allow: groups, groups: ["externalUser"], operations: [read] }
            { allow: groups, groups: ["agent"], operations: [read] }
            { allow: groups, groups: ["engineer"], operations: [read] }
            { allow: groups, groups: ["superAdmin"], operations: [read] }
        ]
    ) {
    userId: ID!
    channelIds: [String!]!
}

This is the same project as another issue that I entered. I don't think they are related. But  just fyi...
https://github.com/aws-amplify/amplify-category-api/issues/725
Expected behavior
@auth rules restrict or allow access based on group.
Reproduction steps
I don't really know what to put here. The auth rules do not work on any type that I push.
GraphQL schema(s)

# Put schemas below this line
type UserChannelPermission
    @model
    @key(fields: ["userId"])
    @auth(
        rules: [
            { allow: groups, groups: ["externalUser"], operations: [read] }
            { allow: groups, groups: ["agent"], operations: [read] }
            { allow: groups, groups: ["engineer"], operations: [read] }
            { allow: groups, groups: ["superAdmin"], operations: [read] }
        ]
    ) {
    userId: ID!
    channelIds: [String!]!
}

Project Identifier
I tried this. The project Identifier was not printed.
Log output

# Put your logs below this line

Additional information
No response

Hey @LeoReubelt :wave: thanks for raising this! From your schema I see you are using version 1 of our GraphQL Transformer (from @key). Version 2 of our transformer provides many enhancements to the existing at-auth directive, at-index, at-primaryKey, and also includes a few new relational directives such as hasMany or hasOne. We recommend migrating in order to leverage the enhancements, however will support v1 for security issues. Would you mind posting the auth resolver for this model?

I have put an example type below but the auth rules don't work on any of the types that i have pushed recently. Auth rules on older types work.

Can you provide more details around this note? When you say older types do you mean models you have pushed in the past vs models you've recently created?

Thanks @josefaidt. Upgrading our resolver is high on my list. The last project i did uses the v2 resolver. We just have to fit it in our timeline. Yes, what I meant is that objects pushed in the past have worked. All of those objects were initially pushed to another environment. And we cloned the environment that they were pushed to, making the environment that i am working on.
These are the resolver mappings for that one of the objects that doesn't work. Is that what you  want?
## [Start] Set default values. **
#set( $createdAt = $util.time.nowISO8601() )
## Automatically set the createdAt timestamp. **
$util.qr($context.args.input.put("createdAt", $util.defaultIfNull($ctx.args.input.createdAt, $createdAt)))
## Automatically set the updatedAt timestamp. **
$util.qr($context.args.input.put("updatedAt", $util.defaultIfNull($ctx.args.input.updatedAt, $createdAt)))
## [End] Set default values. **
## [Start] Set the primary @key. **
#set( $modelObjectKey = {
  "userId": $util.dynamodb.toDynamoDB($ctx.args.input.userId)
} )
## [End] Set the primary @key. **

## [Start] Prepare DynamoDB PutItem Request. **
$util.qr($context.args.input.put("__typename", "UserChannelPermission"))
#if( $modelObjectKey )
  #set( $condition = {
  "expression": "",
  "expressionNames": {},
  "expressionValues": {}
} )
  #foreach( $entry in $modelObjectKey.entrySet() )
    #if( $velocityCount == 1 )
      $util.qr($condition.put("expression", "attribute_not_exists(#keyCondition$velocityCount)"))
    #else
      $util.qr($condition.put("expression", "$condition.expression AND attribute_not_exists(#keyCondition$velocityCount)"))
    #end
    $util.qr($condition.expressionNames.put("#keyCondition$velocityCount", "$entry.key"))
  #end
#else
  #set( $condition = {
  "expression": "attribute_not_exists(#id)",
  "expressionNames": {
      "#id": "id"
  },
  "expressionValues": {}
} )
#end
#if( $context.args.condition )
  #set( $condition.expressionValues = {} )
  #set( $conditionFilterExpressions = $util.parseJson($util.transform.toDynamoDBConditionExpression($context.args.condition)) )
  $util.qr($condition.put("expression", "($condition.expression) AND $conditionFilterExpressions.expression"))
  $util.qr($condition.expressionNames.putAll($conditionFilterExpressions.expressionNames))
  $util.qr($condition.expressionValues.putAll($conditionFilterExpressions.expressionValues))
#end
#if( $condition.expressionValues && $condition.expressionValues.size() == 0 )
  #set( $condition = {
  "expression": $condition.expression,
  "expressionNames": $condition.expressionNames
} )
#end
{
  "version": "2018-05-29",
  "operation": "PutItem",
  "key": #if( $modelObjectKey ) $util.toJson($modelObjectKey) #else {
  "id":   $util.dynamodb.toDynamoDBJson($ctx.args.input.id)
} #end,
  "attributeValues": $util.dynamodb.toMapValuesJson($context.args.input),
  "condition": $util.toJson($condition)
}
## [End] Prepare DynamoDB PutItem Request. **

#if( $ctx.error )
$util.error($ctx.error.message, $ctx.error.type, $ctx.result)
#else
$util.toJson($ctx.result)
#end

Hey @LeoReubelt is this a customized resolver? Running through the reproduction steps here it does appear the groups appear as static values in the associated resolver. I'm also not seeing auth logic in the resolver posted above.
## [Start] Static Group Authorization Checks **
#set($isStaticGroupAuthorized = $util.defaultIfNull(
          $isStaticGroupAuthorized, false))
## Authorization rule: { allow: groups, groups: ["users"], groupClaim: "cognito:groups" } **
#set( $userGroups = $util.defaultIfNull($ctx.identity.claims.get("cognito:groups"), []) )
#set( $allowedGroups = ["users"] )
#foreach( $userGroup in $userGroups )
  #if( $allowedGroups.contains($userGroup) )
    #set( $isStaticGroupAuthorized = true )
    #break
  #end
#end
## Authorization rule: { allow: groups, groups: ["admins"], groupClaim: "cognito:groups" } **
#set( $userGroups = $util.defaultIfNull($ctx.identity.claims.get("cognito:groups"), []) )
#set( $allowedGroups = ["admins"] )
#foreach( $userGroup in $userGroups )
  #if( $allowedGroups.contains($userGroup) )
    #set( $isStaticGroupAuthorized = true )
    #break
  #end
#end

@josefaidt it is not custom. i get why it seems like it is though. i have pushed a ton of new types and never had a problem before. but everything i have pushed on the new environment has a resolver just like this one. as a sanity check, i pushed two new types last night. they both have the same resolver. at lest for create.

@josefaidt I cloned a new environment. It has the same parent as the environment the environment that is the subject of this issue. The @auth rules are working on the parent. But not either of the children. So maybe it has something to do with the new schema types that i am trying to push.
type UserChannelPermission
    @model
    @key(fields: ["userId"])
    @auth(
        rules: [
            { allow: groups, groups: ["externalUser"], operations: [read] }
            { allow: groups, groups: ["agent"], operations: [read] }
            { allow: groups, groups: ["engineer"], operations: [read] }
            { allow: groups, groups: ["superAdmin"], operations: [read] }
        ]
    ) {
    userId: ID!
    channelIds: [String!]!
}

type Channel
    @model
    @key(name: "byStateWorkflow", fields: ["stateWorkflowId", "channelName"])
    @auth(
        rules: [
            { allow: groups, groups: ["externalUser"], operations: [read] }
            { allow: groups, groups: ["agent"], operations: [read] }
            { allow: groups, groups: ["engineer"], operations: [read] }
            { allow: groups, groups: ["superAdmin"], operations: [read] }
        ]
    ) {
    id: ID!
    channelName: String
    channelUsers: [ChannelUserConnection!] @connection(keyName: "byChannel", fields: ["id"])
    stateWorkflowId: ID
    isDefaultChannel: Boolean
}

type Message
    @model
    @key(
        name: "byDateDummy"
        fields: ["dummyFieldUsedForSort", "createdAt"]
        queryField: "getMessagesByDateDummy"
    )
    @auth(
        rules: [
            { allow: groups, groups: ["externalUser"], operations: [] }
            { allow: groups, groups: ["agent"], operations: [] }
            { allow: groups, groups: ["engineer"], operations: [] }
            { allow: groups, groups: ["superAdmin"], operations: [] }
        ]
    ) {
    id: ID!
    channelId: ID!
    senderId: ID!
    channel: Channel @connection(fields: ["channelId"])
    sender: User @connection(fields: ["senderId"])
    createdAt: AWSDateTime!
    text: String
    fileKey: String
    hasBeenReadByIds: [ID!]
    dummyFieldUsedForSort: String
}

type ChannelUserConnection
    @model
    @key(name: "byChannel", fields: ["channelId", "userId"])
    @key(name: "byMember", fields: ["channelId", "userId"])
    @auth(
        rules: [
            { allow: groups, groups: ["externalUser"], operations: [read] }
            { allow: groups, groups: ["agent"], operations: [read] }
            { allow: groups, groups: ["engineer"], operations: [read] }
            { allow: groups, groups: ["superAdmin"], operations: [read] }
        ]
    ) {
    id: ID!
    channelId: ID!
    userId: ID!
    channel: Channel! @connection(fields: ["channelId"])
    user: User! @connection(fields: ["userId"])
}

type PaginatedMessages {
    messages: [Message!]!
    nextToken: String
}

One of my co-workers was able to push a test object. His @auth rules worked fine.

@josefaidt if i add an auth rule with no users in it, it works.  Not sure that is a good way to do it. But for now, I guess it will do.
type UserChannelPermission
@model
@key(fields: ["userId"])
@auth(
rules: [
{ allow: groups, groups: ["externalUser"], operations: [read] }
{ allow: groups, groups: ["agent"], operations: [read] }
{ allow: groups, groups: ["engineer"], operations: [read] }
{ allow: groups, groups: ["superAdmin"], operations: [read] }
{ allow: groups, groups: ["theAntiGroup"], operations: [create, update, delete] }
]
) {
userId: ID!
channelIds: [String!]!
}


SoftLayer API Errors and warden.rb

Hello,
I've been encountering the following errors quite frequently since last week. It looks like it's occurring in warden.rb. I've attached a stack trace with debug flag on.
The stack trace is the same though the exception is different.
>Exception class: EOFError
>Exception message: end of file reached

>Exception class: RuntimeError
>Exception message: Wrong type NilClass. Not allowed!

>Exception class: RuntimeError
>Exception message: Wrong size. Was 1460, should be 2999

>Exception class: RuntimeError
>Exception message: Wrong size. Was 1460, should be 3321

>Exception class: Net::ReadTimeout
>Exception message: Net::ReadTimeout

Sample stack trace below:
==> base01: Creating a new SoftLayer instance...
ERROR warden: Error occurred: Vagrant returned an exception while calling the SoftLayer API.

Exception class: VagrantPlugins::SoftLayer::Errors::SLApiError
Exception message: Vagrant returned an exception while calling the SoftLayer API.

Exception class: Net::ReadTimeout
Exception message: Net::ReadTimeout
 INFO warden: Beginning recovery process...
 INFO warden: Recovery complete.
 INFO warden: Beginning recovery process...
 INFO warden: Recovery complete.
 INFO warden: Beginning recovery process...
 INFO warden: Recovery complete.
 INFO warden: Beginning recovery process...
 INFO warden: Recovery complete.
 INFO warden: Beginning recovery process...
 INFO warden: Recovery complete.
 INFO warden: Beginning recovery process...
 INFO warden: Recovery complete.
ERROR warden: Error occurred: Vagrant returned an exception while calling the SoftLayer API.

Exception class: VagrantPlugins::SoftLayer::Errors::SLApiError
Exception message: Vagrant returned an exception while calling the SoftLayer API.

Exception class: Net::ReadTimeout
Exception message: Net::ReadTimeout
 INFO warden: Beginning recovery process...
 INFO warden: Recovery complete.
ERROR warden: Error occurred: Vagrant returned an exception while calling the SoftLayer API.

Exception class: VagrantPlugins::SoftLayer::Errors::SLApiError
Exception message: Vagrant returned an exception while calling the SoftLayer API.

Exception class: Net::ReadTimeout
Exception message: Net::ReadTimeout
 INFO warden: Beginning recovery process...
 INFO warden: Calling recover: #<Vagrant::Action::Builtin::Call:0x3b43690>
 INFO warden: Beginning recovery process...
 INFO warden: Recovery complete.
 INFO warden: Recovery complete.
 INFO warden: Beginning recovery process...
 INFO warden: Recovery complete.
 INFO warden: Beginning recovery process...
 INFO warden: Recovery complete.
 INFO warden: Beginning recovery process...
 INFO warden: Recovery complete.
 INFO environment: Released process lock: machine-action-dcd514a20cb632f4f705d734b71395b9
 INFO environment: Running hook: environment_unload
 INFO runner: Preparing hooks for middleware sequence...
 INFO runner: 1 hooks defined.
 INFO runner: Running action: #<Vagrant::Action::Builder:0x244dca0>
ERROR vagrant: Vagrant experienced an error! Details:
ERROR vagrant: #<VagrantPlugins::SoftLayer::Errors::SLApiError: Vagrant returned an exception while calling the SoftLayer API.

Exception class: VagrantPlugins::SoftLayer::Errors::SLApiError
Exception message: Vagrant returned an exception while calling the SoftLayer API.

Exception class: Net::ReadTimeout
Exception message: Net::ReadTimeout>
ERROR vagrant: Vagrant returned an exception while calling the SoftLayer API.

Exception class: VagrantPlugins::SoftLayer::Errors::SLApiError
Exception message: Vagrant returned an exception while calling the SoftLayer API.

Exception class: Net::ReadTimeout
Exception message: Net::ReadTimeout
ERROR vagrant: C:/Users/sercan.aktas/.vagrant.d/gems/gems/vagrant-softlayer-0.4.0/lib/vagrant-softlayer/util/warden.rb:31:in `rescue in sl_warden'
C:/Users/sercan.aktas/.vagrant.d/gems/gems/vagrant-softlayer-0.4.0/lib/vagrant-softlayer/util/warden.rb:17:in `sl_warden'
C:/Users/sercan.aktas/.vagrant.d/gems/gems/vagrant-softlayer-0.4.0/lib/vagrant-softlayer/action/create_instance.rb:18:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/warden.rb:34:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/builtin/synced_folders.rb:84:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/warden.rb:34:in `call'
C:/Users/sercan.aktas/.vagrant.d/gems/gems/vagrant-hostmanager-1.5.0/lib/vagrant-hostmanager/action/update_all.rb:33:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/warden.rb:34:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/warden.rb:34:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/builtin/provision.rb:80:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/warden.rb:34:in `call'
C:/Users/sercan.aktas/.vagrant.d/gems/gems/vagrant-softlayer-0.4.0/lib/vagrant-softlayer/action/setup_softlayer.rb:34:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/warden.rb:34:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/warden.rb:95:in `block in finalize_action'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/warden.rb:34:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/warden.rb:34:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/builder.rb:116:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/runner.rb:66:in `block in run'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/util/busy.rb:19:in `busy'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/runner.rb:66:in `run'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/builtin/call.rb:53:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/warden.rb:34:in `call'
C:/Users/sercan.aktas/.vagrant.d/gems/gems/vagrant-softlayer-0.4.0/lib/vagrant-softlayer/action/setup_softlayer.rb:34:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/warden.rb:34:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/builtin/config_validate.rb:25:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/warden.rb:34:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/builtin/handle_box.rb:56:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/warden.rb:34:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/builder.rb:116:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/runner.rb:66:in `block in run'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/util/busy.rb:19:in `busy'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/action/runner.rb:66:in `run'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/machine.rb:196:in `action_raw'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/machine.rb:173:in `block in action'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/environment.rb:474:in `lock'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/machine.rb:161:in `call'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/machine.rb:161:in `action'
C:/HashiCorp/Vagrant/embedded/gems/gems/vagrant-1.6.5/lib/vagrant/batch_action.rb:82:in `block (2 levels) in run'
 INFO interface: error: Vagrant returned an exception while calling the SoftLayer API.

Exception class: VagrantPlugins::SoftLayer::Errors::SLApiError
Exception message: Vagrant returned an exception while calling the SoftLayer API.

Exception class: Net::ReadTimeout
Exception message: Net::ReadTimeout
Vagrant returned an exception while calling the SoftLayer API.

Exception class: VagrantPlugins::SoftLayer::Errors::SLApiError
Exception message: Vagrant returned an exception while calling the SoftLayer API.

Exception class: Net::ReadTimeout
Exception message: Net::ReadTimeout
 INFO interface: Machine: error-exit ["VagrantPlugins::SoftLayer::Errors::SLApiError", "Vagrant returned an exception while calling the SoftLayer API.\n\nException class: VagrantPlugins::SoftLayer::Errors::SLApiError\nException message: Vagrant returned an exception while calling the SoftLayer API.\
n\nException class: Net::ReadTimeout\nException message: Net::ReadTimeout"]

What value do you currently have set for sl.api_timeout?

It is not set, so I assume it is using the default 60 secs.

Try increasing that but given the Runtime exceptions above it looks like it may occasionally be loosing packets as well so it might be a general network issue.

Hi again,
I increased the sl.api_timeout to 500 secs and I am still getting the same errors. Though a new one is added;
Exception class: #Class:0x4277eb0
Exception message: execution expired

Is this a windows box? Seems similar to this upstream issue, see if increasing the winrm timeout helps if you are building a windows box. If not theres also the sl.provision_timeout you can try setting.

It is CentOS

$ ./vagrant-softlayer-credentials software list -y
ERROR: Failed to perform action list on credential category software: execution expired

I started hitting this myself this morning and have never actually seen it before it, happened when my internet went down due to the snowstorm in the North East so it probably is just a general network issue.

I would like to place some 'puts' commands to see which API calls are made by vagrant-softlayer and what are their responses. Any suggestions where in which file I should do that?

Theres a few calls it puts in depending on what options you select but the general:

Check if machine id exists if already created
Verify order selection and place order
Wait for server provision to finish
Get IP Address
Wait for IP to become reachable on ssh (internal vagrant code).
sync folders over ssh

There may also be the joining of the load balancer if you have that configured. But the above would be the important points where it interfaces with softlayer_api and the API itself.

Thanks I will look into it more and keep this updated

More information for you;
When I run vagrant from a linux machine, it runs without a problem.
Sadly our development machines are Windows and we are using Windows Command Prompt since Cygwin has problems with softlayer. And in Windows machine I get these problems.
Do you have any insight on this?

Cygwin has problems with softlayer

I havent used SL or vagrant-softlayer on Windows at all so Im not sure what you are hitting here. Other than a possible difficultly in ensuring you have all the right dependencies installed or possible issues with Cygwin -> Windows path transformations I dont see why it wouldnt work.
With the info provided above all I can tell is it looks like a general network issue, but if you want to provide further details we can reopen and try to continue troubleshoot.


DataStreamSender take a long time under high concurrent query

Describe the bug
   I1122 09:44:58.566290 78779 plan_fragment_executor.cpp:568] Fragment 30f51ba7ecd34f3c-8faada0cd071d2a5:(Active: 19s384ms, % non-ch
ild: 0.00%)
   - AverageThreadTokens: 1.00
   - PeakReservation: 0
   - PeakUsedReservation: 0
   - RowsProduced: 110.59K
  BlockMgr:
     - BlockWritesOutstanding: 0
     - BlocksCreated: 0
     - BlocksRecycled: 0
     - BufferedPins: 0
     - BytesWritten: 0
     - MaxBlockSize: 8.00 MB
     - MemoryLimit: 2.00 GB
     - TotalBufferWaitTime: 0.000ns
     - TotalEncryptionTime: 0.000ns
     - TotalIntegrityCheckTime: 0.000ns
     - TotalReadBlockTime: 0.000ns
  DataStreamSender (dst_id=35):(Active: 19s375ms, % non-child: 0.00%)
     - BytesSent: 926.40 KB
     - IgnoreRows: 0
     - NetworkThroughput(*): 0.00 /sec
     - OverallThroughput: 47.81 KB/sec
     - SerializeBatchTime: 7.103ms
     - ThriftTransmitTime(*): 0.000ns
     - UncompressedRowBatchSize: 1.69 MB

1 The scan node time is very short，and the exchange node parent node time is also very short.
2 There are a lot of timeout querys
3 The CPU, IO, Mem ,Network resource  utilization is normal
4 The bprc transmit_data latency is very high.

5 DataStreamRecvr cancel_stream maybe take a lone time
W1122 20:55:12.063787  6106 data_stream_mgr.cpp:252] cancel cancel_stream time 56
W1122 20:55:12.063791  6112 data_stream_mgr.cpp:159] transmit_data time 53
W1122 20:55:12.068578  6116 data_stream_mgr.cpp:252] cancel cancel_stream time 57

W1122 20:49:50.773799  6119 data_stream_mgr.cpp:252] cancel cancel_stream time 851

This issue is related with https://github.com/apache/incubator-doris/issues/1726


How about playing sound inbuilt in PDF?

I have many PDF with inbuilt sound. By clicking on a graphic in Adobe Reader, it will play the wav file I linked to the musical notation. HTML5 can do sound too. Why not pdf.js? That would be awesome!
Here's a sample:
http://www.guitar.ch/pdf/en/demo/GuitarLovers-LicksRiffs-4v0-en-demo.pdf

Requires the /Screen annotation to be implemented, so I'm also labeling this as annotation work.

Thank you for taking notice! Are there any plans to incorporate sound playback in the near future?

@thosch6 We first have to refactor the annotation layer in order to properly implement this. The refactoring work has already started, but is is not clear yet when it will be done.

Any update on this?  I might be willing to implement if there was a clear path.

@yurydelendik, is it possible for me to do something similar to https://github.com/mozilla/pdf.js/pull/4376 with this?  I really need audio support in PDFs on the web.

@humphd Yes, something that is preffed off by default and comes with tests would help it to be merged more quickly.

Is there any progress on this funtion of playing inbuilt sound or updating information?

Here's a sample:
http://www.guitar.ch/pdf/en/demo/GuitarLovers-LicksRiffs-4v0-en-demo.pdf

Given that the file is no longer available there, nor on the Internet Archive, I'd suggest closing this issue as a duplicate of #2787 since the two features ought to be very similar.
(Also, the simplest way to "support" these Annotation-types might be to treat them basically like FileAttachment ones...)

Here is the new link:
http://www.guitar.ch/pdf/en/demo/GuitarLovers-LicksRiffs-5v0-en-demo.pdf http://www.guitar.ch/pdf/en/demo/GuitarLovers-LicksRiffs-5v0-en-demo.pdf
And no, audio and video are not the same! It is rather annoying that pdf.js does not support audio...

Am 03.04.2021 um 09:28 schrieb Jonas Jenwald @.***>:
Here's a sample:
http://www.guitar.ch/pdf/en/demo/GuitarLovers-LicksRiffs-4v0-en-demo.pdf http://www.guitar.ch/pdf/en/demo/GuitarLovers-LicksRiffs-4v0-en-demo.pdf
Given that the file is no longer available there, nor on the Internet Archive, I'd suggest closing this issue as a duplicate of #2787 https://github.com/mozilla/pdf.js/issues/2787 since the two features ought to be very similar.
(Also, the simplest way to "support" these Annotation-types might be to treat them basically like FileAttachment ones...)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/mozilla/pdf.js/issues/6078#issuecomment-812827341, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABWTVLCMZODL3VBWMJAF5ODTG27RHANCNFSM4BG4253Q.


[idea][ios] Send Native Events to Server

In order to obtain realtime data without user interaction.
For example:

GPS realtime position
Magnetometer
Audio Recording
Video/Image recording
Bluetooth Low Energy Devices
Push Notifications
Wifi

And so on.
It is needed a bridge to communicate native App events that do not require direct user input
but the server can be listening to.
Some ideas is using bindings
https://hexdocs.pm/phoenix_live_view/bindings.html
<button phx-click="app:gps">+</button>

def handle_event("app:gps", _value, socket) do
  {:ok, new_temp} = Thermostat.inc_temperature(socket.assigns.id)
  {:noreply, assign(socket, :temperature, new_temp)}
end

Maybe a subset of the form?
https://liveviewnative.github.io/liveview-client-swiftui/documentation/phoenixliveviewnative/formcontrols
In my experience a way of doing this is using the service locator pattern (http://gameprogrammingpatterns.com/service-locator.html)  of listener objects in the app. Just a simple dictionary with instances created in AppDelegate
And these object will call a specific method or send a specific request to the server when an event in the app happens.
this would require some modifications in the native app. The user would have to choose which events would be sent to the server.
Since,  these events would require special configs for requesting permissions. They have to edit the app anyways.
Just some ideas.

That's an interesting idea with the app:gps but wouldn't that mean that you want to match the value rather than ignore it?
As far as the form controls go, that API is not guaranteed at the moment. It got us to ElixirConf but we'll be extracting that out into its own library instead of being included in the standard swift ui client. We'll need more time to determine if the API is stable or not.

Ok so here are some ideas for the event system.
The native app can trigger two different type of events.

Events that require UX responses (trigger a message, update a table)
Events that are just important to the server and will not trigger an update to the user ui.

Classes
classDiagram
class IHandleEvents {
        - status :: Integer.t
        - options :: Map.t
        - handle(params :: Map.t)
        - init(with: status :: Integer.t, options: options :: Map.t)
        - set(options: options :: Map.t)
}

A interface or protocol that defines the behaviour of an event handler.
classDiagram
class PushNotificationEventHandler~IHandleEvents~ { }

An example event handler that will focus on Push Notifications.
This is the Dispatcher of Server Events.
classDiagram
class EventHandlersLocator{
   - add(event :: Any~IHandleEvents~)
   - get(event :: Any~IHandleEvents~.class.t)
   - remove(event :: Any~IHandleEvents~.class.t)
    - handlers :: Map.t(key:  Any~IHandleEvents~.class.String.t, value: Any~IHandleEvents~)
}

This service will store a single intance of each event handler. These instances are created in AppDelegate.
So a Single Handler can be responsible for dispatching events. The handlers registry will store
the string representation of the class of each handler as its key.
classDiagram
class ServerEvent {
       - name :: String.t
       - params :: Map.t
       - init(name :: String.t, params :: Map.t)
       - send()
       - send(params :: Map.t)
}

The object that will send the information to the server and maybe send a message to
the NotificationCenter
UI Events
These events will need to be defined inside the LiveView template, because they will trigger a LiveView handle_event function.
Example
<app.event name="app:gps:realtime" opts={ [every: :seconds30] }>

In a normal LiveView this maybe a hidden input which will trigger on change event.
<input type="hidden" data-every="seconds30" phx-change="app:gps:realtime" value="">

It will configure the native app event. Enabling it and sending its options as Json.
{
 "name": "app:gps:realtime",
  "opts": {
      "every": "seconds30"
   }
}

In the  LiveView component
def handle_event("app:gps:realtime", value, socket) do
  # server handling code
end

The value would be a json object containing the data.
If the data is an Image, Sound, File or another not JSON encodable,
it must be encoded as base64 string.
{
  "name" : "app:audio",
  "data" :  "bGlnaHQgdw=="
}

Server Events
These events will send a POST request to a specific endpoint in the server. So they will not be defined in the
template. Since they will not be related to LiveView. The configuration must be done inside the Native App.

Why can't pushEvent on the coordinator be used if no user interaction is required?

It is not planned. Rejecting. We have other way to do that: https://liveview-native.github.io/liveview-client-swiftui/documentation/liveviewnative/liveviewcoordinator/pushevent(type:event:value:target:)


Change documentation from Confluence to something open

I really enjoy the open-source nature of this project - but perhaps the documentation can be enhanced by not using Confluence. I was trying to take all the markdown and put it into another product, and WikiJS seemed the best. MDwiki may also be a viable option.
Thanks for all your work.

Hello @MachLearnPort !
Thank you for interesting proposals.
WikiJS really looks cool. Since we are using jinja templates it wouldn't be a problem to develop export to it. Maybe this is something you could help us with?
Regarding MDwiki — I was personally using it for a few Knowledge bases in the past (like this one), but it is not supported anymore and has a few XSS vulnerabilities, discovered by a few folks in russian security chats. So we decided to not use it at all.


Repo sync for protected CLA branch

The pull request is created from main637757930345449153 to main to fix git push error for protected CLA branch

@ghogen : Thanks for your contribution! The author(s) have been notified to review your proposed change.


Add commentary on sufficiency of rv128

A little bit whimsical :-)
Could add comment that even quite sparse usage will not exhaust it quickly.
Calculation (Python):
((2**128 # bytes
*8 # bits
/6.022e23 # moles
*12 # grams
/3.51 # CCs
/(4.0/3*3.1416*0.5**3) # volume of a sphere inscribed in a unit cube
)**(1.0/3) # cm
/100 # m
/1609.344) #miles

2 miles?  I'd prefer 3.2km in a technical document.

That would certainly be more natural for me, but I'm not American :-) Or "over 3000m"

It is a little too glib of a remark.   Because one application of the extra bits is tagging, rather than addressing, it's not abundantly clear that it will be enough for any purpose.

If you use 32 bits (!!) for tags or sparse address space, 96 bits still needs a diamond 1.9m in diameter under the same assumptions.
OK, so it could be a bit less whimsical and/or glib (if you mean the reference to Bill Gates' 640 KB remark), but I think there should be some kind of words about whether and why 128 bits is probably enough for a long long time.
256 bits would need a diamond 2.2 light years in diameter, .

Why do I suddenly want to use a Twinkie? (Ghostbusters reference)
In all seriousness, what can we do here? Could go with a bit/maximum integer table, but that doesn't really express the size, which is what I think you are getting at here?
Don't want to link to Wikipedia, but is there an idea in here that we can grab? (nice call out to RISC-V as well).
Or we could do a "wanna learn more about 128 bit, see .... )  But I'm not sure what is a good, "official" location to link to.
Else we can just close this PR.
@brucehoult thoughts? I'm trying to clear out zombie PRs.

Bit
Maximum Integer

16-bit
65,535

32-bit
4,294,967,295

64-bit
18,446,744,073,709,551,615

128-bit
340,282,366,920,938,463,463,374,607,431,768,211,455

The way things are trending, I think we'll need to significantly rework the RV128 proposal.  In the meantime, it is causing some confusion and clutter in the specs.  So I think we might end up excising it altogether, then reviving it later.  With that in mind, we can probably close the RV128-centric PRs, as fun as they are.

Closing due to other rv128 work being discussed.


Info panel state names fix

The transition function and table have been swapped - the function increases in size a lot longer and I've found myself most frequently accessing the table
The state names are now shown instead of the ids (to reflect any renamed states)

An image of the updates are shown below:

Closes #243

You were pretty much there when I wrote it so I thought whoever saw it could review 🤷


Auto limit removal

For #599, this PR implements a simple tool to remove limits from the existing Solidity sources.  The generated sources will be compiled in the next PRs.

@hackaugusto @rakanalh and I had a quick chat.

deployed sources will be visible in deploy_x.y.z/ directories next to deployment_***.json
in these deployed sources, the TAGS /* LIMITED_VERSION_START */ and /* LIMITD_VERSION_END */ will not be visible.

I'll first create a PR that adds the sources next the deployment data.  After that I come back here.


Foundry VTT became hung when trying to import Bestiary 3 - Mizzinastre Vexidyre

Foundry VTT became hung when trying to import statblock:
Mizzinastre Vexidyre      CR 18
XP 153,600
Female annis hag witch 15 (Pathfinder RPG Advanced Player's Guide 65, Pathfinder RPG Bestiary 3 16)
CE Medium monstrous humanoid
Init +4; Senses darkvision 60 ft.; Perception +16
Defense
AC 31, touch 16, flat-footed 26 (+4 armor, +1 deflection, +4 Dex, +1 dodge, +11 natural)
hp 145 (22 HD; 15d6+7d10+103)
Fort +8, Ref +9, Will +10
DR 5/bludgeoning; SR 17
Offense
Speed 40 ft.
Melee bite +9 (1d4+2), 2 claws +9 (1d4+2 plus grab)
Special Attacks hexes (blight[APG], cackle[APG], evil eye[APG], flight[APG], major healing[APG], misfortune[APG], retribution[APG], waxen image[APG]), rend (2 claws, 1d8+3)
Spell-Like Abilities (CL 7th; concentration +10)
3/day—alter self, fog cloud
Witch Spell-Like Abilities (CL 15th; concentration +22)
At will—feather fall (self only), fly (self only)
1/day—levitate (self only)
Witch Spells Prepared (CL 15th; concentration +22)
8th—horrid wilting (DC 25)
7th—harm (DC 24), power word blind, reverse gravity
6th—cone of cold (DC 23), greater dispel magic, mass fester[APG] (DC 23), mislead (DC 23)
5th—feeblemind (DC 24), mind fog (DC 24), suffocation[APG] (DC 22), teleport, waves of fatigue
4th—crushing despair (DC 23), cure serious wounds, dimension door, enervation, phantasmal killer (DC 21)
3rd—bestow curse (DC 20), dispel magic, major image (DC 20), ray of exhaustion (DC 20), screech[APG] (DC 20), vampiric touch
2nd—alter self, alter self, false life, mirror image, spectral hand, web (DC 19)
1st—charm person (DC 20), chill touch (DC 18), ill omen[APG], mage armor, obscuring mist, ray of enfeeblement (DC 18)
0 (at will)—bleed (DC 17), daze (DC 19), read magic, touch of fatigue (DC 17)
Patron Trickery
Statistics
Str 14, Dex 18, Con 16, Int 24, Wis 12, Cha 16
Base Atk +14; CMB +2 (+6 grapple); CMD 24
Feats Arcane Strike, Craft Staff, Craft Wondrous Item, Dodge, Greater Spell Focus (enchantment), Improved Familiar, Spell Focus (enchantment), Toughness
Skills Acrobatics +4 (+8 to jump), Intimidate +21, Knowledge (arcana) +25, Knowledge (dungeoneering) +22, Knowledge (nature) +32, Knowledge (nobility) +22, Knowledge (planes) +32, Knowledge (religion) +22, Perception +16, Spellcraft +25, Swim +6
Languages Abyssal, Aklo, Dwarven, Elven, Necril, Terran, Undercommon
SQ witch's familiar (quasit named Malkasar)
Combat Gear staff of cackling wrath[APG]; Other Gear amulet of natural armor +1, headband of vast intelligence +4, ring of protection +1, spell component pouch, wooden unholy symbol of Shax, 158 pp
Special Abilities
Arcane Strike As a swift action, add +1 damage, +1 per 5 caster levels and your weapons are treated as magic for the purpose of overcoming damage reduction.
Blight (DC 24) (Su) Curse animals, plants, or land, causing it to wither and die.
Cackle (Su) As a move action, extend the duration of other hexes by 1 rd.
Damage Reduction (5/bludgeoning) You have Damage Reduction against all except Bludgeoning attacks.
Darkvision (60 feet) You can see in the dark (black and white only).
Deliver Touch Spells Through Familiar (Su) Your familiar can deliver touch spells for you.
Empathic Link with Familiar (Su) You have an empathic link with your Arcane Familiar.
Evil Eye -4 (10 rounds, DC 24) (Su) Foe in 30 ft takes penalty to your choice of AC, attacks, saves, ability or skill checks (Will part).
Familiar Bonus:  You gain the Alertness feat while your familiar is within arm's reach.
Grab: Claw (Medium) (Ex) You can start a grapple as a free action if you hit with the designated weapon.
Greater Spell Focus (Enchantment) +1 to the Save DC of spells from one school.
Major Healing (4d8+15) (Su) Use cure critical wounds once per day/person.
Misfortune (2 rounds, DC 24) (Su) Foe in 30 ft must take the lower of 2d20 for rolls (Will neg).
Retribution (7 rounds, DC 24) (Su) Foe in 60 ft suffers half the melee damage it deals, ignoring resist, for duration (Will neg).
Scry on Familiar (1/day) (Sp) You can scry on your familiar, as the spell.
Share Spells with Familiar Can cast spells with a target of "You" on the familiar with a range of touch.
Speak with Animals (Ex) Your familiar can communicate with animals similar to itself.
Speak with Familiar (Ex) You can communicate verbally with your familiar.
Spell Focus (Enchantment) Spells from one school of magic have +1 to their save DC.
Spell Resistance (17) You have Spell Resistance.
Waxen Image (7 uses, DC 24) (Su) As a full rd action, create wax figure used to force to specific actions (Will neg).
Witch's Familiar (Ex) Gain the services of a special familiar that stores spells.
Mizzinastre has long worshiped Shax and considers herself a direct conduit of her lord’s murderous will. The irony of shedding a venerable but still beautiful drow body for a young but aged-looking hag’s body vexes the witch, so she often uses alter self to change back to a more beautiful form, similar to the one she bore in her previous life.

Used your latest module that was available as of this afternoon at about 2:30pm.  I used Hero Lab to produce this statblock. For greater accuracy, you might want to use one of the online resources. Hero Lab can truncate some of the text. All the other statblocks from Bestiary 3 imported -- a total of more than 86% imported!

Wow, that one is evil. No Error Message at all, but a complete freeze of Foundry.
That will take some time.

There was a "?" in the RegEx checking the class level that shouldn't be there. How and why that brought Foundry to crash i have no idea. But it should be fixed in 1.0.3


Automatic updates of WOF neighbourhood data

If data "becomes" invalid somehow, like it starts missing a particular property that we depend on, or the type of data becomes invalid (it's a string where we expect a number), I'm thinking that it's better to not add/update it. This is the safest thing to do, because if we start missing certain required properties, like name or label position, there's not really much we could do to perform an update anyway. Does that sound like the right approach? This also means that if a feature "becomes" funky where it wasn't before, we aren't going to update it, and leave it in its pre-funky state. Of course once the funky state is removed we would update the feature again. At the moment we're not importing any features that are marked funky.
An alternative is to remove these feature from our model. Is that better to do? One edge case here that could make this dangerous would be if something like the label position property names changed. With a "remove policy" we would end up removing all features.
I recognize that both of these are pedantic cases, but I just wanted to communicate what the behavior would end up being so we're at least aware of it if it happens.

Does that sound like the right approach? This also means that if a feature "becomes" funky where it wasn't before, we aren't going to update it, and leave it in its pre-funky state.

I think there might be some corner-cases with this, for example if the new, funky state is also superseded then we might end up displaying the old, non-funky version along with the new, non-funky, superseding item.
I feel like the more that we do to try and curate the WOF data in our pipeline, the more trouble it's going to give us with weird edge cases, and the more time we'll end up spending to investigate why there's a discrepancy between what's in the spelunker and what's on the tiles.
Also, I feel like we'll get more people reporting issues against and fixing the WOF data if we show it "unvarnished". How we encourage people to do that from the tiles is another issue, though...

I think there might be some corner-cases with this, for example if the new, funky state is also superseded then we might end up displaying the old, non-funky version along with the new, non-funky, superseding item.

Good catch on the superseded_by flag! It looks like we're currently only checking against that in the meta csv file. In the scenario where it becomes funky and superseded, we would then drop it, which I think is correct.
But, if it's not marked as superseded in the meta csv file, and marked as superseded in the json file, we would continue to try and process it, with the same "funky" edge cases. In this case I think we should check for the superseded_by flag in the raw json as well, and treat these cases as candidates for removal from our model.

I feel like the more that we do to try and curate the WOF data in our pipeline, the more trouble it's going to give us with weird edge cases, and the more time we'll end up spending to investigate why there's a discrepancy between what's in the spelunker and what's on the tiles.

I agree as well, since we're starting to grow more and more of these scenarios. Part of the increased complexity though is that I've been trying to keep out records from our database that we would never want to show, like those that have been superseded or are funky. We could alternatively push this logic downstream into the queries. The update logic would then be a little bit simpler, at the expense of having more complex queries, which would possibly be a little slower too.
That being said, I still think that having this logic applied during writes is a better trade-off though.
Ultimately, imho it still feels like there ought to be something in between the raw WOF data and our process to import the data. Whether that's a service, library, pre-processed data that is clean and bundled by place type, or some other sort of interface, putting the logic in at that layer means that our import process could be simpler, implying that all other consumers of the data won't need to learn these smarts either. The option to pull down the raw data could still be there, but this could just present an alternate, simpler option, at least for some use cases.

Updated superseded_by handling done in https://github.com/mapzen/tilequeue/pull/48.


tokio || async-std, parsing of arbitrary responses

Tokio runtime is still the default.
Regarding parsing of arbitrary responses, here's how to get screenshots from Rigol DS1000Z series.
    async fn parse_binary_response(mut reader: Pin<&mut (dyn futures_util::io::AsyncBufRead + Send)>) -> Result<Vec<u8>, tokio_lxi::Error> {
        let mut buf = vec![0u8; 9];
        reader.read_exact(&mut buf[0..2]).await?;
        if buf[0] != b'#' {
            return Err(anyhow::Error::msg("First character is not '#' in binary response").into());
        }
        if !buf[1].is_ascii_digit() {
            return Err(anyhow::Error::msg(format!("Second character is not in 0..9 range: {} ({:02x})", char::from(buf[1]), buf[1])).into());
        }
        let digits = (buf[1] - b'0') as usize;
        reader.read_exact(&mut buf[0..digits]).await?;
        let length = u32::from_str_radix(String::from_utf8_lossy(&buf[0..digits]).as_ref(), 10)
            .map_err(|_| {
                anyhow::Error::msg("Can't parse data length")
            })?;

        let mut buf = vec![0u8; length as usize];
        reader.read_exact(&mut buf).await?;

        Ok(buf)
    }

    pub async fn get_screen(&mut self) -> Result<Vec<u8>, Error> {
        let conn: &mut LxiDevice = self.get_conn().await?;
        let image = conn.request_data("DISPLAY:DATA? OFF,OFF,BMP24", Self::parse_binary_response).await?;

        println!("Got screen data of {} bytes", image.len());
        Ok(image)
    }

Also, it's not tokio-lxi anymore. It's more like an async-lxi

Thank you for your contribution. I'm sorry for responding after so much time passed.
Glad to see you've added support for async-std runtime. I agree that async-lxi for now is more correct name of the crate. I'll change the name of this repo and will make a new crate at Crates.io. Do you think it makes sense to make a runtime-async-std a default feature for async-lxi crate?
Handling non-text responses is a kind of a problem I encountered like you. I've found a similar solution of using a response parser as a generic parameter. I used this approach it in another crate that also implements LXI communication but in a synchronous manner.
But maybe it is better to parametrize the LxiDevice itself rather
than send.../receive... methods? In this case LxiDevice could be parametrized by type derived from some trait (e.g. LxiParser) rather than closures. It would make possible to store arbitrary state between requests that may be useful for some devices. What do think about this?
Anyway I'll merge your pull request.
Also don't you mind if I add you as a crate owner to tokio-lxi/async-lxi at Crates.io?


Change focus when the arrow is disabled.

Change focus when the arrow is disabled in the carousel component.

Can I take this issue?

Sure @jsebax! Thanks for helping!


GPFS startup fix for cloud expanision requirement for maintenance node

GPFS startup fix for cloud expanision requirement for maintenance node
Added implementation for state="maintenance"
Now  playbook will not try to startup gpfs is state="maintenance"
Signed-off-by: Rajan Mishra<EMAIL_ADDRESS>

Does this mean we're introducing a new variable state? Which other values are possible? Or is this variable already used elsewhere?
state without any prefix doesn't follow our naming convention... is this on purpose?

@acch  We are already using this for cloud requirement for expansion and contraction i.e similar to ansible state . state can be present , absent or maintenance.

I understand that this is being used (locally) in cloud_playbook.yml. But if we introduce it to roles/core then I'd suggest to follow the naming convention and call it something like scale_daemon_state or so... otherwise it will be very likely that we'll see naming collision similar to #189 ...

@acch Thanks . yes agree.  we can change this to scale_daemon_state.


Code | Numeric Pattern 287

I would like to work on this.Would you like to assign this to me?

@aryashah2k can I work on this?

@aryashah2k this is for Numeric pattern 287 or symbol pattern 1 because when I clicked on symbol pattern1 contribute , it directs me here and its name is Numeric pattern 287, please let me know this is for which pattern.

Yes, so the issues have not been linked currently for the c++ repo, please have alook at the website for pattern and then search that pattern number in the issues tab, sorry for the inconvenience

Okay


Fix PbrMetallicRoughness default case

Currently the Material type has Option<PbrMetallicRoughness> instead of PbrMetallicRoughness with default values. This is an oversight which needs to be fixed.

Fixed in the next release.


Review Jupyter notebook

Task to do
Review Jupyter notebook
Reason
See what else needs to be done
Result
A list of tasks/issues is created
Branch
https://github.com/AgPipeline/template-rgb-plot/tree/earthcube

branch updated: https://github.com/AgPipeline/template-rgb-plot/tree/earthcube


docs(readme): Add Vietnamese README.md translation

Added Vietnamese README.md translation
Moved README files into readme_i18n/

Thank you


Add bromolow (XPEnology) to working list

I have built the package for bromolow and installed the .spk. Works fine for now by using following command docker run --rm --env PACKAGE_ARCH=bromolow --env DSM_VER=6.2 -v $(pwd)/artifacts:/result_spk synobuild

Interesting, I wasn't aware of this project. I think I want to keep it in a second table rather than mixing it in with the normal Synology NASes.


core dump - segmentation fault

It seems a bit tricky to do trigger but this is what i did:
node Version: 16.16, also tried 16.13
better-sqlite3-multiple-ciphers: 7.6.2
OS: Kernel 5.15.0-41-generic, x86_64 GNU/Linux
node                                                                                [22/07/19| 5:28PM]
Welcome to Node.js v16.16.0.
Type ".help" for more information.
>
> sq = require('better-sqlite3-multiple-ciphers')
[Function: Database] { SqliteError: [Function: SqliteError] }
> db = sq(':memory:')
Database {
  name: ':memory:',
  open: true,
  inTransaction: false,
  readonly: false,
  memory: true
}
> db.pragma("rekey='password'")
[1]    179170 segmentation fault (core dumped)  node

This does not happen always. Most of the time it works.
Jul 19 17:32:01 julian systemd-coredump[179207]: Process 179170 (node) of user 1000 dumped core.#012#012Stack trace of thread 179170:#012#0  0x00007f8538a6ef84 _int_malloc (libc.so.6 + 0x97f84)#012#1  0x00007f8538a71299 __GI___libc_malloc (libc.so.6 + 0x9a299)#012#2  0x00007f8538e00b39 _Znwm (libstdc++.so.6 + 0xaab39)#012#3  0x0000000000d8a68e n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x98a68e)#012
#4  0x0000000000d464fd n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x9464fd)#012#5  0x0000000000d5269d n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x95269d)#012#6  0x0000000000d52adb n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x952adb)#012#7  0x0000000000d90923 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x990923)#012#8  0x0000000000d93f56 n/a (/home/julian/.nvm/versions/n
ode/v16.16.0/bin/node + 0x993f56)#012#9  0x0000000000d45003 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x945003)#012#10 0x0000000000d452e3 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x9452e3)#012#11 0x00000000011fc655 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0xdfc655)#012#12 0x00000000011fe037 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0xdfe037)#012#13 0x00000000015f2
d39 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11f2d39)#012#14 0x00000000016888c5 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x12888c5)#012#15 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#16 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#17 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#18 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#19 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#20 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#21 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#22 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#23 0x00007f852a06f936 n/a (n/a + 0x0)#012#24 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#25 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#26 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#27 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#28 0x00007f852a04f433 n/a (n/a + 0x0)#012#29 0x00007f852a067834 n/a (n/a + 0x0)#012#30 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#31 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#32 0x00007f852a053f97 n/a (n/a + 0x0)#012#33 0x00007f852a051983 n/a (n/a + 0x0)#012#34 0x00007f852a04e7ee n/a (n/a + 0x0)#012#35 0x00007f852a04f29b n/a (n/a + 0x0)#012#36 0x00007f852a067834 n/a (n/a + 0x0)#012#37 0x00007f852a06efaa n/a (n/a + 0x0)#012#38 0x00000000015b72b0 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11b72b0)#012#39 0x00007f852a0508cc n/a (n/a + 0x0)#012#40 0x00007f852a04f29b n/a (n/a + 0x0)#012#41 0x00007f852a067834 n/a (n/a + 0x0)#012#42 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#43 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#44 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#45 0x00000000015857ea n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11857ea)#012#46 0x00007f852a072550 n/a (n/a + 0x0)#012#47 0x00000000015839f8 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11839f8)#012#48 0x0000000001583783 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x1183783)#012#49 0x0000000000e250c8 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0xa250c8)#012#50 0x0000000000e25f1f n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0xa25f1f)#012#51 0x0000000000d15dc2 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x915dc2)#012#52 0x0000000000a43316 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x643316)#012#53 0x0000000000a54029 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x654029)#012#54 0x0000000000be74a8 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x7e74a8)#012#55 0x0000000000be9efa n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x7e9efa)#012#56 0x0000000000bf46c8 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x7f46c8)#012#57 0x0000000001571307 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x1171307)#012#58 0x0000000001571b30 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x1171b30)#012#59 0x0000000001577d14 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x1177d14)#012#60 0x0000000001565f38 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x1165f38)#012#61 0x0000000000a43dd5 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x643dd5)#012#62 0x0000000000b4ca06 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x74ca06)#012#63 0x0000000000ace5a2 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x6ce5a2)#012#012Stack trace of thread 179171:#012#0  0x00007f8538af646e epoll_wait (libc.so.6 + 0x11f46e)#012#1  0x0000000001577eb7 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x1177eb7)#012#2  0x0000000001565f38 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x1165f38)#012#3  0x0000000000b7c2ab n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x77c2ab)#012#4  0x00007f8538bd1609 start_thread (libpthread.so.0 + 0x8609)#012#5  0x00007f8538af6133 __clone (libc.so.6 + 0x11f133)#012#012Stack trace of thread 179173:#012#0  0x00007f8538bd8376 futex_wait_cancelable (libpthread.so.0 + 0xf376)#012#1  0x0000000001574489 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x1174489)#012#2  0x0000000000b7716b n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x77716b)#012#3  0x00007f8538bd1609 start_thread (libpthread.so.0 + 0x8609)#012#4  0x00007f8538af6133 __clone (libc.so.6 + 0x11f133)#012#012Stack trace of thread 179175:#012#0  0x00007f8538bd8376 futex_wait_cancelable (libpthread.so.0 + 0xf376)#012#1  0x0000000001574489 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x1174489)#012#2  0x0000000000b7716b n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x77716b)#012#3  0x00007f8538bd1609 start_thread (libpthread.so.0 + 0x8609)#012#4  0x00007f8538af6133 __clone (libc.so.6 + 0x11f133)#012#012Stack trace of thread 179176:#012#0  0x00007f8538bdb454 futex_abstimed_wait_cancelable (libpthread.so.0 + 0x12454)#012#1  0x00007f8538bdb548 __new_sem_wait_slow (libpthread.so.0 + 0x12548)#012#2  0x00000000015742c2 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x11742c2)#012#3  0x0000000000c15215 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x815215)#012#4  0x00007f8538bd1609 start_thread (libpthread.so.0 + 0x8609)#012#5  0x00007f8538af6133 __clone (libc.so.6 + 0x11f133)#012#012Stack trace of thread 179174:#012#0  0x00007f8538bd8376 futex_wait_cancelable (libpthread.so.0 + 0xf376)#012#1  0x0000000001574489 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x1174489)#012#2  0x0000000000b7716b n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x77716b)#012#3  0x00007f8538bd1609 start_thread (libpthread.so.0 + 0x8609)#012#4  0x00007f8538af6133 __clone (libc.so.6 + 0x11f133)#012#012Stack trace of thread 179172:#012#0  0x00007f8538bd8376 futex_wait_cancelable (libpthread.so.0 + 0xf376)#012#1  0x0000000001574489 n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x1174489)#012#2  0x0000000000b7716b n/a (/home/julian/.nvm/versions/node/v16.16.0/bin/node + 0x77716b)#012#3  0x00007f8538bd1609 start_thread (libpthread.so.0 + 0x8609)#012#4  0x00007f8538af6133 __clone (libc.so.6 + 0x11f133)#012#012Stack trace of thread 179179:#012#0

Jul 19 17:11:54 julian systemd-coredump[174735]: Process 174650 (node) of user 1000 dumped core.#012#012Stack trace of thread 174650:#012#0  0x00007fcd11a1ff84 _int_malloc (libc.so.6 + 0x97f84)#012#1  0x00007fcd11a22299 __GI___libc_malloc (libc.so.6 + 0x9a299)#012#2  0x00000000012599be n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0xe599be)#012#3  0x0000000001265b85 n/a (/home/julian/.nvm/versions/node
/v16.13.0/bin/node + 0xe65b85)#012#4  0x0000000001265e15 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0xe65e15)#012#5  0x00000000011db1ad n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0xddb1ad)#012#6  0x00000000011dcafe n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0xddcafe)#012#7  0x00000000011dec1c n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0xddec1c)#012#8  0x000000000108b511
 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0xc8b511)#012#9  0x00000000011f9253 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0xdf9253)#012#10 0x00000000015e7879 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x11e7879)#012#11 0x0000000001679d9d n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x1279d9d)#012#12 0x000000000157a32a n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x117a32a)#012#13 0x000000000157a32a n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x117a32a)#012#14 0x00007fcd03079ad6 n/a (n/a + 0x0)#012#15 0x000000000157a32a n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x117a32a)#012#16 0x00007fcd0308955e n/a (n/a + 0x0)#012#17 0x00007fcd030896de n/a (n/a + 0x0)#012#18 0x000000000157a32a n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x117a32a)#012#19 0x00007fcd030753f3 n/a (n/a + 0x0)#012#20 0x00007fcd03086134 n/a (n/a + 0x0)#012#21 0x000000000157a32a n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x117a32a)#012#22 0x000000000157a32a n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x117a32a)#012#23 0x00007fcd0309f297 n/a (n/a + 0x0)#012#24 0x00007fcd030974c3 n/a (n/a + 0x0)#012#25 0x00007fcd030747ae n/a (n/a + 0x0)#012#26 0x00007fcd0307525b n/a (n/a + 0x0)#012#27 0x00007fcd03086134 n/a (n/a + 0x0)#012#28 0x00007fcd0309c26a n/a (n/a + 0x0)#012#29 0x00000000015abdf0 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x11abdf0)#012#30 0x00007fcd0305df2c n/a (n/a + 0x0)#012#31 0x00007fcd0307525b n/a (n/a + 0x0)#012#32 0x00007fcd03086134 n/a (n/a + 0x0)#012#33 0x00007fcd03077f18 n/a (n/a + 0x0)#012#34 0x00007fcd03084bbd n/a (n/a + 0x0)#012#35 0x00007fcd0305f1aa n/a (n/a + 0x0)#012#36 0x00007fcd030836ef n/a (n/a + 0x0)#012#37 0x00007fcd0304d330 n/a (n/a + 0x0)#012#38 0x0000000001578538 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x1178538)#012#39 0x00000000015782c3 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x11782c3)#012#40 0x0000000000e19208 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0xa19208)#012#41 0x0000000000e19f4f n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0xa19f4f)#012#42 0x0000000000d09eb2 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x909eb2)#012#43 0x0000000000a3f546 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x63f546)#012#44 0x0000000000a50289 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x650289)#012#45 0x0000000000bde618 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x7de618)#012#46 0x0000000000be166a n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x7e166a)#012#47 0x0000000000beb9c8 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x7eb9c8)#012#48 0x0000000001565e47 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x1165e47)#012#49 0x0000000001566680 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x1166680)#012#50 0x000000000156c804 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x116c804)#012#51 0x000000000155acc8 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x115acc8)#012#52 0x0000000000a3ffe5 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x63ffe5)#012#53 0x0000000000b45056 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x745056)#012#54 0x0000000000ac67e2 n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x6c67e2)#012#55 0x00007fcd119ac083 __libc_start_main (libc.so.6 + 0x24083)#012#56 0x0000000000a3bfbc n/a (/home/julian/.nvm/versions/node/v16.13.0/bin/node + 0x63bfbc)

Hi @julianhille,
I've done some tests and you're right. The issue is reproducible and it's intermittent. Take a look at these tests. Very inconsistent. And that's a HUGE problem. Intermittent issues are the worst because those are pretty much impossible to debug. Also way harder to do trial and error tests.
I'm currently on a very tight schedule and this needs a lot more debugging to be able to track down anything that might be causing the issue. I'll keep this issue as open for third-party help for the moment.

I have a system at hand were this always happens.
I could try to debug this and come back.
I had similar issues with my CPP/node pdf extension and so got a bit of experience debugging things like that.
Thanks for the quick response and confirmation.

Segmentation fault comes from sqlite3.c from sqlite3mcGetReservedWriteCipher.
codec->m_hasWriter is a big negative number (which should not) and this ends up in an out of bounds when accessing: codecDescriptorTable[codec->m_writeCipherType-1].
What fixes it for me is do not use rekey for an empty in memory database. If one yses key it works totally fine.
If i fix the line to be
int reserved = (codec->m_hasWriteCipher == 1 && codec->m_writeCipher != NULL) ? codecDescriptorTable[codec->m_writeCipherType-1]->m_getReserved(codec->m_writeCipher) : -1;

(see check against == 1 as intended) the seg fault is gone BUT we still get an abort with
sqlite3BtreeSetPageSize: Assertion `nReserve>=0 && nReserve<=255' failed.

somehow the codec value breaks in sqlite3_rekey_v2

This is all great information. sqlite3_rekey_v2 is a SQLite3MultipleCiphers function. Would you be willing to test an in-memory DB via the SQLite3MultipleCiphers CLI on that same system you managed to debug from? That would help us clarify if the issue's actually in SQLite3MultipleCiphers and not when it's executed through NAN.

Yes. Will get to it.

SQLite version 3.39.2 2022-07-21 15:24:47 (SQLite3 Multiple Ciphers 1.4.7)
Enter ".help" for usage hints.
sqlite> pragma prekey="123"
   ...> ;
sqlite> ATTACH DATABASE ':memory:' AS aux1;
sqlite> PRAGMA rekey='password';
[1]    301894 segmentation fault (core dumped)  ./sqlite3shell ":memory:"

build locally and only rekey it.

As documented here transient (i.e. in-memory) databases and temporary databases will not be encrypted. Only file-based databases (and journals) can be encrypted,
That said, it is simply a misuse to try to encrypt an in-memory database.
Nevertheless, an application should certainly not crash, even in case of misuse of a feature. So, I will address this issue soon.
Regarding the error analysis:

Segmentation fault comes from sqlite3.c from sqlite3mcGetReservedWriteCipher.
codec->m_hasWriter is a big negative number (which should not) and this ends up in an out of bounds when accessing: codecDescriptorTable[codec->m_writeCipherType-1].

The reason for this is that function sqlite3_rekey currently doesn't explicitly check for an in-memory database. Internally, the function tries to allocate a codec and link it to the associated database file (which does not exist for in-memory databases). Therefore, the codec is deleted (because usually the underlying vfs takes ownership of the codec structure). However, function sqlite3_rekey makes use of the codec after it has been deleted - and this leads to a crash, of course.

If i fix the line to be
int reserved = (codec->m_hasWriteCipher == 1 && codec->m_writeCipher != NULL) ? codecDescriptorTable[codec->m_writeCipherType-1]->m_getReserved(codec->m_writeCipher) : -1;

(see check against == 1 as intended) the seg fault is gone BUT we still get an abort with

This is not a fix, but only hides the real cause of the problem ...

sqlite3BtreeSetPageSize: Assertion `nReserve>=0 && nReserve<=255' failed.

... as this assert proves.
For the time being do not use "pragma key" or "pragma rekey" on an in-memory database - it isn't supported anyway.

Thanks @utelle for the insights.

This is not a fix, but only hides the real cause of the problem ...

Yes you are right and it was not meant as a fix for the issue more a test to see that the segfault is really originating in that piece.
@m4heshd what we want to do with this outcome?

ignore for now and wait for rekey to be fixed in the base lib?
and / or add a bit more prominent documentation about that

i don't see other possible todos. Checking for pragma rekey or key if the database is an in-memory would be to error prone, i guess.

Commit https://github.com/utelle/SQLite3MultipleCiphers/commit/e1b82c1f9d09ba9576aadfcca1b23d4e0fdd28fe should fix the problem.
Since this project uses the SQLite3 Multiple Ciphers amalgamation sources, you probably want to wait for the next release. This may take a day or two.

@julianhille Thank you for debugging and your findings. @utelle Much appreciated you chiming in and providing a solution right away.

add a bit more prominent documentation about that

This IMO is a very rare case and unlikely to happen again 🤔. Since the functionality is already documented in SQLite3MC and a proper error is being thrown now, I'll keep the bs3mc documentation simple as it is for now.
Just merged the latest release of SQLite3MC and added some tests too. Changes are available in the latest beta release (v7.6.3-beta.0). Closing the issue.

This IMO is a very rare case and unlikely to happen again . Since the functionality is already documented in SQLite3MC and a proper error is being thrown now, I'll keep the bs3mc documentation simple as it is for now.

Not sure how likely/unlekliy that is, just wanted to say that i always use in-memory DBs for unit tests that is why i found this.


Login process overhaul

Pull request checklist:

[x] CHANGELOG.md was updated, if applicable
[x] Tested on Linux
[ ] Tested on Windows

[ ] Firewall (?)

[ ] Tested on MacOS

Description
tl;dr login will be 100% client-side, no copypaste bullshit, everything will be automatic.
Made use of QHttpServer to create a HTTP server listening on localhost:52107 (port opens only after user tries to authenticate).
The module itself is still not fully added to Qt itself (thus the blocked label), but it's very usable and there're rumors of adding it to Qt soon, so I'm guessing it's worth waiting. And if not, it's possible to migrate to boost:beast.
For ArchLinux friends who wanna test this, AUR package qthttpserver-git does the job.
UX
Removed basic and advanced sections in favour of clean layout with description, warning and two buttons as before. "Paste token" button completely deprecates "Advanced login" because now we're able to fetch all other details from token.
Example:

How does process of logging in work now?

Chatterino opens id.twitch.tv directly (constructed here https://github.com/Chatterino/chatterino2/pull/3065/files#diff-e181c6b227ade04c67338317f8c6988921b020dbf3f29402f2b08efac573fa73R28-R36)
We get redirected to GET localhost:52107/redirect - Chatterino's server is listening on this port and will return to the user an HTML page containing improved version of script from https://github.com/chatterino/login, which will make the user extract access_token from URL fragment and call PUT localhost:52107/token
On PUT localhost:52107/token, Chatterino checks if request contains X-Access-Token header containing access token we received from Twitch.
We validate the token (https://dev.twitch.tv/docs/authentication#validating-requests) and get all the details we need from it this way (without having to call Helix's Get Users).
After validation, we respond to the request that was earlier made to PUT localhost:52107/token with empty body but either 200 for success or 400 in any kind of failure.
If everything was successful, we update settings and change the current user (just like we did before).

The process for pasting token from clipboard is the same, except steps 1-3 and 5 are omitted.

Requiring opening an http server can cause problems with the users firewall.

work on windows but you have to verify to change firewall settings so for non-admin users not so nice

work on windows but you have to verify to change firewall settings so for non-admin users not so nice

How do you know this, did you build that on Windows? How did you do it, there's no package for httpserver on Windows as far as I am aware.

work on windows but you have to verify to change firewall settings so for non-admin users not so nice

How do you know this, did you build this PR on Windows? How did you do it, there's no package for httpserver on Windows as far as I am aware.

https://www.npmjs.com/package/http-server

That's a node module you linked, not the Qt httpserver module

I think I'm gonna rewrite the implementation in boost:beast instead, so I'm putting this one into draft until I learn that framework.

Sorry but I this is not a good solution. Chatterino shouldn't open a tcp port like that. The same thing was done in Chatterino 1 and it creates too many issues.

Sorry but I this is not a good solution. Chatterino shouldn't open a tcp port like that. The same thing was done in Chatterino 1 and it creates too many issues.

This is common practice for most apps when doing auth. What concerns do you have about opening a port for it?
I could understand if the port was open all the time so ideally it would only be open when the user needs to add an account. Haven't read the code so not sure if that's the case.

I could understand if the port was open all the time so ideally it would only be open when the user needs to add an account.

I am aiming for listening on a port only on localhost interface and only for the short period of time - Chatterino would open login page and start listening, user would login, be redirected to the localhost:port, - Chatterino would send back either 200 or 400 and immediately stop listening.

I believe this to be the best solution available to us. There is no middle man that can get compromised or steal your tokens, we can refresh the tokens in the app. Yes it does come at a cost of having to temporarily open a localhost port on the firewall. Could there be another way of getting the token locally that wouldn't require a web server?
I'm excited for not having to deal with Login authentication expired. Login authentication expired. disconnected

Could there be another way of getting the token locally that wouldn't require a web server?

Actually, yes there is fallback way to do it. Since we don't change the authorization flow that's used to obtain the token, we can just open Twitch's authorization page (id.twitch.tv) and after user authorizes application, he'd be redirected to localhost:port/redirect#token=xxxxxx. Copy the token and then pressing new "Paste token from clipboard" button Chatterino will try to validate the pasted token and if the token is valid, all the required properties (user id, login, client id) will be obtained upon successful validation, successfully adding user's account to Chatterino.
I know this may be stupid and perhaps unintuitive, for - some users, but in case the port cannot be opened, - we'd detect the failure and inform user about it, instructing users on how to copy and paste their token - into Chatterino.
With proper instructions in place I believe this should be fine and even Swedish roleplayers should be able to understand it since they already somehow understand how to copy and paste from /client_login.


add etcd watch design documentation

Hope for your advice, @xiang90 @AdoHe

👍 this should be a good beginning of watch.

@StupidHod @AdoHe
Maybe we can start with the goals to have EtcdWatcher, and then talk about details about how to implement each?

@hongchaodeng yes, we should examine the goals before we start the implementation.

@StupidHod So this works pretty similar to the go version?

The design doc looks good. Let's clean it up and get it merged.

@xiang90 , thanks for your advice, I have cleared it up with your tips.

LGTM
Let's move forward with implementation.
Thanks @StupidHod !


remove unnecessary files

See this PR for reference: https://github.com/apache/incubator-openwhisk-runtime-python/pull/34#event-1731280368

Done with 1bef0b30121e03e9ac496682b895f41677443271


[Incremental] Another attempt to get the driver to honor -driver-use-frontend-path

This one only subverts the frontend target check if the new frontend path is a Python script. That way, the option still should work with swiftpm. Don't merge until the swiftpm unit tests pass with this.

@artemcm What do you think about this approach? @DougGregor Do you want to weigh in? Is there a better alternative?

@swift-ci please test


Let futures on the Actix loop go outside and back again

In #28 @piegamesde mentioned a usecase where an actor would spawn a future on the GTK loop (specifically - outside Actix) that would interact with GTK and when done send a message back to the actor. This future cannot use Actix' ActorFuture to retake the actor, because that would make the future execute inside Actix and thus it won't be able to do GTK operations that trigger signals - unless it uses woab::schedule_outside, which does offer a way to get back to the future.
We need a way to allow futures spawned on the Actix loop to yield execution to something outside the Actix loop and await for that something to finish.

The API I want is something like this:
ctx.spawn(async move {
    // Do stuff inside Actix
    woab::outside(async {
        // Do stuff outside Actix
    }).await;
    // do stuff inside Actix again
}.into_actor(self))

This will be based on a little more manual function:
async fn outside(fut: Future) {
    woab::wake_from_outside(|waker| {
        glib::MainContext::ref_thread_default().spawn_local(async move {
            fut.await;
            waker.wake();
        });
    }).await;
}

(Not actual Rust code (don't have all the required annotations), I'll try to get return values to work, and I'll also try to ensure woab::outside does not require move)
The reason for wake_from_outside is so we can write things like
let button: gtk::Button = get_the_button_from_somewhere();
ctx.spawn(async move {
    woab::wake_from_outside(|waker| {
        button.connect_clicked(|_| {
            waker.wake();
        });
    }).await;
    // Do things after the button was clicked
}.into_actor(self))

Actually, I think maybe woab::wake_from_outside should not actually go outside Actix:

If I run the closure immediately, it will not need to be move.
You'd usually use it to connect a signal, so you don't need it to run from outside.
If for whatever reason you do need to run it from outside you can just use schedule_outside (or spawn_outside, which I'm probably going to replace schedule_outside with)

Also, instead of using a waker I can probably just use Tokio's oneshot - which will also make it easy to pass back data on wakeup. This will also make it trivial to pass back the async block's result in woab::outside.


WhereLead and WhereLag operators

Lead and Lag operators are useful, but there are times when you only want to filter based on the next or previous value, but do not need the combined value after filtering. This would be a shortcut for source.Lead(...).Where(...).Select(x => x.current).

Do you have a vision for this implementation and an example of the problem it's trying to overcome? I'd be more than glad to assist with it given some additional clarity!

Do you have a vision for this implementation and an example of the problem it's trying to overcome? I'd be more than glad to assist with it given some additional clarity!

Sure!
So technically, the behavior already exists in SuperLinq, but it's awkward and unwieldy, which is why I'm suggesting additional operators for these.
The existing behavior would be as such:
var source = Enumerable.Range(0, 30).Select(x => DateTime.Today.AddDays(x));

var result = source
  .Lead(1)
  .Where(x => x.current.Month != x.lead.Month)
  .Select(x => x.current);

This would return a list of the last day of the month. I recognize that there are many easier ways to get the last day of the month, but the point is to highlight a potential use-case for .Lead().Where().Select(), if this code were generalized to an input sequence trying to identify certain conditions.
The simpler solution would look like:
var result = source.WhereLead(1, x => x.current.Month != x.lead.Month);

where result is still an IEnumerable<TSource>.
For how to implement, you should be able to take notes from the Lead and Lag operators, though due to the uncertain lengths, the ListIterator optimizations here will not be viable for WhereLead and WhereLag.

Thanks for the swift reply! Given the example, WhereLead reads a bit clunky to me at the moment based on the existing behavior. Is there a reason to prefer WhereLead over the inverse of LeadWhere? For example:
var result = source.LeadWhere(1, x => x.current.Month != x.lead.Month);

I'm open to argument, but my perspective is that WhereLead is a more accurate title for the operator. The goal of the operator is to filter, so it is a Where operator; the Lead/Lag is the mechanism. To me, LeadWhere indicates that the Lead is the goal and the Where is the mechanism.  Thoughts?

That makes more sense to me now, thank you for the clarity!


Uncomplete sentence in patterns

@marvinbuss the syntax {"IS_SENT_START": False, "OP": "*"} works only for words that come before the word in the pattern right? what about the sentences where we can't make any assumptions about how the sentence will end but still want to match as soon as the sentence starts with a certain set of words. One example: "I thought I share our newest features with you" Now the Words we match is "I thought" and "our". Now do I need to add something to make sure it matches even though there are words coming after the words in pattern3?
`    def _you_centered21(
self,
rule_id="R0061",
description="Put your recipient in the center of your sentence. Simply replace the I focused sentence with a You focused one.That's how you catch their attention.",
info="Try to be very precise on what you want and use the present form if possible.",
example_pos="You can find a guide on how you can benefit from the most recent product update attached.",
example_neg="I thought I share our newest features with you"
) -> None:
""" Check for negative and positive tone.
    rule_id (str): The ID of the rule.
    description (str): The description of the rule.
    info (str): Additional info about the rule.
    example_pos (str): Positive example for this rule.
    example_neg (str): Negative example for this rule.
    """
    # Define matcher
    matcher = Matcher(self.nlp.vocab)

    # Define patterns to match
    pattern1 = [
        {"LOWER": "I"}
    ]

    pattern2 = [
        {"LOWER": "thought"}
    ]

    pattern3 = [
        {"LOWER": "our"} ,{"IS_SENT_START": False, "OP": "*"}
    ]

    # Add the patterns to the matcher and apply the matcher to the doc
    matcher.add("positivetone_pattern", [pattern1, pattern2, pattern3])
    matches = matcher(self.doc)

    # Merge the matched intervals
    intervals = [(start, end) for match_id, start, end in matches]
    merged = merge_intervals(intervals)

    # Add recommendations
    if len(merged) > 0:
        self._add_recommendation(
            rule_id=rule_id,
            description=description,
            info=info,
            example_pos=example_pos,
            example_neg=example_neg,
            token_intervals=merged
        )`

@Mochahintwain Can you please add this to the correct repository? This was opened in an Azure repository.
The first pattern will not work since "I" is not lowercase.

Thanks for noticing....I looked up how to change the repository but can't find it...only found it for project boards but we don't have a board. How does that work?
And what do I have to write in the pattern to match for "I thought I share our newest features with you" and ANY other sentence that includes I THOUGHT and the word OUR?


Update PDFReader

Description
Added the ability to use PDFPlumber as the underlying library extracting text, because from my own testing it performed better than PyPDF, albeit a little slower. A user will need to install the PDFPlumber library to use this. The reason behind including this functionality is because it was able to extract more/much higher quality text from PDFs than the PyPDF implementation.
This would also now match the PDFReader from llama-index.
Fixes # (issue)
Type of Change
Please delete options that are not relevant.

[x] New Loader/Tool
[x] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[x] I stared at the code and made sure it makes sense
[x] Tested Locally

Suggested Checklist:

[x] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[x] My changes generate no new warnings
[x] New and existing unit tests pass locally with my changes
[x] I ran make format; make lint to appease the lint gods

Rather than changing the base PDF reader, I would suggest you create a new PDF reader using PDFPlumber

Sounds good, can do!


It shouldn't be possible to bookmark incomplete Analyse queries

For instance this query: https://openprescribing.net/analyse/#org=CCG&orgIds=08D&denom=nothing&selectedTab=summary
which has no numerator selected.
On this page the "Show me the data!" button is disabled, but not the "Subscribe" button.

Update: the name of this bookmark is "Spend on Paracetamol by NHS HARINGEY CCG and other CCGs" which suggests that a numerator was originally selected and that something else has gone wrong.


Can I use velocyto.py to get the velocyto plot on seurat dimensionality reduction

Hi ,
my scRNA data is get the cluster results by seurat(all 6 samples),and I want used the tools to calculate all cluster's velocyto results. I already get the loom files,and merged it.And then what can I do it?
the memory of  merged.loom is 4.5G，I’m a little concerned about the amount of data.I tried used velocyto.R to load the loom file ,my systerm memory is 128G,then I got the memory error,I wonder if it would be better to use Python.

Hello, I did something similar by exporting the UMAP values for each cell from Seurat in R and importing them in Python and adding them as metadata to the velocity object created using a package called scvelo. Hope this leads you in a direction you're interested in.


Missing metadata from kernel on parameter

The analyzer test FixProcessorTest.test_addMissingRequiredArg_multiple() is failing because the VariableDeclarations for the parameters of a top-level function that come from kernel do not have any metadata associated with them. The code being analyzed is
import 'package:meta/meta.dart';

test({@required int a, @required int bcd}) {}
main() {
  test(a: 3);
}

The CFE integration work is suspended, so this will not be addressed.