Feat: access token

This PR creates a resource to create and revoke access tokens in Artifactory.
As Artifactory does not support Read or Update operations on an Access Token, the resource does not support this either.
As there is no read operation, it cannot support an import of an existing access token.
It implements all the features as per the API specs:

https://www.jfrog.com/confluence/display/JFROG/Artifactory+REST+API#ArtifactoryRESTAPI-CreateToken
https://www.jfrog.com/confluence/display/JFROG/Artifactory+REST+API#ArtifactoryRESTAPI-RevokeToken

I have changed the inputs for setting the token expiration so that it can be used in different ways - specified either by end_date_relative or end_date. See the documentation for examples.
Although you can specify a token to be refreshable, the resource does not support the refresh action, as this does not align to CRUD operations.
I've also updated a number of the tests so that they reference the correct client instance.
Cheers,
Josh

@josh-barker-coles I'll close my access_token (#4) implementation then 😄 ?

Oh, I didn't realise you'd implemented it too @donovanmuller. 😄
I should have mentioned that I'd started working on this.
Do you have any feedback on this PR?

@josh-barker-coles LGTM 👍

Does this handle refresh of access tokens?

Test refresh of tokens.

@josh-barker-coles I am starting to take over this PR and repo from John. I will be posting updates today. We are currently seeing what appears to be a race condition. I will post a reproducible script

To proceed further we need a test tf that's repeatable. This TF has been tried with .12 and .13 - I basically have to run apply 2-3 times before it actually takes.
Here is what's been put together on our side - feel free to take it and run with it
# Required for Terraform 0.13 and up (https://www.terraform.io/upgrade-guides/0-13.html)
terraform {
  required_providers {
    artifactory = {
      source  = "registry.terraform.io/jfrog/artifactory"
      version = "2.2.5"
    }
  }
}
provider "artifactory" {
  url = "https://christianb.jfrog.tech/artifactory"
  username = "admin"
  password = "Password1"
}
resource "artifactory_group" "test-group" {
  name             = "terraform"
  description      = "test group"
  admin_privileges = false
}
//# Create a new Artifactory group called terraform
resource "artifactory_user" "test-user" {
  name     = "terraform"
  email    =<EMAIL_ADDRESS>  groups   = ["terraform", "readers"]
  password = "Password1"
}

//# Create a new Artifactory user called terraform

resource "artifactory_access_token" "existing_user" {
  username          = "terraform"
  end_date_relative = "5m"
}
resource "artifactory_access_token" "new_user" {
  username          = "terraform"
  end_date_relative = "1h"
  groups = [
    "readers",
    "terraform"
  ]
}
resource "artifactory_access_token" "no_expiry" {
  username          = "terraform"
  end_date_relative = "0s"
}
resource "artifactory_access_token" "refreshable" {
  username          = "terraform"
  end_date_relative = "1m"
  refreshable = true
  groups = [
    "readers",
    "terraform"
  ]
}
resource "artifactory_access_token" "audience" {
  username          = "terraform"
  end_date_relative = "1m"
  audience = "jfrt@*"
  refreshable = true
}
resource "artifactory_access_token" "fixeddate" {
  username = "terraform"
  end_date = "2021-01-01T01:02:03Z"
  groups = [
    "readers",
    "terraform"
  ]
}
resource "time_rotating" "now_plus_1_hours" {
  rotation_hours = "1"
}

I run
go build
go install # install isn't actually installing anything; infact, it's deleting the binary. I had to manually put it where it should be:

ls ~/.terraform.d/plugins/registry.terraform.io/jfrog/artifactory/2.2.5/darwin_amd64/terraform-provider-artifactory
/Users/christianb/.terraform.d/plugins/registry.terraform.io/jfrog/artifactory/2.2.5/darwin_amd64/terraform-provider-artifactory

When I run the above terraform script against a kube cluster install of jfrog artifactory in gcp I get:
christianb@christianb-mac terraform-provider-artifactory % terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # artifactory_access_token.audience will be created
  + resource "artifactory_access_token" "audience" {
      + access_token      = (sensitive value)
      + audience          = "jfrt@*"
      + end_date          = (known after apply)
      + end_date_relative = "1m"
      + id                = (known after apply)
      + refresh_token     = (sensitive value)
      + refreshable       = true
      + username          = "terraform"
    }

  # artifactory_access_token.existing_user will be created
  + resource "artifactory_access_token" "existing_user" {
      + access_token      = (sensitive value)
      + end_date          = (known after apply)
      + end_date_relative = "5m"
      + id                = (known after apply)
      + refresh_token     = (sensitive value)
      + refreshable       = false
      + username          = "terraform"
    }

  # artifactory_access_token.fixeddate will be created
  + resource "artifactory_access_token" "fixeddate" {
      + access_token  = (sensitive value)
      + end_date      = "2021-01-01T01:02:03Z"
      + groups        = [
          + "readers",
          + "terraform",
        ]
      + id            = (known after apply)
      + refresh_token = (sensitive value)
      + refreshable   = false
      + username      = "terraform"
    }

  # artifactory_access_token.new_user will be created
  + resource "artifactory_access_token" "new_user" {
      + access_token      = (sensitive value)
      + end_date          = (known after apply)
      + end_date_relative = "1h"
      + groups            = [
          + "readers",
          + "terraform",
        ]
      + id                = (known after apply)
      + refresh_token     = (sensitive value)
      + refreshable       = false
      + username          = "terraform"
    }

  # artifactory_access_token.no_expiry will be created
  + resource "artifactory_access_token" "no_expiry" {
      + access_token      = (sensitive value)
      + end_date          = (known after apply)
      + end_date_relative = "0s"
      + id                = (known after apply)
      + refresh_token     = (sensitive value)
      + refreshable       = false
      + username          = "terraform"
    }

  # artifactory_access_token.refreshable will be created
  + resource "artifactory_access_token" "refreshable" {
      + access_token      = (sensitive value)
      + end_date          = (known after apply)
      + end_date_relative = "1m"
      + groups            = [
          + "readers",
          + "terraform",
        ]
      + id                = (known after apply)
      + refresh_token     = (sensitive value)
      + refreshable       = true
      + username          = "terraform"
    }

  # artifactory_group.test-group will be created
  + resource "artifactory_group" "test-group" {
      + admin_privileges = false
      + auto_join        = (known after apply)
      + description      = "test group"
      + id               = (known after apply)
      + name             = "terraform"
      + realm            = (known after apply)
    }

  # artifactory_user.test-user will be created
  + resource "artifactory_user" "test-user" {
      + admin                      = (known after apply)
      + disable_ui_access          = (known after apply)
      + email                      =<EMAIL_ADDRESS>      + groups                     = [
          + "readers",
          + "terraform",
        ]
      + id                         = (known after apply)
      + internal_password_disabled = (known after apply)
      + name                       = "terraform"
      + password                   = (sensitive value)
      + profile_updatable          = (known after apply)
    }

  # time_rotating.now_plus_1_hours will be created
  + resource "time_rotating" "now_plus_1_hours" {
      + day              = (known after apply)
      + hour             = (known after apply)
      + id               = (known after apply)
      + minute           = (known after apply)
      + month            = (known after apply)
      + rfc3339          = (known after apply)
      + rotation_hours   = 1
      + rotation_rfc3339 = (known after apply)
      + second           = (known after apply)
      + unix             = (known after apply)
      + year             = (known after apply)
    }

Plan: 9 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

time_rotating.now_plus_1_hours: Creating...
time_rotating.now_plus_1_hours: Creation complete after 0s [id=2020-11-18T18:28:35Z]
artifactory_access_token.refreshable: Creating...
artifactory_access_token.no_expiry: Creating...
artifactory_access_token.new_user: Creating...
artifactory_access_token.existing_user: Creating...
artifactory_access_token.fixeddate: Creating...
artifactory_access_token.audience: Creating...
artifactory_user.test-user: Creating...
artifactory_group.test-group: Creating...
artifactory_group.test-group: Creation complete after 1s [id=terraform]
artifactory_access_token.fixeddate: Creation complete after 1s [id=2840873941]
artifactory_access_token.new_user: Creation complete after 1s [id=3997111315]
artifactory_access_token.refreshable: Creation complete after 1s [id=3573456991]

Error: PUT https://christianb.jfrog.tech/artifactory/api/security/users/terraform: 400 [{Status:400 Message:Could not create user}]

  on test.tf line 21, in resource "artifactory_user" "test-user":
  21: resource "artifactory_user" "test-user" {

Error: you must specify at least 1 group when creating a token for a non-existant user - terraform, or correct the username

  on test.tf line 30, in resource "artifactory_access_token" "existing_user":
  30: resource "artifactory_access_token" "existing_user" {

Error: you must specify at least 1 group when creating a token for a non-existant user - terraform, or correct the username

  on test.tf line 42, in resource "artifactory_access_token" "no_expiry":
  42: resource "artifactory_access_token" "no_expiry" {

Error: you must specify at least 1 group when creating a token for a non-existant user - terraform, or correct the username

  on test.tf line 55, in resource "artifactory_access_token" "audience":
  55: resource "artifactory_access_token" "audience" {

christianb@christianb-mac terraform-provider-artifactory % terraform apply
time_rotating.now_plus_1_hours: Refreshing state... [id=2020-11-18T18:28:35Z]
artifactory_group.test-group: Refreshing state... [id=terraform]
artifactory_access_token.new_user: Refreshing state... [id=3997111315]
artifactory_access_token.fixeddate: Refreshing state... [id=2840873941]
artifactory_access_token.refreshable: Refreshing state... [id=3573456991]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # artifactory_access_token.audience will be created
  + resource "artifactory_access_token" "audience" {
      + access_token      = (sensitive value)
      + audience          = "jfrt@*"
      + end_date          = (known after apply)
      + end_date_relative = "1m"
      + id                = (known after apply)
      + refresh_token     = (sensitive value)
      + refreshable       = true
      + username          = "terraform"
    }

  # artifactory_access_token.existing_user will be created
  + resource "artifactory_access_token" "existing_user" {
      + access_token      = (sensitive value)
      + end_date          = (known after apply)
      + end_date_relative = "5m"
      + id                = (known after apply)
      + refresh_token     = (sensitive value)
      + refreshable       = false
      + username          = "terraform"
    }

  # artifactory_access_token.no_expiry will be created
  + resource "artifactory_access_token" "no_expiry" {
      + access_token      = (sensitive value)
      + end_date          = (known after apply)
      + end_date_relative = "0s"
      + id                = (known after apply)
      + refresh_token     = (sensitive value)
      + refreshable       = false
      + username          = "terraform"
    }

  # artifactory_user.test-user will be created
  + resource "artifactory_user" "test-user" {
      + admin                      = (known after apply)
      + disable_ui_access          = (known after apply)
      + email                      =<EMAIL_ADDRESS>      + groups                     = [
          + "readers",
          + "terraform",
        ]
      + id                         = (known after apply)
      + internal_password_disabled = (known after apply)
      + name                       = "terraform"
      + password                   = (sensitive value)
      + profile_updatable          = (known after apply)
    }

Plan: 4 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

artifactory_access_token.audience: Creating...
artifactory_access_token.existing_user: Creating...
artifactory_user.test-user: Creating...
artifactory_access_token.no_expiry: Creating...
artifactory_user.test-user: Creation complete after 1s [id=terraform]

Error: you must specify at least 1 group when creating a token for a non-existant user - terraform, or correct the username

  on test.tf line 30, in resource "artifactory_access_token" "existing_user":
  30: resource "artifactory_access_token" "existing_user" {

Error: you must specify at least 1 group when creating a token for a non-existant user - terraform, or correct the username

  on test.tf line 42, in resource "artifactory_access_token" "no_expiry":
  42: resource "artifactory_access_token" "no_expiry" {

Error: you must specify at least 1 group when creating a token for a non-existant user - terraform, or correct the username

  on test.tf line 55, in resource "artifactory_access_token" "audience":
  55: resource "artifactory_access_token" "audience" {

christianb@christianb-mac terraform-provider-artifactory % terraform apply
time_rotating.now_plus_1_hours: Refreshing state... [id=2020-11-18T18:28:35Z]
artifactory_access_token.new_user: Refreshing state... [id=3997111315]
artifactory_access_token.refreshable: Refreshing state... [id=3573456991]
artifactory_user.test-user: Refreshing state... [id=terraform]
artifactory_access_token.fixeddate: Refreshing state... [id=2840873941]
artifactory_group.test-group: Refreshing state... [id=terraform]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # artifactory_access_token.audience will be created
  + resource "artifactory_access_token" "audience" {
      + access_token      = (sensitive value)
      + audience          = "jfrt@*"
      + end_date          = (known after apply)
      + end_date_relative = "1m"
      + id                = (known after apply)
      + refresh_token     = (sensitive value)
      + refreshable       = true
      + username          = "terraform"
    }

  # artifactory_access_token.existing_user will be created
  + resource "artifactory_access_token" "existing_user" {
      + access_token      = (sensitive value)
      + end_date          = (known after apply)
      + end_date_relative = "5m"
      + id                = (known after apply)
      + refresh_token     = (sensitive value)
      + refreshable       = false
      + username          = "terraform"
    }

  # artifactory_access_token.no_expiry will be created
  + resource "artifactory_access_token" "no_expiry" {
      + access_token      = (sensitive value)
      + end_date          = (known after apply)
      + end_date_relative = "0s"
      + id                = (known after apply)
      + refresh_token     = (sensitive value)
      + refreshable       = false
      + username          = "terraform"
    }

Plan: 3 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

artifactory_access_token.audience: Creating...
artifactory_access_token.no_expiry: Creating...
artifactory_access_token.existing_user: Creating...
artifactory_access_token.audience: Creation complete after 0s [id=917914487]
artifactory_access_token.existing_user: Creation complete after 0s [id=3728992805]
artifactory_access_token.no_expiry: Creation complete after 0s [id=239402867]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
christianb@christianb-mac terraform-provider-artifactory % git branch
* (HEAD detached at jbarker/feat/access-token)
  master
christianb@christianb-mac terraform-provider-artifactory % git remote -v
jbarker git@github.com:colesgroup/terraform-provider-artifactory.git (fetch)
jbarker git@github.com:colesgroup/terraform-provider-artifactory.git (push)
origin  git@github.com:jfrog/terraform-provider-artifactory.git (fetch)
origin  git@github.com:jfrog/terraform-provider-artifactory.git (push)
christianb@christianb-mac terraform-provider-artifactory %

Hi @chb0github,
Here is the updated code to create the user tokens and avoid the race conditions.
The missing piece was linking the resources together.
I've also updated the resource documentation and examples to clarify this.
Another reference point is in the test code to see how I'm creating users dynamically.
# Required for Terraform 0.13 and up (https://www.terraform.io/upgrade-guides/0-13.html)
terraform {
  required_providers {
    artifactory = {
      source  = "registry.terraform.io/jfrog/artifactory"
      version = "2.2.5"
    }
  }
}
provider "artifactory" {
  url = "https://christianb.jfrog.tech/artifactory"
  username = "admin"
  password = "Password1"
}
resource "artifactory_group" "test-group" {
  name             = "terraform"
  description      = "test group"
  admin_privileges = false
}
//# Create a new Artifactory group called terraform
resource "artifactory_user" "test-user" {
  name     = "terraform"
  email    =<EMAIL_ADDRESS>  groups   = [
    artifactory_group.test-group.name, # depend on group before creating user
    "readers",
  ]
  password = "Password1"
}

//# Create a new Artifactory user called terraform

resource "artifactory_access_token" "existing_user" {
  username          = artifactory_user.test-user.name  # depend on user before creating the token
  end_date_relative = "5m"
}
resource "artifactory_access_token" "new_user" {
  username          = artifactory_user.test-user.name  # depend on user before creating the token
  end_date_relative = "1h"
  groups = [
    "readers",
    artifactory_group.test-group.name, # depend on group before creating the token
  ]
}
resource "artifactory_access_token" "no_expiry" {
  username          = artifactory_user.test-user.name  # depend on user before creating the token
  end_date_relative = "0s"
}
resource "artifactory_access_token" "refreshable" {
  username          = artifactory_user.test-user.name  # depend on user before creating the token
  end_date_relative = "1m"
  refreshable = true
  groups = [
    "readers",
    artifactory_group.test-group.name, # depend on group before creating the token
  ]
}
resource "artifactory_access_token" "audience" {
  username          = artifactory_user.test-user.name  # depend on user before creating the token
  end_date_relative = "1m"
  audience = "jfrt@*"
  refreshable = true
}
resource "artifactory_access_token" "fixeddate" {
  username = artifactory_user.test-user.name  # depend on user before creating the token
  end_date = "2021-01-01T01:02:03Z"
  groups = [
    "readers",
    artifactory_group.test-group.name,  # depend on group before creating the token
  ]
}
resource "time_rotating" "now_plus_1_hours" {
  rotation_hours = "1"
}

I will give it a run through again today

The updates to your TF work fine on create. However, when I go to destroy I get an error:
Error: {
  "error" : "invalid_request",
  "error_description" : "Provided access token with ID 'c041a147-ea91-452e-9b1f-166775d9e18c' was issued by a different service with ID 'jfrt@01eqc45mh93tbd16q8zck51zs4' (current service ID: 'jfrt@01eqvjaaf84mvj1890ttww16rr')"
}

Now that I know this problem occurs I am going to make sure it wasn't something on my end

Ok, I have tested it on my end and it looks good.

@chb0github @peters95 : Is there anything else I need to do for the PR?

@chb0github @peters95 : Is there anything else I need to do for the PR?

Yes. If you scroll up you will see that @peters95 requested a change:

Please clarify is this creating a new user in a group or not?

It requires other to approve - I can't approve on my own. They will be back
from vacation Monday
On Thu, Nov 26, 2020 at 1:50 PM josh-barker-coles<EMAIL_ADDRESS>wrote:

@josh-barker-coles commented on this pull request.
In docs/resources/artifactory_access_token.md
https://github.com/jfrog/terraform-provider-artifactory/pull/5#discussion_r531233883
:

+~> Note: Access Tokens will be stored in the raw state as plain-text. Read more about sensitive data in
+state.
+
+
+## Example Usages
+### Create a new Artifactory Access Token for an existing user
+```hcl
+resource "artifactory_access_token" "exising_user" {

username          = "existing-user"
end_date_relative = "5m"
+}
+```

+### Creates a new token for groups
+```hcl
@peters95 https://github.com/peters95 @chb0github
https://github.com/chb0github, Yes, this is creating a transient user
in the group.
I've updated the documentation to explain.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/jfrog/terraform-provider-artifactory/pull/5#discussion_r531233883,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AALIRUEFT2KEWN5N5JEEMXTSR3ESVANCNFSM4TAJ7PNQ
.

--
Christian Bongiorno

any chance this will be officially release soon?

I am working on releasing it today.

released:
https://registry.terraform.io/providers/jfrog/artifactory/latest


Issue with node v22.0.0

Environment
node 22.0.0
repack 3.7.0 also in 4.0.0-rc.2
react-native 0.74.1
macos ventura 13.6.1
xcode 15.0.1
Description
Can´t build app using node v22.0.0. Work´s fine if switch to node 20.0.0 (using nvm). The same problem on Android. Both tests are on real devices.
I tried a clean and empty new project and have the same problem.
[DevServer] Server listening at http://<IP_ADDRESS>:8081
[DevServer] GET 200 /status request completed { responseTime: 5.667741000652313 }
[LoggerPlugin] Compiling ios: 3% setup (watch run)
[LoggerPlugin] Compiling ios: 23% building (2/8 entries 100/164 dependencies 4/80 modules)
[LoggerPlugin] Compiling ios: 23% building (2/8 entries 794/900 dependencies 82/296 modules)
[LoggerPlugin] Compiling ios: 30% building (3/8 entries 1364/1468 dependencies 160/364 modules)
[LoggerPlugin] Compiling ios: 30% building (3/8 entries 1580/1680 dependencies 200/436 modules)
[LoggerPlugin] Compiling ios: 30% building (3/8 entries 1827/1900 dependencies 242/503 modules)
[LoggerPlugin] Compiling ios: 30% building (3/8 entries 2639/2668 dependencies 493/600 modules)
node:internal/event_target:1096
  process.nextTick(() => { throw err; });
                           ^

Error [ERR_UNHANDLED_ERROR]: Unhandled error. ({})
    at Compiler.emit (node:events:509:17)
    at Worker.<anonymous> (/Users/hebert/App/AwesomeProject/node_modules/@callstack/repack/dist/webpack/Compiler.js:97:14)
    at Worker.emit (node:events:520:28)
    at MessagePort.<anonymous> (node:internal/worker:263:53)
    at [nodejs.internal.kHybridDispatch] (node:internal/event_target:822:20)
    at MessagePort.<anonymous> (node:internal/per_context/messageport:23:28) {
  code: 'ERR_UNHANDLED_ERROR',
  context: {}
}

i have same issue. Anyone has any solution?

Thanks for the report!
Fix merged, will be available later this week.

i have same issue. Anyone has any solution?

While the fix not merged to main project you can switch to any older node version. If using nvm this is very simples as "nvm use 20.0.0" ...


Generate InetAdresses for ipv4 and ipv6

The PR adds generation of InetAddress objects both for IPv4 and IPv6

Nice one!


Fix failing CI test - loosen requirements in a too demanding test

Signed-off-by: Tomasz Baranski<EMAIL_ADDRESS>
What this PR does / why we need it:
The failing test suggest a better way to get DV's StorageClass. This PR makes the VolumeSnapshotStatus to be calculated without waiting for the underlying PVC.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #
Special notes for your reviewer:
Release note:

NONE

/cc @mhenriks

@tomob looks like some unit tests fail now. Also the other lanes don't seem yet to be happy about the change.

/retest

/retest

/test pull-kubevirt-e2e-k8s-1.17

/retest

/retest

/retest

/retest

/retest

/lgtm
/approve

/retest

/override pull-kubevirt-e2e-k8s-1.17

This one looks interesting, is this new or was it there before: https://prow.apps.ovirt.org/view/gcs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/4462/pull-kubevirt-e2e-k8s-1.17/1328617864038453248 ?

This one looks interesting, is this new or was it there before: https://prow.apps.ovirt.org/view/gcs/kubevirt-prow/pr-logs/pull/kubevirt_kubevirt/4462/pull-kubevirt-e2e-k8s-1.17/1328617864038453248 ?

I tried it a couple of times. I cannot reproduce the test.
> export | grep KUBE
KUBECONFIG /home/tbaransk/work/repos/cnv/kubevirt/_ci-configs/k8s-1.17/…
KUBEVIRT_PROVIDER k8s-1.17

> make functest
[...]
Running Suite: Tests Suite
==========================
Random Seed:<PHONE_NUMBER>
Will run 1 of 694 specs

JUnit report was created: /home/tbaransk/work/repos/cnv/kubevirt/_out/artifacts/junit.functest.xml

Ran 1 of 694 Specs in 23.604 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 693 Skipped
PASS | FOCUSED

Ginkgo ran 1 suite in 25.448289919s
Test Suite Passed
Detected Programmatic Focus - setting exit status to 197

/retest

/retest


feat(store): snapshot

:tada: This PR is included in version 1.0.0-alpha.2 :tada:
The release is available on:

GitHub release
v1.0.0-alpha.2

Your semantic-release bot :package::rocket:


Ignore empty array normalized values

Empty array normalized values should be ommited, otherwise it end up included to query

Do you have a real-world query that fails?

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (66cd95d) 100.00% compared to head (88fd8b9) 100.00%.
Report is 4 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff             @@
##                main       #11   +/-   ##
===========================================
  Coverage     100.00%   100.00%
- Complexity       674       675    +1
===========================================
  Files            121       121
  Lines           1962      1962
===========================================
  Hits            1962      1962

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Do you have a real-world query that fails?

@shyim not really to be honest.. I'm migrating my app from Elastic to Opensearch and Ongr dsl library to yours dsl and some internal tests started to fail which test Search->toArray() output. Even though queries works fine, IMO it shouldnt pass empty data. WDYT?

P.S. ongr has array_filter which removes empty values: https://github.com/ongr-io/ElasticsearchDSL/blob/master/src/Search.php#L779

Makes sense. Maybe I broke it while I removed Symfony Serializer.

@shyim thanks! Would it be possible to tag a new release?

I tagged a release :)


Handle small issues with BioModels XML files.

Some BioModel files are missing part the of string for the "metaid" tag in their XML. For example, model 433 has this XML for its second line

AWE (via libsbml) is unable to read this. The correct format is

This error can easily be detected and resolved by AWE.

Hi! I looked at the what I assumed was model 433 (BIOMD0000000433), and wasn't able to identify where the error is. I also tried looking at the converted SBML version.
Is this the correct model and would it be possible to provid a screenshot of the issue?
Thanks!


ThreeFold Connect app - undefined is not an object (evaluating 'e.find') error message

I was getting this error message from the ThreeFold Connect mobile app on ios 14.4 app version 2.3.1 multiple times today while was trying to make a transaction to deploy a new VDC on vdc.devnet.grid.tf

the message displayed after scanning the QR and confirm.
worth mention that trying the deployment on testnet at that time failed too with an error message shown in the screenshot below

@JonasWijne
Originally logged as critical but it was just service being down, might be worth having a look at the error handling on our end though. Could you see anything that could explain an error like this being returned from our end?
Maybe a wrong value returned from the payment service? Something we don't handle correctly?


ImportError: No module named vnctpmd

环境

操作系统: macos
Anaconda版本: 如Anaconda 4.0.0 Python 2.7.13 |Anaconda 4.0.0 (x86_64)
vn.py版本: Version: 1.9.0
python run.py
Traceback (most recent call last):
File "run.py", line 28, in
from vnpy.trader.gateway import (ctpGateway, ibGateway)
File "/Users/fanren/.pyenv/versions/anaconda2-4.0.0/lib/python2.7/site-packages/vnpy/trader/gateway/ctpGateway/init.py", line 5, in
from .ctpGateway import CtpGateway
File "/Users/fanren/.pyenv/versions/anaconda2-4.0.0/lib/python2.7/site-packages/vnpy/trader/gateway/ctpGateway/ctpGateway.py", line 16, in
from vnpy.api.ctp import MdApi, TdApi, defineDict
File "/Users/fanren/.pyenv/versions/anaconda2-4.0.0/lib/python2.7/site-packages/vnpy/api/ctp/init.py", line 4, in
from .vnctpmd import MdApi
ImportError: No module named vnctpmd

进入vnpy/api/ctp目录，运行bash build.sh，build目录 下 找不到vnctpmd.so和vnctptd.so

运行bash build.sh报错了吗？

运行bash build.sh报错了吗？

sudo bash build.sh
Password:
~/LH/chinahadoop/vnpy-master/vnpy/api/ctp/build ~/LH/chinahadoop/vnpy-master/vnpy/api/ctp
-- The C compiler identification is AppleClang <IP_ADDRESS>00039
-- The CXX compiler identification is AppleClang <IP_ADDRESS>00039
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Under unix: 8
CMake Warning at /usr/local/Cellar/cmake/3.12.1/share/cmake/Modules/FindBoost.cmake:577 (message):
Imported targets and dependency information not available for Boost version
(all versions older than 1.33)
Call Stack (most recent call first):
/usr/local/Cellar/cmake/3.12.1/share/cmake/Modules/FindBoost.cmake:959 (_Boost_COMPONENT_DEPENDENCIES)
/usr/local/Cellar/cmake/3.12.1/share/cmake/Modules/FindBoost.cmake:1618 (_Boost_MISSING_DEPENDENCIES)
CMakeLists.txt:83 (find_package)
CMake Warning at /usr/local/Cellar/cmake/3.12.1/share/cmake/Modules/FindBoost.cmake:577 (message):
Imported targets and dependency information not available for Boost version
(all versions older than 1.33)
Call Stack (most recent call first):
/usr/local/Cellar/cmake/3.12.1/share/cmake/Modules/FindBoost.cmake:959 (_Boost_COMPONENT_DEPENDENCIES)
/usr/local/Cellar/cmake/3.12.1/share/cmake/Modules/FindBoost.cmake:1618 (_Boost_MISSING_DEPENDENCIES)
CMakeLists.txt:83 (find_package)
CMake Warning at /usr/local/Cellar/cmake/3.12.1/share/cmake/Modules/FindBoost.cmake:577 (message):
Imported targets and dependency information not available for Boost version
(all versions older than 1.33)
Call Stack (most recent call first):
/usr/local/Cellar/cmake/3.12.1/share/cmake/Modules/FindBoost.cmake:959 (_Boost_COMPONENT_DEPENDENCIES)
/usr/local/Cellar/cmake/3.12.1/share/cmake/Modules/FindBoost.cmake:1618 (_Boost_MISSING_DEPENDENCIES)
CMakeLists.txt:83 (find_package)
CMake Warning at /usr/local/Cellar/cmake/3.12.1/share/cmake/Modules/FindBoost.cmake:577 (message):
Imported targets and dependency information not available for Boost version
(all versions older than 1.33)
Call Stack (most recent call first):
/usr/local/Cellar/cmake/3.12.1/share/cmake/Modules/FindBoost.cmake:959 (_Boost_COMPONENT_DEPENDENCIES)
/usr/local/Cellar/cmake/3.12.1/share/cmake/Modules/FindBoost.cmake:1618 (_Boost_MISSING_DEPENDENCIES)
CMakeLists.txt:83 (find_package)
CMake Warning at /usr/local/Cellar/cmake/3.12.1/share/cmake/Modules/FindBoost.cmake:577 (message):
Imported targets and dependency information not available for Boost version
(all versions older than 1.33)
Call Stack (most recent call first):
/usr/local/Cellar/cmake/3.12.1/share/cmake/Modules/FindBoost.cmake:959 (_Boost_COMPONENT_DEPENDENCIES)
/usr/local/Cellar/cmake/3.12.1/share/cmake/Modules/FindBoost.cmake:1618 (_Boost_MISSING_DEPENDENCIES)
CMakeLists.txt:83 (find_package)
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE
CMake Error at /usr/local/Cellar/cmake/3.12.1/share/cmake/Modules/FindBoost.cmake:2044 (message):
Unable to find the requested Boost libraries.
Unable to find the Boost header files.  Please set BOOST_ROOT to the root
directory containing Boost or BOOST_INCLUDEDIR to the directory containing
Boost's headers.
Call Stack (most recent call first):
CMakeLists.txt:83 (find_package)
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
CTPAPI_MD_LIBRARY
linked by target "vnctpmd" in directory /Users/fanren/LH/chinahadoop/vnpy-master/vnpy/api/ctp
CTPAPI_TD_LIBRARY
linked by target "vnctptd" in directory /Users/fanren/LH/chinahadoop/vnpy-master/vnpy/api/ctp
-- Configuring incomplete, errors occurred!
See also "/Users/fanren/LH/chinahadoop/vnpy-master/vnpy/api/ctp/build/CMakeFiles/CMakeOutput.log".
☁  ctp

not available for Boost version(all versions older than 1.33)

直接 brew install boost 这个吗 还是
下面哪个
brew search boost
==> Formulae
boost                 boost-build           boost-python<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>boost-bcp             boost-mpi             boost-python3<EMAIL_ADDRESS><EMAIL_ADDRESS>==> Casks
boostnote                 focus-booster             iboostup                  mongobooster              turbo-boost-switcher
☁  ~

brew install boost 按装之后
又运行
sudo bash build.sh
Password:
~/LH/chinahadoop/vnpy-master/vnpy/api/ctp/build ~/LH/chinahadoop/vnpy-master/vnpy/api/ctp
-- The C compiler identification is AppleClang <IP_ADDRESS>00039
-- The CXX compiler identification is AppleClang <IP_ADDRESS>00039
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Under unix: 8
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE
CMake Error at /usr/local/Cellar/cmake/3.12.1/share/cmake/Modules/FindBoost.cmake:2044 (message):
Unable to find the requested Boost libraries.
Boost version: 1.67.0
Boost include path: /usr/local/include
Could not find the following Boost libraries:
      boost_python

Some (but not all) of the required Boost libraries were found.  You may
need to install these additional Boost libraries.  Alternatively, set
BOOST_LIBRARYDIR to the directory containing Boost libraries or BOOST_ROOT
to the location of Boost.
Call Stack (most recent call first):
CMakeLists.txt:83 (find_package)
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
CTPAPI_MD_LIBRARY
linked by target "vnctpmd" in directory /Users/fanren/LH/chinahadoop/vnpy-master/vnpy/api/ctp
CTPAPI_TD_LIBRARY
linked by target "vnctptd" in directory /Users/fanren/LH/chinahadoop/vnpy-master/vnpy/api/ctp
-- Configuring incomplete, errors occurred!
See also "/Users/fanren/LH/chinahadoop/vnpy-master/vnpy/api/ctp/build/CMakeFiles/CMakeOutput.log".

CTP的API不支持MACOS

mini的2个gateway好像也的注释到，和ctp互斥，否则也会报错ImportError: No module named vnctpmd


Add: OpenBCI Wifi direct to Laptop/Phone

Directly connect the OpenBCI to a laptop with WIFI instead of going through a router.
Some university and hospital WIFI networks might be difficult work with.
Please +1 if you're interested!

By "direct" do you mean with the shield being the AP?

Yes exactly!

I think this would be the easiest option for end-users to work with the AP and would vastly simplify the setup. If you want I can ask some neuro-researchers I know.

great! I will plan to add this in the first software update 1.1.0!
Thanks @pawelchojnacki I'm so excited to simply your experience. For now, keep in mind you may use a cell phone hotspot to create a little local network that will allow your computer to still have internet access while at the same time give a simple setup. I heard you had trouble finding the device on the network, the new GUI will be released soon which uses SSDP to discover the device on the network.

My problem was with network sharding, where Copernicus Science Center in Warsaw created subnets where participants weren't able to see each other.
Please keep in mind that some phones have a hard limitation to allow only one machine to connect to its network, and tethering the internet to the PC over the wire may put it in a different subnet as well.

Never knew that about the phones, I have an iPhone that allows multiple wireless connections.
That's also really fascinating to hear about the way you were on the same network and you couldn't take to a device within that same network. SSDP is the same protocol printers use so something is not right here, if you're on the same network you should be able to broadcast search for it. None the less, makes this issue more important.

Some places (like conference centers) deliberately block you from seeing other parts of the network / other guests to limit the possibility of scamming / MITM attacks. I think it can be a frequent problem when showcasing OpenBCI at different hackathons / conferences.

:+1: I'm interested, and might know how to fix this :) :rocket:

Hey this is in progress! If you checkout the dev branch here, there is wifi direct code in the examples/WifiShield_TCPUDP/WifiShield_TCPUDP and the connection proposal is on a flow chart on google drive

@aj-ptw
I think this issue should be closed since you merged: #57  for direct wifi


Feature: make Keycloak realm configurable

Make Keycloak realm configurable in dev setup in order to enable usage of admin with other realms than SpringBootKeycloak of shogun-boot.
Plz review @terrestris/devs

Nice.
Maybe we could replace (in a follow up PR) the <<INSERT_…>> stuff with the use of env variables: https://webpack.js.org/guides/environment-variables/

See #49


Font is getting disturbed in Chrome

Link to PDF file (or attach file here):
1.http://www.rightprospectus.com/documents/Nuveen/SUM_AllAmericanMunicipalBondFund.pdf
2.http://www.rightprospectus.com/documents/Nuveen/PRO_AllAmericanMunicipalBond.pdf
Configuration:

Web browser and its version:Chrome
Operating system and its version:Windows 7
PDF.js version:1.1.469
Is an extension:

Steps to reproduce the problem:

Open First link with Pdf Js Viewer in Chrome
2.In the same tab Open Second link with Pdf Js Viewer.(Chrome)
3.We will able to see the disturbed font as in attached Picture.
4.But if we open the second link for the first time , there is no issue.
5.A User in our system always navigate to documents in same tab , so we are facing this issue.
6.Issue is in second link provided above.

What is the expected behavior? (add screenshot)

What went wrong? (add screenshot)

Link to a viewer (if hosted on a site other than mozilla.github.io/pdf.js or as Firefox/Chrome extension):

We Tried applying fix by https://github.com/mozilla/pdf.js/pull/6715/commits/87762afec477b0951681df65e6a8286f9d5a1c6a , provided by pdf js team.
This does not work for us. Can you please provide any suggestions or code changes that we can apply here to solve the issue and please let us know any problem with the pdf.
Thanks in Advance!!!

You are using a very old version of PDF.js. Many font fixes have been applied since. Try to open the file with https://mozilla.github.io/pdf.js/web/viewer.html (Open File button in the toolbar) to see if the problem remains.

Hi, We integrated latest version of PDF.js(v1.4.20), issue still persist. It happens only in chrome when we open below two documents in same tab one after another.
1.http://www.rightprospectus.com/documents/Nuveen/SUM_AllAmericanMunicipalBondFund.pdf
2.http://www.rightprospectus.com/documents/Nuveen/PRO_AllAmericanMunicipalBond.pdf
Could you please verify if something is wrong with above PDF documents related to font settings?
We tried to open these documents using demo URL-  https://mozilla.github.io/pdf.js/web/viewer.html it works fine. Demo URL uses open file button in toolbar whereas we pass PDF url in query string. When we use open file button, its working with our old version also.
Do we have any way to open PDF in demo site using query string?

FYI, I've also encountered this issue a couple of times in the past month when I was reading papers for research. After opening the PDF Viewer (Chromium 51, Linux), the fonts looked a bit weird (sometimes thin, sometimes doubled like the picture above).
@gsampath127 Can you try the beta version? If that doesn't solve the issue, can you apply the patch at #7347 and see whether it resolves the problem (note: this is just a font issue, I didn't check whether it is related to your test case).

Hi, We tried by integrating with beta version, issue is still present. We also tried to apply the patch at #7347 but it did not work for us.
Please suggest.

Hi, Could you please look into above issue? Please let us know if something is wrong in PDF so that we can get those PDF's corrected.
Thanks.

Hi Team, Could you please look into above issue? or  Please let us know if something is wrong in PDF so that we can get those PDF's corrected.
This issue is a kind of work stopper for us and we have tried all options from our side.
Thanks.

I cannot reproduce the issue using Firefox 47.0.1 on Arch Linux with the latest PDF.js development version. I really think this is some kind of regression in Chrome that should be reported upstream if it cannot be reproduced with other browsers.

Base on the conversation above, problem occurs for custom solution and reported didn't provide steps to reproduce the issue. Closing as incomplete. The workaround was found at https://github.com/mozilla/pdf.js/issues/7462#issuecomment-232102161, however it does not guarantee a correct solution and without steps to reproduce it's impossible to validate/verify it.


Got error when creating a new space  -- 413 Request Entity Too Large

just reporting   not for you tofix.
413 Request Entity Too Large

HTTP/1.1 413 Request Entity Too Large
Server: nginx/1.18.0 (Ubuntu)
Date: Wed, 14 Jun 2023 13:24:14 GMT
Content-Type: text/html
Content-Length: 192
Connection: close

Description field has charachter limit of 100, there should be no problem a space with these specifications.
I have created a replica and not encountered with an issue as shown below.


CAMEL-9570: Change CamelNamespaceHandler so that is doesn't force Blu…

…eprint Container to create objects too early

Hold off on this one for a bit - I think I've found a better way that will ensure backward compatibility and not required the temporary CamelContext

Never mind - I tried injecting the Camel Context into the CamelDependenciesFinder, but that causes the same issue with the Blueprint proxies.  Stick with this PR as-is.
I've tested it with my local projects, and I think it's good to go

I get an error in camel-test-blueprint with this PR
Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 5.309 sec <<< FAILURE! - in org.apache.camel.test.blueprint.BlueprintPackageScan2Test
testRoute(org.apache.camel.test.blueprint.BlueprintPackageScan2Test)  Time elapsed: 3.196 sec  <<< ERROR!
java.lang.RuntimeException: Unable to initialize bean .camelBlueprint.factory.camel-1
Caused by: org.osgi.service.blueprint.container.ComponentDefinitionException: Unable to initialize bean .camelBlueprint.factory.camel-1
Caused by: org.apache.camel.RuntimeCamelException: Error during post processing of bean Routes: []
Caused by: org.apache.camel.NoSuchBeanException: Cannot lookup: foo from registry: org.apache.camel.blueprint.BlueprintContainerRegistry@54cef2f8 with expected type: interface org.apache.camel.Endpoint due: Unable to initialize bean .camelBlueprint.bean.factory.foo
Caused by: org.osgi.service.blueprint.container.ComponentDefinitionException: Unable to initialize bean .camelBlueprint.bean.factory.foo
Caused by: org.apache.aries.blueprint.di.CircularDependencyException: [BeanRecipe[name='camel-1'], BeanRecipe[name='.camelBlueprint.factory.camel-1'], BeanRecipe[name='foo'], BeanRecipe[name='.camelBlueprint.bean.factory.foo'], BeanRecipe[name='camel-1']]

Do you see that error also? I will run this again to double check, and then also with the PR reverted as well.

Okay first run got these errors
Failed tests:
org.apache.camel.test.blueprint.PropertyInjectTest.testPropertyInject(org.apache.camel.test.blueprint.PropertyInjectTest)
  Run 1: PropertyInjectTest.testPropertyInject:39->CamelTestSupport.assertMockEndpointsSatisfied:875 mock://result Body of message: 0. Expected: <Hello> but was: <null>
  Run 2: PropertyInjectTest.testPropertyInject:39->CamelTestSupport.assertMockEndpointsSatisfied:875 mock://result Body of message: 0. Expected: <Hello> but was: <null>
  Run 3: PropertyInjectTest.testPropertyInject:39->CamelTestSupport.assertMockEndpointsSatisfied:875 mock://result Body of message: 0. Expected: <Hello> but was: <null>

Tests in error:
org.apache.camel.test.blueprint.BlueprintPackageScan2Test.testRoute(org.apache.camel.test.blueprint.BlueprintPackageScan2Test)
  Run 1: BlueprintPackageScan2Test>CamelBlueprintTestSupport.setUp:241->CamelBlueprintTestSupport.createBundleContext:175 » Runtime
  Run 2: BlueprintPackageScan2Test>CamelBlueprintTestSupport.setUp:241->CamelBlueprintTestSupport.createBundleContext:175 » Runtime
  Run 3: BlueprintPackageScan2Test>CamelBlueprintTestSupport.setUp:241->CamelBlueprintTestSupport.createBundleContext:175 » Runtime

org.apache.camel.test.blueprint.management.ManagedEndpointInjectRefEndpointTest.testRef(org.apache.camel.test.blueprint.management.ManagedEndpointInjectRefEndpointTest)
  Run 1: ManagedEndpointInjectRefEndpointTest.testRef:58 » CamelExecution Exception occ...
  Run 2: ManagedEndpointInjectRefEndpointTest.testRef:58 » CamelExecution Exception occ...
  Run 3: ManagedEndpointInjectRefEndpointTest.testRef:58 » CamelExecution Exception occ...

Tests run: 102, Failures: 1, Errors: 2, Skipped: 0

Yes I get the same errors on the 2nd run as well with this PR

Let me look at that - it's strange that what I've done would cause a circular dependency in the blueprint.

Yeah you are likely almost there, but osgi and blueprint is hard. Without this PR all the unit tests passes.

Also maybe some of those test failures may surfaced an issue in the test with your PR. However I haven't looked at those tests why the fail, but it has known in the past to happen.

An alternative way could maybe be in that dependency finder to collect those "links" and then later do some kind of osgi service reference or something to make the blueprint graceful wait, or some other way of waiting until they are there, and make blueprint be in grace wait mode.

Well I'm stuck :-)
After pounding my head on this one test failure for the past couple of days, I don't really know if I can "fix" this.  The one test that fails looks like a completely valid condition/setup/route to me, but the PR breaks the test.  Somehow the change introduces a cyclic dependency - the endpoint depends on the camel context which depends on the endpoint - BOOM.
Digging through the code a bit, I think I see how this is happening.  There are several classes in the org.apache.camel.blueprint package that do something like blueprintContainer.getComponentInstance(camelContextId).  I think this may be causing the circular reference - Blueprint is trying to create the object (in this case, and endpoint) in order to inject it into the Camel Context, and then the object factory tries to get the Camel Context from Blueprint to create the object.
From my earlier experiments, it appeared that if blueprintContainer.getComponentInstance(...) was called during parsing/initialization phase of Blueprint lifecycle, my dynamic services were broken.  With these calls embedded in multiple factory classes (I counted at least 8), this problem just got much bigger.  I'll have to look at the whole parse/construction process again to see if I can come up with a good way to fix my services issue in a backward compatible fashion, but this one may be out of my league.  I'm very comfortable working with the OSGi/Blueprint pieces, but the tangle of the Camel pieces wrapped around the OSGi/Blueprint pieces is a little beyond me right now.
Personally, I think the circular dependency in the code is a real issue, regardless of my services issue.  If you have any suggestions on how to setup the factories to get rid of the dependency problem, I'd love to hear about it.

I think you are continued this effort on
https://github.com/apache/camel/pull/1297
If so maybe this PR can be closed

This PR is trying to address a separate issue - the issue in CAMEL-9570 with Blueprint service proxies.  The 1297 PR is for improvement CAMEL-10513, which is different.  I had mixed the two of them up in CAMEL-9570 before, but I created a separate JIRA and PR just for the startup behavior.
Although this PR does fix CAMEL-9570, it breaks other functionality - so it obviously can't be merged to master.  However, I was hoping someone could take a look at it and give me some ideas on how to proceed.  CAMEL-9570 is one of the most problematic issues I'm facing from an operational standpoint.

Closing old PRs


Clean up abandoned jobs

If a job fails to complete (because girder is stopped, for instance), an item is marked as waiting for a large_image.  The UI doesn't allow cancelling (the large_image information can be deleted via a REST call, though).
Add an endpoint that would clean up all jobs that are in-progress and over a given age, since they are probably stuck.

Added endpoint in PR #113.


Getting Error Variable names are not unique when using .read_10x_h5 function

Good Evening,
My goal here is to get either a Gene barcode or dense matrix from a .h5 file from 10x. I'm currently trying to use the .read_10x_h5() function to help me achieve this. To my understanding I just need to input the file name into the function. When I run my code (See below), I get an error stating that "Variable names are not unique. To make them unique, call .var_names_make_unique."  From the documentation I don't see a way that I can call .var_names_make_unique(). Is there some preprocessing that I'm missing?
user_input = input("Enter the path of your file: ")
def convert_h5_to_adata(filename):
    filename = str(filename)
    if os.access(filename, os.R_OK):
        sc.read_10x_h5(filename)
        return
convert_h5_to_adata(user_input)

Hi @knapii-developments,
You are missing a return value for the sc.read_10x_h5() function. That function will return your anndata object. You can call .var_names_make_unique on that object like this:
adata = sc.read_10x_h5(filename)
adata.var_names_make_unique()

Thanks @LuckyMD.
When I ran the script, I noticed that the output of adata.X is 0 based and not 1 based. Is that expected? When reading about the MEX format, 10X does mention that the index in the MEX file is 1-based. It was surprising to see when I ran the function against a sample dataset from 10x to see 0s  in anndata.X. Can you provide your thoughts?

I got this same problem, how to doing?


No Logger set

Hey,
I always get these messages when the task runs. Any hints?
PHP Deprecated:  No logger set for RoboFtp\FtpDeployTask. Use $this->task(Foo::class) rather than new Foo() in loadTasks to ensure the builder can initialize task the task, or use $this->collectionBuilder()->taskFoo() if creating one task from within another. in \vendor\consolidation\robo\src\Common\TaskIO.php
on line 43
Deprecated: No logger set for RoboFtp\FtpDeployTask. Use $this->task(Foo::class) rather than new Foo() in loadTasks to ensure the builder can initialize task the task, or use $this->collectionBuilder()->taskFoo() if creating one task from within another. in \vendor\consolidation\robo\src\Common\TaskIO.php on li
ne 43

It looks like Robo probably deprecated the API this plugin was written with. I'm not able to maintain this plugin right now, but it should be a case of looking for recent changes in Robo and then opening a pull request to update to the newer API.


Discussion: Additional AQL DATE functions

It started out at https://github.com/arangodb/arangodb/issues/317#issuecomment-136006829, continued in the comments of a gist and is to be continued here.
Implementation status (aql-date-extra branch):

[x] DATE_LEAPYEAR():
true if divisible by 4, but not 100, except 400; false otherwise
[x] DATE_DAYOFYEAR():
1..365 in a regular year, 1..366 in a leap year
[x] DATE_ISOWEEK():
1..53, week date according to ISO8601 (monday=1st day of week, no fractional weeks)
[x] DATE_QUARTER():
1..4; Jan/Feb/Mar=1, Apr/May/Jun=2, Jul/Aug/Sep=3, Oct/Nov/Dec=4
[ ] DATE_CALC():
add or subtract a given amount of time units from date.
TODO: rename to DATE_ADD() / DATE_SUBTRACT(), support ISO durations, discuss if mixed-sign ISO durations are useful enough to be added
[ ] DATE_DIFF():
Calculate difference between two dates, in a given time unit, optionally with decimal fraction.
TODO: Testing, reconsider time unit naming (singular vs. plural, min vs. i)
[ ] Additional DATE_*() functions:
TODO: Assess usefulness of functions like "is birthday", "last day of month/week"...
Favor generic solutions, so that AQL queries stay semantically meaningful even if the functions are used differently than anticipated. Counter example: DATE_IS_BIRTHDAY(DATE_ADD(date, 7, "days")) to compare month and day (better would be: DATE_PART(), DATE_MATCH() or DATE_COMPARE()).

remove "maybe this isn't what we want to do?" from my comment

Done locally, will be in next commit.

with the ambiguity of "year" as a unit, I think this is a better implementation, and moment.js does the same. It just needs to be documented.

The diff between last day of Feb 2015 and last day of Feb 2016 should be exactly 1 year, because this is probably what is expected ("end of Feb", not +365 days). That should be documented of course.

We then generally prefer to apply the same patterns to different situations [...]

I fully agree that we should offer a few, but powerful functions instead of dozens of specific functions that everyone keeps forgetting about. It also reduced the "risk" of nonsensical queries like your examplary DATE_IS_BIRTHDAY(DATE_ADD(e.date, 7, "d")) for general purpose month and day comparisons.
If I understand SQL's DATEPART() right, than it's basically a generic version of all the DATE functions in AQL, that return a single component of a date (DATE_YEAR, DATE_MONTH, ...). I don't think we need it, because it's redundant and less expressive. On the other hand, it would allow for a static query that returns the date component based on a bind parameter - but that doesn't sound like a real-world use case.
Your suggestion to have a function like DATE_PART() that returns multiple components is interesting though. I believe the following would work:
LET birthday = "2000-03-15"
// DATE_NOW() = "2015-03-07"
DATE_PART(birthday, ["month", "day"]) == DATE_PART(DATE_NOW(), ["month", "day"])
// [3, 15] == [3, 7] ?

Downside: two calls to DATE_PART() with identical unit arrays. Maybe this is a better solution:
DATE_COMPARE(birthday, DATE_NOW(), ["month", "day"])

It could also replace:
 DATE_YEAR(date) == DATE_YEAR(DATE_NOW())
// by
DATE_COMPARE(date, DATE_NOW(), "year")

It would only work for "is on ... ?" style queries though. We might also provide a better alternative / syntactically sugared way for:
LET now = "2015-09-07 15:30:00.000"
DATE_HOUR(now) >= 12 AND DATE_HOUR(now) < 0 // PM (or is it AM until 1PM?)

This isn't needed for range checks like "after 2015-09-01 and before 2015-10-01", because of the sort properties of ISO strings, date > "2015-09-01" AND date < "2015-10-01" works. If you want to exclude years (for season check for instance), than SUBSTRING(date, 5) > ... should work. But that may not be apparent (same goes for sortability of ISO strings).
Thus, a semantically more meaningful function would help. It would allow for complex range checks like "around noon in spring", which is really tough to do in current AQL (day > 20 and day < 21 doesn't work, several combined conditions are required):
(DATE_MONTH(date) == 3 AND DATE_DAY(date) >= 20) OR (DATE_MONTH(date) == 6 AND DATE_DAY(date) < 21) OR (DATE_MONTH(date) > 3 AND DATE_MONTH(date) < 6) ...

I'm not sure if DATE_COMPARE() should have a secondary signature to take 3 dates + an array of units, or if it should be a distinct function called DATE_BETWEEN() or DATE_IN_RANGE(), but it would look like this:
DATE_IN_RANGE(DATE_NOW(), "0000-03-20 11:00", "0000-06-20 14:00", ["month", "day", "hour"])

There are a lot of things to consider here: should the range be inclusive? If years, month or days aren't to be compared, is zero-ing them really the best way? If they aren't compared anyway, than it shouldn't matter what there value is. So zeroes would be just a convention to emphasize that a certain date component is ignored. On the other hand, you might want to fully exclude them - but than we can't be ISO8601 compliant and not accept actual dates as 2nd and 3rd parameter. Instead, we would pass arrays like:
DATE_IN_RANGE(DATE_NOW(), [3, 20, 11], [6, 20, 14], ["m", "d", "h"])

The 4th argument describes what every value at he corresponding index means (which time unit it is). But it may be a little cryptic after all.
Instead of ["m", "d", "h"], we could support "mdh" as a shortcut. This would obviously work for abbreviated time units only. We would need minutes to be i instead of (or in addition to) min. Not sure if we should support q for quarter and w for week here... it would require the 2nd and 3rd argument to be arrays, because we can't express quarters and weeks in normal ISO date strings. A syntax similar to ISO durations could be developed however: DATE_IN_RANGE(DATE_NOW(), "3M20D11H/6M20D14H"), 1Q/3Q (for 1st to 3rd or 1st to 2nd quarter, depending on whether it would be inclusive or not) etc. But it's super cryptic for human eyes, could be confused with ISO durations (prefixed with P) and not all combinations are possible (quarters and months easily conflict).
Speaking of ISO durations: A range can also be expressed as "2015-09-07/P3M23D". I don't think it's worth to implement however, because you could do "2015-09-07", DATE_ADD(..., "3M23D") or the more verbose variants thereof.

I think time units as well as other units are typically expressed as singular outside a given context, [...] but I agree days just looks better

We can also have both, that shouldn't make a huge difference in terms of performance and memory consumption. Since I switched away from switch (haha) to object key lookups, the time to resolve a time unit should be constant as in O(1). Adding a few more aliases / duplicates to the lookup objects would require like 200 bytes of additional memory. Not very worrisome. It's more of a cosmetical issue if people write queries like 1, "days" or 5, "year".

Have you considered "i" for minute? That would make DATE_DIFF()s unit arguments compatible with php's date_format, and MySql also uses "%i" for minutes.

No, but that's a reasonable and simple change. Should we keep "min" / "mins" though?

DATE_ADD(date, amount, unit) feels more intuitive to me, and it's more similar to mysql. [...] Not sure if you're talking about an extra line of code in Executor.cpp, better handling of memory, or if it's plain impossible with ISO duration.

It definitely possible. It would be a slight change in Executor.cpp from ns,n,s to ns,ns|s to allow strins and numbers as 2nd argument and optionally a string as 3rd. It will require type checking in aql.js however. A call date, string1, string2 would mean that string1 is an iso duration, and that a possible 3rd parameter will be silently ignored - although one could also make it raise an error. In a call like date, number, string, number would be the amount and string the time unit - and we need to ensure that string !== undefined, because the 3rd parameter is optional by definition, yet required for this signature (I don't think it's possible to define independent signatures in Executor.cpp). If this is any bad for optimization, I'm as clueless as you.

I've reconsidered slightly and now I'm in favor of
DATE_COMPARE(birthday, DATE_NOW(), "month", "day")

I think you are right that DATE_COMPARE() is a better name than DATE_MATCH(). It reduces the risk that people would get online dating search results looking for the AQL documentation anyway... :)
Regarding switching from array to two arguments: I think most people would prefer a range of units, such as "hour" to "second", rather than "hour" and "second" (but not minute). Two separate (both optional) arguments would make that easier to express. Also if you want just one unit it would be more natural than ["year"] (although you suggested that anyway).
If someone wants to check if the year and day part matches, but not the month, they would have to do two DATE_COMPARE(). I think that's not a big deal considering how unlikely the scenario sounds.

DATE_IN_RANGE() / DATE_BETWEEN()
I think these are confusing and my head isn't ready to suggest anything. ISO durations might not have been helping ;). A full date can be range checked using date >= before_date && date <= after_date, and I think that will be a much more common use case than range-checking partial dates. A query such as "is between march 17 and october 17" is slightly vague without a year. Any annual event takes place between march 17 and october 17 (unless you exclude march 17 and october 17). I understand what you mean though, and the given the opposite order (between october 17 and march 17) is complex to express with operators, maybe this calls for a function?

MS SQL's DATEPART() works much like MySql's EXTRACT(), except EXTRACT() also takes a limited predefined set of constants representing a range between two units, for example YEAR_MONTH or DAY_SECOND (day, hour, minute and second). MONTH_DAY isn't among them it seems. I'm not too keen on predefined constants like that because apart from being less flexible, possibly requiring CONCAT() they also have to be learned.
I would use DATE_PART() for filtering, not formatting. Until I thought of the "is between october and march example" I thought DATE_PART() would suffice for range checks, but I guess not. You're also right that SUBSTRING() can do the same, so yeah, I guess we don't need it?
"Your suggestion to have a function like DATE_PART() that returns multiple components is interesting though. I believe the following would work."
I didn't actually mean returning multiple components, just taking multiple components as arguments. Returning multiple components looks better, but in JS [] == [] yields false. Maybe == in arango isn't internally == in JS though?

Good point about DATE_MATCH() :smile:
I agree that from unit to unit is a little nicer, while still allowing for complex partial comparisons by calling it twice. Plus the common use case to check birthdays is handled very well. A technical note though: it would also be possible to allow a variable amount of parameters, like DATE_COMPARE(date, DATE_NOW(), "month", "day", "minute", "ms") - which is nicer than having square brackets around the units. But the usefulness seems really low. A range is probably better, given as two arguments. Otherwise you would have to specify all units in between the start and end unit.

(between october 17 and march 17) is complex to express with operators, maybe this calls for a function?

You got a point, a comparison will usually match a certain range from year to ms. A seasonal check like the following should work:
LET now = DATE_NOW()
date >= DATE_TIMESTAMP(DATE_YEAR(now), 10, 17) AND
    date < DATE_TIMESTAMP(DATE_YEAR(now)+1, 3, 17)
    // could also DATE_ADD() an ISO duration to calculate the end date!

A specialized date-in-range function might be slightly faster though, because you can tell it to ignore the time components. In above example, it would compare date against each character in 2015-10-17 00:00:00.000Z and 2016-03-17 00:00:00.000Z - but checking the time is not really necessary here. But it might also be slower, because a partial comparison in JS may take longer than a native(?) string comparison (C++?). Alltogether, it's probably not worth to implement a range function until people yell for it.
I wasn't aware that array checks aren't possible like that, it should be more like _.isEqual(arr1, arr2) (underscore.js). But I don't think we need it anyway. The implementation of DATE_COMPARE() can be as simple as subtring-ing the right range of characters from both ISO strings and do simple string comparison - correct?

The diff between last day of Feb 2015 and last day of Feb 2016 should be exactly 1 year, because this is probably what is expected ("end of Feb", not +365 days). That should be documented of course."

This isn't how I've applied it. I don't agree with that logic generally, but this case might be an exception where it makes sense. By my diff method march 30 and april 30 is a month apart, and march 31 and april 30 is less than a month apart (even if they are both the last day of the month). The first draft I wrote compared the offset since the start of the month divided in the length for the respective month, but then 2015-01-02 and 2015-02-02 would be more than a month apart. This was actually the only thing I changed.
Maybe you should add this to the date_diff() todo? I think it's more important to handle leap days in date_compare. You wouldn't want those leaplings to miss their birthday vouchers.

Alltogether, it's probably not worth to implement a range function until people yell for it.

Yeah, that's what I was thinking. Then we (or more likely someone else) would see some real world scenarios, which would help in solving them.

I wasn't aware that array checks aren't possible like that, it should be more like _.isEqual(arr1, arr2) (underscore.js).

Yeah, this is a limitation of dynamic languages and JS in particular. Recommended watch: https://www.destroyallsoftware.com/talks/wat

The implementation of DATE_COMPARE() can be as simple as subtring-ing the right range of characters from both ISO strings and do simple string comparison - correct?

I haven't considered all angles, but I think that's the way to go. We probably want to handle leap days as either feb 28 or mar 1?

Watman! I can watch that over and over again :laughing:
Ah well, I'm not convinced about my earlier statement that the difference between Feb 28 2015 and Feb 29 2016 should be exactly a year, because it would be equal for both, 29th and 28th of Feb 2016... really hard to make a "good" decision here IMO. If your code does the same as moment.js, then it's probably fine as it is.
As for the DATE_COMPARE() implementation, what do you think of this?
// 0123_56_89_12_45_78_012_
var unitsSubstr = {
  year: [0, 3],
  month: [5, 6],
  day: [8, 9],
  hour: [11, 12],
  minute: [14, 15],
  second: [17, 18],
  millisecond: [20, 22]
}
// add aliases here

start = unitSubstr[unit1][0] // + error checking
end = unitSubstr[unit2][1] // + error checking
var dateStr1 = MAKE_DATE(value1).toISOString();
var dateStr2 = MAKE_DATE(value2).toISOString();
return dateStr1.slice(start, end) == dateStr2.slice(start, end)

Leaplings are indeed a problem and need to be handled in above code :unamused:
This is especially tricky because it seem to depend on your country when your birthday is in non-leap years. I would rather go for the 28th instead of the 1st of March, because I would want my voucher rather too early than too late.
But can we just make a 02-29 a 02-28? For both input dates? Only in non-leap years?

You're fast!
I don't mind either way, but considering many date handlers probably leave them out we might want to go with earlier.
I think this implementation works with the addition for leaplings, week and quarter. But since week and quarter doesn't have to be supported in the range, we could support them as single units only?

:athletic_shoe:
I would actually leave out weeks and quarters in DATE_COMPARE(), since they aren't expressible in ISO date strings. If you want to compare them, just do DATE_ISOWEEK(date1) == DATE_ISOWEEK(date2) or DATE_QUARTER(date1) == DATE_QUARTER(date2).

:+1: Good thinking!

Oh, and I missed this one: For consistency I prefer if all units had one-letter abbreviations, or none had it (speaking about all methods where it applies). So i rather than min to fit the pattern.

I'm with you, but how to handle milliseconds? Typing it out is cumbersome, ms is very well remembered but uses two letters. MySQL and Python use %f for milliseconds, although it's actual millliseconds whereas .000Z in an ISO string is more a fraction of a second, limited to 3 decimal places. Not sure if it's named after Fraction? Makes sense to use it then. Or rather miLlisecond?
If we make it f, do we also want ms as alias?

Full name (plural)
Full name (singular)
Abbreviation
Additional alias

years
year
y

months
month
m

days
day
d
dates / date?(meh, but JS calls it date...)

hours
hour
h

minutes
minute
i
min?

seconds
second
s

milliseconds
millisecond
f
ms?

A crazy idea I just had: If we wanted to compare years, months and hours, couldn't we zero out the days, minutes, seconds and ms and do a full string comparison?
LET now = DATE_NOW() // 2015-09-08 20:17:35.265Z
LET date = "2015-09-19 15:30:00.000Z"
LET d1 = DATE_PART(now, "year", "month", "hour") // 2015-09-00 20:00:00.000Z
LET d2 = DATE_PART(date, "year", month", "hour") // 2015-09-00 15:00:00.000Z
d1 == d2 // could also be >, >= etc.

The comparison itself would be coded in AQL instead of JS.
Pro: allows for flexible range checks on user-side
Contra:

user needs to handle leap day
DATE_PART() needs to be called twice for the common use case
####-00-00... is not a valid ISO date (but it's not supposed to be used other than for comparisons)

Uploaded a new Windows build with DATE_ADD(), DATE_SUBTRACT() (both support ISO durations) and DATE_COMPARE():
https://github.com/CoDEmanX/ArangoDB/releases/tag/aql-date-extra
Seems to work as expected, except the leapling issue. Not sure if we should really handle it unasked. Should be trivial for user queries to account for leap days:
FOR y IN 2012..2020
    LET birthday = DATE_ISO8601("1984-02-29")
    LET now = CONCAT(y, "-02-28")
    RETURN {
        year: y,
        leaplingBirthday: LEFT(
            (DATE_COMPARE(birthday, "0000-02-29", "m", "d") AND NOT DATE_LEAPYEAR(now)) ?
            DATE_SUBTRACT(birthday, "P1D") :
            birthday
        , 10)
    }

@friday: Do you run Linux or OSX? If so, would you be able and willing to run the JS unit tests? I tried it  on Windows, without success so far - I struggle at the very basics (fully working Ruby, suitable arangod, proper commands to start it...)
Feel free to add missing unit tests or refine (or fix?) existing ones. The documentation could also use a few more examples, like the leap year birthday thing I posted above with some explanation.

OS X. I have to deal with some other things on my agenda before I can can get to it (tonight).

No problem. If you find some time to compile the code and run the tests in the next couple days, I would greatly appreciate it. It's way too frustrating on Windows. Had not much fun with Linux either, although it's a different breed of problems - access rights, VM folder sharing, getting VirtualBox to run a certain distro at all (had to downgrade like 12 versions to make a vagrant container work), picking a distro that runs the required tools without hacking the kernel... :smile:
BTW: I added a note and a TODO to DATE_ADD() / DATE_SUBTRACT(). Do you think weeks should be supported? It's not a big deal to implement, but it will require an additional check on the unit type on every call. Probably a premature optimization concern, shouldn't really hurt performance.

Late comment on your table: I had completely overseen ms. That one is harder. f is used for microseconds, although fff/FFF is used in some languages to mean "only the 3 first", meaning milliseconds. If it's short for "fraction" I think we could use it, but perhaps it's still confusing for people coming from MySql, Python, or Java. I think l is less confusing, and we'd also know what it was short for. Mongodb, who are also limited by the js date object, and hence to milliseconds rather than micro uses %L for ms in $dateToString (had to look hard to find something to justify with this time.. haha). I also prefer keeping ms and min now, because ms seems to be the best alias for ms.
I'd expect date to either not work or represent the year - day range, rather than just day. And if we support date as a range, I think we should also support time as a hour to ms range.
I think plural forms make sense in DATE_ADD() / DATE_SUBTRACT(), but not so much in DATE_COMPARE(). What do you think?
Regarding AQL based string comparison, I'm not quite following. Is there a functional benefit of doing it in AQL instead of JS? I'm sure performance would be better, but this doesn't seem like a heavy operation?
I'm not sure returning "2015-09-00 20:00:00.000Z" is better than "20150920". We also have to consider that people might be returning DATE_PART() instead of using it for comparisons.

@CoDEmanX : having a look at the table in the first post, it looks like most items are checked (i.e. done). Or the TODO markers there still relevant, given the checkmarks?
I think I can provide a fix for the server-side caches, but would wait with that until the branched has been merged into ArangoDB devel.
Regarding additional date functions (DATE_IS_BIRTHDAY was mentioned above): I think there should be a basic set of date functions, and the functions above cover the most basic needs. I wouldn't make it more complicated by adding more and very specialized functions.
I am also confused about DATE_COMPARE. Isn't that a simple substring comparison of two date strings? Does it require a function of its own?

Some facts:

MySQL: %f for Microseconds (000000..999999), %l (L) for hours in 12-hour format
.NET: fff for 3 digits of milliseconds
Python: %f for milliseconds, %w for day of week, %j for day of year, %W for iso date (week number)
MSSQL: Cryptic numbers for various pre-defined formats (the hell?!), although there's also a 2-3 letter way for formatting - mi for minutes, mmm for milliseconds.
Java: Case sensitive, MM for months, mm for minutes and SSS for milliseconds.
PHP: w for weekday (there's also N for 1-based weekdays starting at Monday), W for ISO week date (number of week), t for number of days in month (interesting), L for leap year (0 or 1), B for Swatch internet time, i for minutes, u for microseconds, U for seconds since Unix epoch and I (upper-case i) for DST.
CouchDB: no native support?
MongoDB: %L for Millisecond (3 digits, zero padded), %j for  Day of year, %w for Day of week and %UWeek of year.

f for ms is quite popular. MongoDB's L is reasonable. PHP's U seems to be inspired by getutime(), but dunno how they came up with u for ms.
Pros / Cons:
f: there's no F in milliseconds, but it's F like fractions of a second.
L: contained in milliseconds, but can be confused with upper-case i if written in lower case, and conflicts with MySQL's l (L) for hours in am/pm format and also PHP's L for leap year indication.
My favorite is therefore f, but since we don't need l (L) for anything else, I wouldn't mind to add it as another alias.
w as a shorthand for week should be okay, because we don't do formatting and don't support addition of / assigning weekdays. (k is a bit counter-intuitive, yet not the worst either.)

I'd expect date to either not work or represent the year - day range, rather than just day. And if we support date as a range, I think we should also support time as a hour to ms range.

True. There's setUTCHours(hour, minute, second, ms) in JS, but there's no equivalent function for year, month and day. Moreover, there's no real benefit in using setUTCHours() - 4 getters need still be called and something added / subtracted. You would save 3 setter calls, but that's not a performance gain I suppose - and I don't know if it's as robust as the individual setters on overflows. Will scratch date as alias for day.

I think plural forms make sense in DATE_ADD() / DATE_SUBTRACT(), but not so much in DATE_COMPARE(). What do you think?

Well, the unit is e.g. month, but we basically compare two months. My preference is to have both - we don't need to run ArangoDB on a C64 and can afford that extra kilobyte.

Regarding AQL based string comparison, I'm not quite following. Is there a functional benefit of doing it in AQL instead of JS? I'm sure performance would be better, but this doesn't seem like a heavy operation?

Performance is probably negligible, but there would be a functional benefit: DATE_COMPARE() does only ==, whereas a DATE_PART()-like function could DATE_PART(...) > DATE_PART(...), >=, <, <=, == and !=. It gives the user a bit more responsibility, because you will usually need to call DATE_PART() on both dates with identical unit ranges. The usefulness is anyone's guess.

I'm not sure returning "2015-09-00 20:00:00.000Z" is better than "20150920". We also have to consider that people might be returning DATE_PART() instead of using it for comparisons.

Same sort properties, but the former is an ISO string and better human-readable, whereas the latter is more compact (so slightly faster?) but less legible. A user should actually not store either, but use it for sorting / comparing alone (on the other hand, there might be specific use cases in which you would want to store a compact representation of date + month - which you can still achieve like CONCAT(DATE_YEAR(...), DATE_MONTH(...)).

@jsteemann: Checked means "mainly implemented", as in fundamentally working, but may still need some reconsideration, assessment or testing. I remove TODOs as they are resolved, so yes, they are relevant, even if the checkbox is ticked ("done, but...").
Much interested in your cache solution! Will leave the current solution as-is for now.
DATE_COMPARE() does nothing more than substring comparison indeed. We had a different design in mind when we started, but this seemed more useful. You could do that with plain SUBSTRING() in AQL, but you don't need two calls and don't have to count index positions (pure syntactic sugar). If we added built-in leapling support, then it would be more than only string comparisons. But I'm not sure if it's a good idea.
Will let you know when it's ready for code review!

@CoDEmanX : +1 of f and w and Commodore 64. I strongly prefer single-letter units as opposed to multi-letter ones.
Regarding DATE_COMPARE: I see, and I think it's no harm keeping it.

dunno how they came up with u for ms

Probably µ, which is used as a prefix for micro.
Python's %f is actually for microseconds too, but the .NET implementation where fff is milli- and ffffff is microseconds supports your theory that it's short for fraction and not having any semantic tie to microseconds. I confused the Java one SSS for FFF. My bad! So :+1:  for f, and I don't think we want an l alias.

Performance is probably negligible, but there would be a functional benefit: DATE_COMPARE() does only ==, whereas a DATE_PART()-like function could DATE_PART(...) > DATE_PART(...), >=, <, <=, == and !=. It gives the user a bit more responsibility, because you will usually need to call DATE_PART() on both dates with identical unit ranges. The usefulness is anyone's guess.

Now I follow you. Yes, this is why I was considering DATE_PART() before, along with the fact that MS SQL and MySql having similar functionality. However I wasn't then considering you could just as well use SUBSTRING() as long as the units are in range (and I think they usually will be).
Thinking further on the leap day issue: We wouldn't want to make both date1 and date2 leap-day compensating. If we did, those greedy bastards born on feb 28 would get an extra voucher every 4th year. I can think of a couple of "solutions":

Privilege date1 - Still quirky. DATE_COMPARE(date1, date2) and DATE_COMPARE(date2, date1) would not be symmetric. It's likely to be misimplemented due to the fact that it would "just work" in 1460 days out of 1461 if the order is wrong. Doing some tests in moment.js I just realized this is what they do in moment.diff(), but it's not documented. We might not want to go down that road?
Add optional 5th boolean argument - More "safe", but not really good imo.

I can think of at least one more "creative" solution, but I think we might have been overthinking this already. Maybe just leave it as it is?

Oh right, I hope I didn't mix up milli- and microseconds of other languages / DBMS too...
What do we do about DATE_PART() then? Users can write SUBSTRING() based solutions, or CONCAT(DATE_*(), "-", ...), so we don't really need to provide this, other than for the same reason we provide DATE_COMPARE() (syntactic sugar).
DATE_COMPARE() is quite useful because it covers is-birthday, except for leap days (+1 for leaving it as is and add an example to docs how to compensate leaplings manually).
We could add DATE_FORMAT() after all, which would be a more flexible version of DATE_PART(). It should support numeric values only however, because we would need locales for dozens of languages otherwise, or support English only, but that's again something for the client-side. Mapping would be something like:

Abbreviated unit
Result of function
comment

o / 8 / 1?
DATE_ISO8601()
numeric string units :laughing:

l?
DATE_LEAPYEAR()
do we want this? as boolean?!

y
DATE_YEAR()

q
DATE_QUARTER()

m
DATE_MONTH()

??
number of days in month
simple lookup, would make it an AQL function

v / k?
DATE_ISOWEEK()

w?
DATE_DAYOFWEEK()
different from w in DIFF_ADD() / ISO durations!

a / x / r?
DATE_DAYOFYEAR()

d
DATE_DAY()

h
DATE_HOUR()

i
DATE_MINUTE()

s
DATE_SECOND()

f
DATE_MILLISECOND()

t?
DATE_TIMESTAMP()

n?
DATE_NOW()
probably the least useful in this list...

And probably used like:
DATE_FORMAT(date, "%m/%d/%y") // "09/18/2015"

Things to consider:

be case sensitive here to allow for w = day of week / W = week number? (would be inconsistent with other date functions, but other functions don't support half of these units on the other hand)
automatically zero-padded? (9th -> 09)
placeholder for two-digit year?
placeholder for 12-hour format hours / for AM/PM?
use a 3rd party lib for the formatting?

I just noticed that moment.js is already part of the repo:
arangodb\js\node\node_modules\joi\node_modules\moment\min\moment-with-locales.js

Haha... that's a dep for joi (hapi) though. I think it's for foxx apps?

Yeah, must be Foxx.
Here' a demo of DATE_FORMAT()

Just for info, the codepen snippets posted so far seem to be broken in Firefox:

Content Security Policy: The page's settings blocked the loading of a resource at self ("script-src http://codepen.io 'unsafe-eval' http://assets.cdpn.io http://assets.codepen.io http://preprocessor.codepen.io https://assets.codepen.io http://.pubnub.com https://.pubnub.com http://platform.twitter.com http://connect.facebook.net http://www.google-analytics.com https://.google-analytics.com https://.google.com http://wufoo.com https://wufoo.com http://.wufoo.com https://.wufoo.com https://.stripe.com https://.instapaper.com http://.instapaper.com http://dc8na2hxrj29i.cloudfront.net http://s3.buysellads.com http://srv.buysellads.com/ https://.keen.io https://d26b395fwzu5fz.cloudfront.net/").

Not a problem for me and no action required.
Yes, moment.js is required by joi, and is therefore part of the repo already. If that's not the version required by the new date functions, we can add the required version again, specifically for AQL.

By the way, we plan building a 2.7.0-RC1 at some point this Friday (CET time). Do you want the date function improvements to be included in this? Then we would need to pull the changes from the repo until Friday noon (CET time). Do you think that's reasonable or should we wait with that?

Oh, forgot a curly brace. Maybe deleted it accidentally (was broken in Chrome too). Updated it.
Our DATE functions have no dependencies. My guess is that it's faster than full-blown integrated moment.js. On the other hand, there's no support for month and weekday names etc. But that's something the client should do anyway.
The patch is basically ready for inclusion, although the documentation isn't complete, nor did we actually run any of the unit tests (which are also incomplete). I'll try to make it ready til tomorrow.
A brief test of the ISO duration cache showed interesting results: It claims a lot of memory indeed, if every duration is dfferent from previous ones, and is not released anymore. The cache can't be utilized unless the same ISO durations are used again. What I didn't expect however, is that the query time is almost identical if the cache is used, i.e. only a single ISO duration:
FOR i IN 1..1000000 // 1 million
    RETURN DATE_ADD(DATE_NOW(), CONCAT("P", i, "H")) // roughly 5 seconds

FOR i IN 1..1000000 // 1 million
    RETURN DATE_ADD(DATE_NOW(), "P1H" // roughly 5 seconds

Ok, should I pull then?
Can you confirm the repo is CoDEmanX/ArangoDB, and branch name is aql-date-extra?
We can pull missing documentation later if that's not ready.
If the cache does not provide any benefit, I also think it can be removed. We can also have some coarse cache invalidation (i.e. at start of each AQL query) so it doesn't grow unbounded, but still provides a benefit if date functions are called multiple times inside a query.

I'm doing some clean-up and will provide another branch for pulling (aql-date) with my commits squashed together.
I'm interested in how you would implement a cache invalidation on query start, but we won't need it if the cache doesn't provide any performance gain anyway. I'll remove it from my code.

Invalidation on query start would be trivial, because when an AQL query uses a V8 context, it will generate some initialization code for it anyway. We could add some code to reset the code to this init code, which can be as simply as resetting the cache variable back to an empty object. So this would be simple, but I agree, if it's not providing any benefit, let's don't do it.

Looking forward to pulling from the aql-date branch. I think it's easiest if you open a PR when you think the branch is ready.
If we include your work in the 2.7.0RC-release, what kind of mentions do you think would be appropriate. Do you want to be mentioned as the contributor with your Github username or with a "real" name?

Not sure what I did yesterday there... but the cache performence seems to be more like:

1 mio times the ISO duration: 11 seconds
1 mio times different ISO durations (not in cache): 19 seconds
1 mio times different ISO durations (cached from earlier): 19 seconds

So the cache does seem to speed it up if it's always the same ISO duration (up to 1.7x faster), but not for varying ISO durations, eventhough it should be faster if the same query is run again... I suspect the cache lookup takes longer the larger it becomes (after several runs with different ISO durations, running with the same duration takes 13-14 seconds instead of the initial 11 it seems).
New aql-date branch: https://github.com/CoDEmanX/ArangoDB/commits/aql-date
Pull request: https://github.com/arangodb/arangodb/pull/1481
I'm re-building Arango to verify that I didn't break anything by rebasing the branch. Will force-push fixes to the branch if necessary.


Convert Create/Edit to Mid Fi (Fuel Flows )

Definition of Ready

[x] Does this meet the INVEST criteria?
[x] Does this have a clearly defined user?
[x] Does this have an estimate?
[x] Does this have acceptance criteria?
[x] Does this have sufficient context (wireframes, business rules, etc.)?
[x] Is this traceable to a feature, epic, or PI?

User Story
As a user
I want to view MP Components using the mid fidelity design
So that I have a better experience
Acceptance Criteria
Given a user has accessed MP Components
When it is created or edited
Then it is displayed according to the wireframes
Context
https://xd.adobe.com/view/df8c27d8-1cff-494d-946d-7c8cf1a2bdc3-8adf/
https://xd.adobe.com/view/eb6c3176-9c62-46e0-af54-8be9a2771b7a-be3f/
Testing
Test Case:
Tester:
Definition of Done

[ ] Have all Acceptance Criteria been agreed to and validated by the scrum team?
[ ] Has the code been unit tested and peer reviewed?
[ ] Have the functional tests been executed?
[ ] Have all defects been reviewed and dispositioned, resolved, or deferred?
[ ] Has the user story been accepted by the Product Owner?
[ ] Is the user story ready to be deployed to the test/staging environments?

Modal not built yet. Will address responsiveness when modal is built.


impl Encodable2718, Decodable2718 for TransactionSigned

First step towards https://github.com/paradigmxyz/reth/issues/11153
Implements eip2718 traits for TransactionSigned, removes some of the method and uses methods provided by the trait instead:

encode_enveloped -> encode_2718
decode_enveloped -> decode_2718
length_without_header -> encode_2718_len
envelope_encoded -> encoded_2718
encode_inner(buf, false) -> encode_2718
encode_inner(buf, true) -> network_encode

This now uses decode_signed_fields methods which don't yet include partiy decoding fix from https://github.com/alloy-rs/alloy/pull/1305, so we should probably merge this on next alloy release

the ci is currently failing because of the issue addressed in https://github.com/alloy-rs/alloy/pull/1367
@mattsse are we OK with including this after next alloy release?

sorry about the conflicts -.-
let's ignore the test for now and track, doing a new alloy release asap anyway

changed pooled tx decoding a bit, tests should pass now https://github.com/paradigmxyz/reth/pull/11218/commits/230f53c4aa8618dbe8926f0024d878d20a700a1c

ci is still failing due to lack of this check in 2718 impl
https://github.com/paradigmxyz/reth/blob/ba4e41110b87cccda21d588e0a59bc94ee910e80/crates/primitives/src/transaction/mod.rs#L1436-L1438
looks like it was added in https://github.com/paradigmxyz/reth/pull/8296 and is only required for engine? should we instead only perform it there? adding it to decode_2718 would result in it being applied in Decodable impl too which might result in issues with decoding of eg lists of transactions

addressed in https://github.com/paradigmxyz/reth/pull/11218/commits/b5c2d1556b896cef0b38b4ebc4fe5b14bfaf3c3d by removing those tests and adding this checks to engine code handling decoding of transactions


Segfault when running server in CI

See the build result here.
sassc gem 2.2.1
ruby 2.5.5
rails <IP_ADDRESS>
I see some other issues related to segfaults with sprockets 4, but none that match these specifics: test mode, assets not precompiled, running on CI. Also, there are no CI failures for other versions of ruby/rails, even with the same version of this gem (you can see this in the other jobs on the build linked above).

The build in the above comment was triggered by a pull request. The build of the same commit triggered from my own Github repo passed. Not sure what the difference is.

We're observing something similar to this - we have multiple 'bad docker builds' - reinstalling sassc within the image resolves the problems
This has been observed with both Ruby 2.6 and 2.5 base image(s)


ValueError: Cannot feed value of shape

Thanks for your share,I want to use your model to train my data,but raise some error:
ValueError: Cannot feed value of shape (1, 256, 512, 3) for Tensor 'instance_input_label:0', which has shape '(?, 256, 512)',my works include:

draw different color on black background for instance segmentation(three channel),for example(10,10,10),(20,20,20),(30,30,30)...
draw white line on lane otherwise black(0,0,0),(255,255,255),(I try to use json file to draw line and then use cv2.imwrite to save,when i read the image,inspecte it's shape,it's channel is 3 even if the black backgroud is zeros(720,1280),3 channel image data is ok?).
generator data list  for example:/xxx/image /xxx/gt_gray/gt_instance in train.txt and val.txt
when i try to train from scrach,the code raise the error,if you can you tell me which step is error,i will appreciate for you help,thanks for your share too.
@MaybeShewill-CV

@bleedingfight The label is divided into binary label and instance label. You may check the data_provider file

@MaybeShewill-CV  thank for your help,i thinks i found the reason
i read the instance demo image of your gt_image_instance,if i use IMREAD_UNCHANGED,the image be imread as (720,1280),Is it that grayscale image be read with flag IMREAD_UNCHANGED,it will be 1 channel,because of the flag changed default IMREAD_COLOR,so imread can read as image's bit instead of BGR image.

@bleedingfight Yes the default cv2.imread type is color image

@MaybeShewill-CV  my code raise error for loss is nan,i check my image,i found that my resized image's pixle be changed for interpolation,i think it may be the reason of my error,so i have two question:
1,if my image have different size i can use the code?(1920x1080,1280x1080)
2. if my image too small I can use the code?(224,224)
I'am sorry to waste your precious time,thanks for your help

@bleedingfight Since the model is based on fully convolution network the image shape will not be the problem you can feed the model with any image of any shape. Maybe you should use the nearest interpolation method when you change the size of your label image.

@MaybeShewill-cv，thanks for your kind。
I try it in train-lanenetshell.py，alter the only not is nearest interpolate to nearest 。but the result is same。loss nan。I try to
decease the lrto0.00000001,itcan. output. some epoch and then. loss nan。have some other way to avoid it.by the. way,tools include three train*,all can be
use to. train. my. model? or. i. can. delete. instanse. segmentation?

@bleedingfight You can also only train the binary segmentation branch if you don not care the lane instance

@MaybeShewill-CV
It's means that. i use. git. checkout. to a branch(binary) to train or just. run. lanenet_binary_segmentation.py

@bleedingfight What i mean is the model's binary segmentation brance can be trained alone to detect the lanes but it can not recognize each lane instances. You can read the origin paper for details

@bleedingfight
请问：为什么训练您的数据集时，在图片预处理
label_img = cv2.imread(gt_label_path, cv2.IMREAD_UNCHANGED)时，可以将
(720, 1280, 3)变为(720, 1280)
我用自己的训练集，处理之后
(720, 1280, 3)还是(720, 1280，3)
这是什么原因呢？

@y11203090135 数据是作者MaybeShewill-cv的，他说要处理成灰度图，你说的处理后的是指哪里？查看一下文档说明，我好久没弄这个了，按照说明一步步弄是可以的，我之前试过只是损失nan没法解决后来没弄了！

@bleedingfight  可以了，thank you!

@y11203090135 can you translate in english?


No wheels for tf-nightly Macos arm64

Click to expand!
Issue Type
Build/Install
Have you reproduced the bug with TF nightly?
Yes
Source
binary
Tensorflow Version
tf-nightly 2.13.0.dev20230224
Custom Code
No
OS Platform and Distribution
Macos M1
Mobile device
No response
Python version
3.9
Bazel version
No response
GCC/Compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current Behaviour?
I'm not able to install tf-nightly on Macs running arm.

Standalone code to reproduce the issue
> pip install tf-nightly==2.13.0.dev20230224
ERROR: Could not find a version that satisfies the requirement tf-nightly==2.13.0.dev20230224 (from versions: none)
ERROR: No matching distribution found for tf-nightly==2.13.0.dev20230224

Relevant log output
No response

@eapolinario Thanks for reporting the issue.
I have tried it on Mac M1 with arm64 and I was successfully able to install tf-nightly==2.13.0.dev20230224. Please refer to the below screenshots.

Can you please check for different nightly versions and let us know if the issue persists?

@pjpratik , the screenshot above is poitning to the x86_64 wheel, right?

@eapolinario AFAIK, arm64 don't have nightly wheels. I did check with 2.12 nightly and 2.11 as well but the nightly was pointing to x86_64 wheels. The documentation for stable release for CPU on MacOS also has x86_64 wheels as well.
Have you tried with different nightly versions and observed the same behaviour?
Thanks.

@pjpratik , yeah, I observe the same behavior. Can you clarify why arm64 wheels are not being built? I tried to search for a github issue but couldn't find any.

@eapolinario Please find the similar issue #47782. As per this comment, the official pip packages are built for x86_64. For ARM, we have to build from source.
Please refer to this for source installation on MacOS.
Thanks.

Closing as stale. Please reopen if you'd like to work on this further. Thanks!