storefront installer seems to halt

Hi there,
I am trying to install vue-storefront but experience this error
Environment details

Browser: N/a
OS: MacOS 10.14.3
Node: v11.9.0

Current behavior
Build storefront npm...
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ ERROR                                                                                │
│                                                                                │
│ Error: Can't build storefront npm.                                                                                │
│     at Promise (/vue-storefront-demo/vue-storefront/core/scripts/installer.js:478:16)             │
│     at new Promise (<anonymous>)                                                                                │
│     at Storefront.npmBuild (/vue-storefront-demo/vue-storefront/core/scripts/installer.js:474:12) │
│                                                                                │
│ Please check log file for details: /vue-storefront-demo/vue-storefront/var/log/install.log

install.log
Cloning into '../vue-storefront-api'...
npm WARN lifecycle The node binary used for scripts is /var/folders/2m/g2s_zt790dv06fzrg_qn28bw0000gn/T/yarn--1552386594369-0.8316919989310463/node but npm is using /usr/local/Cellar/node/11.9.0/bin/node itself. Use the `--scripts-prepend-node-path` option to include the path for the node binary npm was executed with.

><EMAIL_ADDRESS>install /Users/odp/Desktop/vue-storefront-demo/vue-storefront-api/node_modules/fsevents
> node install

node-pre-gyp WARN Using request for node-pre-gyp https download
[fsevents] Success: "/Users/odp/Desktop/vue-storefront-demo/vue-storefront-api/node_modules/fsevents/lib/binding/Release/node-v67-darwin-x64/fse.node" is installed via remote

><EMAIL_ADDRESS>install /Users/odp/Desktop/vue-storefront-demo/vue-storefront-api/node_modules/sharp
> (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)

info sharp Using cached /Users/odp/.npm/_libvips/libvips-8.7.0-darwin-x64.tar.gz

><EMAIL_ADDRESS>postinstall /Users/odp/Desktop/vue-storefront-demo/vue-storefront-api/node_modules/nodemon
> node bin/postinstall || exit 0

npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>|| ^0.13.0 || ^14.0.2 but none is installed. You must install peer dependencies yourself.
npm WARN<EMAIL_ADDRESS>requires a peer of<EMAIL_ADDRESS>but none is installed. You must install peer dependencies yourself.

added 985 packages from 698 contributors and audited 11012 packages in 20.954s
found 1 low severity vulnerability
  run `npm audit fix` to fix them, or `npm audit` for details
npm WARN lifecycle The node binary used for scripts is /var/folders/2m/g2s_zt790dv06fzrg_qn28bw0000gn/T/yarn--1552386594369-0.8316919989310463/node but npm is using /usr/local/Cellar/node/11.9.0/bin/node itself. Use the `--scripts-prepend-node-path` option to include the path for the node binary npm was executed with.

><EMAIL_ADDRESS>restore /Users/odp/Desktop/vue-storefront-demo/vue-storefront-api
> node scripts/elastic.js restore

Tue, 12 Mar 2019 10:30:59 GMT | starting dump
Tue, 12 Mar 2019 10:30:59 GMT | got 100 objects from source file (offset: 0)
Tue, 12 Mar 2019 10:31:00 GMT | sent 100 objects to destination elasticsearch, wrote 100
Tue, 12 Mar 2019 10:31:00 GMT | got 132 objects from source file (offset: 100)
Tue, 12 Mar 2019 10:31:00 GMT | sent 132 objects to destination elasticsearch, wrote 132
Tue, 12 Mar 2019 10:31:00 GMT | got 102 objects from source file (offset: 232)
Tue, 12 Mar 2019 10:31:01 GMT | sent 102 objects to destination elasticsearch, wrote 102
Tue, 12 Mar 2019 10:31:01 GMT | got 104 objects from source file (offset: 334)
Tue, 12 Mar 2019 10:31:01 GMT | sent 104 objects to destination elasticsearch, wrote 104
Tue, 12 Mar 2019 10:31:01 GMT | got 109 objects from source file (offset: 438)
Tue, 12 Mar 2019 10:31:01 GMT | sent 109 objects to destination elasticsearch, wrote 109
Tue, 12 Mar 2019 10:31:01 GMT | got 170 objects from source file (offset: 547)
Tue, 12 Mar 2019 10:31:01 GMT | sent 170 objects to destination elasticsearch, wrote 170
Tue, 12 Mar 2019 10:31:01 GMT | got 106 objects from source file (offset: 717)
Tue, 12 Mar 2019 10:31:01 GMT | sent 106 objects to destination elasticsearch, wrote 106
Tue, 12 Mar 2019 10:31:01 GMT | got 101 objects from source file (offset: 823)
Tue, 12 Mar 2019 10:31:02 GMT | sent 101 objects to destination elasticsearch, wrote 101
Tue, 12 Mar 2019 10:31:02 GMT | got 109 objects from source file (offset: 924)
Tue, 12 Mar 2019 10:31:02 GMT | sent 109 objects to destination elasticsearch, wrote 109
Tue, 12 Mar 2019 10:31:02 GMT | got 108 objects from source file (offset: 1033)
Tue, 12 Mar 2019 10:31:02 GMT | sent 108 objects to destination elasticsearch, wrote 108
Tue, 12 Mar 2019 10:31:02 GMT | got 105 objects from source file (offset: 1141)
Tue, 12 Mar 2019 10:31:02 GMT | sent 105 objects to destination elasticsearch, wrote 105
Tue, 12 Mar 2019 10:31:02 GMT | got 115 objects from source file (offset: 1246)
Tue, 12 Mar 2019 10:31:02 GMT | sent 115 objects to destination elasticsearch, wrote 115
Tue, 12 Mar 2019 10:31:02 GMT | got 103 objects from source file (offset: 1361)
Tue, 12 Mar 2019 10:31:02 GMT | sent 103 objects to destination elasticsearch, wrote 103
Tue, 12 Mar 2019 10:31:02 GMT | got 108 objects from source file (offset: 1464)
Tue, 12 Mar 2019 10:31:03 GMT | sent 108 objects to destination elasticsearch, wrote 108
Tue, 12 Mar 2019 10:31:03 GMT | got 151 objects from source file (offset: 1572)
Tue, 12 Mar 2019 10:31:03 GMT | sent 151 objects to destination elasticsearch, wrote 151
Tue, 12 Mar 2019 10:31:03 GMT | got 125 objects from source file (offset: 1723)
Tue, 12 Mar 2019 10:31:03 GMT | sent 125 objects to destination elasticsearch, wrote 125
Tue, 12 Mar 2019 10:31:03 GMT | got 109 objects from source file (offset: 1848)
Tue, 12 Mar 2019 10:31:03 GMT | sent 109 objects to destination elasticsearch, wrote 109
Tue, 12 Mar 2019 10:31:03 GMT | got 107 objects from source file (offset: 1957)
Tue, 12 Mar 2019 10:31:03 GMT | sent 107 objects to destination elasticsearch, wrote 107
Tue, 12 Mar 2019 10:31:03 GMT | got 177 objects from source file (offset: 2064)
Tue, 12 Mar 2019 10:31:03 GMT | sent 177 objects to destination elasticsearch, wrote 177
Tue, 12 Mar 2019 10:31:03 GMT | got 103 objects from source file (offset: 2241)
Tue, 12 Mar 2019 10:31:03 GMT | sent 103 objects to destination elasticsearch, wrote 103
Tue, 12 Mar 2019 10:31:04 GMT | got 107 objects from source file (offset: 2344)
Tue, 12 Mar 2019 10:31:04 GMT | sent 107 objects to destination elasticsearch, wrote 107
Tue, 12 Mar 2019 10:31:04 GMT | got 120 objects from source file (offset: 2451)
Tue, 12 Mar 2019 10:31:04 GMT | sent 120 objects to destination elasticsearch, wrote 120
Tue, 12 Mar 2019 10:31:04 GMT | got 28 objects from source file (offset: 2571)
Tue, 12 Mar 2019 10:31:04 GMT | sent 28 objects to destination elasticsearch, wrote 28
Tue, 12 Mar 2019 10:31:04 GMT | got 0 objects from source file (offset: 2599)
Tue, 12 Mar 2019 10:31:04 GMT | Total Writes: 2599
Tue, 12 Mar 2019 10:31:04 GMT | dump complete

npm WARN lifecycle The node binary used for scripts is /var/folders/2m/g2s_zt790dv06fzrg_qn28bw0000gn/T/yarn--1552386594369-0.8316919989310463/node but npm is using /usr/local/Cellar/node/11.9.0/bin/node itself. Use the `--scripts-prepend-node-path` option to include the path for the node binary npm was executed with.

><EMAIL_ADDRESS>migrate /Users/odp/Desktop/vue-storefront-demo/vue-storefront-api
> node node_modules/migrate/bin/migrate

Elasticsearch INFO: 2019-03-12T10:31:05Z
  Adding connection to http://localhost:9200/

  up : 1513602693128-create_new_index.js
Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  starting request {
    "method": "DELETE",
    "path": "/*/_alias/vue_storefront_catalog",
    "query": {}
  }

Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  Request complete

Public index alias does not exists [aliases_not_found_exception] aliases [vue_storefront_catalog] missing, with { resource.type="aliases" & resource.id="vue_storefront_catalog" }
Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  starting request {
    "method": "DELETE",
    "path": "/vue_storefront_catalog",
    "query": {}
  }

Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  Request complete

{ acknowledged: true }
Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  starting request {
    "method": "PUT",
    "path": "/vue_storefront_catalog",
    "body": {
      "settings": {
        "analysis": {
          "tokenizer": {
            "comma": {
              "type": "pattern",
              "pattern": ","
            }
          },
          "analyzer": {
            "comma": {
              "type": "custom",
              "tokenizer": "comma"
            }
          }
        }
      }
    },
    "query": {}
  }

Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  Request complete

{ acknowledged: true,
  shards_acknowledged: true,
  index: 'vue_storefront_catalog' }
  up : 1513602868543-change_field_types.js
Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  starting request {
    "method": "PUT",
    "path": "/vue_storefront_catalog/_mapping/product",
    "body": {
      "properties": {
        "sku": {
          "type": "keyword"
        },
        "url_key": {
          "type": "keyword"
        },
        "url_path": {
          "type": "keyword"
        },
        "slug": {
          "type": "keyword"
        },
        "size": {
          "type": "integer"
        },
        "size_options": {
          "type": "integer"
        },
        "price": {
          "type": "float"
        },
        "has_options": {
          "type": "integer"
        },
        "special_price": {
          "type": "float"
        },
        "color": {
          "type": "integer"
        },
        "color_options": {
          "type": "integer"
        },
        "pattern": {
          "type": "text"
        },
        "id": {
          "type": "long"
        },
        "status": {
          "type": "integer"
        },
        "weight": {
          "type": "integer"
        },
        "visibility": {
          "type": "integer"
        },
        "created_at": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        },
        "updated_at": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        },
        "special_from_date": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        },
        "special_to_date": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        },
        "news_from_date": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        },
        "news_to_date": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        },
        "description": {
          "type": "text"
        },
        "name": {
          "type": "text"
        },
        "configurable_children": {
          "properties": {
            "url_key": {
              "type": "keyword"
            },
            "sku": {
              "type": "keyword"
            },
            "has_options": {
              "type": "boolean"
            },
            "price": {
              "type": "float"
            },
            "special_price": {
              "type": "float"
            }
          }
        },
        "configurable_options": {
          "properties": {
            "attribute_id": {
              "type": "long"
            },
            "default_label": {
              "type": "text"
            },
            "label": {
              "type": "text"
            },
            "frontend_label": {
              "type": "text"
            },
            "store_label": {
              "type": "text"
            },
            "values": {
              "properties": {
                "default_label": {
                  "type": "text"
                },
                "label": {
                  "type": "text"
                },
                "frontend_label": {
                  "type": "text"
                },
                "store_label": {
                  "type": "text"
                },
                "value_index": {
                  "type": "keyword"
                }
              }
            }
          }
        },
        "category_ids": {
          "type": "long"
        },
        "eco_collection": {
          "type": "integer"
        },
        "eco_collection_options": {
          "type": "integer"
        },
        "erin_recommends": {
          "type": "integer"
        },
        "tax_class_id": {
          "type": "integer"
        },
        "position": {
          "type": "integer"
        },
        "required_options": {
          "type": "integer"
        },
        "Size_options": {
          "type": "keyword"
        },
        "Color_options": {
          "type": "keyword"
        }
      }
    },
    "query": {}
  }

Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  Request complete

{ acknowledged: true }
Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  starting request {
    "method": "PUT",
    "path": "/vue_storefront_catalog/_mapping/taxrule",
    "body": {
      "properties": {
        "id": {
          "type": "long"
        },
        "rates": {
          "properties": {
            "rate": {
              "type": "float"
            }
          }
        }
      }
    },
    "query": {}
  }

Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  Request complete

{ acknowledged: true }
Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  starting request {
    "method": "PUT",
    "path": "/vue_storefront_catalog/_mapping/attribute",
    "body": {
      "properties": {
        "id": {
          "type": "long"
        },
        "attribute_id": {
          "type": "long"
        },
        "options": {
          "properties": {
            "value": {
              "type": "text",
              "index": "not_analyzed"
            }
          }
        }
      }
    },
    "query": {}
  }

Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  Request complete

{ acknowledged: true }
Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  starting request {
    "method": "PUT",
    "path": "/vue_storefront_catalog/_mapping/cms_page",
    "body": {
      "properties": {
        "id": {
          "type": "long"
        },
        "identifier": {
          "type": "keyword"
        },
        "creation_time": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        },
        "update_time": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        }
      }
    },
    "query": {}
  }

Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  Request complete

{ acknowledged: true }
Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  starting request {
    "method": "PUT",
    "path": "/vue_storefront_catalog/_mapping/cms_block",
    "body": {
      "properties": {
        "id": {
          "type": "long"
        },
        "identifier": {
          "type": "keyword"
        },
        "creation_time": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        },
        "update_time": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        }
      }
    },
    "query": {}
  }

Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  Request complete

{ acknowledged: true }
Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  starting request {
    "method": "PUT",
    "path": "/vue_storefront_catalog/_mapping/category",
    "body": {
      "properties": {
        "url_key": {
          "type": "keyword"
        },
        "url_path": {
          "type": "keyword"
        },
        "slug": {
          "type": "keyword"
        },
        "is_active": {
          "type": "boolean"
        },
        "product_count": {
          "type": "integer"
        },
        "parent_id": {
          "type": "integer"
        },
        "created_at": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        },
        "updated_at": {
          "type": "date",
          "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
        }
      }
    },
    "query": {}
  }

Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  Request complete

{ acknowledged: true }
  up : 1513603161654-reindex.js
Elasticsearch DEBUG: 2019-03-12T10:31:05Z
  starting request {
    "method": "POST",
    "path": "/_reindex",
    "body": {
      "source": {
        "index": "vue_storefront_catalog_temp"
      },
      "dest": {
        "index": "vue_storefront_catalog"
      }
    },
    "query": {}
  }

Elasticsearch DEBUG: 2019-03-12T10:31:07Z
  Request complete

{ took: 2080,
  timed_out: false,
  total: 2599,
  updated: 0,
  created: 2599,
  deleted: 0,
  batches: 3,
  version_conflicts: 0,
  noops: 0,
  retries: { bulk: 0, search: 0 },
  throttled_millis: 0,
  requests_per_second: -1,
  throttled_until_millis: 0,
  failures: [] }
  up : 1513603418700-delete_old_index.js
Elasticsearch DEBUG: 2019-03-12T10:31:07Z
  starting request {
    "method": "DELETE",
    "path": "/vue_storefront_catalog_temp",
    "query": {}
  }

Elasticsearch DEBUG: 2019-03-12T10:31:07Z
  Request complete

{ acknowledged: true }
  up : 1530101328854-local_es_config_fix.js
Searching for deprecated parameters in file '/Users/odp/Desktop/vue-storefront-demo/vue-storefront-api/config/custom-environment-variables.json'...
File '/Users/odp/Desktop/vue-storefront-demo/vue-storefront-api/config/custom-environment-variables.json' updated.
Searching for deprecated parameters in file '/Users/odp/Desktop/vue-storefront-demo/vue-storefront-api/config/local.json'...
File '/Users/odp/Desktop/vue-storefront-demo/vue-storefront-api/config/local.json' updated.
  migration : complete
Cloning into 'var/magento2-sample-data'...

Is it timing out?
Since vue-storefront want node version <11.x I tried to downgrade via brew. That does not seem to change the scenario.
Environment details Updated

Browser: N/a
OS: MacOS 10.14.3
Node: v10.15.2
Yarn: v1.13.0

Please full the issue template at least with your env details otherwise we can't help you

@filrak apologies,
I didn't think it would fit either a bug or FR.
I've updated OP with env details

I've added some more info related to downgrading node via homebrew

Hi @0dp how are things with this one?

Hey @pkarw
Yarn was just updated on brew to v 1.15.2, so I updated and everything went smooth with the installation
┌──────────────────────────────────────────────────────────────────────┐
│ Congratulations!                                                     │
│                                                                      │
│ You've just successfully installed vue-storefront.                   │
│ All required servers are running in background                       │
│                                                                      │
│ Storefront: http://localhost:3000                                    │
│ Backend: http://localhost:8080                                       │
│                                                                      │
│ Logs: /Users/odp/Desktop/vue-storefront-demo/vue-storefront/var/log/ │
│                                                                      │
│ Good Luck!                                                           │
└──────────────────────────────────────────────────────────────────────┘

✨  Done in 172.72s.

Great!

Just a note for anyone else running into this ticket when finding this problem, without brew being the issue. I mistakenly believed that I could npm install instead of yarn, so the dependencies didn't get properly installed (as this project is using workspaces) and the build broke when it got to the compilation step.


Add garbage collector mechanism for unneeded resources

Need to create a function, tool, or process that cleans up unneeded datasets, models, endpoints, training pipelines, batch predictions, model evaluations, etc.
When we run tests, sometimes the tests fail in a way that the after hooks aren't called. We need to ensure that all resources are destroyed when they aren't needed.

Labeled this as a bug, as I consider leaking resources when tests fail a major issue (and I'd like this work to be captured by our SLOs).


Support hbase connector

Support the hbase connector which can write data to the hbase.

@liuxiaocs7 feel free to take it!


Support multiple providers

This PR reworks the way how providers are created. It is now possible to add arbitrary providers with very little overhead. Azure needs verification - I cannot test that.
This PR affects

https://github.com/gini/dexter/pull/22
https://github.com/gini/dexter/pull/27

Closes https://github.com/gini/dexter/issues/23

Hey @cblims! Would it be possible for you to verify that the new azure provider works as expected? Your help would be highly appreciated

Hi, just testet and it works. However "auto-configuration" mode does not. Normally I run dexter without any parameters to read current configuration from ~/.kube/config. Is this one removed?

@cblims Thanks for giving it a try. It should work. Could you share the logs and maybe even a anonymized snippet from you kube config so I could verify locally?

Running dexter auth azure starts authentication, but seems like it doesn't pick up the tenant id from idp-issuer-url. Here is a .kube/config dump. UUIDs  in client-id and idp-issuer-url are fake. The UUID in idp-issuer-url is a tenant id.
- name<EMAIL_ADDRESS>  user:
    auth-provider:
      config:
        client-id: a1209219-e181-41f3-9671-17bac336b8ae
        client-secret: redacted
        id-token: redacted
        idp-issuer-url: https://login.microsoftonline.com/096bd15c-4e79-47f8-b3d6-15519ec857c2/v2.0
      name: oidc

Hey @cblims.
azure issue should be fixed now. Would you mind giving it another spin?

in func AzureCommand azureProvider.tenant is always common when running without --tenant. Auto pilot mode kics in after microsoft.AzureADEndpoint is set. I can look into that sometime during this weekend.
Btw, before autopilot mode was initiated by only running dexter without any options. My initial thought introducing autopilot was to autodetect provider and its configuration based on kubectl context. Are you planning to remove this?

I really like the autopilot functionality. I think i will refactor it into a dedicated provider (something like dexter auth auto. Need to think about this a little more... Thanks

Autopilot is back to normal operation. OAuth2 endpoints for azure are constructed as expected.
@cblims: You could still use it as before. You just have to use the new azure subcommand


Last is always null

When graphing metrics from collectd the displayed last value is always null. Not sure if this is expected behaviour when using collectd.
I'm using 0.3.0rc1.

sorry for the noise, I accidentally created a PR in my own fork..
now, everything should be nice.

Fixed in e1e0353f3bddb26d51a3cee3eea374ba30720b3c.


Happy new year :sparkles:

Happy new year :sparkles:

Very happy new year to all

Very happy new year to all


Thread.wait_to_finish() second call is an error, is this desired?

Godot version:
3.0.5 Stable Official
OS/device including version:
Linux Mint 18.3 64-bit Nvidia Drivers 384.130 GPU=Nvidia GTX950M
Issue description:
Call wait_to_finish() twice on the same thread, an accidental call usually, results in an error that stops the rest of the function from executing.
Steps to reproduce:
Please see the simple attached program.
Minimal reproduction project:
ThreadsWaitToFinish.zip

The answer is probably a yes and with some more careful programming I have been able to avoid this potential scenario. But I am still curious if this is what the developers want, but it may be impossible to avoid given the context of threads. so I am closing but at least  others will find this. The key is to put some guards in place to make sure you only call wait_to_finish() once.


Unauthenticated request to company's projects board shows 404 instead of login page

Description
For private instances of Gitea, ie. where https://GITEA/explore/repos redirects to the login page rather than allowing users to display any content anonymously, I expect all unauthenticated requests to be redirected to the login page.
This for example works fine for following URLs:

https://GITEA/MyCompany
https://GITEA/org/MyCompany/members
https://GITEA/org/MyCompany/teams

But when try to access the projects URL https://GITEA/MyCompany/-/projects I'll see a 404 error page.
I am aware that that page reads "The page you are trying to reach either does not exist or you are not authorized to view it." and that you could argue that it is correct, but given that the other URLs behave differently, this behaviour is inconsistent and should thus be considered a bug.
BTW, this problem cannot be reproduced via https://try.gitea.io/ since it allows anonymous access.
Gitea Version
1.19.3
Can you reproduce the bug on the Gitea demo site?
No
Log Gist
No response
Screenshots
No response
Git Version
No response
Operating System
No response
How are you running Gitea?
downloaded version
Database
None

It might be an expected behavior to prevent probing for names present. If it redirected access to private repository, it'd be possible to determine if a private repository does exist or not.

Yes this looks like a bug, for private instances all pages should redirect to login

Fixed with version 1.20.0


Using VS remote test environment results in System.ArgumentNullException : Value cannot be null. (Parameter 'directory')

Description
Verify throws System.ArgumentNullException : Value cannot be null. (Parameter 'directory') when trying to run tests in Visual Studio on Windows using remote test environment targeting  Ubuntu-20.04 via WSL2.
How to reproduce

Create a new xUnit test project.
Setup a remote test environment using WLS2.
Create a test that uses Verify.
Run the test.
Test fails with  System.ArgumentNullException : Value cannot be null. (Parameter 'directory') exception.

Unless this is not something you are willing to support, I would expect that instead of the exception I would get the usual Verify flow generating received files and comparing them with verified files.
System
This happens when using the following NuGet package versions:
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.3.1" />
<PackageReference Include="Verify.Xunit" Version="17.10.2" />
<PackageReference Include="xunit" Version="2.4.2" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.4.5">
<PackageReference Include="coverlet.collector" Version="3.1.2">
<PackageReference Include="Verify.Xunit" Version="17.10.2" />

in the following system:
.NET SDK (reflecting any global.json):
 Version:   6.0.400
 Commit:    7771abd614

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.22000
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.400\

More info
I also tested using the latest Verify pre-release version: 18.0.0-beta.18.
It should be straightforward to reproduce this issue using the steps outlined above but if it helps, I could create a repo to reproduce this.
testenvironments.json
This was the test environment file I used:
{
  "version": "1",
  "environments": [
    {
      "name": "WSL-Ubuntu-20.04",
      "type": "wsl",
      "wslDistribution": "Ubuntu-20.04"
    }
  ]
}

Verify 17.10.2 exception stack trace
  Message:
    System.ArgumentNullException : Value cannot be null. (Parameter 'directory')

  Stack Trace:
    Guard.AgainstEmpty(String value, String argumentName) line 99
    Guard.BadDirectoryName(String name, String argumentName) line 50
    PathInfo.ctor(String directory, String typeName, String methodName) line 15
    <.cctor>b__42_2(String sourceFile, String projectDirectory, Type type, MethodInfo method) line 23
    VerifierSettings.GetPathInfo(String sourceFile, Type type, MethodInfo method) line 29
    ReflectionFileNameBuilder.FileNamePrefix(MethodInfo method, Type type, String sourceFile, VerifySettings settings, String uniquenessForReceived, String uniquenessForVerified) line 11
    <>c__DisplayClass0_0.<GetVerifier>b__0(String uniquenessForReceived, String uniquenessForVerified) line 16
    InnerVerifier.ctor(String sourceFile, VerifySettings settings, GetFileConvention fileConvention) line 17
    Verifier.GetVerifier(VerifySettings settings, String sourceFile) line 12
    <<Verify>b__0>d.MoveNext() line 30
    --- End of stack trace from previous location ---
    ReadDataFromDifferentGitHubWorkflowCommandDependencyErrorTests.FailedHttpToDownloadArtifact() line 102
    --- End of stack trace from previous location ---

Verify 18.0.0-beta.18 exception stack trace
  Message:
    System.ArgumentNullException : Value cannot be null. (Parameter 'directory')

  Stack Trace:
    Guard.AgainstEmpty(String value, String argumentName) line 108
    Guard.BadDirectoryName(String name, String argumentName) line 59
    PathInfo.ctor(String directory, String typeName, String methodName) line 15
    <.cctor>b__26_0(String sourceFile, String projectDirectory, Type type, MethodInfo method) line 10
    Verifier.GetPathInfo(String sourceFile, Type type, MethodInfo method) line 18
    Verifier.GetVerifier(VerifySettings settings, String sourceFile, Boolean useUniqueDirectory) line 23
    <<Verify>b__0>d.MoveNext() line 63
    --- End of stack trace from previous location ---
    ReadDataFromDifferentGitHubWorkflowCommandDependencyErrorTests.FailedHttpToDownloadArtifact() line 102
    --- End of stack trace from previous location ---

happy to support wsl. can u submit a PR that achieves this

This one seems trickier than the issue with strict JSON I was able to debug but I can give it a try. If I figure it out I'll submit a PR.

any update on a PR? or should i close this?

Things got busy and I didn't manage to progress with this. Feel free to close.


Problematic default lane_change=no value for virtual lines

In
https://github.com/fzi-forschungszentrum-informatik/Lanelet2/blob/master/lanelet2_core/doc/LinestringTagging.md#as-lane-boundary
it is described that it is not possible to execute a lane change over a virtual line by default.
We had multiple discussions about this and so far could not come up with a situation where this would be a problem. Could you describe one?
On the other hand we can imagine multiple cases where a lane change across a virtual lane border would make sense. For example cases, where the lane markings are "missing" in the real world and virtual lines have been used to model this.

Virtual lines are frequently used in intersections if there are not enough road markings. Since things like overtaking are usually not allowed in intersections and lane changes are at least not a good idea, we chose to disallow lane changes by default. And I personally think it is a good idea to be too restrictive rather than too lax when it comes to choosing default behaviors.
But the document just describes the behaviour of the default-provided traffic rules. Nothing keeps you from either writing your own traffic rules that allow this or writing a small script that adds an override to every virtual line.

How do you handle then situations like this:
Consider a single right turn lane and two lanes in the road going out. Now the vehicle may use both of those lanes when exiting the intersection/junction. Do you then always add two lanelets for those two possible paths?
Another situation is two left turn connecting to two outgoing lanes where a lane change is possible in the middle of the intersection. Do you then build 4 lanelets for that? Usually when there is no other vehicle nearby a lane change is not a problem in this kind of situation and neither is it illegal to the best of my knowledge. Of course attention must be paid and no lane change shall be attempted otherwise.
Here a real world example:

I admit that this particular situation on the left with the two turn lanes would probably have no virtual lines since there are lane markings but I can post another example without lane markings if needed.
Regarding overtaking I think that is nothing which would be possible since there is just not enough distance/space for that in order to even consider that usually. So that is nothing we ever had in mind.

Consider a single right turn lane and two lanes in the road going out. Now the vehicle may use both of those lanes when exiting the intersection/junction. Do you then always add two lanelets for those two possible paths?

Yes

Another situation is two left turn connecting to two outgoing lanes where a lane change is possible in the middle of the intersection. Do you then build 4 lanelets for that?

I would create two lanelets and if I really want make lane changes possible and there is no dashed marking I would tag it with lane_change=yes.

Alright this can technically work but I still don't understand why it is the right default. Just to wrap it up, could you describe a typical concrete situation where lane_change=no on virtual lines is really necessary (not just to simplify the path planing)? So far I honestly couldn't come up with one.


Decrease Memory Consumption

It's important to work on ways to decrease memory consumption by Vinum.
Some important suggestions could be:

Disregard/release values hold by Computes that compute primitive datatypes, and compute fast.
Relax the rules of SEDMM concerning State structs must always hold their values in Traits. Or completely disregard Traits and pass in a scope to structs like Compute that cleanup before a new update is ran.
Work on "blurring" the line between handling constant values (local x = data) and state structs, which enable some cool tricks such as dissolving state structs into their values when we are sure that it will always evaluate to a constant. (dynamic dependencies overcomplicate this).

This issue is blocking.

While the second suggestion (that relates to SEDMM) is a general optimization (we decrease memory footprint on every struct creation), the other two are mostly use-case-dependent optimizations, and could be implemented in various and different ways.

Specifically speaking on disabling value caching by state structs, this could be implemented by adding a Flow state struct that mostly sends down values towards its dependents, so that whenever a dependency of it updates, the Flow struct will send down the value to the dependents, but release the value after the update process ends.
Example:
local x = Value(1)
local x_power2 = Flow(function(node)
    return Use(node, x) ^ 2
end)

On(x_power2, function()
    -- since On's listener are ran during an update, x_power2
    -- still holds the computed value, but in the case it doesn't,
    -- it will recompute the value, which is fine since the operation
    -- is mostly free.
   some_instance.some_property = Read(x_power2)
end)

While this works well (the original condition of not holding values is met), it could be painfully hard to decide whenever one should use Flow or Compute, and ultimately forces you to be always mentally aware of everything which isn't ideal and practical.
However, deciding to make Computes smarter about releasing their values when a condition is met is also met with a challenge, which is mainly deciding when a process is "fast enough" and the produced value is "lightweight enough". Flows moved this responsibility from Vinum to the developer, which as previously mentioned, isn't practical.
But, we could change the condition (or the heuristic, if you want to call it that) so that when there are only Listener(internal objects created by On operator) dependents present, we could always release the value, as it'll only be used in side-effects and not calculations.

It's important to note that the condition where when only listeners are present relates to how Flow functions, as while when it updates, its values are present, they are released when its update process is finished, so any other independent update process is launched, the dependents will force the Flow to calculate which isn't also practical (and it would practically functioning as a Compute, just with added costs).

The decision for the previous issue has been finalized, which is to extend Computes' functionality to act as something like Flow when their dependents are only Listeners, where it will release its value when the update process is done.

There is an important aspect of heuristic/conditional value caching that should be carefully understood. Since #37 implemented a relatively simple and semi-generalized (it doesn't work for Values, and shouldn't anyway) implementation where if there are only iListener dependents (internal objects created by On), it's safe to assume we shouldn't hold the value after the update process ends.
This created three cases:

First case, the object has other dependent types (Computes, Maps, etc) additionally to the iListener dependents. In this case, the value is cached.
Second case, the object only has iListener, as such, the value is dismissed after the update process, however, the cost of reading is still the same as the first case under the condition that Read is called during the update process, and not after it is finished (this can happens in On's listener function if you used task.defer or task.spawn and yielded.
Third case, which doesn't happen often, but when it does, it is usually in perf-critical code, which when the object doesn't have any dependents, and is often read by loops (RenderStepped, etc). The cost of recomputing can be avoided by marking objects that don't have dependents at all as "unreleasables", so that its cached normally.

This is in. The optimization that is conditional value caching should only work with "Flow-like" objects that only care about sending values to iListeners, as such, we shouldn't care about recomputing values, and should perhaps only error/warn when the value is dismissed at the point of reading (because according the previous three cases, it's impossible to read when the value is released, unless you use yielding, which at that point, you should cache the value yourself in the function body.

An important question to be made is should we cancel all CTasks of an object when its marked as releasable? Technically, speaking, InnerScopes overcomplicate this issue since while yes we "dismissed" the value, the object's RNode is still holding it in its cancelableTasks field.
However, an as equally important question as the previous one is, should we actually cancel all types of CTasks, or the ones that are only constructed by the InnerScope operator? Consider the following:
local y = scope:Compute(function(node)
    Asynk(node, function()
        task.wait(2)
        asynkWrite(node, 10)
    end)
    return 0
end)

On(y, function()
end)

Technically speaking, this tree will be marked as releasable and y will dismiss its value and cancel the asynk operation as well. As such, its important to mark CTasks created by InnerScope by some sort of a tag so that we can identify the CTasks we want to cancel in the event of dismissing a value.

Update: this is in.
Not sure if i want to pursue the last one, at the moment, but as a preparation step, I'll work on renaming objects and types, and after that, I will do a little memory consumption comparison between master and pr-decrease-memory.
After that, #37 is ready to be merged.


D50, and not D65, being used to perform Chromatic adaption from XYZ to Lab*?

If we want to see the Colour volume of a D65 display, would we not want to convert XYZ in to Lab* using the D65 white point [0.95047, 1, 1.08883] - line 116 of CIELabGamut.m
Changing this variable to D65 causes a change in ring shape if plotted in isolation. Then, when using the intersection view of BT.2020, which uses D65 white, this again distors the ring shape of the test gamut, leaving the reference Synthetic gamut outline unchanged.

So the point is that the normalisation of Lab is now deemed to be wrong.  The chromatic adaptation used by CIELab, that is just ratioing to the white point, causes inconsistencies.  So, the approach generally adopted to get around this, including by the CIE themselves (see CIE168), but also the IEC and the ICDM (disclaimer - I'm both an IEC expert and a member of the ICDM), is to first chromatically adapt the XYZ points using Bradford adaptation (generally considered the most appropriate chromatic adaptation transform, or CAT), and then to convert to CIELab, in order to try and ensure apples-to-apples comparisons.  The consensus was to use D50 as the reference white point (it didn't really matter so long as everybody agrees to use the same one) for comparative assessments of gamut, in particular colour gamut volume (this was set by CIE168, and everybody else followed).
So, if you want to get a measure of gamut volume which can be compared to other reproduction media, first use Bradford CAT to adapt the XYZ to D50, then shift to CIELab space and measure the volume.

See the Wikipedia color appearance model page, particularly the section on CIELab

One of the limitations of CIELAB is that it does not offer a full-fledged chromatic adaptation in that it performs the von Kries transform method directly in the XYZ color space (often referred to as “wrong von Kries transform”), instead of changing into the LMS color space first for more precise results. ICC profiles circumvent this shortcoming by using the Bradford transformation matrix to the LMS color space (which had first appeared in the LLAB color appearance model) in conjunction with CIELAB.
Due to the "wrong" transform, CIELAB is known to perform poorly when a non-reference white point is used, making it a poor CAM even for its limited inputs. The wrong transform also seems responsible for its irregular blue hue, which bends towards purple as L changes, making it also a non-perfect UCS.

@bendoswell I'm happy to discuss this further, however this is intentional behaviour and follows all standards, both for colour gamut volume and for gamut rings, and is also the approach the original author of Gamut RIngs, Dr Masaoka of the NHK, follows.
I'll close the issue in a week or so.

Thanks for taking the time to respon Euan. This makes sense now.
It converts to D50 because that is what was standardised. And while it would technically be possible to produce rings where measured data is adapted to D65 XYZ, the rings / volume calculations wouldn't be all that helpful if you wanted to compare to another set of published rings / volume percentage, which were produced using the reccommended / agreed D50 adaptation...

@bendoswell  no problem, glad to help.  I'll close this issue now.


Numba support for Python 3.10

Dear all,
as you may know, we encountered some serious issues when attempting to port
Numba to Python 3.10. This issue will serve as a summary of existing challenges
and also as a central reference issue with links to all existing branches and
PRs.
The story so far
Supporting new Python versions for Numba is always somewhat involved. The main
reason for this is that Numba interfaces to Python via the bytecode, which is
not a stable interface/contract. In the 3.10 case, the change that caused the
most headaches was:
https://bugs.python.org/issue44626
Specifically, the part of that change that now causes the CPython bytecode
optimizer to inline return statements. I.e. the following feature:
/* If this block ends with an unconditional jump to an exit block,
 * then remove the jump and extend this block with the target.
 */
static int
extend_block(basicblock *bb) {
    if (bb->b_iused == 0) {
        return 0;
    }
    struct instr *last = &bb->b_instr[bb->b_iused-1];
    if (last->i_opcode != JUMP_ABSOLUTE && last->i_opcode != JUMP_FORWARD) {
        return 0;
    }
    if (last->i_target->b_exit && last->i_target->b_iused <= MAX_COPY_SIZE) {
        basicblock *to_copy = last->i_target;
        last->i_opcode = NOP;
        for (int i = 0; i < to_copy->b_iused; i++) {
            int index = compiler_next_instr(bb);
            if (index < 0) {
                return -1;
            }
            bb->b_instr[index] = to_copy->b_instr[i];
        }
        bb->b_exit = 1;
    }
    return 0;
}

Ref: https://github.com/python/cpython/blob/d41abe8/Python/compile.c#L7533-L7558
This change was noticed early during the porting process. It turned out to
cause the most headaches for the with objmode context manager:
https://numba.readthedocs.io/en/stable/user/withobjmode.html?highlight=objmode#the-objmode-context-manager
Perhaps the easiest way to think about this is with the following Python source
example (importantly: note that CPython optimizes this at the Bytecode level, not at the
syntax or AST levels! This is for illustration only):
with objmode:
    if a == 1:
      b = 1
    elif a == 2:
      b = 2
    else a == 3:
      b = 3
return b

with objmode:
    if a == 1:
      b = 1
      return b
    elif a == 2:
      b = 2
      return b
    else a == 3:
      b = 3
      return b

Unfortunately, in the with objmode we can not allow return statements,
since the code within the statement will be lifted out and executed in
object-mode so as to get access to interpreter features. So this bytecode
optimization renders this Numba feature useless, since the return will be
inlined! Worse yet: there is no way for Numba to tell if this was a bytecode
optimizer action or something the user intended! Incidentally, there is a way
to defeat this optimization, by making the return block big enough (see the
code snippet above for MAX_COPY_SIZE) however this is too rudimentary and can
probably only be used to adapt a few failing tests.
Effectively, Numba needed a way to undo the optimization caused by CPython and
as a precursor to that, would need to understand where a with-context begins
and where it ends. However much of this logic was based on heuristics and so
had to be overhauled to facilitate better control flow analysis.
Running the tests
Anaconda provides (at the time of writing) a Python 3.10 binary and associated
compilers on at least Linux and OSX so that can be used for testing.
Compiling Numba requires Numpy. However, the only Numpy's available (at the
time of writing) for 3.10 were 1.21.3 and 1.21.4 via conda-forge. Note however,
that both of these releases do appear buggy:
https://github.com/numpy/numpy/issues/20376
Live branches
At the time of writing the following branches were live:

Python 3.10 changes for things unrelated to with-statements:
https://github.com/numba/numba/pull/7557

With-detection for <= 3.9:
https://github.com/numba/numba/pull/7498

With-detection for == 3.10:
https://github.com/numba/numba/pull/7560

If you need it (due to lack of 1.20 binaries): Python 1.21.
https://github.com/numba/numba/pull/7483

Dead branches

https://github.com/numba/numba/pull/7544
https://github.com/numba/numba/pull/7529
https://github.com/numba/numba/pull/7381

Latest test run is at: https://github.com/numba/numba/pull/7563

All relevant branches merged, Python 3.10 is available from master at the time of writing! Thank you all for your patience and patches! EOM


Project page not purged when new release is made

I uploaded a release on Pypi using twine, but not all files at once so twine was run multiple times.
The "Download files" page on warehouse now shows fewer files than the page on old pypi:
https://pypi.org/project/scipy/1.1.0rc1/#files
vs.
https://pypi.python.org/pypi/scipy
It may be some cache effect, but on the other hand it's been several hours now and it has not updated.
(If it'll eventually sync, that's ok with me, but opening an issue here just in case this is some issue with warehouse.)

unfortunately logs for that celery task are long gone. anecdotally I haven't seen any purge api calls fail to Fastly, though we should be retrying in that case.
this project uploaded a large number of files, and at least one of the purges "stuck".
we may want to ask Fastly if they begin debouncing at some point if we issue 10+ purges for the same key in quick succession.

Yea I know what this is, the ORIGIN_CACHE variable is still pointing to the old service_id, we need to update it in the configuration to point to the new one. That's a secure variable so I can't recover it, so whenever @ewdurbin wakes up he can do it :)
We can also probably delete FASTLY_KEY and FASTLY_SERVICE from the configuration, the bin/release used that, but it doesn't anymore.

Can somebody do the magic manual purge for my package ipaddress as well?

configuration updated. it's deploying now. after that I'll issue a purge all, after which purges should be working with the new service id.

Thanks a lot

Hi all,
I appear to have the same issue with my package 'canari'. Version 3.3.1 is not showing up as a candidate when using pip. It works when using easy_install for some reason.


[PROJECT] Cleanup basic concepts & tests

We need a good clean up of basic concepts. And have tests for subsumption.
EDIT (jfalcou)
After discussion:

[ ] built_in_scalar_value (name is questionable - ints and floats)
[ ] arithmetic_scalar_value ( + product types)
[ ] predeclare wide <arithmetic_scalar_value>
[ ] arithmetic_simd_value = instance_of_wide
[ ] builtin_simd_value = arithmetic_simd_value && builtin_scalar_value<element_type>
[ ] predeclare logical<built_in_scalar_value>
[ ] predeclare logical<wide<built_in_scalar_value>>
[ ] logical_scalar_value = instance_of_logical && the T inside is built_in_scalar_value
[ ] logical_simd_value   = instance_of_logical && the T inside is built_in_simd_value
[ ] scalar_value = arithmetic_scalar_value || logical_scalar_value
[ ] simd_value   = arithmetic_simd_value || logical_simd_value
[ ] value = scalar_value || simd_value

Note for later: we need to have product_type types somewhere so we can ask for SIMD product types proper.
This is a use case for complex for ex.

Merged in 40e4615


Request for sui::vec_map::count_value function

Been studying @juzybits code base.
He has a helper function "count_value" to count how many times a value appears in a VecMap.
Now I wondered, should this function be added to sui::vec_map?
Who is the decision maker for this part of the code base, and what is the reasoning to add or reject such functions?

I probably can take this, seems a easy feature to add as a first contribution

I'm not sure something that narrow should belong in the file itself. Though we will try to add some more general utility helpers to these collections :)

Thanks for your reply.

✅


TRUST4 failing with 10x data

Hi! I have been running into some issues with TRUST4 and I'm not entirely sure what's going on. To preface, I am more of a wet lab scientist and am quite new to bioinformatics and using bash, so apologies for naive questions!
I've tried numerous iterations to try to run this with paired-end fastq files generated from a 10x sequencing run, but everything has failed to run to completion. Some listed here:
run-trust4 -t 3 -f hg38_bcrtcr.fa --ref human_IMGT+C.fa -1 tube5_fastq/*_R1_*.fastq.gz -2 tube5_fastq/*_R2_*.fastq.gz

run-trust4 -t 3 -f hg38_bcrtcr.fa --ref human_IMGT+C.fa -1 tube5_fastq/*_R1_*.fastq.gz -2 tube5_fastq/*_R2_*.fastq.gz --barcodeRange 0 15 + --barcode tube5_fastq/*_R1_*.fastq.gz

Most of the errors I get say "failed: 9 at /Users/kkearns/miniconda3/bin/run-trust4 line 48." I've been running these codes locally but will move to our cluster soon. I also tried running the code with the example files and that worked.
Some bioinformaticians who work here suggested that I would need to generate my own -f and --ref files, but my impression was that the files provided here should be applicable to the data I'm working with. Is this not the case?
I've also tried to run TRUST4 with a bam file that was also in the directory that I was given access to (named all_contig.bam), but it failed. I took a look at the bam file I have as well as the example.bam file and they look different, but again, I'm not sure if this is just my inexperience working with these types of data.
I would appreciate any help you can provide! Thank you in advance!

Could you please provide all the message output by TRUST4? Thanks.

Sure, like this?
Read in and count kmers for 35100000 reads.
system /Users/kkearns/minicoda3/bin/trust4 -t 3 -f hg38_bcrtcr.fa -o TRUST_HIPC_Pilot_Hu_5_8D_TCR_S15_L004_R1_001 -1 TRUST_HIPC_Pilot_Hu_5_8D_TCR_S15_L004_R1_0001_toassemble_1.fq -2 TRUST_HIPC_Pilot_Hu_5_8D_TCR_S15_L004_R1_001_toassemble_2.fq failed: 9 at /Users/kkearns/miniconda3/bin/run-trust4 line 48.

It seems your sample has many candidate reads from VDJ region (35 million) so it may require a lot of memory to store the information.  The return message 9 seems to be out of memory error. If you are running this on a compute server, you may need to allocate more memory.
I also found an issue with your command. It seems the R1 is for barcode, and R2 is for sequenced reads. The short read in R1 may be misclassified as candidate reads which could cause many false positives. So if you are using a single-end 10X genomics data, the command should be:
run-trust4 -t 3 -f hg38_bcrtcr.fa --ref human_IMGT+C.fa -u tube5_fastq/R2.fastq.gz --barcodeRange 0 15 + --barcode tube5_fastq/R1.fastq.gz

Ah okay I will try running with more memory when I can try it on the cluster.
The people who ran these samples told me it was paired-end, so would I use -1 and -2 for R1 and R2 and keep the --barcode option with R1?

Can you show me the first few lines for the R1, R2 files? For 10x paired-end data, there should be three read files.

(base) kkearns@01-4123 tube5_fastq % zmore HIPC_Pilot_Hu_5_8D_TCR_S15_L004_R1_001.fastq.gz
@A00475:266:HCYF5DSXY:4:1101:1118:1016 1:N:0:NGGAACCCAA+NATTCGAGGA
CNAGTTGAGAGTAAGGTCCCACGTTGTTTCTGAGATGGGAGAGAAGGTGGTGTGAGGCCATCACGGAAGATGCTGCTGCTTCTGCTGCTTCTGGGGCCAGG
+
F#FFFFFFFFFFFFFFFFFFFFFFFFF,FFF,F,FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF,FFFFFFFFFFFF:FFFFFF
@A00475:266:HCYF5DSXY:4:1101:2962:1016 1:N:0:NGGAACCCAA+NATTCGAGGA
CNGTTTAAGGCCGAATTCTTCGATAAGTTTCTTATATGGGGTTCTGTTCACAGTGACACTGATCTGGTAAAGCCCCCATCCTGGCCTGACCCTGCCATGGG
+
F#FFFFFFFFFFFFFFFFFFFFFFFF,:,F,:,:FF,,FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
@A00475:266:HCYF5DSXY:4:1101:3974:1016 1:N:0:NGGAACCCAA+NATTCGAGGA
GNAATCACACCGTTGGCCGATATTCATATCTTATATGGGGATCTCAGACCCGAGGCTAGCATGGGCTGCAGGCTGCTCTGCTGTGCGGTTCTCTGTCTCCT
+
F#FFFFFFFFFFFFFFFFFFFFFFFFF,FF,FFF,F,:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFFFFFF
@A00475:266:HCYF5DSXY:4:1101:4336:1016 1:N:0:NGGAACCCAA+NATTCGAGGA
TNGGAACGTCTCGTTCGATTTATCTAGGTCTTATCTGGGGACCCCTATGGCTGCCAGAGGAGAGAAGAGACAACCTGATGATAGAAGTAACTCTTATAACT
+

(base) kkearns@01-4123 tube5_fastq % zmore HIPC_Pilot_Hu_5_8D_TCR_S15_L004_R2_001.fastq.gz
@A00475:266:HCYF5DSXY:4:1101:1118:1016 2:N:0:NGGAACCCAA+NATTCGAGGA
NTGATGGCTCAAACACAGCGACCTCGGGTGGGAACACCTTGTTCAGGTCCTCTACAACTGTGAGTCTGGTGCCTTGTCCAAAGAAAGCTTCAGTGTTTCCT
+
#FFFFFFFFFFF:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFFFFFFFFFFFFF,FF
@A00475:266:HCYF5DSXY:4:1101:2962:1016 2:N:0:NGGAACCCAA+NATTCGAGGA
NCCTAGGAAACCCAGGACCACCCAGCAGAGGAGCCTGGTGCCCATGGCAGGGTCAGGCCAGGATGGGGGCTTTACCAGATCAGTGTCACTGTGAACAGAAC
+
#FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
@A00475:266:HCYF5DSXY:4:1101:3974:1016 2:N:0:NGGAACCCAA+NATTCGAGGA
NACAAACATGAGCTCCAGTGGCTTCTTAGCACTTTGCTTGTACCAATACATAGCGTTATGCCCCAGATGTTGTTCACATTTCAAAGACTTCTTATTTGTCA
+
#FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFFFFFFF
@A00475:266:HCYF5DSXY:4:1101:4336:1016 2:N:0:NGGAACCCAA+NATTCGAGGA
NGTCTCTCAGCTGGTACACGGCAGGGTCAGGGTTCTGGATATTTGGGATGACTTGGAGCTTTGTTCCAGTTCCAAAGGTAACTTTCTGGTAACCCTTTGCT
+

Thanks for providing the snapshot of read files. Here is the command that should work on your data:
run-trust4 -t 3 -f hg38_bcrtcr.fa --ref human_IMGT+C.fa -1 tube5_fastq/R1.fastq.gz -2 tube5_fastq/R2.fastq.gz --barcode tube5_fastq/R1.fastq.gz --barcodeRange 0 15 + --read1Range 16 -1 +

I tried running that code as is, but it immediately stopped and said "Unknown parameter +". I ran the code without the last + and I think it worked!
run-trust4 -t 8 \
-f /home/kkearns/hipc_trust4/hg38_bcrtcr.fa \
--ref /home/kkearns/hipc_trust4/human_IMGT+C.fa \
-1 /home/kkearns/hipc_trust4/tube5_fastq/*_R1_*.fastq.gz \
-2 /home/kkearns/hipc_trust4/tube5_fastq/*_R2_*.fastq.gz \
--barcode /home/kkearns/hipc_trust4/tube5_fastq/*_R1_*.fastq.gz \
--barcodeRange 0 15 + \
--read1Range 16 -1

Please find below the first few lines of >5000 lines from report.tsv.
#count	frequency	CDR3nt	CDR3aa	V	D	J	C	cid	cid_full_length
978	9.560117e-02	TGCGCCAGCAGCCAAGGGCCCGCTGAGGGCCCTGGGGCCAACGTCCTGACTTTC	CASSQGPAEGPGANVLTF	TRBV4-3*01	TRBD1*01	TRBJ2-6*01	TRBC	CCCATACGAGAGGCCC_15592	1
744	7.272727e-02	TGTGCCAGCAGCTACCGGACAGGTCCGGGTAATTCACCCCTCCACTTT	CASSYRTGPGNSPLHF	TRBV4-2*01	TRBD1*01	TRBJ1-6*02	TRBC	AGGGTGACAGACACTT_19774	1
657	6.422287e-02	TGTGCCAGCAGCCAACCTAGCGGGACCTACGAGCAGTACTTC	CASSQPSGTYEQYF	TRBV3-1*01	TRBD2*02	TRBJ2-7*01	TRBC	TGATGGCTCAAACACA_19	1
523	1.092086e-01	TGTGTGGTGAGCGACAGAGGCTCAACCCTGGGGAGGCTATACTTT	CVVSDRGSTLGRLYF	TRAV10*01	.	TRAJ18*01	TRAC	TTAGGCAGATCTCTGC_28758	1
476	9.939445e-02	TGTGCTCTAAGTGGAAGTAACAGCTGGGGGAAATTGCAGTTT	CALSGSNSWGKLQF	TRAV16*01	.	TRAJ24*02	TRAC	TTGCGTCAGTTTAGGA_53692	1

My bad, there is no need to flip the read sequence so the read1Range option takes two parameters. Your command is correct. Glad it works!

Since your data is single-cell, you may want to use the results in trust_barcode_report.tsv file.

Thank you so much for your help! I also wanted to ask if you had any recommendations on next steps to analyze this data? I see many people using VDJTools, but the format of trust_barcode_report.tsv is different than the format that VDJTools wants.

VDJTools is for bulk data analysis. For that, you can use the trust_report.tsv file (though you may need to remove entries such as "out_of_frame").
For single-cell data, you may want to try tools like scRepertoire or immunarch. I think they support TRUST4 output format now. If not, you can run the "scripts/trust-barcoderep-to-10X.pl" to convert the barcode report file to 10X VDJ format, which should be compatible with more tools.


Publish to NPM

Expected Behavior
I should be able to install the plugin from NPM.
Current Behavior

It is not available in NPM.
$ npm install eslint-plugin-crc
npm ERR! Darwin 16.7.0
npm ERR! argv "/Users/jeffrose/.nvm/versions/node/v6.10.2/bin/node" "/Users/jeffrose/.nvm/versions/node/v6.10.2/bin/npm" "install" "eslint-plugin-crc"
npm ERR! node v6.10.2
npm ERR! npm  v4.3.0
npm ERR! code E404

npm ERR! 404 Registry returned 404 for GET on https://registry.npmjs.org/eslint-plugin-crc
npm ERR! 404
npm ERR! 404  'eslint-plugin-crc' is not in the npm registry.
npm ERR! 404 You should bug the author to publish it (or use the name yourself!)
npm ERR! 404
npm ERR! 404 Note that you can also install from a
npm ERR! 404 tarball, folder, http url, or git url.

Hi, @jeffrose , thanks for your interest! To be honest, I'm not sure whether this is ready for NPM: the codebase is for a whitepaper I'm writing. It also needs refactoring.
If, however, you'd like me to publish it, I'm open to the suggestion.

Hello @gregswindle , I guess it depends on the functionality that's currently available. I am mostly interested in goals #3 and #5. If you think it's not quite ready for public consumption, I understand. If part of the functionality works, I don't seem harm in publishing as long as the version number reflects that.

I appreciate the prompting, @jeffrose. I'll dust off eslint-plugin-crc this weekend and see how far I can get. In the meantime, if you're interested in some of the mechanics under the hood, please take a look at the GitBook Refactoring JavaScript I'm using (with RunKit examples) to make notes as I work on this.
I'll keep you posted and shoot for a 0.1.0 version. Thanks again.

Hi, @jeffrose. Update, as promised: I didn't get a chance to work on eslint-plugin-crc this weekend. I will, however, take a look this week. Thanks for your patience.

Hey @gregswindle , you've written one of the more thorough explorations of inheritance I've seen. Unless I overlooked it, one thing that might be worth calling out is how the different kinds of inheritance produce functionally equivalent but different results.
function FooBar(){
    this.bar = function(){}
}
FooBar.prototype.foo = function(){}

console.log( new FooBar().foo === new FooBar().foo ); // true
console.log( new FooBar().bar === new FooBar().bar ); // false

When doing prototypal inheritance, the class instances are sharing a function. When doing constructor inheritance, the class instances are getting their own copy.
In terms of best practices, you typically want to inherit properties via constructor inheritance and methods via prototypal inheritance.
No worries about the plugin. While ESLint provides a rule for cyclomatic complexity, there is a gap to be filled for other kinds of analysis.

Excellent point, @jeffrose! Thanks for putting a finer point to prototypal vs. constructor inheritance. If you don't mind, I'll incorporate your example (and attribute it to you) in the documentation. In fact, I think I smell a rule called prefer-constructor-inheritance somewhere....
And thank you for pointing out the gap; that's motivating.

Hi, @jeffrose.
Status
I'm gonna try and deliver MVP1 of eslint-plugin-crc by end-of-year 2017. The refactor/#41-rule-based-crc-models branch is coming along nicely, and I've pruned tons of unnecessary dependencies.
Request
Would you be willing to review PRs? If so, I'd love to add you as a collaborator. If not, that's cool: I understand and value your time.


Write tokens/credentials endpoint for getting scoped AWS credentials

For local file uploads, the plan is to return scoped, temporary AWS credentials that allow a user to upload files to a scoped prefix in a bucket.
See the swagger spec for some more details.

Some documentation that might be useful:
http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html
http://boto3.readthedocs.io/en/latest/reference/services/sts.html#STS.Client.get_session_token

The boto3 ones might be helpful because it's more familiar than the java docs, but we'll need to use java here


[OAL refactor step 1]Refactor dispatcher manager

This is the first step of OAL related refactor. The major point of this refactor is moving the oal-tool back to the main repo, and make it easier to change and use.
There are my expected steps.

[This PR] Make hard codes in DispatcherManager goes away.
Move genereated package to an maven module(a.k.a. a jar in oal-libs), named generated-analysis
Move the oal-tool back into the main repo, and could generate codes during mvn compile.

Coverage increased (+0.001%) to 11.996% when pulling 6766fd6d37929d470c298bfdf61d90dd3f21e3ed on refactor-dispatcher-manager into 5f5be776015e16ccad1184730c23f6276022ee08 on master.


Update to pom.xml to prevent java.lang.NoSuchMethodError: java.nio.By…

…teBuffer.rewind()Ljava/nio/ByteBuffer
Issue #, if available:
The code when compiled and run in emr-notebook environment, fails with the above message during writeStream.
Description of changes:
The pom.xml was updated to include maven.compiler.release of 8 in its properties.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@pankajmisra thank you for your PR!  Did you build the connector with JDK 9+?  I don't think it will have java.lang.NoSuchMethodError, if it is built with JDK8.

I understand your point, thank you. I'd suggest to follow the recommendation in the reference link I previously shared so that pom.xml can work for both JDK8 and JDK9+.
<profile>
  <id>java-8-compatible</id>
  <activation>
    <jdk>[9,)</jdk>
  </activation>
  <properties>
    <maven.compiler.release>8</maven.compiler.release>
  </properties>
</profile>

I updated the pom file to support both JDK8 and JDK9+ build.
Thank you again for your contribution and pointing out the JDK9+ build problem.


Add option to fail registration on duplicate methods

This allows detecting duplicates at path matcher building time and fail early and loudly instead of silently later.
Add unit tests to validate both behavior for both configurations of the option.
Fixes #88.

@nareddyt @qiwzhang hi, can please have a look, please?


CUDA out of memory. Tried to allocate 384.00 MiB (GPU 0; 15.90 GiB total capacity; 14.69 GiB already allocated; 291.75 MiB free; 14.76 GiB reserved in total by PyTorch)

any fixes for this?

Try reducing the batch size.

here are some possible solution for the above issue:-

Reduce Batch Size:
The most straightforward way is to reduce the batch size. This decreases the amount of memory needed for each forward and backward pass.
sample snippet:-

batch_size = 16  # Try reducing this value
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)

clear cache
Use Mixed Precision Training:
Mixed precision training can significantly reduce memory usage by using half-precision (float16) instead of full-precision (float32).
Sample snippet:-

from torch.cuda.amp import autocast, GradScaler

scaler = GradScaler()
for epoch in range(num_epochs):
    for inputs, labels in train_loader:
        optimizer.zero_grad()
        with autocast():
            outputs = model(inputs.float())
            loss = criterion(outputs, labels.long())
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()

Gradient Accumulation:
Accumulate gradients over several mini-batches to simulate a larger batch size

accumulation_steps = 4
for epoch in range(num_epochs):
    optimizer.zero_grad()
    for i, (inputs, labels) in enumerate(train_loader):
        outputs = model(inputs.float())
        loss = criterion(outputs, labels.long())
        loss = loss / accumulation_steps
        loss.backward()
        if (i + 1) % accumulation_steps == 0:
            optimizer.step()
            optimizer.zero_grad()

Optimize Data Loading:
Ensure that data loading is efficient and not causing memory overheads.

train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, num_workers=4, pin_memory=True)

Example with Reduced Batch Size and Mixed Precision
Here is how you can combine reducing batch size and using mixed precision training:

import torch
from torch.cuda.amp import autocast, GradScaler
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader

# Assuming you have your dataset and model defined
train_loader = DataLoader(train_dataset, batch_size=16, shuffle=True)  # Reduced batch size
model = CNN_LSTM(input_dim, hidden_dim, num_classes).cuda()
optimizer = optim.Adam(model.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss()
scaler = GradScaler()

for epoch in range(num_epochs):
    model.train()
    for inputs, labels in train_loader:
        inputs, labels = inputs.cuda(), labels.cuda()  # Ensure data is on GPU
        optimizer.zero_grad()
        with autocast():
            outputs = model(inputs.float())
            loss = criterion(outputs, labels.long())
        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()

    # Evaluate on validation set if needed
    # Calculate accuracy, etc.

# Example evaluation loop
model.eval()
total_correct = 0
total_samples = 0

for inputs, labels in test_loader:
    inputs, labels = inputs.cuda(), labels.cuda()
    with torch.no_grad():
        outputs = model(inputs.float())
        _, predicted = torch.max(outputs, 1)
        total_correct += (predicted == labels).sum().item()
        total_samples += labels.size(0)

accuracy = total_correct / total_samples
print(f'Accuracy on test set: {accuracy}')

Hope, this helps
Thanks


Removed composer option as it this will download an old version.

closing in favor of ongr-io/ongr-sandbox#146


clientside encryption perf with existing tests

Adds support for client-side encryption options to be set on clients.

Separates transfer options and client options into interfaces.
Current overarching concrete options class inherits both the above interfaces, renamed class to scope it to SDK-partitioned transfers.
Blobs ServiceTest does not require options implement client options interface but will honor it if it does.
Created options folder based on namespacing of existing classes.

/azp run net - storage-blob - perf


Added delay between requests.

May help with the pacing violation.

Hey, thanks for the PR! I think you might want to change yahoo-earnings-calendar/scraper.py instead of creating a scraper.py. Hopefully your problem is fixed by #22.


Update translations to Punjabi.json

ENS: Randhawa.eth

@prabhchintan thanks for submitting! please change your pull request so you're adding a new folder (I believe the two letter abbreviation for Punjabi is "pa"), rather than overwriting the English translation

Thanks @brantlymillegan -- just created new folders for Punjabi (pa) and Hindi (hi) translations


Dictionaryies>>Find Class leads to a MessageNotUnderstood occurred (error 2010), a ClassOrganizer class does not understand  #'cachedOrganizer'

MessageNotUnderstood >> defaultAction @2 line 3
MessageNotUnderstood (AbstractException) >> _signalWith: @5 line 25
MessageNotUnderstood (AbstractException) >> signal @2 line 47
ClassOrganizer class (Object) >> doesNotUnderstand: @9 line 10
ClassOrganizer class (Object) >> _doesNotUnderstand:args:envId:reason: @7 line 12
SystemOrganizer >> categoryDict @3 line 3
JadeServer64bit32 (JadeServer) >> sbFindClassPackageMap @18 line 8

running this version of Jadeite:

I'm logged in as SystemUser:

but I am running in a stone that has tODE installed as DataCurator ... which leads to believe that Jade is looking at the state of DataCurator (even when running as SystemUser) to determine whether or not GLASS is installed, which is not kosher ...

this no longer reproduces


Package request: discord-canary

Discord Canary
I just don't know why we can update the non-linux version of canary but not both.
There is some compatibility issue?
ref: https://github.com/NixOS/nixpkgs/commit/315a7b36caa56f11d0428f79ed686c494eb85531
Metadata

Existent package: https://github.com/NixOS/nixpkgs/commit/315a7b36caa56f11d0428f79ed686c494eb85531
platforms: unix, linux, darwin, ...

Can you explain a bit more what you mean? The commit your are referring is updating the dawin canary and belongs to this PR that also updates the Linux Variant, but just not in one commit.
So at least I am not sure what is the request and a bit more explanation may help (at least me).

Can you explain a bit more what you mean? The commit your are referring is updating the dawin canary and belongs to this PR that also updates the Linux Variant, but just not in one commit. So at least I am not sure what is the request and a bit more explanation may help (at least me).

I see, my fault. I didn't search for related PRs.
Maybe i should have opened a forum thread instead, sorry.
Thanks for help me.


Update module src-d/go-git to v4.7.1

This PR contains the following updates:

Package
Type
Update
Change
References

gopkg.in/src-d/go-git.v4
require
minor
v4.0.0-20180807092216-43d17e14b714 -> v4.7.1
source

Release Notes

src-d/go-git
v4.7.1
Compare Source
From this release you can us go-git as a go module. Also, this release includes a bunch of fixes in different packages, take a look at the change log
Change Log

all: convert into a Go module #943
blame: fix edge case with missing \n in content length causing mismatched length error #986 (mcuadros)
repository: improve CheckoutOption.Hash doc #985 (mcuadros)
repository: allow open non-bare repositories as bare #984 (mcuadros)
repository: use remote name in fetch while clone, Fixes #827 [#828]
repository: allow ResolveRevision to resolve annotated tags #772 (novas0x2a)
storage: expose Storage cache. #949 (kuba--)
storage/filesystem: keep packs open in PackfileIter #962 (jfontan)
config: Add test for Windows local paths. Fixes #681 #977 (filipnavara)
git: Fix Status.IsClean() documentation #978 (urld)
git: use time.IsZero in Prune #974 (u5surf)
git: Add tagging support #928 (vancluever)
plumbing: object, Add support for Log with filenames. Fixes #826 #979 (gnithin)
plumbing: tree, add a Size() method for getting plaintext size #982 (strib)
plumbing: improve test for urlencoded user:pass #968 (smola)
plumbing: diff, fix fatal: corrupt patch error in unified diff format #932 (distorhead)
plumbing: cache, fix potential LRU cache size issue. #958 (kuba--)

v4.7.0
Compare Source
Here is a new release of go-git, containing a mix of performance improvements (#929, #942), new features and issue fixing, take a log to the change log.
Change Log

storage/dotgit: add KeepDescriptors option #942 (jfontan)
plumbing/storer: add ExclusiveAccess option to Storer #941 (jfontan)
plumbing: object, Clamp object timestamps before unix epoch to unix epoch #939 (taruti)
config: add commentChar to core config struct #937 (zaquestion)
storage/dotgit: search for incoming dir only once #935 (jfontan)
storage/dotgit: Remove empty dirs when cleaning with Dir opt. #933 (kuba--)
plumbing, storage: add bases to the common cache #929 (jfontan)
plumbing/idxfile: object iterators returns entries in offset order #927 (jfontan)
plumbing/format: gitignore, fixed an edge case for .gitignore #924 (fkorotkov)

v4.6.0
Compare Source
Finally here is a new release of go-git, coming with big performance improvements thanks to the new packfile representation #899 (erizocosmico) and parser  #898 (jfontan), speeding up the execution of read operation from 25% to almost a 300% depending on the repository topology.
Also, this release comes with some new features and bug fixes, take a look at the change log.
Change Log

plumbing/format: packfile, new packfile representation  #899 (erizocosmico)
plumbing/format: packfile, new packfile parser #898 (jfontan)
plumbing/format: packfile, improve packfile reading performance #906 (erizocosmico) #916 (jfontan)
plumbing/transport/internal: common, support Gogs for ErrRepositoryNotFound #888 (jeromedoucet)
plumbing/format/idxfile: add new Index and MemoryIndex #896 (erizocosmico)
plumbing/object: fix PGP signature encoder/decoder #892 (jfontan)
git: Add ability to PGP sign commits #920 (vancluever)
git: fix cloning of a single tag #910 (fkorotkov)
plumbing: object, return ErrFileNotFound in FindEntry. Fixes #883 #885 (jsravn)

v4.5.0
Compare Source
This release includes a variety of different changes, some performance improvements for packafile index decoding, but as well as some functions at storage were made public, plus some other fixes and improvements.
Change Log

plumbing: add context to allow cancel on diff/patch computing #874 (smola)
utils: diff, skip useless rune->string conversion #873 (smola)
Remote.Fetch: error on missing remote reference #870 (mcuadros)
packfile: optimise NewIndexFromIdxFile for a very common case #869 (dsymonds)
storage/filesystem: avoid norwfs build flag #864 (smola)
plumbing/transport: http, Adds token authentication support [Fixes #858] #859 (ebilling)
storage: filesystem, make ObjectStorage constructor public #857 (erizocosmico)

v4.4.1
Compare Source
Here is a new version of go-git including some memory optimizations by dsymonds and a couple of bug fixes, #480 and #793.
Change Log

plumbing: packfile, Don't copy empty objects. Fixes #840 #856 (kuba--)
plumbing: object, adds tree path cache to trees. Fixes #793 #855 (jfontan)
dotgit: Move package outside internal. #854 (ajnavarro)
config: modules, worktree: Submodule fixes for CVE-2018-11235 #848 (josephvusich)
packfile: improve Index memory representation to be more compact #846 (dsymonds)
idxfile: optimise allocations in readObjectNames #845 (dsymonds)

v4.4.0
Compare Source
New go-git version including a new functionality to support excludes on the Worktree and also a big improvement of performance for repositories with thousands of references.
Change Log

git: remote, Do not iterate all references on update. #833 (jfontan)
Worktree: Provide ability to add excludes to worktree #825 (maguro)
Fix for "Worktree Add function adds ".git" directory" #815 (kuba--)

v4.3.1
Compare Source
Maintenance release including purely bug fixes.
Change Log

git: remote, Add shallow commits instead of substituting. Fixes #412 #810 (jfontan)
storage: dotgit, init fixtures in benchmark. Fixes #770 #808 (jfontan)
dotgit: ignore filenames that don't match a hash #807 (strib)
Resolve full commit sha #706 (antham)

v4.3.0
Compare Source
A small feature release including the new PlainOpenWithOptions function able to locate the .git folder by @mvdan and a basic feature that was missing, the tracking branches on Clone by @TheHipbot
Change Log
Implemented enhancements:

config: adds branches to config for tracking branches against remotes… #803 (TheHipbot)
add PlainOpen variant to find .git in parent dirs #784 (mvdan)

v4.2.1
Compare Source
Changes
Merged pull requests:

Add commit hash to blame result #795 (shanedasilva)
*: Use CheckClose with named returns and fix tests #794 (jfontan)
Resolve HEAD if symRefs capability is not supported #792 (ajnavarro)
*: skip time consuming tests #786 (mcuadros)
Fix RefSpec.Src() #783 (maguro)
Worktree.Checkout: handling of symlink on Windows #778 (fkollmann)
plumbing: format: pktline, Accept oversized pkt-lines up to 65524 bytes #797 (jvusich)

v4.2.0
Compare Source
After a very busy week, at Git Merge 2018, here is a new release of go-git, doesn't include anything very relevant besides the new Worktree.[AddGlob|RemoveBlob] methods and the new traversal order for Repository.Log done by @ilius
Changes
Fixed bugs:

Basic Auth: Encode special characters in password #723
Plain clone throws error even when repo doesn't exist #755

Merged pull requests:

repository.Log: add alternatives for commit traversal order #771 (ilius)
storage/filesystem: optimize packfile iterator #769 (dennwc)
Unused params, unused code, make Go tip's vet happy #766 (mvdan)
plubming: transport, Escape the user and pswd for endpoint. Fixes #723 #762 (zkry)
plumbing: ssh, return error when creating public keys from invalid PEM #759 (mdelillo)
blame: Add blame line data #754 (shanedasilva)
new methods Worktree.[AddGlob|RemoveBlob] and recursive Worktree.[Add|Remove] #739 (mcuadros)

v4.1.1
Compare Source
Maintenance release including purely bug fixes. The only API change is  Clone.NoCheckout option #721 beeing added.
Changes
Fixed bugs:

transport: http, fix services redirecting only info/refs #751 (mcuadros)
plumbing: diff, fix crash when a small ending equal-chunk #749 (irias)
plumbing: format/packfile, fix panic retrieving object hash. #744 (jfontan)
storage/filesystem/shallow: fix error checking #740 (ferhatelmas)
plumbing: format/packfile, fix crash with cycle deltas #731 (jfontan)

Merged pull requests:

plumbing: packfile, Add a buffer to crc writer #724 (jfontan)
git: added Clone.NoCheckout option #721 (rykov)

v4.1.0
Compare Source
Here is a new release of go-git, it's focused on a performance improvement at clone operation with medium to big repositories (bigger than 200mb).
The clone operation it's now 2x times faster and uses 3x-5x times less memory.
Changes
Fixed bugs:

config: set default pack window size in config #712 (jfontan)

Merged pull requests:

plumbing: cache, modify cache to delete more than one item to free space #720 (jfontan)
plumbing: packfile, clean reconstructed objects outside pack window #716 (jfontan)
examples: add branch add/remove example #714 (marwan-at-work)

v4.0.0
Compare Source

go-git is a highly extensible git implementation library written in pure Go.
After almost 18 months of development, more than +700 commits, and 15 release candidates we are pleased to release the v4 covering almost all the commonly used git functionalities.
The project license has changed to Apache License Version 2.0 and Developer Certificate of Origin. Also now provide Contributing Guidelines in order to make easier contribute to the project.
Stable API
The API is considered stable in all the functions reachable from the main package. This means that the plumbing structs and functions from plumbing package that are being used as arguments or returned by functions in the main package are stable too. Plumbing structures used under the hood may change in following releases but in ways that won't break compatibility.
Changes
Fixed bugs:

panic in object.getFileStatsFromFilePatches() for commits updating submodules: both from and to are nil #654
RemoveReference doesn't always work #648
Checking out over https fails #623
git commit -a should not refer to the storage but to the filesystem #571
SIGSEGV in fillREFDeltaObjectContent #568
Rare SIGSEGV in fillREFDeltaObjectContent #682
repo.Reference(plumbing.ReferenceName("refs/remotes/origin/branch"), true) results in "reference not found" error #676
Commit Modified Lines #622
PlainClone not fully populating tree on Windows #389

Merged pull requests:

License upgrade, plus code of conduct and contributing guidelines #680 (mcuadros)
Worktree.Grep() support multiple patterns and pathspecs #695 (darkowlzz)
Worktree, add Grep() method for git grep #686 (darkowlzz)
Worktree, add Clean() method for git clean #675 (darkowlzz)
storage: filesystem, add support for git alternates #663 (darkowlzz)
storage: add new functions for garbage collection #669 (strib)
fix Repository.ResolveRevision for branch and tag #660 (antham)
*: update to go-billy.v4 and go-git-fixtures.v3 #655 (mcuadros)
storage: dotgit, handle refs that exist in both packed-refs and a loose ref file #666 (strib)
storage: dotgit, remove ref cache for packed refs #651 (erizocosmico)
utils: merkletrie, filesystem fix symlinks to dir  #640 (dimonomid)
config: support a configurable, and turn-off-able, pack.window #587 (strib)
Remote, fix a range loop can break in advance #641 (sunfishgao)
Remote, support for non-force, fast-forward-only fetches #665 (strib)
Remote, add the last 100 commits for each ref in haves list #610 (strib)
Remote, add support for ls-remote #609 (darkowlzz)
Remote, iterate over references only once #580 (erizocosmico)
Worktree.Add, Support Add deleted files, fixes #571 #577 (grunenwflorian)
plumbing: cache, enforce the use of cache in packfile decoder #698 (jfontan)
plumbing: packp/capability, skip argument validations for unknown capabilities #626 (orirawlings)
plumbing: packfile, fix DecodeObjectAt when Decoder has type #646 (mcuadros)
plumbing: packfile, delete index maps from memory when no longer needed #632 (strib)
plumbing: packfile, use buffer pool for diffs #631 (strib)
plumbing: packfile, improve delta reutilization #697 (ajnavarro)
plumbing: object, add Stats() to Commit #613 (darkowlzz)
plumbing: object, add support for signed commits #616 (darkowlzz)
plumbing: object/tag, add signature and verification support #658 (darkowlzz)
plumbing: object, new Commit.Verify method #653 (darkowlzz)
plumbing: object, do not eat error on tree decode #652 (ferhatelmas)
plumbing: object: patch, fix stats for submodules (fixes #654) #677 (krylovsk)
plumbing: transport, add port to SCP Endpoints #608 (balkian)
plumbing: transport/http, Close http.Body reader when needed #657 (ajnavarro)
plumbing: transport/ssh, check .ssh/config for host and port overrides; fixes #629 #687 (smithrobs)
plumbing: transport, converts Endpoint interface into a struct #650 (mcuadros)
plumbing: transport, made public all the fields and standardized AuthMethod #649 (mcuadros)
plumbing: revlist, do not revisit ancestors as long as all branches are visited #588 (erizocosmico)
plumbing: the commit walker can skip externally-seen commits #586 (strib)
plumbing: packfile, improve performance of delta generation #582 (erizocosmico)
plumbing: object, commit.Parent() method #534 (josharian)

v4.0.0-rc9
Compare Source
Documentation updates and some minor fixes
v4.0.0-rc8
Compare Source

transport/http: fix partial request with haves. Fix #216. (#221)
packfile: cache undeltified objects to improve decode performance (#218)
plumbing/object: adds Tree method to Tree (#224)
config: modules, marshal and unmarshal implementation 4e78181
new repository constructors and worktree  24c1878
object: modes in TreeEntry as os.FileMode, and not the git internal   cf9efc6
storage: IndexStorer implementation           de1c0bf
Repository.Progress moved as a field in *Options (#237)           a48bc6e
new srcd.works/go-git.v4 import path

v4.0.0-rc7
Compare Source
Public API

remote: add Push (#178)
server: add git server implementation (#190)
doc: clarify return values of Pull/Fetch. (#205)
remote: fix empty-git-upload-pack error in fetch, when the reference points
to a non-commit object (#209)
repository: fix pull when fetch returns up-to-date (#207)
blame: make line and its members public so they can be actually used (#213)

Plumbing

plumbing: fix signature with > before < parsing (#204)

Formats

packfile/decoder: speed up packfile iterator when specific type (#200)
packfile/scanner: reset zlib reader instead of new one (#201)

Transport

transport: remove SetAuth, fixes #206 (#210)

Test

fixtures: new fixture with multiple root commits (#212)
fixtures: initialize fixtures into separated methods (#214)
test: restore default protocol. (#215)

Internals

fix typo in docs for plumbing/difftree/internal/merkletrie/iter.go (#192)

v4.0.0-rc6
Compare Source

fix plumbing/difftree package name (#191)
utils/fs: extract billy (#173)

v4.0.0-rc5
Compare Source
Public API:

remote: make Fetch atomic. (#185)
remote: sideband support (#164)
move plumbing from top level package to plumbing (#183)
repository: Ref -> Reference; Refs -> References. (#168)
file: return error on File.Contents() if reader fails (#150)
rev-list command implementation for objects (#135)
difftree: merkletrie internal package with iterator (#133)

Config:

bare flag (#177)
support delete refspecs (IsDelete). (#166)

Storage:

filesystem, clean close when the packfile is not used (#140)
shallow storage (#180)

Protocols:

packp: add report status message. (#162)
packp: add reference update request encoder. (#147)
packp: add upload pack response message (#161)
capabilities: implementation  (#144, #151, #153, #170)
sideband: muxer and demuxer (#143)

Formats:

packfile: packfile write (#131)
packfile: delta calculation (#182, #186, #172, #159)
pktline: fix readPayloadLen err handling (#148)

Utils:

fs: memory fix read offset
fs: memory fix ReadDir, error getting filename

Transports:

http: Add configurable http client factory (fixes #120) (#121)
git: new git protocol (#175)
ssh: keeping the original path (#189)
local/ssh: add git-send-pack support (#163)
local: add local transport (#145)
create Client interface (#132)

v4.0.0-rc4
Compare Source
Global:

storage interface refactor (#112)
new plumbing package (#118)

Fixes:

utils/fs: add ReadAt to memory file and tests. (#122)
format/packfile: Fix nil tx bug (#124)

v4.0.0-rc3
Compare Source
Utils:

utils/fs: Fix O_CREATE flag check in OpenFile #116
utils/fs: utils/fs: added test for open-read-seek #117

v4.0.0-rc2
Compare Source
Core

added Blobs, Trees and Objects iterators to Repository #114

Clients

ssh: fix ssh client #96

Formats

packp: package advrefs encodes and decodes advertised references messages #92
packp: package ulreq encodes and decodes upload-request messages #106
index: added index format support #91 #105
config: package to encode, decode and manipulate git config files #97

Utils

binary: new package that collect all the spare helper functions about binary operations #102
fs: move 'os' and 'test' to separate packages #93
fs: add Remove() #94
fs: add OpenFile method to filesystem interface #104
fs: new memory file system #108

v4.0.0-rc15
Compare Source
Fixed bugs:

Worktree.Clone can't checkout commit pointed to by tag, but Git works #574
Worktree.Checkout:Force bug? #435
Worktree Checkout and Reset call to fetch operations on submodules #415

Merged pull requests:

packfile: improve performance a little by reducing gc pressure #578 (erizocosmico)
Repository.Clone added Tags option, and set by default AllTags  #576 (mcuadros)
Add sideband support for push #573 (orirawlings)
Worktree.Reset refactor and Soft, Merge, Hard and Mixed modes #572 (mcuadros)
dotgit: avoid duplicated references returned by Refs #569 (erizocosmico)

v4.0.0-rc14
Compare Source
Implemented enhancements:

Support cancellation of long-running repository operations (with context.Context) #483

Fixed bugs:

Can't clone annotated tags with ReferenceName in Clone #557
PlainClone w/ tag returning reference not found #529
Character normalization bug #495
plumbing.Reference.IsBranch is not working well on windows #462
windows support #345
packp: AdvRefs is not encoded correctly when missing Head pointer #152
user.Current() causes error when cross compiled #537
Unable to push when there are a lot of merged merge requests history in ref #530
Panic when calling Patch.String #523
CloneOptions should reuse Auth method when recursing submodules #521

Merged pull requests:

remote: avoid expensive revlist operation when only deleting refs #567 (strib)
plumbing: use sliding window in delta calculations, like git CL #565 (strib)
plumbing: use seen map in tree walker #564 (strib)
dotgit: rewrite the way references are looked up #563 (erizocosmico)
grammar correction #561 (zachgersh)
repository: Resolve commit when cloning annotated tag, fixes #557 #558 (orirawlings)
_examples: context #555 (mcuadros)
plumbing: use LookPath instead of Stat to fix Windows executables #554 (strib)
packp: fixed encoding when HEAD is not a valid ref #552 (mcarmonaa)
examples: add example for pulling changes #546 (orirawlings)
fix race condition on ObjectLRU #544 (erizocosmico)
prevent PackWriter from using Notify if nothing was written #543 (erizocosmico)
serialized remotes in alphabetical order #542 (mcarmonaa)
*: windows support, skipped receive_pack_test for git transport #540 (mcarmonaa)
Avoid using user.Current() #538 (kimh)
*: windows support, some more fixes (2) #536 (mcarmonaa)
*: windows support, some more fixes #533 (mcarmonaa)
Remote.Clone fix clone of tags in shallow mode #532 (mcuadros)
plumbing: moved Reference.Is\* methods to ReferenceName.Is\* #531 (mcuadros)
plumbing: fix pack commands for the file client on Windows #527 (strib)
Normalize filenames before comparing. #526 (joshbetz)
reuse Auth method when recursing submodules, fixes #521 #522 (devonbarrett)
config: multiple values in RemoteConfig (URLs and Fetch) #501 (smola)

v4.0.0-rc13
Compare Source
Implemented enhancements:

Adding SSH ciphers when connecting #393
Walking through tree of non gitignored files #307

Fixed bugs:

No way to specify Auth for SubmoduleUpdateOptions #520
panic when using custom SSH client #519
Pack files >2GB create corrupted indexes #502
Different behavior on Fetch and Pull #470
git push new branch with depth=1 #455
Upon repo.Push standard git still sees commits as unpushed #443
Fetch remote tags to local #371
Very slow for large repos #505

Merged pull requests:

filesystem: reuse cache for packfile iterator #517 (smola)
revlist: ignore all objects reachable from ignored objects #516 (smola)
storage: reuse deltas from packfiles #515 (smola)
cache: reuse object cache for delta resolution, use LRU policy #514 (smola)
worktree: expose underlying filesystem #513 (mcuadros)
format: idxfile, support for >2Gb packfiles #512 (mcuadros)
remote: pull refactor to match default behavior of cgit #511 (mcuadros)
packfile: create packfile.Index and reuse it #510 (smola)
*: package context support in Repository, Remote and Submodule #509 (mcuadros)
transport: context package support allowing cancellation of any network operation #507 (mcuadros)
Add AuthMethod based on HTTP Authorization header #504 (strin)
move Repository.Pull to Worktree.Pull #499 (mcuadros)
remote: push, update remote refs on push #498 (mcuadros)
Implement a NoTags mode for fetch that mimics git fetch --no-tags #496 (lupine)
*: several windows support fixes #493 (mcuadros)
packfile: Avoid panics patching corrupted deltas. #492 (ajnavarro)
*: add more IO error checks #491 (smola)
repository: allow push from shallow repositories #489 (mcuadros)
worktree: checkout, create branch #487 (mcuadros)
Add example code for listing tags #484 (orirawlings)

v4.0.0-rc12
Compare Source
Implemented enhancements:

transport: push with HTTPS : receive-pack not supported yet #406

Fixed bugs:

Push: RefSpec deleting reference does not work #466
data races in plumbing/transport/internal/common #463
Git status silently fails in a repository with no commits. #459
capabilities parsing fails when non-standard capability is found (was: Cloning from github broken?) #450
Chroot boundary crossed error when cloning symlinks pointing outside the repo #445
submodules path in gitdir is wrong (was: Submodules not having any objects) #413
References() returns forward slashes on Windows #372
Fetch remote tags to local #371
Race detected during PlainClone #351
Reference IsTag() is broken #461

Improvements

remote: fix Worktree.Status on empty repository #480 (mcuadros)
remote: avoid duplicate haves #479 (mcuadros)
worktree: test improvemnts on empty worktree #478 (mcuadros)
git: remove ErrObjectNotFound in favor of plumbing.ErrObjectNotFound #477 (taralx)
transport/server: add asClient parameter #476 (smola)
plumbing: protocol, fix handling multiple ACK on upload-pack and test… #475 (mcuadros)
Use buffered IO for decoding index files. #473 (taralx)
improve delete support on push #472 (smola)
plumbing: protocol, fix handling multiple ACK on upload-pack #469 (mcuadros)
remote: fix push delete, closes #466 #467 (smola)
fix reference shortening #465 (smola)
transport/file: avoid race with Command.Wait, fixes #463 #464 (smola)
fix race on packfile writer, fixes #351 #453 (smola)
storage/filesystem: Fix nil dereference in Shallow() #452 (taralx)
capability: accept unknown capabilities, fixes #450 #451 (smola)
transport: http push #432 (ajnavarro)
transport/ssh: allow passing SSH options #423 (smola)
Fixed modules directory path #414 (Cromel)
remote: fetch, correct behavior on tags #485 (mcuadros)

v4.0.0-rc11
Compare Source
Highlights

worktree: adds .gitignore support (#429) (#444)
worktree: symlink support (#436)
*: upgrade to go-billy.v3

Bug fixes

client: ssh, fix push on git and ssh (#418)
revlist: ignore treeEntries that are submodules. (#440)
repository: update local remote references during fetch even if no pack needs to be received (#434)
worktree: Add create and push the blob objects to the storer (#441)
storage/filesystem: call initialization explicitly, fixes (#408) (#409)

Improvements:

internal/dotgit: rewrite code to avoid stackoverflow errors (#442)
client: ssh, support SSH Agent Auth on Windows (#405)
partial windows support (#369)
packfile: copy operation limited to 64kb (#411)
object: fix naming of NewCommit{Pre,Post}Iterator (#421)

v4.0.0-rc10
Compare Source
Highlights
v4.0.0-rc10 introduces all major functionalities to be expected in v4.0.0:

Working tree manipulation: status, reset, checkout, add, remove, move and commit.
Diff for different objects and printing to unified diff format.
Better submodules support.
Improved git log API.

The release also contains major improvements in some performance areas, APIs to work with repositories over SSH and tons of bugfixes.
Notes

Go 1.8 is the minimum supported version for go-git.
API for the top-level package, plumbing/object and plumbing/transport are frozen with this release candidate. Backwards incompatible changes from this release until v4.0.0 should be minimal and affect only very low-level packages.
History has been filter-branched and does not match v4.0.0-rc9 and older.

Features

Submodules init and update (#270)
plumbing/storer: add RemoveReference (#269)
Add Repository.Log() method (fix #298) (#337)
worktree: status and reset implementation (#339)
worktree: reset and checkout support for submodules ([#344](https://renovatebot.com/gh/src-d/go-git/issue

Renovate configuration
:date: Schedule: At any time (no schedule defined).
:vertical_traffic_light: Automerge: Disabled by config. Please merge this manually once you are satisfied.
:recycle: Rebasing: Whenever PR becomes conflicted, or if you modify the PR title to begin with "rebase!".
:no_bell: Ignore: Close this PR and you won't be reminded about this update again.

This PR has been generated by Renovate Bot.

:warning: Lock file problem
Renovate failed to update a lock file. You probably do not want to merge this PR as-is.
:recycle: Renovate will retry this branch, including lockfile, only when one of the following happens:

any of the package files in this branch needs updating, or
the branch becomes conflicted, or
you rename this PR's title to start with "rebase!" to trigger it manually

The lock file failure details are included below:
go.sum
Command failed: go get
go: finding gopkg.in/src-d/go-git.v4 v4.7.1
go: finding github.com/mitchellh/go-homedir v1.0.0
go: finding github.com/kevinburke/ssh_config v0.0.0-20180830205328-81db2a75821e
go: finding github.com/alcortesm/tgz v0.0.0-20161220082320-9c5fe88206d7
go: finding github.com/gliderlabs/ssh v0.1.1
go: finding github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568
go: finding github.com/jessevdk/go-flags v1.4.0
go: finding github.com/google/go-cmp v0.2.0
go: finding gopkg.in/src-d/go-billy.v4 v4.2.1
go: finding gopkg.in/src-d/go-git-fixtures.v3 v3.1.1
go: finding github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239
go: finding golang.org/x/net v0.0.0-20180906233101-161cd47e91fd
go: finding gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127
go: finding golang.org/x/text v0.3.0
go: finding github.com/kr/pretty v0.1.0
go: finding golang.org/x/sys v0.0.0-20180903190138-2b024373dcd9
go: finding github.com/kr/text v0.1.0
go: finding github.com/kr/pty v1.1.1
go<EMAIL_ADDRESS>git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /var/folders/1y/nbqcg2hn73dg7bc9vpkbn18r0000gn/T/renovate/cache/go/pkg/mod/cache/vcs/67698a57fcf1e2e2f3930542ab0263846a1ad71c488399c086e4929ccb6a539b: exit status 128:
	error: RPC failed; curl 18 transfer closed with outstanding read data remaining
	fatal: The remote end hung up unexpectedly
	fatal: early EOF
	fatal: index-pack failed
go: error loading module requirements
 `go get` (exited with error code 1)