RatInABox data saving

We should provide a standardised way to export RiaB data (times, trajectories and cell data). This follows on from a conversation I had with @niksirbi and @sfmig about integrating some functionality from RiaB into movement and vica versa.
The two ways this could be done are either each cell/agent has its own .save_history() function. Or else a single utils function could take all active agents / cells and save all data together into one file.
Option would be to then have a .load_history() function.

I'm cross-linking this to other relevant issues/discussions for cross-reference:

https://github.com/neuroinformatics-unit/movement/issues/341
https://github.com/freemocap/freemocap/issues/653

We, in the movement project, are also in the process of figuring out export formats and the above threads have contain some relevant points.
Probably the most low-effort file format for exchanging trajectories between movement and RatInABox would be a tabular .csv file in a "tidy"/"tall" format. A contrived example follows:

time
agent_id
x
y

0
Alice
35
22

0
Bob
100
120

1
Alice
37
25

1
Bob
102
121

Other columns can be added as needed, as long as we stick to the principle of "one variable - one column", and the valid column names are documented somewhere.
The advantage of csv is that it's simple, cross-language, and wouldn't necessitate adding any new dependencies to RatInABox. csv files can be written with native Python libraries - see example here.


Add Expressions from Ibis

Is your feature request related to a problem? Please describe.
We would like to add more expressions and kernels for functionality to eventually have parity with Ibis (https://ibis-project.org/reference/expression-numeric).
Generic
https://ibis-project.org/reference/expression-generic

[ ] .between
[ ] .fill_null
[ ] .hash

Numeric
https://ibis-project.org/reference/expression-numeric

[ ] .abs
[ ] .cos/cot/sin/tan/acos/asin/atan/atan2
[x] .ceil
[ ] .clip
[ ] .degrees
[ ] .exp
[ ] .floor
[ ] .ln
[ ] .log
[ ] .log10
[ ] .log2
[ ] .radians
[ ] .round
[ ] .sign
[ ] .sqrt

Strings
Single-argument kernels:

[ ] .str.capitalize()
[ ] .str.upper()
[ ] .str.lower()
[ ] .str.lstrip()
[ ] .str.rstrip()
[ ] .str.reverse()

Multi-argument kernels:

[ ] .str.find()
[ ] .str.left()
[ ] .str.right()
[ ] .str.like()
[ ] .str.ilike()
[ ] .str.lpad()
[ ] .str.rpad()
[ ] .str.replace()
[ ] .str.split()
[ ] .str.substr()

Taking care of this issue.

Thanks @NormallyGaussian for ceil #1867 !

Would like to take care of a few of these as an intro to this repo !

Would like to take care of a few of these as an intro to this repo !

Yes, some of the string functionality should be a pretty good introduction! Just make sure to comment in this PR which functions you are taking to avoid duplicate with :)
Here's a really awesome PR to show you how to get started: https://github.com/Eventual-Inc/Daft/pull/1867

Thanks @jaychia ! I will take the single-argument string kernels to start:

.str.capitalize()
.str.upper()
.str.lower()
.str.lstrip()
.str.rstrip()
.str.reverse()

Exploded all the checkboxes into individual issues so they can be assigned and tracked better.


SDK REFACTO + COSTON INTEGRATION

I had to redo everything because there were bugs, normally this branch is tested and approved. I put it at version 2.0.1 since the current version of the package is at 2.0.0. This PR contains :

The SDK refacto so that it can monitor all the files
The integration of COSTON

please apply these patch changes to solve your typescript issue. I have shared patch file with you @antiyro in discord. here is raw version. you need to run git apply changes.patch and then commit those changes. I was not able to commit these changes to your branch
diff --git a/package.json b/package.json
index af1a646..91f0db4 100644
--- a/package.json
+++ b/package.json
@@ -1,7 +1,7 @@
 {
   "name": "@pangolindex/sdk",
   "license": "MIT",
-  "version": "2.0.1",
+  "version": "2.0.0",
   "description": "🛠 An SDK for building applications on top of Pangolin.",
   "main": "dist/index.js",
   "typings": "dist/index.d.ts",
diff --git a/src/constants.ts b/src/constants.ts
index e8464cd..0d3920a 100644
--- a/src/constants.ts
+++ b/src/constants.ts
@@ -1,16 +1,9 @@
 import JSBI from 'jsbi'
-import { CHAINS } from './chains'
+import { ChainId, CHAINS } from './chains'

 // exports for external consumption
 export type BigintIsh = JSBI | bigint | string

-export enum ChainId {
-  FUJI = 43113,
-  AVALANCHE = 43114,
-  WAGMI = 11111,
-  COSTON = 16
-}
-
 export enum TradeType {
   EXACT_INPUT,
   EXACT_OUTPUT
diff --git a/src/entities/fractions/currencyAmount.ts b/src/entities/fractions/currencyAmount.ts
index 5691c7d..4e5abb3 100644
--- a/src/entities/fractions/currencyAmount.ts
+++ b/src/entities/fractions/currencyAmount.ts
@@ -4,9 +4,10 @@ import invariant from 'tiny-invariant'
 import JSBI from 'jsbi'
 import _Big from 'big.js'
 import toFormat from 'toformat'
-import { BigintIsh, Rounding, TEN, SolidityType, ChainId } from '../../constants'
+import { BigintIsh, Rounding, TEN, SolidityType } from '../../constants'
 import { parseBigintIsh, validateSolidityTypeInstance } from '../../utils'
 import { Fraction } from './fraction'
+import { ChainId } from '../../chains'

 const Big = toFormat(_Big)

diff --git a/src/entities/fractions/price.ts b/src/entities/fractions/price.ts
index e0310b9..0e1e57a 100644
--- a/src/entities/fractions/price.ts
+++ b/src/entities/fractions/price.ts
@@ -4,11 +4,12 @@ import { currencyEquals } from '../token'
 import invariant from 'tiny-invariant'
 import JSBI from 'jsbi'

-import { BigintIsh, Rounding, TEN, ChainId } from '../../constants'
+import { BigintIsh, Rounding, TEN } from '../../constants'
 import { Currency } from '../currency'
 import { Route } from '../route'
 import { Fraction } from './fraction'
 import { CurrencyAmount } from './currencyAmount'
+import { ChainId } from '../../chains'

 export class Price extends Fraction {
diff --git a/src/entities/pair.ts b/src/entities/pair.ts
index 100527c..85652ca 100644
--- a/src/entities/pair.ts
+++ b/src/entities/pair.ts
@@ -14,9 +14,9 @@ import {
   ONE,
   FIVE,
   _997,
-  _1000,
-  ChainId
+  _1000
 } from '../constants'
+import { ChainId } from '../chains'
 import { sqrt, parseBigintIsh } from '../utils'
 import { InsufficientReservesError, InsufficientInputAmountError } from '../errors'
 import { Token } from './token'
diff --git a/src/entities/route.ts b/src/entities/route.ts
index 71c8440..653dacf 100644
--- a/src/entities/route.ts
+++ b/src/entities/route.ts
@@ -1,4 +1,4 @@
-import { ChainId } from '../constants'
+import { ChainId } from '../chains'
 import invariant from 'tiny-invariant'
 import { Currency, CAVAX } from './currency'
 import { Token, WAVAX } from './token'
diff --git a/src/entities/token.ts b/src/entities/token.ts
index 35401ff..d715d8d 100644
--- a/src/entities/token.ts
+++ b/src/entities/token.ts
@@ -1,5 +1,5 @@
 import invariant from 'tiny-invariant'
-import { ChainId } from '../constants'
+import { ChainId } from '../chains'
 import { validateAndParseAddress } from '../utils'
 import { Currency } from './currency'
 import { CHAINS } from '../chains'
diff --git a/src/entities/trade.ts b/src/entities/trade.ts
index b1c7232..03c4a29 100644
--- a/src/entities/trade.ts
+++ b/src/entities/trade.ts
@@ -1,6 +1,7 @@
 import invariant from 'tiny-invariant'
+import { ChainId } from '..'

-import { ChainId, ONE, TradeType, ZERO } from '../constants'
+import { ONE, TradeType, ZERO } from '../constants'
 import { sortedInsert } from '../utils'
 import { Currency, CAVAX } from './currency'
 import { CurrencyAmount } from './fractions/currencyAmount'
diff --git a/src/fetcher.ts b/src/fetcher.ts
index a88277c..030b587 100644
--- a/src/fetcher.ts
+++ b/src/fetcher.ts
@@ -6,7 +6,7 @@ import { Pair } from './entities/pair'
 import IPangolinPair from<EMAIL_ADDRESS> import invariant from 'tiny-invariant'
 import ERC20 from './abis/ERC20.json'
-import { ChainId } from './constants'
+import { ChainId } from './chains'
 import { Token } from './entities/token'

 let TOKEN_DECIMALS_CACHE: { [chainId: number]: { [address: string]: number } } = {
diff --git a/src/index.ts b/src/index.ts
index 1f7e3a2..857e0ad 100644
--- a/src/index.ts
+++ b/src/index.ts
@@ -3,7 +3,6 @@ export { JSBI }

 export {
   BigintIsh,
-  ChainId,
   TradeType,
   Rounding,
   FACTORY_ADDRESS,


Cilium fails to route pod through Egress Gateway

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
Using the following CiliumEgressGatewayPolicy, Cilium somehow fails to match and route traffic through the Gateway.
% kubectl get ciliumegressgatewaypolicies.cilium.io -o yaml
apiVersion: v1
items:
- apiVersion: cilium.io/v2
  kind: CiliumEgressGatewayPolicy
  metadata:
    creationTimestamp: "2023-07-14T07:30:22Z"
    generation: 1
    name: hath-egress
    resourceVersion: "32475226"
    uid: cfd0bf18-5934-4086-a471-209f196fbeed
  spec:
    destinationCIDRs:
    - <IP_ADDRESS>/0
    egressGateway:
      egressIP: 107.172.x.x
      nodeSelector:
        matchLabels:
          node.kubernetes.io/name: bbb.net
    selectors:
    - podSelector:
        matchLabels:
          io.kubernetes.pod.namespace: hath
kind: List
metadata:
  resourceVersion: ""

Cilium Version
Daemon: 1.14.0 b5013e15 2023-07-26T10:28:38+02:00 go version go1.20.5 linux/amd64
Kernel Version
Linux 6.1.0-0.deb11.7-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.20-2~bpo11+1 (2023-04-23) x86_64 GNU/Linux
Kubernetes Version
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.6+k3s1", GitCommit:"3b1919b0d55811707bd1168f0abf11cccc656c26", GitTreeState:"clean", BuildDate:"2023-06-26T17:51:14Z", GoVersion:"go1.19.10", Compiler:"gc", Platform:"linux/amd64"}
Sysdump
No response
Relevant log output
% cilium status --verbose
KVStore:                Ok   Disabled
Kubernetes:             Ok   1.26 (v1.26.6+k3s1) [linux/amd64]
Kubernetes APIs:        ["EndpointSliceOrEndpoint", "cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEgressGatewayPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "cilium/v2alpha1::CiliumCIDRGroup", "core/v1::Namespace", "core/v1::Pods", "core/v1::Service", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:   Strict   [eth0 107.172.x.x]
Host firewall:          Disabled
CNI Chaining:           none
Cilium:                 Ok   1.14.0 (v1.14.0-b5013e15)
NodeMonitor:            Disabled
Cilium health daemon:   Ok
IPAM:                   IPv4: 5/254 allocated from <IP_ADDRESS>/24,
Allocated addresses:
  <IP_ADDRESS> (kube-system/nginx-ingress-nginx-controller-885pl [restored])
  <IP_ADDRESS> (router)
  <IP_ADDRESS> (kube-system/metrics-server-67444bb9c-sdpdz [restored])
  <IP_ADDRESS> (kube-system/coredns-c949dd448-5kn29 [restored])
  <IP_ADDRESS> (health)
IPv4 BIG TCP:           Disabled
IPv6 BIG TCP:           Disabled
BandwidthManager:       Disabled
Host Routing:           BPF
Masquerading:           BPF   [eth0]   <IP_ADDRESS>/24 [IPv4: Enabled, IPv6: Disabled]
Clock Source for BPF:   ktime
Controller Status:      35/35 healthy
  Name                                          Last success   Last error   Count   Message
  bpf-map-sync-cilium_lxc                       5s ago         never        0       no error
  cilium-health-ep                              9s ago         never        0       no error
  dns-garbage-collector-job                     19s ago        never        0       no error
  endpoint-1134-regeneration-recovery           never          never        0       no error
  endpoint-1155-regeneration-recovery           never          never        0       no error
  endpoint-372-regeneration-recovery            never          never        0       no error
  endpoint-573-regeneration-recovery            never          never        0       no error
  endpoint-812-regeneration-recovery            never          never        0       no error
  endpoint-gc                                   19s ago        never        0       no error
  ipcache-inject-labels                         10s ago        25m17s ago   0       no error
  k8s-heartbeat                                 17s ago        never        0       no error
  metricsmap-bpf-prom-sync                      4s ago         never        0       no error
  resolve-identity-1155                         9s ago         never        0       no error
  restoring-ep-identity (1134)                  25m10s ago     never        0       no error
  restoring-ep-identity (372)                   25m10s ago     never        0       no error
  restoring-ep-identity (573)                   25m10s ago     never        0       no error
  restoring-ep-identity (812)                   25m10s ago     never        0       no error
  sync-host-ips                                 10s ago        never        0       no error
  sync-lb-maps-with-k8s-services                25m10s ago     never        0       no error
  sync-policymap-1134                           4s ago         never        0       no error
  sync-policymap-1155                           4s ago         never        0       no error
  sync-policymap-372                            5s ago         never        0       no error
  sync-policymap-573                            4s ago         never        0       no error
  sync-policymap-812                            5s ago         never        0       no error
  sync-to-k8s-ciliumendpoint (1134)             0s ago         never        0       no error
  sync-to-k8s-ciliumendpoint (1155)             9s ago         never        0       no error
  sync-to-k8s-ciliumendpoint (372)              0s ago         never        0       no error
  sync-to-k8s-ciliumendpoint (573)              0s ago         never        0       no error
  sync-to-k8s-ciliumendpoint (812)              0s ago         never        0       no error
  sync-utime                                    10s ago        never        0       no error
  template-dir-watcher                          never          never        0       no error
  waiting-initial-global-identities-ep (1134)   25m10s ago     never        0       no error
  waiting-initial-global-identities-ep (372)    25m10s ago     never        0       no error
  waiting-initial-global-identities-ep (812)    25m10s ago     never        0       no error
  write-cni-file                                25m19s ago     never        0       no error
Proxy Status:            No managed proxy redirect
Global Identity Range:   min 256, max 65535
Hubble:                  Disabled
KubeProxyReplacement Details:
  Status:                 Strict
  Socket LB:              Enabled
  Socket LB Tracing:      Enabled
  Socket LB Coverage:     Full
  Devices:                eth0 107.172.x.x
  Mode:                   SNAT
  Backend Selection:      Random
  Session Affinity:       Enabled
  Graceful Termination:   Enabled
  NAT46/64 Support:       Disabled
  XDP Acceleration:       Disabled
  Services:
  - ClusterIP:      Enabled
  - NodePort:       Enabled (Range: 30000-32767)
  - LoadBalancer:   Enabled
  - externalIPs:    Enabled
  - HostPort:       Enabled
BPF Maps:   dynamic sizing: on (ratio: 0.002500)
  Name                          Size
  Auth                          524288
  Non-TCP connection tracking   65536
  TCP connection tracking       131072
  Endpoint policy               65535
  IP cache                      512000
  IPv4 masquerading agent       16384
  IPv6 masquerading agent       16384
  IPv4 fragmentation            8192
  IPv4 service                  65536
  IPv6 service                  65536
  IPv4 service backend          65536
  IPv6 service backend          65536
  IPv4 service reverse NAT      65536
  IPv6 service reverse NAT      65536
  Metrics                       1024
  NAT                           131072
  Neighbor table                131072
  Global policy                 16384
  Session affinity              65536
  Sock reverse NAT              65536
  Tunnel                        65536
Encryption:                        Wireguard       [NodeEncryption: Disabled, cilium_wg0 (Pubkey: j/5Eidipjk8KhMcDW/lMh5BOgo+LKcCHQI0p4yEJTDo=, Port: 51871, Peers: 1)]
Cluster health:                    2/2 reachable   (2023-07-29T21:28:32Z)
  Name                             IP              Node        Endpoints
  bbb.net (localhost)   107.172.x.x   reachable   reachable
  aaa.net               64.71.x.x   reachable   reachable

% cilium bpf egress list
Source IP    Destination CIDR   Egress IP   Gateway IP
<IP_ADDRESS>   <IP_ADDRESS>/0          <IP_ADDRESS>     Not Found

% helm -n kube-system get values cilium
bpf:
  masquerade: true
egressGateway:
  enabled: true
encryption:
  enabled: true
  type: wireguard
hubble:
  enabled: false
kubeProxyReplacement: strict
l7Proxy: false

Anything else?
No response
Code of Conduct

[X] I agree to follow this project's Code of Conduct

Thanks for this report, would you be able to grab a sysdump file and upload it please so we can better debug the issue? (Using https://docs.cilium.io/en/stable/operations/troubleshooting/#automatic-log-state-collection)


Fix tooltip focus management, #40

This PR fixes an issue with the management of the focus : when the tooltip contained interactive elements, triggering events on them might be consider as target focus-out. As there is a callback attached to it, it was called and closed the tooltip.
We now check whether the event target is not an element within the tooltip before executing the content of the focus-out callback.
Closes #40

:tada: This PR is included in version 2.7.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


thinkpad x1 gen6

First stab at Thinkpad x1 gen6 profile.

I think they're both maybe a little opinionated? It would be good if we had clearer guidelines about which options you should set vs. leave to users to set. I also think it would be nice if we could share as much as possible between the thinkpad configurations otherwise there's a ton of copy pasting.

@michaelpj i found things not to be to opinionated in #60. i think @yegortimoshenko has an idea about guidelines, but it would be nicer to see them written down.
sharing between thinkpad, with current setup this is definitely possible. i personally see this as an optimization step. once we notice that things duplicate we move the to "lower" placing default.nix.

closing this since this PR was reviewed in #60 (which just got merged) and integrated non-opinionated configuration from this PR.


[tsp-client] tsp-client init --local-spec-repo flag not working

Running the command:
tsp-client init -c <URL> --local-spec-repo <PATH TO LOCAL>
Any changes I make to my local tspconfig.yaml are not being used. I must commit and push the changes then update my URL.
Oddly enough, if I have an error in my local tspconfig.yaml, an error will be thrown when generating.
So somehow the command is throwing an error when the local project isn't valid, but it's also not using any changes made in my local project.

Thank you!!


SyntaxError when using ' in values list

Hi all,
I found today your library and liked it a lot.
Sadly I only found a little bug when I tested it on a bigger database:
When adding \' into a string in an insert values list the following error is thrown:
SyntaxError: Syntax error found near VALUES Clause (SELECT Statement)
It can be tested with an updated version of the basic-insert.sql test:
INSERT INTO concessions (item, size, id, price)
VALUES ('Lay\'s Nachos', 'Regular', null, null),
       ('Pizza', null, 8, 2.00);

I tried to fix it in the grammar.pegjs but I gave up. I hope you can fix it soon.

@philippsimon, according to the SQLite spec, \' is not considered valid string escape syntax.

A string constant is formed by enclosing the string in single quotes ('). A single quote within the string can be encoded by putting two single quotes in a row - as in Pascal. C-style escapes using the backslash character are not supported because they are not standard SQL.
reference

You can get your query to parse correctly by escaping single quotes as '' within a string literal:
INSERT INTO concessions (item, size, id, price)
VALUES ('Lay''s Nachos', 'Regular', null, null),
       ('Pizza', null, 8, 2.00);

@nwronski thanks! Didn't know that before.


Redownloading model on run command

What is the issue?
I downloaded llama3 with the ollama run llama3 command, and after it downloaded, first it failed with the error:
Error: llama runner process has terminated: exit status 0xc0000005
Then to see if the error reproduces, I ran ollama run llama3 but it started to download it all over again. I don't understand why its doing that. And its repeatedly doing that, everytime I run the command it just goes to redownloading.
Note: I changed model directory by creating environment variable OLLAMA_MODELS as per the instructions in F.A.Qs. Also, model is in both the C drive and my other drive that I chose.
Foremost, I would like it to stop redownloading and use what is on my system already. That is top priority.
Please help guys.
OS
Windows
GPU
Intel
CPU
Intel
Ollama version
0.1.38

Same here.
Intel i9 Macbook Darwin 23.1.0
ollama 0.1.38

Im experiencing this issue from https://github.com/ollama/ollama/issues/2551#issuecomment-2198366862
#2551

Looks like this issue slipped through the cracks.
We don't currently support Intel GPUs, so I presume you're running on CPU.  We should try to understand the crash, so the first thing I'd recommend is upgrade to the latest version, and if you're still seeing a crash, please share the server log so we can see why it crashed.
Occasionally we update models with fixes for templates, etc., and when you perform ollama pull we'll check to see if it has changed and pull only if it did.  ollama run should not do this, and will use the existing pulled model even if it has changed on ollama.com.  It's possible there's some filesystem corruption, running out of disk space, or other failure that's causing the model to be corrupted or removed and leading to the re-pull.  Again, server logs will help diagnose.


feat: make set method unavailable on router instance

Calling set method directly could cause state not in sync with actual location.
This includes both fix in types and in runtime.
Introduced new generic type StoreLike with minimal subset of properties most utility functions expect to work with.

ESLint issue need to be fixed.

@ai fixed

I find a better way d7735f7

More clear type names: ReadableStore/WritableStore
I hide router.set only in development to save size
?. removal was wrong. It was an error in ReadonlyIfObject type.

But thanks for raising the problem of read-only stores.

Perfect


Create fields.md on the documentation

Closes DOC-103
Change Summary

Rename schema.md to json_schema.md.
Create initial fields.md documentation.

please review


Bug Fixes: Maven Classifier, SemVer

Add support for maven dependency classifiers

    classifier:
    The classifier distinguishes artifacts that were built from the same POM but differ in content. It is some optional and arbitrary string that - if present - is appended to the artifact name just after the version number.

    As a motivation for this element, consider for example a project that offers an artifact targeting Java 11 but at the same time also an artifact that still supports Java 1.8. The first artifact could be equipped with the classifier jdk11 and the second one with jdk8 such that clients can choose which one to use.

    Another common use case for classifiers is to attach secondary artifacts to the project's main artifact. If you browse the Maven central repository, you will notice that the classifiers sources and javadoc are used to deploy the project source code and API docs along with the packaged class files.

Build Identifiers should not be used in SemVer comparisions

Dependabot was not ignoring the build identifier when determing if a dependency update was a semver update

This would be a great addition to dependabot! Thanks for making this @a1flecke!

@feelepxyz All checks pass. My employer is encountering both of these bugs which is lowering the utility of dependabot for us.  Please let me know if there is something that I need to do for this PR.

@feelepxyz I could use some assistance here.  I have been fighting what appears to be some flaky tests with VCR and requests to rubygems  Are these tests known to have issues?
Example results when running locally is VS Code devcontainer using the same rspec seed:
Finished in 2 minutes 38.6 seconds (files took 1.29 seconds to load)
680 examples, 0 failures

Randomized with seed 50232

[dependabot-core-dev] ~/dependabot-core/bundler $ pwd
/home/dependabot/dependabot-core/bundler

@feelepxyz It seems like the backports gem is missing a spec fixture in "fixtures/ruby/rubygems_responses/.  Can you provide direction on how I generate the info-backports file?
I believe this helper will then setup the missing HTTP stub
The CircleCI failure.
     Failure/Error: raise VCR::Errors::UnhandledHTTPRequestError.new(vcr_request)

     Dependabot::SharedHelpers::ChildProcessFailed:

       Child process raised VCR::Errors::UnhandledHTTPRequestError with message:

       ================================================================================

       An HTTP request has been made that VCR does not know how to handle:

         GET https://index.rubygems.org/info/backports

@feelepxyz It seems like the backports gem is missing a spec fixture in "fixtures/ruby/rubygems_responses/. Can you provide direction on how I generate the info-backports file?

We currently don't have any tooling but if you copy this https://index.rubygems.org/info/backports into a file named info-backports and add it https://github.com/dependabot/dependabot-core/blob/master/bundler/spec/fixtures/ruby/rubygems_responses/ it should work.
These stubs are pretty annoying as it seems like VCR/webmock don't always pick up that they are needed. Possibly because we run Bundler in a forked process 🤷♂

Looks like maven, common and bundler are passing so going to :shipit: this 🎉

@feelepxyz   Thanks for your help with this.

Sorry for digging into this old thread. This is a great improvement btw. Thanks for doing this.
We hit a corner case of this change so would like to clarify whether it is intended or not.
When checking whether a dependency is internal, meaning some submodule of a project, classifier is appended to the full name; however when building up the list of internal dependencies, there is no classifier appended because it is hard if not impossible to known. So the problem we have now is Dependabot would try to check newer version of a submodule if in parent pom.xml we have:
<dependencyManagement>
  <dependencies>
    <groupId>foo</groupId>
    <artifactId>bar</artifactId>
    <version>${project.version}</version>
    <classifier>foobar</classifier>
  </dependencies>
</dependencyManagement>

Because it believes foo:bar:${project.version}:foobar is not an internal dependency.
Due to another no-so-standard setup we have, a submodule would have an auto-bumped version of artifacts published every time CI passes in main branch (without changing source code), and the result of that is the published artifacts have higher version than the project's. In this case, for this classified dependency, Dependabot would try to bump the project version in parent pom.xml because it is a property, and that is not what we expect since it results an invalid parent pom.xml because all children still refer to the old version in parent section.
My question is, whether this is intended change? Would it be better we exclude classifier when checking internal dependencies? A possible consequence of skipping classifier is that the classified artifact is indeed not published by a submodule of the current project, but the probability of this setup would be very low.

@honnix Unfortunately a lot of time has passed since I created this PR.  Given the complexity of Maven I cannot be 100% certain, but you are likely accurate that using a classifier for intra-project dependencies is not going to work since the classifier comes into play when an artifact is published.

@a1flecke Thanks for replying and the confirmation. I will try to work on a PR to address this.


[Regression] Snapshot doesn't work with Xcode 7.3.1 (w/ workaround)

New Issue Checklist

[🚀] Updated fastlane to the latest version
[🚀] I have read the Contribution Guidelines

Issue Description
The Swift syntax of SnapshotHelper.swift has been upgraded to the newer Swift language, which breaks compatibility with older versions of Xcode (7.3.1.).
This would be fine but the snapshot command fails if you're using an older version of the SnapshotHelper.swift.
Workaround
Manually changing the // SnapshotHelperVersion [1.x] to 1.3, but with the 1.2 helper syntax bypasses the check and the screenshots run.
Suggested Fix
If an older version of Xcode is being used don't say the 1.2 Snapshot helper is out of date which currently fails the build.
Appreciate that Xcode 7.3.1 won't work forever with snapshot - but it'd be nice to have this one fixed in the core.

✅ fastlane environment ✅
Stack

Key
Value

OS
10.12.4

Ruby
2.3.0

Bundler?
false

Git
git version 2.7.4 (Apple Git-66)

Installation Source
~/.rbenv/versions/2.3.0/bin/fastlane

Host
Mac OS X 10.12.4 (16E195)

Ruby Lib Dir
~/.rbenv/versions/2.3.0/lib

OpenSSL Version
OpenSSL 1.0.2g  1 Mar 2016

Is contained
false

Is homebrew
false

Is installed via Fabric.app
false

Xcode Path
/Applications/Xcode-7.3.1.app/Contents/Developer/

Xcode Version
7.3.1

System Locale

Variable
Value

LANG
en_GB.UTF-8
✅

LC_ALL

LANGUAGE

fastlane files:
No Fastfile found
No Appfile found
fastlane gems

Gem
Version
Update-Status

fastlane
2.24.0
✅ Up-To-Date

Loaded fastlane plugins:
No plugins Loaded
Loaded gems

Gem
Version

did_you_mean
1.0.0

slack-notifier
1.5.1

i18n
0.7.0

json
1.8.3

thread_safe
0.3.5

tzinfo
1.2.2

minitest
5.8.3

activesupport
<IP_ADDRESS>

CFPropertyList
2.3.4

claide
1.0.1

colored
1.2

nanaimo
0.2.3

xcodeproj
1.4.2

rouge
1.11.1

xcpretty
0.2.4

terminal-notifier
1.7.1

unicode-display_width
1.1.2

terminal-table
1.7.3

plist
3.2.0

public_suffix
2.0.5

addressable
2.5.0

multipart-post
2.0.0

word_wrap
1.0.0

tty-screen
0.5.0

babosa
1.0.2

highline
1.7.8

commander-fastlane
4.4.4

excon
0.54.0

faraday
0.10.1

unf_ext
<IP_ADDRESS>

unf
0.1.4

domain_name
0.5.20161129

http-cookie
1.0.3

faraday-cookie_jar
0.0.6

fastimage
2.0.1

gh_inspector
1.0.3

uber
0.0.15

representable
2.3.0

retriable
2.1.0

mime-types-data
3.2016.0521

mime-types
3.1

hurley
0.2

little-plugger
1.1.4

multi_json
1.12.1

logging
2.1.0

jwt
1.5.6

memoist
0.15.0

os
0.9.6

signet
0.7.3

googleauth
0.5.1

httpclient
2.8.3

google-api-client
0.9.20

mini_magick
4.5.1

multi_xml
0.6.0

rubyzip
1.2.0

security
0.1.3

xcpretty-travis-formatter
0.0.4

dotenv
2.1.1

bundler
1.13.7

faraday_middleware
0.10.1

io-console
0.4.5

generated on: 2017-04-03

@bencollier checkout this PR: https://github.com/fastlane/fastlane/pull/8751 - setting skip_helper_version_check  should ignore the version check of the SwiftHelper - can you give it a try?

Installed and working a treat - thanks for the swift fix!

https://github.com/fastlane/fastlane/pull/8751 is merged.
i thought posting memes is a thing of the past? 👍

It is a thing of the past! My bad! The pun just got the better of me!

i am still for the return of the meme's 😆


[FLOC-2785] Docker plugin mount API

Relevant documentation: https://github.com/docker/docker/blob/master/docs/extend/plugins_volume.md#volumedrivermount

Marking as "Accepted" for now, but bounce back for review if major changes. Must fix tests before landing.


Disllow clicks outside MissThreshold for GameNotes

Closes #383

I moved Story.Notes.HasClickedNote = true; down to ClickNote() because I think it makes more sense for that flag to be applied there rather than a function meant for checking if a note is clickable


Lights brightness level and colors

Problem
the domain lights has multiple kind of lights but they all shown like switches

Why are you creating this issue?

Expected behavior
should provide brightness level and color to supported lights

Can you provide some details about your lights?
Please open the developer tools in home assistant and go to the states tab. Search for your light entity and share the attribute column shown there.

effect_list: blink, breathe, okay, channel_change, finish_effect, stop_effect
supported_color_modes: brightness
effect: null
color_mode: null
brightness: null
last_seen:<PHONE_NUMBER>250
level_config:
on_level: previous
linkquality: 43
update:
installed_version: 16909584
latest_version: 16909584
state: idle
update_available: false
friendly_name: Osram Light
supported_features: 44
brightness: 3
color:
h: 240
hue: 240
s: 100
saturation: 100
x: 0.136
'y': 0.04
color_mode: xy
color_options:
execute_if_off: false
color_temp: 575
color_temp_startup: 65535
last_seen:<PHONE_NUMBER>423
linkquality: 109
power_on_behavior: previous
update:
installed_version: 587806257
latest_version: 587806257
state: idle
update_available: false
device_class: update
friendly_name: Ikea Z Bedroom Corridor
min_color_temp_kelvin: 2202
max_color_temp_kelvin: 4000
min_mireds: 250
max_mireds: 454
effect_list: blink, breathe, okay, channel_change, finish_effect, stop_effect
supported_color_modes: color_temp
effect: null
color_mode: null
brightness: null
color_temp_kelvin: null
color_temp: null
hs_color: null
rgb_color: null
xy_color: null
color_options:
execute_if_off: false
color_temp_startup: 65535
last_seen:<PHONE_NUMBER>613
linkquality: 47
power_on_behavior: previous
update:
installed_version: 587814449
latest_version: 587814449
state: idle
update_available: false
friendly_name: Berandah_2
supported_features: 44

these lights via zigbee2mqtt not philips hue

What matter controller do you use, that shows them as switches only? Alexa? Google Home?
The attributes are a bit hard to read, because you copied all into one message without splitting them by light 😁
But as far as I understand:

the Osram light supports brightness. This should work with this addon. If not, it’s definitely a bug.
the IKEA light does not have supported_color_modes which is required for my current implementation. It only has supported_features which is the old/unsupported way. I could add support for supported_features but that’s more an enhancement than a bug. I’ll need to prioritize it against other bugs.
The Berandah light does not support color, but color temp. The current implementation only supports color temperature with colored lights. I was not thinking of color temperature without other colors. That’s my bad, but I assume it’s more a feature request than a bug.

i did put some spaces between the 3 lights , sorry if it appears connected together , and regarding the controller I'm using  other home assistant setup , because its the most showing details


Stacking Merge on top of flattened convolutional layers

Used this example as a starting point
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Dense, Flatten, Merge
from keras.layers.convolutional import Convolution2D

(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train = X_train.reshape(X_train.shape[0], 1, 28, 28)
X_train = X_train.astype("float32")

left = Sequential()
left.add(Convolution2D(5, 1, 3, 3))
left.add(Flatten())
left.add(Dense(3380, 10))

right = Sequential()
right.add(Convolution2D(5, 1, 3, 3))
right.add(Flatten())
right.add(Dense(3380, 10))

left.compile(loss='categorical_crossentropy', optimizer='sgd')
right.compile(loss='categorical_crossentropy', optimizer='sgd')

prediction_left = left.predict(X_train)
prediction_right = right.predict(X_train)

print(prediction_left.shape)
print(prediction_right.shape)

model = Sequential()
model.add(Merge([left, right], mode='sum'))
model.add(Dense(10, 10))

model.compile(loss='categorical_crossentropy', optimizer='sgd')
prediction_model = model.predict(X_train)
print(prediction_model.shape)

This code is yielding the following error for me:
     33
     34 model.compile(loss='categorical_crossentropy', optimizer='sgd')
---> 35 prediction_model = model.predict(X_train)
     36 print(prediction_model.shape)

/home/*/venv/lib/python2.7/site-packages/Keras-0.1.2-py2.7.egg/keras/models.pyc in predict(self, X, batch_size, verbose)
    468     def predict(self, X, batch_size=128, verbose=0):
    469         X = standardize_X(X)
--> 470         return self._predict_loop(self._predict, X, batch_size, verbose)[0]
    471
    472     def predict_proba(self, X, batch_size=128, verbose=1):

/home/*/venv/lib/python2.7/site-packages/Keras-0.1.2-py2.7.egg/keras/models.pyc in _predict_loop(self, f, ins, batch_size, verbose)
    247             ins_batch = slice_X(ins, batch_ids)
    248
--> 249             batch_outs = f(*ins_batch)
    250             if type(batch_outs) != list:
    251                 batch_outs = [batch_outs]

/home/*venv/lib/python2.7/site-packages/theano/compile/function_module.pyc in __call__(self, *args, **kwargs)
    579                     raise TypeError("Missing required input: %s" %
    580                                     getattr(self.inv_finder[c], 'variable',
--> 581                                             self.inv_finder[c]))
    582                 if c.provided > 1:
    583                     raise TypeError("Multiple values for input: %s" %

TypeError: Missing required input: <TensorType(float32, 4D)>

The Merge layer is supposed to work with mode='sum'when using two input layers with the same dimension. The way I see this code only differs from the example shown here by having convolutional layers before the dense ones
You can see with the prints that left and right sides of the network are working as expected and have exactly the same dimension
Am I doing something wrong here?

Edit: so the problem isn't even the convolutional layers, I literally copy-pasted the Merge example and this does not work
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers.core import Activation, Dense, Flatten, Merge

(X_train, Y_train), (X_test, Y_test) = mnist.load_data()

X_train = X_train.reshape(X_train.shape[0], 28*28)
X_test = X_test.reshape(X_test.shape[0], 28*28)
X_train = X_train.astype("float32")
X_test = X_test.astype("float32")

left = Sequential()
left.add(Dense(784, 50))
left.add(Activation('relu'))

right = Sequential()
right.add(Dense(784, 50))
right.add(Activation('relu'))

model = Sequential()
model.add(Merge([left, right], mode='sum'))

model.add(Dense(50, 10))
model.add(Activation('softmax'))

model.compile(loss='categorical_crossentropy', optimizer='rmsprop')

model.predict(X_train)

It is ok if I compile right and left alone so it is not a problem with the inputs I suppose

Edit 2: Oops, just realized I'm not using a tuple of inputs

Can you please show the exact code you used, in the end, which worked? Thanks!


feat: implement input module

Please provide detailed description and e2e tests

Or convert it to draft, if it's not review-ready

In order to test against WPT, remove the __dir__ files within input.

Are you intending to break this PR down into smaller pieces? http://go/small-cls

@thiagowfx Depends. We could separate the schema and parser changes, but for the main implementation, do we want to have partial implementation?

The splits that you did recently are a good start.
It's OK to keep the main implementation in a single PR as it's unnecessary burden for you to try to split them.
I was mostly referring to the helper/aux files and to the protocol/schema.


Stop using non-standard protocols in URLs and switch to native URL() over url-parse

We'd like to drop usage of url-parse and instead use the standard URL.  However, these two do not have compatible behavior for parsing URLs with non-standard protocols, and we currently use non-standard protocols in a few places (e.g. fluid:, fluid-odsp:, fluid-test:).  To successfully move to the standard URL for parsing, we'll want to eliminate this non-standard protocol usage (which we really shouldn't be doing of its own merit anyway).
To eliminate the usage, there are dependent usages we'll have to change accordingly:

[ ] Deprecate and remove .protocolName from IDocumentServiceFactory
[ ] Remove or redesign MultiDocumentServiceFactory to avoid depending on protocol for factory selection
[ ] Update url resolvers to resolve urls with standard protocols
[ ] Ensure partners have a path forward after the above changes
[ ] Update fluid-fetch to avoid relying on protocol
[ ] Update other references to the non-standard protocols throughout the repo (tests, etc.)
[ ] ...others?
[ ] Swap to native URL and drop url-parse

This is still a good idea.


[bulk] Validation: absolute links

Docs Build status updates of commit f467b53:
:warning: Validation status: warnings

File
Status
Preview URL
Details

docs-ref-services/preview/key-vault-index.md
:warning:Warning
View (azure-java-preview)
Details

docs-ref-services/preview/key-vault-index.md

Line 2, Column 1: [Warning: ms-prod-and-service - See documentation] Only one of the following attributes can exist: 'ms.prod', 'ms.service'. Use ms.prod for on-premise products, or ms.service for cloud services.
Line 11, Column 16: [Warning: ms-prod-technology-invalid - See documentation] Invalid value for 'ms.technology': 'azure' is not valid with 'ms.prod' value 'azure'.
Line 3, Column 14: [Suggestion: duplicate-descriptions - See documentation] Attribute 'description' with value 'Overview of the Azure Key Vault libraries for Java' is duplicated in 'docs-ref-services/preview/key-vault-index.md(3,14)', 'docs-ref-services/preview/key-vault.md(3,14)'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 82bf95a:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error: RuningBuildFailed] Some unexpected errors happened when running build, please open a ticket in https://aka.ms/SiteHelp and include the error report for our team to troubleshoot

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 82bf95a:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error: RuningBuildFailed] Some unexpected errors happened when running build, please open a ticket in https://aka.ms/SiteHelp and include the error report for our team to troubleshoot

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Global effort to fix build validation errors
@sima-zhu
The Content & Learning team is fixing build validation errors on docs.microsoft.com for the rest of H2. This will eliminate potential accessibility, security, and usability issues. This PR includes only build validation fixes and does not change other content. Please review within five business days and merge or comment in the PR with any changes you’d like to see. Thank you!

Global effort to fix build validation errors
@sima-zhu
The Content & Learning team is fixing build validation errors on docs.microsoft.com for the rest of H2. This will eliminate potential accessibility, security, and usability issues. This PR includes only build validation fixes and does not change other content. Please review within five business days and merge or comment in the PR with any changes you’d like to see. Thank you!


Phase out the use of the non compressed runtimes

There is no need to remove the non compressed information but it is required to ensure the summary does use the compressed versions now.

Currently, we show both so there is no emergency. However, the compressed runtimes being now the established defaults, the following:

prop
ipfs
sha256
wasm path

in the json head should be adjusted.


Re-generating index/map/toc files to support 3 new profiles

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

FromYmlToTOC-INDEX.ps1
:white_check_mark:Succeeded

advocates/index.html.yml
:white_check_mark:Succeeded
View

advocates/index.yml
:white_check_mark:Succeeded
View

advocates/map.yml
:white_check_mark:Succeeded
View

advocates/toc.yml
:white_check_mark:Succeeded
View

advocates/yoshio-terada.yml
:white_check_mark:Succeeded
View

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


chore(github-actions): fix on merge workflow

address #1333

Codecov Report

Merging #1337 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1337   +/-   ##
=======================================
  Coverage   76.99%   76.99%
=======================================
  Files         222      222
  Lines        4347     4347
  Branches      840      840
=======================================
  Hits         3347     3347
  Misses        874      874
  Partials      126      126

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0be2757...76617e3. Read the comment docs.


Updating Curriculum Development

Updating curriculum development branch.

Merging without approval because I am the lead developer of the curriculum. If any fixes or changes are needed, please reopen this pull request.
Students: Remember that you're application will need the approval of at least 2 other reviewers (developers). More information is located in the README.MD.

This pull request was created by mistake, please ignore this.


Nearest Neighbors Log

model = tc.nearest_neighbors.create(references) with an SFrame of all strings outputs Starting brute force nearest neighbors model training. with no more logs to follow it. Although I think the model still gets created properly, or at least I hope. But maybe just the log does not reflect the progress and completion.

Does the call to tc.nearest_neighbors.create return? If so, try calling print(model).

You could try it out with this as references
test_references = tc.SFrame({'text1': ['Hi, how are you?', 'I am great, thanks!']}). The model gets created and print shows:
Class                          : NearestNeighborsModel

Attributes
----------
Method                         : brute_force
Number of distance components  : 1
Number of examples             : 2
Number of feature columns      : 1
Number of unpacked features    : 1
Total training time (seconds)  : 1.0089

So as I mentioned before the problem is only the generation of logs. But also I am curios why the Distance is not shown in the print statement. I am curios to know what distance function was used.


Configuration option to set the commit convention

A variety of repos follows certain commit convention (usually Angular's, ESLint's, ...) which is there to enforce a consistent commit message style e.g. (taken from an actual commit):
Update(Python): Added tests and documentation

Added the documentation, unit tests and added the feature that allows sorting in descending order

BREAKING CHANGE: One more parameter

closes #2

The commit log above uses the ESLINT commit convention (obtained using commitizen i.e. git cz).
This is usually done in repos where an automated changelog build process is in place, say using the following NPM scripts:
"scripts": {
  "changelog": "conventional-changelog -p eslint -i CHANGELOG.md -s -r 0",
  "version": "npm run changelog && git add CHANGELOG.md"
}

With the following package.json config:
"config": {
  "commitizen": {
    "path": "cz-adapter-eslint"
  }
}

So if imgbot was to commit to the branch especially with auto-merging enabled (cf. #231 ), it would be ideal that it respects the commit convention (if there's one in place) thus preventing some inconsistencies and some unnecessary manual changes.
Possible solutions
A. Look in package.json for the "config.commitizen" key and then use that for the commit
B. Analyse the commits (maybe like commit-lint would do) and determine the commit convention (assuming there's one) and use that
C. Look for a .czrc file (which could be something like { "path": "cz-adapter-eslint" }) and then use it to commit with the correct style
D. Have a setting in .imgbotconfig that will instruct imgbot on how to write the commit messages.

If I understood the project's contribution guide, this would be stuff going on CommitMessage.cs and PullRequest.cs right?
I suppose that the file that would be reading a settings file say ./.github/.imgbot.yml should be able to modify KnownGitHubs.cs's CommitMessageTitle field.
I suck at C# and haven't really worked on any bots so I'm not sure I'll be able to do this on my own but can help.

Hi @Berkmann18! you're certainly on the right track.
CommitMessage.cs has a parameter that contains the compression result and returns a commit message.
This return of this function typically looks like, for example
[ImgBot] Optimize images

*Total -- 321.19kb -> 123.05kb (7.81%)

/path/image1.png -- 4.19kb -> 0.63kb (4.9%)
/path/image2.png -- 4.19kb -> 0.63kb (84.9%)

where the first line is the tile.
We can keep the constant in KnownGitHubs.cs as a fallback
maybe rename it to DefaultCommitMessageTitle
Then I think you would want additional parameter(s) passed in besides the CompressionResult[] to the Create() method to override the first line in the commit message and affect the title.
You can use CommitMessageTests to help develop the algorithm
Once you have that all working, we can figure out how to get the right parameters populated and passed to the function, depending on what it is you end up needing to pull it off.
And finally, when all that is done and working we can update PullRequest.cs to base the title of the pull request on the first line of the commit, so we keep these in sync. We do a similar thing to generate the markdown table in the pull request body.
Let me know how it goes!

why not .othersconfig?

Hi @MatheusRV thanks for bumping this.
What is .othersconfig ?

Can we reopen this? I just encountered this in one of my repos (https://github.com/keplersj/cises/pull/125.) I used squash and merge to change the commit message, but having it done automatically would be a nice feature to have.

hi @keplersj!
Thanks for linking to a recent example
Re-opening


Allow all 8.* versions of PHP

We will not support any future PHP versions. Only versions we have tested with. Replaced with !338


calculate_necessary_state_changes should detect changes to Application images

Currently the with_cmp decorator on Application compares only Application names. Changes to the image should also be detected.

We are moving our development planning to JIRA. This issue is now being tracked at https://clusterhq.atlassian.net/browse/FLOC-267. You are welcome to file additional issues in GitHub if that's easier for you.


[C++] Use strongly typed enums in generated code

Using enum class could improve type safety in the C++ API. It would also be a API breakage.Something to consider for the future.

Duplicate of https://github.com/real-logic/simple-binary-encoding/issues/690


Added redirect for progressive-web-apps/index

Added redirect from progressive-web-apps/index link to progressive-web-apps-chromium/index page.

Docs Build status updates of commit 3a59e8f:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

.openpublishing.redirection.json
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Flesh out project links and add more specificity

Currently, the README:

does not link to specific projects (either a live demo or the repo)
does not add specific links to training resources, like Teach LA's training tracks or Hackschool

This should be a simple resource-gathering and Markdown fix.
Going to tag in Advaith & Anna given their role on the dev team, and Einar since he has familiarity on the content from Dev Pathways!

ACM does a LOT of stuff - how are we planning on deciding which specific projects and training resources are going to be here? Or is it just going to be all ongoing projects? - this would probably end up cluttering the landing page.

I think we would not feature all of our projects. Instead, we would just pick highlights that are either:

good places to start (in terms of easy to pick up)
very actively maintained and technically interesting

Feel free to jig stuff around, I just made this as a first draft.


[PE-D][Tester E] Formatting issues in UG

The user guide has a lot of pages which are half empty.
examples:

This results in the expected outputs appearing on different pages as the command description.

Labels: severity.Low type.DocumentationBug
original: tanwartushar/ped#5

dup of #240

Closing this since dup


Ia32MtrrDefaultTypeMsr

(Please use this issue template for a bug report. You need not use it for any other type of issues.)
Description
(Please describe)
kIa32MtrrDefType  intel difference？
Expected behavior

(Please describe)
Actual behavior
(Please describe)
Steps to reproduce the problem
(Please describe if applicable)
Specifications

Commit (eg, 2099af7): (Please fill in)

OS version (ie, output of the "ver" command, or N/A): (Please fill in)

Architecture: (Please choose: x86, x64, Both, or N/A)

Hardware (eg, Physical, VMware 14, Bochs, or N/A): (Please fill in)

Details: (Please describe)

--
NOTE: The Details section may include related log files, crash dump files,
impact/importance of the issue, proposed fix and anything you may think useful
for fixing the issue.
It is hugely helpful and speeds up investigation of a bug check or a hang issue
if the following files are provided:

a Debug build version of a compiled SYS file and a PDB file
a full set of source code used to build the SYS file if it has been modified
a log file (i.e., C:\Windows\HyperPlatform.log)
a system crash dump file (i.e., C:\Windows\MEMORY.DMP)

GitHub does not support attaching of any binary files into issues. You may use
any free file hosting services, however. Here are some of those services:

OneDrive
DropBox
GoogleDrive
Sendspace
Mega.nz

I think @a132266 is seeing that the definitions in the table and one in the <IP_ADDRESS> IA32_MTRR_DEF_TYPE MSR section are different. I would suggest

I would suggest asking at the Intel forum (https://software.intel.com/en-us/forums/platform-and-technology-discussion). They are usually rather prompt.

Thanks for answers


Timedimension on IE /Edge - Blink, Flicker behaviour

When you change the value of FPS, the behaviour in IE/Edge is different from Chrome, FF, Opera.
The wms layers transition is not smooth, with a blinky/flickery transition (It could potentially trigger an epilepsy attack on some users).
Do you know were in the Timedimension code we can try to get this behaviour fixed?

Hi Filipe,
it only shows up if you change the value of the FPS? Can you test it with different values at timeDimensionControlOptions.playerOptions.transitionTime?
TimeDimension WMS layers are animated using new hide and show methods at L.TileLayer (see https://github.com/socib/Leaflet.TimeDimension/blob/9ad9ab7355aeb32725a0c579912ebdfff43e200f/src/leaflet.timedimension.layer.wms.js#L380). The same methods for L.NonTiledLayer.
Can you try changing visibility value rather than display?

Hi Biel,
Edge browser - blinks at 1 FPS, and more (tested 5 fps, and 10 fps);
IE 11 - blinks at 1 FPS, and more (tested 5 fps, and 10 fps);
The behaviour is the same changing visibility instead of display in the code.
The flickering also happens when the leaflet view is changing, so, if the user is moving the map in any direction, zooming in and out, a new WMS is requested and the flicker happens, and the transition between wms layers isn't smooth like in Chrome, FF and Opera.

I have a similar problem on FF43 with big NonTiledLayer images with Leaflet 1.0b2
I managed to reduce the flickering by changing hide/show to visibility instead of display but it breaks current NTL code as it uses visibility to hide itself when bounds are out of limits.

I can open a PR with this fix, but it requires a change on NTL.
cc @oliverheilig


Default filter settings

Add a default filter button to allow the end user to save it own preference.

No framework provided for such feature


THORN-2340: update OpenShift REST Client Java to 7.0.0

Motivation
OpenShift REST Client Java 7.0.0.Final has been released,
which depends on OkHttp 3.11.0. This means we can unify
our OkHttp versions once again, after having to split
in THORN-2165.
Modifications
OpenShift REST Client Java was updated to 7.0.0.Final.
The OkHttp library is kept on 3.9.0 to ease dependency
alignment for product.
The Okio library was updated to 1.13.0 because that's
what OkHttp 3.9.0 uses.
Result
Single version of OkHttp across all fractions.
No behavioral difference.

[x] Have you followed the guidelines in our Contributing document?
[x] [v2] Have you created a JIRA and used it in the commit message?
[ ] [v4] Have you created a GitHub Issue and used it in the commit message?
[x] Have you checked to ensure there aren't other open Pull Requests for the same issue?
[ ] Have you built the project locally prior to submission with mvn clean install?

retest this please


z-index on series

I was dealing with a mixed chart and I couldn't find a way where I can control over which line/column shown over the other.
currently the columns bar hiding the lines and I want it to be the other way.
I tried to play with it but couldn't find a way to do it.
is it possible to add a property on the series like order or z-index?
Thanks

@dor272 did you ever find a fix to this?

Found other people on stackoverflow trying to solve this problem, but without success. There doesn't seem to be a way to order the series and decide which one goes over the others.
I tried reordering the series, but the stacking is always the same. The lines go always over series charts.

have same problem

I am from 2024, still have same problem.
...
        series: [{
          name: "Current period",
          data: [...],
          zIndex: 2 // this option only affects line and area charts
        }, {
          name: "Same day past period",
          data: [...],
          zIndex: 1 // this option only affects line and area charts
        }],
...

changing zIndex doesn't change anything...

This is still not working


Fix Bittrex _subCount execption

Addresses issue #99

Fix ordering of events for connection
Fix race condition logic with subscribe

Pull Request Test Coverage Report for Build 356

15 of 16 (93.75%)  changed or added relevant lines in 2 files are covered.
2 unchanged lines in 2 files lost coverage.
Overall coverage increased (+0.07%) to 84.496%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/exchanges/kucoin-client.js
3
4
75.0%

Files with Coverage Reduction
New Missed Lines
%

src/exchanges/ethfinex-client.js
1
94.57%

src/exchanges/kucoin-client.js
1
85.52%

Totals

Change from base Build 352:
0.07%

Covered Lines:
2355

Relevant Lines:
2630

💛  - Coveralls


broadcast region when we use etcdBaseKV to saveRegion

I am reading the core/kv.go and found the following code
// SaveRegion saves one region to KV.
func (kv *KV) SaveRegion(region *metapb.Region) error {
	if atomic.LoadInt32(&kv.useRegionKV) > 0 {
		return kv.regionKV.SaveRegion(region)
	}
	return saveProto(kv.KVBase, regionPath(region.GetId()), region)
}

If useRegionKV=0, we will use kv.KVBase to saveRegion, but at the same time region_syncer is still runing in the
https://github.com/pingcap/pd/blob/59968e186cf23bdc172cd7be3c3c7e22aeb600f0/server/cluster.go#L128
And the leader will broadcast the region to member due to
https://github.com/pingcap/pd/blob/59968e186cf23bdc172cd7be3c3c7e22aeb600f0/server/cluster_info.go#L563
Maybe it could be skipped when we dont use leveldb as kvstore to decrease the network flow?

@bradyjoestar  It's not always broadcast the changed, only if the client has established the stream with the leader. and if not use leveldb, it will not establish stream with the leader.

But there is a problem that the config change only know if PD restart or leader changed, maybe you can help us to improve it.

@nolouch my pleasure if it is not hurry . :grin:

@nolunch  I will try to handle it. :ok_hand:


Improvement

Make a wildcard like --sep that separates docstrings from """ by one spacebar.

This is requesting the same as #39.  Closing as a duplicate.


Custom fields in JWT payload

Is there a way to get the custom fields that i put into the JWT payload?

Hi @long1eu, thanks for reaching out!
We currently do not support accessing these fields directly in the SDK. The StitchUserProfile currently only exposes the metadata fields that are exposed by Google and Facebook login.
We will look into supporting custom fields in a future release of our SDKs. Until then, you can access the custom metadata fields by creating a Stitch function that uses the data field in the user context to return the necessary fields. See https://docs.mongodb.com/stitch/functions/context/#context-user
Let me know if you have any questions about this approach!
I'll also leave this issue open until we have a way to access these fields directly in the SDK.

How do I store Metadata Fields from AWS Cognito's JWT? I am looking at the Auth Providers > Edit custom-token where it says "Specify where metadata is located in your JWT. Dot notation is supported." Could you elaborate how to get the path?


Trouble changing wwwc in modality DX !

I can reset the wwwc of images(DX),but there are no changes in viewerport.JUST the modality DX can not adjust,CT,MR are OK.Someone have the same issue like this?Can someone solve it,thanks!!!

Hi 👋, I would be more than happy to look into this if you were to provide a minimal reproduction.
Ideally, in a GitHub repo, or at a minimum, the DX image(s) you are experiencing this issue with. If the issue is reproducible on our live viewer, a link to a study that reproduces the issue, and the steps required to reproduce it, would suffice.

Hi 👋, I would be more than happy to look into this if you were to provide a minimal reproduction.
Ideally, in a GitHub repo, or at a minimum, the DX image(s) you are experiencing this issue with. If the issue is reproducible on our live viewer, a link to a study that reproduces the issue, and the steps required to reproduce it, would suffice.
Hello，now i have no way import DX image to live viewer.I can change wwwc of DX image in other DICOM viewer，so issue may happen in codes.I email the demo to you or can you email me a demo ，then  i have a try.

It is probably applying a VOI LUT embedded in the DICOM file, which will lock your WW/WC while it is applied. See https://rawgit.com/cornerstonejs/cornerstone/master/example/modalityAndVIOLut/index.html
You can test it here: https://rawgit.com/cornerstonejs/cornerstoneWADOImageLoader/master/examples/dicomfile/index.html
Uncheck the Apply VOI LUT box, and see if you can use W/L.

It is probably applying a VOI LUT embedded in the DICOM file, which will lock your WW/WC while it is applied. See https://rawgit.com/cornerstonejs/cornerstone/master/example/modalityAndVIOLut/index.html
You can test it here: https://rawgit.com/cornerstonejs/cornerstoneWADOImageLoader/master/examples/dicomfile/index.html
Uncheck the Apply VOI LUT box, and see if you can use W/L.

sorry，i can not test the image，when i select the image，it shows 【object object】on the head，i think the example can not cover the type of my image


Fix support for empty attributes

Initial checklist

[x] I read the support docs
[x] I read the contributing guide
[x] I agree to follow the code of conduct
[x] I searched issues and couldn’t find anything (or linked relevant results below)
[x] If applicable, I’ve added docs and tests

Description of changes
This change adds handling for attributes that are explicitly set with an empty string "" or all whitespace "  ".
Prior to this, html comments that contained an empty string attribute were ignored completely.
For example, given the following html commentMarker returns null.
const html = '<!-- action target="1234" field="type" value="" -->'
const marker = commentMarker(node)
// marker === null

Additionally, attributes with whitespace only were coerced to number resulting in unexpected behavior.
For example, the following results in an attribute value of 0.
const html = '<!-- action target="1234" field="type" value=" " -->'
const marker = commentMarker(node)
// marker.parameters === {
//   target: 1234,
//   field: "type",
//   value: 0,
// }

Thanks, released in 2.1.2!!

@wooorm @tmelliottjr thank you both for this! 👏

Thank you for the quick 👀 on this, greatly appreciated!


Using $compile() seems to ignore my directive's "restrict" setting

Please see this JSBin: http://jsbin.com/puxevo/1/edit?js,console
Note that if you change the CSS class in the string being $compile()-ed, it works!

This can probably be closed as won't fix - $compile needs to fetch the DDO, but if it is being called in the factory function that ultimately returns the DDO, one will never get the DDO. Angular needs to get the DDO for the directive with that class name because it needs to determine if it satisfies the restrict settings.

@wesleycho is right. This is working as expected. We need to execute the directive factory in order to get the restrict property (which is less than ideal, but would be too much of breaking change to get rid of - that's a different discussion though).
There are several ways to work around this limitation, e.g. compile inside the compile function (and cache the result after the first time), add the wut class after compiling the element (assuming there are no other directives that match .wut), use ng-class or interpolation etc.


Need instructions on multiple functions in one project

The title indicates that you can develop Azure Functions plural.  The instructions here are for only developing one function.  If you have one local.settings.json file, how should I handle one settings file for multiple functions.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 369aefa6-74da-3001-f835-15abdfffd5d6
Version Independent ID: 2cc2a2db-d059-3252-7a1a-4261b7d3c37c
Content: Develop Azure Functions using Visual Studio
Content Source: articles/azure-functions/functions-develop-vs.md
Service: azure-functions
GitHub Login: @ggailey777
Microsoft Alias: glenga

Hey @adamjones-OIA - I believe the name of the product is Functions (in plural) and that is the context being communicated throughout our docs.
To answer your question around the use of local.settings.json file, it isn't necessarily specific to each of the function you may have running, rather, it contains the settings for the local environment in which you could develop and run one or more functions.

Hello Mike, Thanks for the reply.  Is the following statement is true:  If we have one Function App with multiple functions, the design is to have them all share one configuration file.  Both locally and in the cloud.
For my case, I'd like two Azure Blob Storage trigger functions.  For Example, I have the blob trigger path a configuration because I have multiple environments:
[FunctionName("ProcessOpenOrders")]
public static async Task Run([BlobTrigger("%TriggerBlobPath%/{name}")]Stream myBlob, string name, ILogger logger, [Blob("%OutboundBlobPath%/{name}", FileAccess.Write)] Stream outputStream, ExecutionContext context)

[FunctionName("ProcessMaterialsUpload")]
public static async Task Run([BlobTrigger("%TriggerBlobPath%/{name}")]Stream myBlob, string name, ILogger logger, [Blob("%OutboundBlobPath%/{name}", FileAccess.Write)] Stream outputStream, ExecutionContext context)

This would be illegal because I have two configurations with the name TriggerBlobPath.  Instead, I should have specific keys:
[FunctionName("ProcessOpenOrders")]
public static async Task Run([BlobTrigger("%ProcessOpenOrdersTriggerBlobPath%/{name}")]Stream myBlob, string name, ILogger logger, [Blob("%ProcessOpenOrdersOutboundBlobPath%/{name}", FileAccess.Write)] Stream outputStream, ExecutionContext context)

[FunctionName("ProcessMaterialsUpload")]
public static async Task Run([BlobTrigger("%ProcessMaterialsUploadTriggerBlobPath%/{name}")]Stream myBlob, string name, ILogger logger, [Blob("%ProcessMaterialsUploadOutboundBlobPath%/{name}", FileAccess.Write)] Stream outputStream, ExecutionContext context)

(With one function, I am putting this configuration in the values section of the local.settings.json file locally and the configuration section of the function app in azure)

@adamjones-OIA the function app provides an execution context for the set of functions deployed into it.
You are correct that two Blob storage triggers would need to resolve to two separate paths.
This article has a section that shows how to use VS tooling to add a function to an existing function app.  The tooling creates a new class file for each function, but you can also create a function by adding a new static method to an existing class.
Is there some specific information about using the local.settings.json file that you think we need to add?


Added a MobileNetV3 backend

Based on ConvNeXt backend code, this PR adds the 3.4M param MobileNetV3 backend.
Again, I did not write test functions.

@lucasvandijck , thanks for adding mobilenet. Could you maybe reuse the upsampling block from the convnext to avoid code duplication?

Again, I did not write test functions.

That's okay. All registered backbones get tested here

@lucasvandijck , thanks for adding mobilenet. Could you maybe reuse the upsampling block from the convnext to avoid code duplication?

Again, I did not write test functions.

That's okay. All registered backbones get tested here

This should be fixed. I didn't like the inter-backend dependencies, hence why I re-implemented it, but this should work fine as well.

I didn't like the inter-backend dependencies, hence why I re-implemented it, but this should work fine as well.

yeah the proper way would be to refactor that upsamplingblock, but this is still bettern than duplicating code :+1:


ocropus-gpageseg --> too many connected compenents   /

This is my problem when trying to ocr with my pngs
book/0001.bin.png SKIPPED too many connnected components for a page image (21112 > 1176) (use -n to disable this check)

./ocropus-gpageseg -n book/0001.bin.png
INFO:
INFO:  book/0001.bin.png
INFO:  scale 4.472135955
Traceback (most recent call last):
File "./ocropus-gpageseg", line 423, in safe_process1
process1(job)
File "./ocropus-gpageseg", line 373, in process1
print_error("%s: scale (%g) less than --minscale; skipping\n"%(fname,str(scale)))
TypeError: float argument required, not str

@jmokoistinen when using -n behind the command it'll be skipped and the script will work.
-n disables all checks.

./ocropus-gpageseg '/home/rajeev/workspace/ocr-py/shots-1.jpg' -n

INFO:
INFO:  ########## ./ocropus-gpageseg /home/rajeev/workspace/ocr-py/shots-1.jpg
INFO:
INFO:  /home/rajeev/workspace/ocr-py/shots-1.jpg
INFO:  scale 8.0
Traceback (most recent call last):
File "./ocropus-gpageseg", line 423, in safe_process1
process1(job)
File "./ocropus-gpageseg", line 373, in process1
print_error("%s: scale (%g) less than --minscale; skipping\n"%(fname,str(scale)))
TypeError: float argument required, not str

The initial error
File "./ocropus-gpageseg", line 373, in process1
print_error("%s: scale (%g) less than --minscale; skipping\n"%(fname,str(scale)))
TypeError: float argument required, not str

is corrected by 1c06ec3.
This will now produce an error message for the user, because the computed scale does not seem to be correct, because it is too small (maybe tiny font size? newspaper?). As indicated by @rajeevkannav you can overwrite the default values for these checks.


Adding Serbian community resources

Initial addition of all things I can collect about Serbian projects. Note that we keep programming resources under Gitlab, and I added them under "github" type (since there were no "gitlab" type and I though that generic "osm" type will not do the justice).

since there were no "gitlab" type and I though that generic "osm" type will not do the justice

We could add a gitlab type - FontAwesome supports it:  https://fontawesome.com/icons?d=gallery&q=gitlab&s=brands

since there were no "gitlab" type and I though that generic "osm" type will not do the justice

We could add a gitlab type - FontAwesome supports it:  https://fontawesome.com/icons?d=gallery&q=gitlab&s=brands

Or maybe you could broaden definition of "github" type. Rename it to "git" (everything related to git, including github, gitlab etc.) with git icon (because git != github :D). Or, if you don't want to stop with git, rename it to "sources", such that it represent collection of source codes for a given community (but, I cannot find appropriate FontAwesome icon for it).
(I am just throwing ideas, any approach is fine by me!)

Or maybe you could broaden definition of "github" type. Rename it to "git" (everything related to git, including github, gitlab etc.) with git icon (because git != github :D). Or, if you don't want to stop with git, rename it to "sources", such that it represent collection of source codes for a given community (but, I cannot find appropriate FontAwesome icon for it).
(I am just throwing ideas, any approach is fine by me!)

The purpose of the type is to match it to a reasonable icon, so I'll just add a gitlab one.
I don't think things like github and gitlab are really the kinds of "communities" that you want to push onto people who are just casuallly editing around Serbia, but if you want to do that we support it.

The purpose of the type is to match it to a reasonable icon, so I'll just add a gitlab one.
I don't think things like github and gitlab are really the kinds of "communities" that you want to push onto people who are just casuallly editing around Serbia, but if you want to do that we support it.

No, didn't want to push anything to people (especially explaining them why github != git) :smile: . But, I guess approach with "gitlab" type (as you did it) makes sense (otherwise osm-community-index wouldn't have matrix, telegram, irc... as types, you would have only one named "messaging" :smile: ). At least this is how I rationalized it to myself:) Anyway, thanks for very quick response and addition!

No, didn't want to push anything to people (especially explaining them why github != git) :smile: . But, I guess approach with "gitlab" type (as you did it) makes sense (otherwise osm-community-index wouldn't have matrix, telegram, irc... as types, you would have only one named "messaging" :smile: ). At least this is how I rationalized it to myself:) Anyway, thanks for very quick response and addition!

GitLab resources have been added to https://community.osm.be/ 🎉

GitLab resources have been added to https://community.osm.be/ 🎉


how to programmatic usage ?

https://the-guild.dev/graphql/codegen/docs/advanced/programmatic-usage
I failed to add your plugin

Hello @productdevbook , could you please share what you have tried please? 🙂

Your structure does not support the structure in the link I posted. Can you create a plugin structure

Hello @productdevbook 👋
Sorry, it took longer than I wish to get to your message.
It looks like the programmatic usage link you posted is mainly for a normal plugin. Here's roughly what's needed to run codegen programmatically for a preset:
// generate.mjs
import * as fs from 'fs/promises';
import * as path from 'path';
import * as url from 'url';
import { codegen } from '@graphql-codegen/core';
import { preset } from '@eddeee888/gcg-typescript-resolver-files';
import { loadFilesSync } from '@graphql-tools/load-files';
import { mergeTypeDefs } from '@graphql-tools/merge';
import { loadSchema } from '@graphql-tools/load';
import { GraphQLFileLoader } from '@graphql-tools/graphql-file-loader';

const __dirname = path.dirname(url.fileURLToPath(import.meta.url));

const typeDefs = mergeTypeDefs(loadFilesSync('libs/graph-backend/**/*.graphqls'));
const schemas = await loadSchema('libs/graph-backend/**/*.graphqls', {
  loaders: [new GraphQLFileLoader()],
  includeSources: true, // Note: This is important as the preset relies on sources to do smart stuff
});

const generateOptions = await preset.buildGeneratesSection({
  baseOutputDir: 'src/schema',
  presetConfig: {},
  schema: typeDefs,
  schemaAst: schemas,
  config: {},
  documents: [],
});

Promise.all(
  generateOptions.map(async (option) => {
    const output = await codegen(option);
    await fs.writeFile(path.join(__dirname, option.filename), output, 'utf8');
  })
);

Then, you can run the following to generate the files:
$ node generate.mjs

Note: As you can see from the above, we are loading .graphql files twice to generate typeDefs and schemas. I'm not sure what's the best way to load typeDefs and schemas once. Maybe something to check with the folks in GraphQL Discord?

Thank you working, but example 2 please folder check.

cd example2
pnpm install
cd templates/nuxt
tsx index.ts

Dont export example2/src It takes the name of the road.

I think your baseOutputDir has to be relative to your current folder.
So, assuming your current working directory is /Users/productdevbook/testalani/programatic-graphql, then it'd be something like this: example2/src

but I need to be able to give it where I want, I'm building a dynamic structure. You need to send it accordingly.
I think you need to count something like cwd and let us change it.

Sorry for not getting to this earlier. I think that could be a good feature request in https://github.com/dotansimha/graphql-code-generator 🙂
Server Preset is basically a plugin for the generator and if it's actioned there, it'd work here too

Closing this as it looks solved upstream. Thank you!


Fix EOL npm:file-loader:6.2.0 in Web Console

Description

file-loader in used in web-console - Ref
It is deprecated and is not recommended.

Proposed Fix

Migrate to asset modules

working on it


How can we support depending on JS libraries (on NPM)

There's a lot of valuable JS libraries out there developers would love to use in Titanium apps and as part as modules.
Alloy comes with Underscore, Backbone and Momentjs but the last two need to be modified in order to work in Titanium.

Backbone because we don't have a shim for XMLHttpRequest which it relies on via jQuery/Zepto/ender.
Momentjs because our require() is not Nodejs compliant.

It would be great if we could resolve TIMOB-16078 and/or integrate Titaniumifier in the app/module build process to wrap Nodejs dependencies into a single CommonJS module including common shims.

Author and maintainer of Titaniumifier here. I’m gonna braindump as much as I can. Please:

forgive me if it looks confused, and
follow up with your doubts.

I think ideally we'd resolve TIMOB-16078 and then we'd only need shims for NodeJS APIs which can then be simply separate CommonJS libs.

First of all we need some lexicon to start with. I hope Appc will starts following CommonJS and now standard terminology.

modules are

akin to compilation units
they import (or require()) stuff from other modules
they export stuff

packages are

a way to distribute a module or a set of modules
the concept of dependencies is delegated to package managers

Given this terminology we build up some more:

package managers are

tools (mostly CLI and registries) that download packages
they enforce their (or someone else’s) dependency declaration system

bundlers are

tools that, by implementing a package manager or environment imports resolution algorithm, can squash packages into a single deliverable module

This dicotomy started when a server-oriented environment (cheap and fast file access, higher memory availability) became the target environment for thousands of libraries/packages and was to be somewhat ported to a remote client (costly and slow resource access, lower memory availability). We are of course talking about Node.js, npm and the browser.
When talking about mobile apps we are in a slightly different environment, “Distributed Client”.

×
Server

Browser

Distr. Cl.

Memory avail.
High

Low
=
Low

Resource access
Fast

Slow
≠
Fast

Space Constraints
None

Tight
=
Tight

Network Access
Good

Unknown
=
Unknown

Cross-target
No

Yes
=
Yes

Cross-envir. ¹
Yes

No
≠
Yes

Note n. 1: I’m referring to the complete different ways to access APIs. On the server we have Window and ＊nix environments. On Distributed Clients we have to add the Mobile Environments

/cc @hansemannn @cb1kenobi @jhaynie

In fact bundlers are similar to what linking used to be. You statically analyze the stuff you have (either sources or half-baked binaries with headers) and create a new source/binary that has all the compilation unit’s dependencies now fully statically linked.
When such a one time compilation phase is not available (or just when bundlers weren’t the cool stuff in the block) you need to use a loader, or more specifically a module loader. See here a good list of module loaders. Cases where you have no other options include plugin-based platforms, where final (obfuscated sometimes) products are installed directly on the target runtime platform.
Sometimes you pair a bundler with a loader to achieve better performances when resource access is slow and you really care about your “heap” memory footprint. Typical is the use of a dynamic System.require call in React Router routes.
The easiest loader in the browser (pre ES2015) is just a list of <script> tags. Horribly inefficient in terms of maintainability, because you need to load stuff in order (mostly by trial and error), and I’m pretty sure anyone here has felt that pain.
The typical browser pipeline looks like the following:

define your app and you app dependencies using your environment of choice;
install (read: download and configure) your dependencies;
eventually compile your code through your transpiler/compiler of choice;
run a static analysis tool to define the module’s dependency graph;
eventually bundle your app entry points ² so that they can be consumed by your loader;
serve that stuff to the client with a loader.

Note n.2: the ability for a bundler to generate multiple entries is incredibly important when including Web Workers in the game.
Most of this could be ditched with HTTP2, but it is still sometimes more efficient.
You could also simplify the pipeline if your loader and server can handle dynamic combo-loading. While now deprecated, YUI 3 got this right many years ago, they also had a modules (not packages) registry called The YUI Gallery.

How much of all this does apply to us? Well not so much… and a lot.
Let’s have a look at constraints in distributed clients/mobile apps:

they must be “distributed” as much as possible (no interpreted source download as in the browser);
the smaller they are the better it is (and no ask-at-runtime-and-I’ll-give loading as in the browser)
the faster the bootstrap the better (so a middle ground between long living processes a-la Node.js and small clients a-la browser)

The good side is that:

the file system is as fast as in a server, more or less;
we have not a lot of memory but nothing to fear about regarding the “heap”;
we almost always have a compilation bottleneck (in the neutral sense).

With this in mind we can go straight to the point:

do we need to support npm packages?
If yes, can we do it?
If yes, how?

Let‘s answer the first question: yes. Javascript is a mess, Node.js is ok, npm rocks. Can we please all agree that it is the combination of Node.js modernity and the quality of npm that made that ecosystem a success and move along?
The next one is «can we do it». First of all let’s clear that we already support CommonJS modules.
Supporting Node.js resolution algorithm out of the box would be awesome… if package authors were even 10% tidy as they should. The risk is to get a huge bunch of stuff into the app that you don’t really want.
I personally believe we have exclusively the following ways to do it (with relative challenges).

Add support for full Node.js (TIMOB-16078) resolution algorithm

we need node_modules inside the app
we need to discern good stuff—required and used (re)sources—from bad stuff—tests, docs and… stuff

using patterns (**/*.js) we may miss something interesting
using white/blacklist we put some burden to the final developer
using static analysis we could still miss something (see dynamic requires, for example moment language files)
we could mitigate the scenario by mixing in some runtime analysis, but you get env (dev, production) inconsistencies

no issue with multiple entries (:+1: on Android Services and iOS Background Services)

Squash everything with bundlers

very difficult to get right if we implement one ourselves (please no. I repeat: please no)
same issues with static analysis
could help a lot in thinning the app for different environments
sometimes configurations too are difficult to get right (see some webpack config files and you’ll get what I’m talking about)

I leave the «how» to another moment.
What is even more interesting is the built-in Node.js modules support topic. The question is «why do we need to shim util, http etc.?» and then «if yes, how?»
Most of them are relatively easy to shim and already are for the browser. Those shims work more or less also on Titanium SDK ≤ 5. But the most critical (as in «can disrupt the quality of your app in a few milliseconds») are the lower level ones (non comprehensive list follows):

http/https
stream (and their usage around the APIs!!!)
net (Sockets. I mean, real ones. We need them. Really)
fs (asynchronous, for real)
and last but not least Buffer and process.nextTick.

Those are the ones that if ported correctly from Node.js would really be a killer feature.

:mega: This specific braindump has finished, if you want to read another leave your comments down here! :mega:

Thanks a lot @yuchi!
I do think we should keep the NodeJS API discussion separate, while of course once we have NodeJS compliant require() in place would make this a lot easier (with a special exception for global scope NodeJS objects like process).
In regards to:

we need to discern good stuff—required and used (re)sources—from bad stuff—tests, docs and… stuff

I'm not sure if we would need to. How is this different from using packages in a Node.js app or browser? They eat disk space there as well. And if module maintainers manage their .npmignore files correctly then there shouldn't be too much unused files anyway, except for maybe README and docs - which would be relatively easy to filter out - if at all needed.

Thank you for feedback! About Node.js APIs I’m willing to patron them as much as it’s needed, just move the conversation or split it as you guys prefer. IMHO http(s), net, stream and Buffer are must tho.
About filtering stuff… it’s not always the case and whatever we do will be magic in the eyes of developers and stuff could break for unexpected reasons.
Also it’s very difficult to create a filter for such resources, you really don’t know when they will be used without deeply analyzing the code. What about images or fonts? What if they are just screenshots of the component final UI?
You can blow up very easily.
To directly address your point, this is one of the main differences between server, remote client and distributed clients: space.

in the server your hardware’s HDD is your limit
in the remote cliente the server’s HDD is your limit, and loading time for required resources
in a distributed client is the final consumer’s HDD (and download time!)

So no, those stuff doesn’t really harm Node.js developers and browser developers (whether they use a bundler or a loader).

I agree, but either way (bundler or not) you'd have to filter and potentially break stuff. I say we just don't Instead, we might want to introduce a .tiignore file to have Titanium ignore certain files from Resources?

Either we get the full burden or we risk to break stuff. We need:

a set of very well defined and explicit heuristics to reduce the surface of manual management;
a way for maintainers to either whitelist or blacklist resources;
a way for consumers to signal maintainers that something is off;
a way for consumers to overwrite package configuration “in the meanwhile” (or permanently).

The last one is very difficult to solve. In fact how do you specify a resource?
If you try to specify it as an absolute project path you can point to direct dependencies’ resources only. Let me give you an example.
Let’s say that htmlparser2 has a .txt file that contains valid html tags, but for some reason we are completely missing it in out static analysis. The issue is that in a Titanium app I’ll probably want to use html2as instead of doing the transformation manually. I could add this in my .titaniumrc (I‘m assuming npm v3 semantics here):
{ "whitelist": [ "/node_modules/htmlparser2/lib/valid-tags.txt" ] }

I can do this because even if I do not depend on htmlparser2 directly npm v3 will install it in the root node_modules to reduce useless nesting (as npm v2 would do).
But if I install a version of h2m that uses a different version of htmlparser2 I would get the following directories: (versions are fake)

MyAwesomeApp/

…stuff…
node_modules/

ti-html2as/

node_modules/

htmlparser2/ (v1.2.0)

h2m/

node_modules/

htmlparser2/ (v2.0.0)

A solution would be a whitelist in the following format:
{
  "whitelist": {
    "htmlparser2": [ "lib/valid-tags.txt" ]
  }
}

or
{
  "whitelist": {
    "htmlparser2": { "version": "^2.0.0", "files": [ "lib/valid-tags.txt" ] }
  }
}

Even better:
{
  "whitelist": [
     "**/*.ttf",
     {
       "files": [ "**/*.ttf" ]
     },
     {
       "package": "htmlparser2",
       "version": "^2.0.0",
       "files": [ "lib/valid-tags.txt" ]
     }
  ]
}

Since we'd want users to use any Node.js module, including once not designed specifically for Titanium we can't ask the maintainers of those modules to add any Titanium specific instructions. So if the Titanium developer wants to make the app lighter, then he should be responsible for maintaining an ignore list. And I think it should be as simple as possible so ideally standard ignore format (the developer would know if he uses a NPM version that flattens or not).

This is sure, but a good way to define inclusions and exclusions is necessary for good maintainability.
Small technical note: the developer cannot know if dependencies are collapsed in a common one or not. Npm installs results are not deterministic, they change depending on the time of installation. That means that even if the dev could know where the file to include or exclude lays by inspecting the file system, this could change on a different installation (in another dev environment or in the CI). That’s why I proposed that strange syntax.

Minor but vital point
If we are going to support npm packages through hyperloop “modules” (please, call them packages…) then the whole app must support them. The only way out this is the package bundling option, that is to use something along the lines of Titaniumifier on the package creating a collapsed result.
If we go down that line we will get duplicated dependencies in the app.

Also we should think about supporting the npm registry without using the npm directory structure.
This is what jspm does.

We discussed this topic in the Hyperloop meeting today and all those in attendance agreed that supporting this feature is important, but it's outside the scope of Hyperloop. This should be supported in Titanium and thus this topic should be moved elsewhere. We should draft up a PJD (project justification document) and possibly a CSPEC (https://github.com/appcelerator/cspec). I don't want throw away the content in this issue, so any volunteers to get the ball rolling?

Random point as well: But the Ti.Network.HTTPCleint is meant to be effectively API-compatible with XMLHTTPRequest.

I’ve been sponsoring this issue for 2 years now. While I’m a little bit tired, I’d like to see it solved.
Can you guys give me a fresh recap of ① the expected spec submission process ② the approval workflow ③ who in the Appc is going to take technical ownership of this topic these topics?
The topics at hand are:

Support for npm-compatible dependencies semantics for packages in Titanium SDK
Support/porting of core Node.js modules
Support for npm-compatible dependencies semantics for hyperloop modules packages

Also, what’s your expected ETA?

(And when I say 2 years I mean it: http://yuchi.github.io/ticonf-2014-titaniumifier/ use ← and → )

@yuchi I hear you. I don't know enough to answer your CSPEC questions, but I believe we, Appc, should probably be the ones to kick off the CSPEC.
If there are tooling changes required to support packages, then it's highly likely I'll be the one to work on them. Otherwise it'll be up to each platform team to implement it. And who knows, maybe some of these APIs will be implemented as Hyperloop modules.
Don't hold your breathe for Mobile Web to support this. :)
When you say supporting core Node.js modules, I assume you mean the APIs. This is a huge task. As much as I'd love to see support for it in Titanium SDK 6, I'd be surprised if we had time. We have a ton of stuff packed in for v6 and we don't know what Apple has up their sleeve at WWDC this year.
We can get started on a spec and start defining which Node APIs we have to support and which ones does it make sense to not support. Node's docs is a good place to start. We can also take Node's unit tests and fix them up to work in the Titanium environment.

Thank you @cb1kenobi.

@yuchi I hear you. I don't know enough to answer your CSPEC questions, but I believe we, Appc, should probably be the ones to kick off the CSPEC.

That’s ok with me, but I’m actually interested in the specc’ing process you established, not only regarding this specific topic. Could you invoke someone to give me some directions?
At least I could be more helpful by giving the kind of support and resources you could need.

If there are tooling changes required to support packages, then it's highly likely I'll be the one to work on them. Otherwise it'll be up to each platform team to implement it. […]

Ok! :+1:

[…] And who knows, maybe some of these APIs will be implemented as Hyperloop modules.

I actually expect it to happen!

Don't hold your breathe for Mobile Web to support this. :)

Node.js built-in modules apart this is the easiest platform to support at all! :P

When you say supporting core Node.js modules, I assume you mean the APIs. This is a huge task. As much as I'd love to see support for it in Titanium SDK 6, I'd be surprised if we had time. We have a ton of stuff packed in for v6 and we don't know what Apple has up their sleeve at WWDC this year.
We can get started on a spec and start defining which Node APIs we have to support and which ones does it make sense to not support. Node's docs is a good place to start. […]

I don’t think anyone expects full support on day-0 on this. Just having the minimal pure-JS stuff working would be a good step forward.

[…] We can also take Node's unit tests and fix them up to work in the Titanium environment.

This. One-hundred thousands times this.

@yuchi the CSPEC repo has instructions on how to create a CSPEC:
https://github.com/appcelerator/cspec#how-to-submit-a-cspec
Once you have it up you can do a PR on the CSPEC repo to have it linked.
@cb1kenobi on:

all those in attendance agreed that supporting this feature is important, but it's outside the scope of Hyperloop

I agree it's outside the scope of HL core, but of course it is very much related to how we decide architect HL modules right? It would be good to asses what elements of what @yuchi suggests will depend on architecture decisions still to be made for HL (modules) and what can be do not depend on that. Maybe as part of your CSPEC you can look into this @yuchi?

Obviously in draft, but the PR is there waiting to be greenlit.
https://github.com/appcelerator/cspec/pull/1

Great @yuchi, let's get this ball roling!