Token renewal operation failed due to timeout

Please follow the issue template below. Failure to do so will result in a delay in answering your question.

Library

[ ]<EMAIL_ADDRESS>or<EMAIL_ADDRESS>[ ]<EMAIL_ADDRESS>[ ]<EMAIL_ADDRESS>[x]<EMAIL_ADDRESS>[ ]<EMAIL_ADDRESS>
Important: Please fill in your exact version number above, e.g<EMAIL_ADDRESS>Framework
Angular 8
Description
Cannot get access token when app is hosted on azure and everything works in my company's internal server.
I've tested this which seems to genrally work but I'm facing work frames security issues (Blocked a frame with origin "https://MY_URL" from accessing a cross-origin frame, or also Failed to set the 'domain' property on 'Document': Assignment is forbidden for sandboxed iframes.).
I've also tried to test if we are in an iframe
Here is the code which generates errors:
private async getAccessToken(): Promise < string > {
  return new Promise<string>((resolve: Function, reject: Function): void => {
    const isIframe: boolean = window !== window.parent && !window.opener;
    if (!isIframe) { // added as a workaround
      this.msalService.acquireTokenSilent({
        scopes: this.confService.getMicrosoftScopes(),
        redirectUri: `${window.origin}/auth.html`, // added as a workaround
      }).then((authResponse: AuthResponse) => {
        resolve(authResponse.accessToken);
      })
        .catch((error: { name: string }) => {
          if (error.name === "InteractionRequiredAuthError") {
            this.msalService.acquireTokenPopup({
              scopes: this.confService.getMicrosoftScopes(),
            }).then((authResponse: AuthResponse) => {
              resolve(authResponse.accessToken);
            }).catch((acquireTokenPopupError) => {
              console.log(acquireTokenPopupError);
            })
          } else {
            reject('Could not get access token')
          }
        });
    } else {
      setTimeout(() => {
        resolve(null);
      }, 10000)
    }
  });
}

Thank you for your help

Error Message
Token renewal operation failed due to timeout
Security

[ ] Is this issue security related?

Regression

[ ] Did this behavior work before?
Version: 1.0.0

MSAL Configuration
` {
      provide: MSAL_CONFIG,
     useFactory: (): Configuration => ({
       auth: {
        clientId: 'CLIENT_ID',
         authority: 'https://login.microsoftonline.com/common',
        redirectUri: window.location.origin,
      },
      cache: {
        cacheLocation: 'localStorage',
        storeAuthStateInCookie: true, //added to test but worked on CI with false
      },
      system: {
        loadFrameTimeout: 20000 //added to test but worked on CI without
      }
    })
},
{
  provide: MSAL_CONFIG_ANGULAR,
    useFactory: (): MsalAngularConfiguration => ({
      popUp: true,
      consentScopes: [
        "user.read",
        "calendars.read",
        "calendars.readwrite",
        "calendars.readwrite.shared",
        "group.read.all",
        "openid",
        "profile"
      ],
      extraQueryParameters: {
        prompt: 'select_account' //added to test but worked on CI without
      }
    })
},`

// Provide configuration values here.
// For Azure B2C issues, please include your policies.

Reproduction steps
// Provide relevant code snippets here.
// For Azure B2C issues, please include your policies.

Expected behavior
Browsers/Environment

[x] Chrome
[x] Firefox
[ ] Edge
[ ] Safari
[ ] IE
[ ] Other (Please add browser name here)

but I'm facing work frames security issues (Blocked a frame with origin "https://MY_URL" from accessing a cross-origin frame, or also Failed to set the 'domain' property on 'Document': Assignment is forbidden for sandboxed iframes.).

This is potentially the cause of your timeouts. Can you share more details about the errors messages you're seeing?

I've captures the logs for each solution I've tested.
With "normal" scenario the same app works perfectly when deployed on our internal servers.
What I've noticed is that when it works, the app in the hidden frame has not time to load and all the requests to te script are canceled because I guess the iframe was closed. Here is the capture of this (it is written "finished" but it was written "canceled".

As soon as this iframe is closed we can see the requests to graph working (me, calendars, events).
On Azure the same app has a different behaviour. The iframe seems to be never closed, all the app is loaded and the requests to graph are never done.

The only message I have is this one
acquireTokenSilent catch block main.aae7314fd886d26e42fa.js:1
ClientAuthError: Token renewal operation failed due to timeout.
    at t [as constructor] (https://kubectl-ppd03.powimo.com/vendor.308a54bd7208ddbf6258.js:1:1728317)
    at new t (https://kubectl-ppd03.powimo.com/vendor.308a54bd7208ddbf6258.js:1:1728939)
    at Function.t.createTokenRenewalTimeoutError (https://kubectl-ppd03.powimo.com/vendor.308a54bd7208ddbf6258.js:1:1729895)
    at https://kubectl-ppd03.powimo.com/vendor.308a54bd7208ddbf6258.js:1:1752560
    at e.invokeTask (https://kubectl-ppd03.powimo.com/polyfills.2e9dda1f02a2d7a33f1f.js:1:18867)
    at Object.onInvokeTask (https://kubectl-ppd03.powimo.com/vendor.308a54bd7208ddbf6258.js:1:1627587)
    at e.invokeTask (https://kubectl-ppd03.powimo.com/polyfills.2e9dda1f02a2d7a33f1f.js:1:18788)
    at t.runTask (https://kubectl-ppd03.powimo.com/polyfills.2e9dda1f02a2d7a33f1f.js:1:13946)
    at t.invokeTask (https://kubectl-ppd03.powimo.com/polyfills.2e9dda1f02a2d7a33f1f.js:1:19999)
    at invoke (https://kubectl-ppd03.powimo.com/polyfills.2e9dda1f02a2d7a33f1f.js:1:19890)
```

When you receive the "Blocked a frame..." error message, can you try opening the login.microsoftonline.com URL in its own tab?

When I open login.microsoft.com I can log in and I'm redirected to office365 dashboard.
In my app, I have this message
main.aae7314fd886d26e42fa.js:1 DOMException: Blocked a frame with origin "https://kubectl-ppd03.powimo.com" from accessing a cross-origin frame.
    at started.test.window.addEventListener.l._listener (https://fastly.trychameleon.com/assets/chmln/adeaf8112/chmln.js:1:240710)
    at e.invokeTask (https://kubectl-ppd03.powimo.com/polyfills.2e9dda1f02a2d7a33f1f.js:1:18867)
    at Object.onInvokeTask (https://kubectl-ppd03.powimo.com/vendor.308a54bd7208ddbf6258.js:1:1627587)
    at e.invokeTask (https://kubectl-ppd03.powimo.com/polyfills.2e9dda1f02a2d7a33f1f.js:1:18788)
    at t.runTask (https://kubectl-ppd03.powimo.com/polyfills.2e9dda1f02a2d7a33f1f.js:1:13946)
    at t.invokeTask [as invoke] (https://kubectl-ppd03.powimo.com/polyfills.2e9dda1f02a2d7a33f1f.js:1:19999)
    at p (https://kubectl-ppd03.powimo.com/polyfills.2e9dda1f02a2d7a33f1f.js:1:33011)
    at h (https://kubectl-ppd03.powimo.com/polyfills.2e9dda1f02a2d7a33f1f.js:1:33317)

Can you generate a fiddler trace of the timeout error you were seeing previously and send it to the email on my profile?

That's done.
Thank you

@jmckennon have you received my trace ?

@vcs38 The trace that you sent shows an access token being acquired successfully, was that the correct trace that you meant to show us? I was expecting one with errors.
Additionally, loadFrameTimeout just increases the time before getting a timeout error. Sometimes playing with the value can help show if the error is truly a timeout and it just needed more time to complete, but this is usually not the case.

I don't have any network issues you are right.
I acquire the token via the network but it's in the application that I have this error.
For reminding, the same app is deployed on two different environments: one internally and another one on Azure.
What we can see is that the app deployed internally works perfectly maybe because some scripts in the iframe are not loaded (maybe the network is not fast enough ?)

I've seen exactly why I'm having a timeout in msal.js file by debugging.
This fonction creates the timeout:
WindowUtils.monitorWindowForHash = function (contentWindow, timeout, urlNavigate, logger, isSilentCall) {
    return new Promise(function (resolve, reject) {
      var maxTicks = timeout / WindowUtils.POLLING_INTERVAL_MS;
      var ticks = 0;
      logger.verbose("monitorWindowForHash polling started");
      var intervalId = setInterval(function () {
        if (contentWindow.closed) {
          logger.error("monitorWindowForHash window closed");
          clearInterval(intervalId);
          reject(ClientAuthError_1.ClientAuthError.createUserCancelledError());
          return;
        }
        var href;
        try {
          /*
           * Will throw if cross origin,
           * which should be caught and ignored
           * since we need the interval to keep running while on STS UI.
           */
          href = contentWindow.location.href;
        }
        catch (e) { }
        if (isSilentCall) {
          /*
           * Always run clock for silent calls
           * as silent operations should be short,
           * and to ensure they always at worst timeout.
           */
          ticks++;
        }
        else {
          // Don't process blank pages or cross domain
          if (!href || href === "about:blank") {
            return;
          }
          /*
           * Only run clock when we are on same domain for popups
           * as popup operations can take a long time.
           */
          ticks++;
        }
        if (href && UrlUtils_1.UrlUtils.urlContainsHash(href)) {
          logger.verbose("monitorWindowForHash found url in hash");
          clearInterval(intervalId);
          resolve(contentWindow.location.hash);
        }
        else if (ticks > maxTicks) {
          logger.error("monitorWindowForHash unable to find hash in url, timing out");
          logger.errorPii("monitorWindowForHash polling timed out for url: " + urlNavigate);
          clearInterval(intervalId);
          reject(ClientAuthError_1.ClientAuthError.createTokenRenewalTimeoutError());
        }
      }, WindowUtils.POLLING_INTERVAL_MS);
    });
  };

On Azure I'm reaching this block of code:
     else if (ticks > maxTicks) {
          logger.error("monitorWindowForHash unable to find hash in url, timing out");
          logger.errorPii("monitorWindowForHash polling timed out for url: " + urlNavigate);
          clearInterval(intervalId);
          reject(ClientAuthError_1.ClientAuthError.createTokenRenewalTimeoutError());
        }

I reach this blocl because the variable href is in error.
I've set a breakpoint at th specific line and href is undefined.
href is set at this specific line:
try {
          /*
           * Will throw if cross origin,
           * which should be caught and ignored
           * since we need the interval to keep running while on STS UI.
           */
          href = contentWindow.location.href;
        }
        catch (e) { }

What i can see with my breakpoint is that the content of contentWindow.location.href is in error as shown below:

The full mesage is
Exception: DOMException: Blocked a frame with origin "https://kubectl-ppd03.powimo.com" from accessing a cross-origin frame. at https://kubectl-ppd03.powimo.com/vendor.308a54bd7208ddbf6258.js:1:1752205<error: SecurityError: Blocked a frame with origin "https://kubectl-ppd03.powimo.com" from accessing a cross-origin frame.<error> at e.invokeTask (https://kubectl-ppd03.powimo.com/polyfills.2e9dda1f02a2d7a33f1f.js:1:18867) at Object.onInvokeTask (https://kubectl-ppd03.powimo.com/vendor.308a54bd7208ddbf6258.js:1:1627587) at e.invokeTask (https://kubectl-ppd03.powimo.com/polyfills.2e9dda1f02a2d7a33f1f.js:1:18788) at t.runTask (https://kubectl-ppd03.powimo.com/polyfills.2e9dda1f02a2d7a33f1f.js:1:13946) at t.invokeTask (https://kubectl-ppd03.powimo.com/polyfills.2e9dda1f02a2d7a33f1f.js:1:19999) at invoke (https://kubectl-ppd03.powimo.com/polyfills.2e9dda1f02a2d7a33f1f.js:1:19890) at n.args.<computed> (https://kubectl-ppd03.powimo.com/polyfills.2e9dda1f02a2d7a33f1f.js:1:50848)
https://kubectl-ppd03.powimo.com/vendor.308a54bd7208ddbf6258.js:1:1752205 refers to the line href = contentWindow.location.href;
Hope this will really help because my company and our clients are blocked
Thanks

I really need some support. Can someone tell me what else can I do ?

Sorry to pollute but I still need some help

Sorry for the delay here @vcs38. Can you try setting initialNavigation: 'disabled' when inside an iframe (example), while also using the largely increased loadFrameTimeout value?

MSAL Angular v2 alpha is now available: https://github.com/AzureAD/microsoft-authentication-library-for-js/releases/tag/msal-angular-v2.0.0-alpha.0
We do not expect applications to experience the same timeout errors from v1, so please try it out and let us know if you still experience these issues.


The shapedirs of basicmodel_neutral_lbs_10_207_0_v1.1.0.pkl is 300 while the pre-trained model's is 10, which causes size mismatch when loading the provided model.

The shapedirs of basicmodel_neutral_lbs_10_207_0_v1.1.0.pkl is 300 while the pre-trained model's is 10, which causes size mismatch when loading the provided model.
Originally posted by @wanzysky in https://github.com/Jeff-sjtu/HybrIK/issues/17#issuecomment-872872257
There is only 1.10 available on the SMPL website, can you please provide the 1.00 version to replicate the results?

size mismatch for smpl.shapedirs: copying a param with shape torch.Size([6890, 3, 10]) from checkpoint, the shape in current model is torch.Size([6890, 3, 300]).

https://github.com/mkocabas/VIBE gives the 1.0.0 model.

Hi @wanzysky ,
Did the 1.0.0 model from VIBE works for you? I am not sure if I provide the SMPL 1.0.0 model will violate the license or not.

Hi @wanzysky ,
Did the 1.0.0 model from VIBE works for you? I am not sure if I provide the SMPL 1.0.0 model will violate the license or not.

Yes.

Hi @wanzysky, I am not sure what needed to be changed here ... In the demo file, I have
cfg_file = 'configs/256x192_adam_lr1e-3-hrw48_cam_2x_w_pw3d_3dhp.yaml'
CKPT = './pretrained_hrnet.pth'

In the directory model_files, I have all sort of basicmodel (1.0.0 and 1.1.0) but I am not sure what to change here ... Thank you for your help !


remove Mixin/Base suffix from pkg:observe

These suffixes shouldn't be needed as we are in a Pub package, and thus can have a (slightly) looser versioning policy. Also having 6 types instead of 2 is a point of confusion for new users.

This issue has been moved to dart-lang/observe#36.


⚠ Redacted has degraded performance

In f11909a, Redacted ($RED) experienced degraded performance:

HTTP code: 200
Response time: 4816 ms

Resolved: Redacted performance has improved in f1cf25a.


"/usr/bin/sh: rd: command not found" on WINDOWS 7 x64

I installed the Make and CMake in my computer and Enable a 64-Bit Visual C++ Toolset on the Command Line as well. . I follow the steps in the documents on CARLA: https://carla.readthedocs.io/en/latest/how_to_build_on_windows/.  So far so good!
Now, I have an issue when run the Rebuild.bat. There is an issue prompt out said "/usr/bin/sh: rd: command not found" just like you mention in the Note that:
"Take care if you have Cygwin installed. This could cause the errors like
/usr/bin/sh: rd: command not found While executing Reuild.bat. "
However, I didn't install the Cygwin, so any solution on this? I saw some closed issue about this issue but it only related with Linux, so can you help me solve this issue in Windows OS?
Thank you so much!

Hi @NYJiaxing.
This is a common problem, this error means that you have a bash interpreter in your path.
When you run Rebuild.bat, it calls rd that is the Delete folder(s) command for Windows.
However! If you have in your path some bash interpreter, sometimes, somehow, it will try to run the your commands as Linux commands. This is probably because a wrong order of paths in the PATH windows variable. Also check out your user's PATH too ;)
As far as I've seen this may occur when you have installed SourceTree or Cygwin. I'll will update this list if I see more softwares!

Oops! Miss-click closing the issue.

Hi @marcgpuig,
Thank you for your reply, I just reinstall the python and delete all the Cygwin related files I can find in my computer. However, it still report the same issue.

Check if C:\Program Files\Git\bin is in your path. I'm not sure if having it may cause problems.


Coregionalization with heteroscedastic noise

What is the simplest way to include heteroscedastic noise models with coregionalization? Is it just a case of setting the variance of the various kernel components (one per output) to the appropriate noise value?

Hi Matthew,
Just to clarify you mean heteroscedastic across dimensions, not data points
right? Tagging in @ric70x7 who's done the most work with heteroscedastic &
coregionalisation in GPy. Ricardo, did you create a notebook for
coregionalisation with different output noises?
Neil
On Tue, Feb 9, 2016 at 7:32 AM, Matthew Graham<EMAIL_ADDRESS>wrote:

What is the simplest way to include heteroscedastic noise models with
coregionalization? Is it just a case of setting the variance of the various
kernel components (one per output) to the appropriate noise value?
—
Reply to this email directly or view it on GitHub
https://github.com/SheffieldML/GPy/issues/300.

I have a number of different outputs and each output is associated with a separate heteroscedastic noise model - is this dimensions or data points?

dimensions of the data is the number of outputs (denoted p in statistics).
Data points is the number of data observations (denoted n). Although it can
be that you have different numbers of observations for each output.
Heteroscedasticity can be modelled acros the different outputs and/or
across different data observations.
Neil
On Tue, Feb 9, 2016 at 8:05 AM, Matthew Graham<EMAIL_ADDRESS>wrote:

I have a number of different outputs and each output is associated with a
separate heteroscedastic noise model - is this dimensions or data points?
—
Reply to this email directly or view it on GitHub
https://github.com/SheffieldML/GPy/issues/300#issuecomment-181753178.

There is a notebook in the GPy notebook repository over at Github. It is in the GPy section, if I am not completely mistaken. I am not sure how well it describes heteroscedasticity across dimensions, though. Have a look at the notebook and feel free to ask questions here :)

Am 09.02.2016 um 10:45 schrieb Neil Lawrence<EMAIL_ADDRESS>dimensions of the data is the number of outputs (denoted p in statistics).
Data points is the number of data observations (denoted n). Although it can
be that you have different numbers of observations for each output.
Heteroscedasticity can be modelled acros the different outputs and/or
across different data observations.
Neil
On Tue, Feb 9, 2016 at 8:05 AM, Matthew Graham<EMAIL_ADDRESS>wrote:

I have a number of different outputs and each output is associated with a
separate heteroscedastic noise model - is this dimensions or data points?
—
Reply to this email directly or view it on GitHub
https://github.com/SheffieldML/GPy/issues/300#issuecomment-181753178.

—
Reply to this email directly or view it on GitHub.

Hi Matthew,
A coregionalized model with heteroscedastic noise, that sounds interesting.
It shouldn't be difficult to implement. I wonder how it would perform in
practice.
The first step I recommend you is to go through the tutorials of
heteroscedastic regression and coregionalized regression. Then you can
build your own model.
The core ideas of how this models are implented in GPy are the following:
Heteroscedastic noise: Each of the N observations y_i are associated to a
different noise term sigma_w where w <= N.
y_i = f(x_i) + epsilon[w]_i
When you define your likelihood, you have to define the number of noise
terms in the model. You do this with a dictionary (Y_metadata) that
contains a vector of n indices that associates each y_i to one sigma_w.
Something like this:
Y_metadata = {'output_index':np.arange(Ny)[:,None]}
Then the likelihood can be defined as:
likelihood = likelihoods.HeteroscedasticGaussian(Y_metadata)
If you want the noise terms to be non-Gaussian, or some to be Gaussian and
others not... well that is indeed possible, but the implementation needs
more thinking.
Coregionalized regression: Here you have D outputs, so you can index your
observations as y_di. You don't need to have the same number of
observations for each output. You have something like this:
y_di = f_d(x_i) + \epsilon_di
This model uses a different noise term across outputs, but is homoscedastic
within ouputs. Notice that all y_di have the same input space x_i,
regardless the output d. The way this is handled in GPy is with a
coregionalized kernel, that expands the input dimension (x_d) with a set of
indices that tells to which output d the input point x_n will be used:
x_dn = (x_n | d)
Don't use any kernel directly on these inputs, you have to use a
coregionalized kernel:
K=GPy.kern.RBF(1)B = GPy.kern.Coregionalize(input_dim=1,output_dim=2)
coreg_kernel = K.prod(B,name='B.K')
In summary: Create a coregionalized kernel.
Create a likelihood with the Y_metadata defining your different noise terms.
Append to the right of X a column of output indices.
Put everything together in a regression model, and blend...
Don't try to use the GPy.models.GPCoregionalizedRegression, it has some
pre-defined parameters that won't work with your assumptions.
You can use the GPHeteroscedasticeRegression, but passing as parameters of
model a Coregionalized kernel and an already expanded input dimension set X.
Good luck,
Ricardo
On Feb 9, 2016 2:47 AM, "Max Zwiessele"<EMAIL_ADDRESS>wrote:

There is a notebook in the GPy notebook repository over at Github. It is
in the GPy section, if I am not completely mistaken. I am not sure how well
it describes heteroscedasticity across dimensions, though. Have a look at
the notebook and feel free to ask questions here :)

Am 09.02.2016 um 10:45 schrieb Neil Lawrence<EMAIL_ADDRESS>dimensions of the data is the number of outputs (denoted p in
statistics).
Data points is the number of data observations (denoted n). Although it
can
be that you have different numbers of observations for each output.
Heteroscedasticity can be modelled acros the different outputs and/or
across different data observations.
Neil
On Tue, Feb 9, 2016 at 8:05 AM, Matthew Graham<EMAIL_ADDRESS>wrote:

I have a number of different outputs and each output is associated
with a
separate heteroscedastic noise model - is this dimensions or data
points?
—
Reply to this email directly or view it on GitHub
<https://github.com/SheffieldML/GPy/issues/300#issuecomment-181753178
.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
https://github.com/SheffieldML/GPy/issues/300#issuecomment-181811836.

In the coregionalized tutorial, the regression class takes a list of inputs (X_list), one per output, but you seem to be saying that GPHeteroscedasticRegression needs to take in a single set of inputs which is the same for all outputs. Do I need to construct a superset of all the different inputs for heteroscedastic regression and tell the kernel which point goes with each output?

Yes, the coregionalization model does the indexing and expansion of the
input set for you. But that is not the way the upper class of regression
model works.
You need:

an array Y (N x 1), where N = N_1 + N_2 + ... + N_D; N_d is the number
of observations of output d, and D is the total number of outputs.
an array X (N x (q+1) ), where q is the actual input dimension, and the
extra 1 is the index that has the information about the output.

You can use this function to build the arrays:
GPy.util.multioutput.build_XY(input_list,output_list=None,index=None)
2016-02-09 16:19 GMT-08:00 Matthew Graham<EMAIL_ADDRESS>
In the coregionalized tutorial, the regression class takes a list of
inputs (X_list), one per output, but you seem to be saying that
GPHeteroscedasticRegression needs to take in a single set of inputs which
is the same for all outputs. Do I need to construct a superset of all the
different inputs for heteroscedastic regression and tell the kernel which
point goes with each output?
—
Reply to this email directly or view it on GitHub
https://github.com/SheffieldML/GPy/issues/300#issuecomment-182144645.

I'm getting the following error when I try and follow this. Can you also comment on why the GPCoregionalizedRegression is not appropriate when the class description says that it is a "Gaussian Process model for heteroscedastic multioutput regression".
def getTestData():
npts = np.random.randint(300)
dt = np.abs(np.random.normal(25., 10., npts - 1))
t = np.zeros(npts)
t[1:] = np.cumsum(dt)
obs = np.random.normal(0., 1., npts)
errseed = np.random.normal(0., 0.05, npts)
err = np.abs(np.random.normal(0., np.abs(np.random.normal(0., 0.05, npts))))
return np.array([t, obs, err])
Get data
nExp = 2
input_list = []
output_list = []
variance = np.array([])
For each expt
for i in range(len(expt)):
t, obs, err = getTestData()
input_list.append(t[:, None])
output_list.append(obs[:, None])
variance = np.concatenate((variance, err ** 2.))
Construct correct input and output data constructs
X, Y, idx = GPy.util.multioutput.build_XY(input_list, output_list)
Y_metadata = {'output_index': idx}
Define coregionalized kernel
K = GPy.kern.OU(1)
B = GPy.kern.Coregionalize(input_dim = 1, output_dim = nExp)
coreg_kernel = K.prod(B, name = "B.K")
Fit GP OU model
m = GPy.models.GPHeteroscedasticRegression(X, Y, kernel = coreg_kernel, Y_metadata = Y_metadata)

LinAlgError                               Traceback (most recent call last)
 in ()
53
54 # Fit GP OU model
---> 55 m = GPy.models.GPHeteroscedasticRegression(X, Y, kernel = coreg_kernel, Y_metadata = Y_metadata)
56 m['.*het_Gauss.variance'] = variance[:, None] # Set the noise parameters to the error in Y
57 m.het_Gauss.variance.fix() # We can fix the noise term, since we already know it
/Users/mjg/anaconda/lib/python2.7/site-packages/GPy/core/parameterization/parameterized.pyc in call(self, *args, **kw)
25         self.highest_parent._connect_fixes()
26         logger.debug("calling parameters changed")
---> 27         self.parameters_changed()
28         return self
29
/Users/mjg/anaconda/lib/python2.7/site-packages/GPy/core/gp.pyc in parameters_changed(self)
186             this method yourself, there may be unexpected consequences.
187         """
--> 188         self.posterior, self._log_marginal_likelihood, self.grad_dict = self.inference_method.inference(self.kern, self.X, self.likelihood, self.Y_normalized, self.mean_function, self.Y_metadata)
189         self.likelihood.update_gradients(self.grad_dict['dL_dthetaL'])
190         self.kern.update_gradients_full(self.grad_dict['dL_dK'], self.X)
/Users/mjg/anaconda/lib/python2.7/site-packages/GPy/inference/latent_function_inference/exact_gaussian_inference.pyc in inference(self, kern, X, likelihood, Y, mean_function, Y_metadata)
54         Ky = K.copy()
55         diag.add(Ky, likelihood.gaussian_variance(Y_metadata)+1e-8)
---> 56         Wi, LW, LWi, W_logdet = pdinv(Ky)
57
58         alpha, _ = dpotrs(LW, YYT_factor, lower=1)
/Users/mjg/anaconda/lib/python2.7/site-packages/GPy/util/linalg.pyc in pdinv(A, *args)
206
207     """
--> 208     L = jitchol(A, *args)
209     logdet = 2.*np.sum(np.log(np.diag(L)))
210     Li = dtrtri(L)
/Users/mjg/anaconda/lib/python2.7/site-packages/GPy/util/linalg.pyc in jitchol(A, maxtries)
63         diagA = np.diag(A)
64         if np.any(diagA <= 0.):
---> 65             raise linalg.LinAlgError("not pd: non-positive diagonal elements")
66         jitter = diagA.mean() * 1e-6
67         num_tries = 1
LinAlgError: not pd: non-positive diagonal elements

Ok, I think I got confused about what you need. Sorry, it was quite early
here, when I read your email.
Lets go from the beginning. The intention of using the term heteroscedastic
in the descripton of the coregionalized_regression class is to make clear
that each output has a different noise term. So the model is
heteroscedastic across outputs, and there is a noise term for each output.
After looking at your code, I think that this is what you want. If that is
the case, ignore everything I've said and use the coregionalized regression
model.
But, if you want to build a coregionalization model, where you have more
noise terms than outputs, then you have to follow my previous
instructions... and have some good data quality, so that the model is able
to learn what is being asked.
Now, the error you got is because the covariance matrix is not positive
definite. Try playing with the lenghtscale (use a larger one) or add
something to the diagonal.
Cheers,
Ricardo
2016-02-09 18:16 GMT-08:00 Matthew Graham<EMAIL_ADDRESS>
I'm getting the following error when I try and follow this. Can you also
comment on why the GPCoregionalizedRegression is not appropriate when the
class description says that it is a "Gaussian Process model for
heteroscedastic multioutput regression".
def getTestData():
npts = np.random.randint(300)
dt = np.abs(np.random.normal(25., 10., npts - 1))
t = np.zeros(npts)
t[1:] = np.cumsum(dt)
obs = np.random.normal(0., 1., npts)
errseed = np.random.normal(0., 0.05, npts)
err = np.abs(np.random.normal(0., np.abs(np.random.normal(0., 0.05,
npts))))
return np.array([t, obs, err])
Get data
nExp = 2
input_list = []
output_list = []
variance = np.array([])
For each expt
for i in range(len(expt)):
t, obs, err = getTestData()
input_list.append(t[:, None])
output_list.append(obs[:, None])
variance = np.concatenate((variance, err ** 2.))
Construct correct input and output data constructs
X, Y, idx = GPy.util.multioutput.build_XY(input_list, output_list)
Y_metadata = {'output_index': idx}
Define coregionalized kernel
K = GPy.kern.OU(1)
B = GPy.kern.Coregionalize(input_dim = 1, output_dim = nExp)
coreg_kernel = K.prod(B, name = "B.K")
Fit GP OU model
m = GPy.models.GPHeteroscedasticRegression(X, Y, kernel = coreg_kernel,
Y_metadata = Y_metadata)
LinAlgError Traceback (most recent call last)
in ()
53
54 # Fit GP OU model
---> 55 m = GPy.models.GPHeteroscedasticRegression(X, Y, kernel =
coreg_kernel, Y_metadata = Y_metadata)
56 m['.*het_Gauss.variance'] = variance[:, None] # Set the noise
parameters to the error in Y
57 m.het_Gauss.variance.fix() # We can fix the noise term, since we
already know it
/Users/mjg/anaconda/lib/python2.7/site-packages/GPy/core/parameterization/parameterized.pyc
in call(self, *args, **kw)
25 self.highest_parent._connect_fixes()
26 logger.debug("calling parameters changed")
---> 27 self.parameters_changed()
28 return self
29
/Users/mjg/anaconda/lib/python2.7/site-packages/GPy/core/gp.pyc in
parameters_changed(self)
186 this method yourself, there may be unexpected consequences.
187 """
--> 188 self.posterior, self._log_marginal_likelihood, self.grad_dict =
self.inference_method.inference(self.kern, self.X, self.likelihood,
self.Y_normalized, self.mean_function, self.Y_metadata)
189 self.likelihood.update_gradients(self.grad_dict['dL_dthetaL'])
190 self.kern.update_gradients_full(self.grad_dict['dL_dK'], self.X)
/Users/mjg/anaconda/lib/python2.7/site-packages/GPy/inference/latent_function_inference/exact_gaussian_inference.pyc
in inference(self, kern, X, likelihood, Y, mean_function, Y_metadata)
54 Ky = K.copy()
55 diag.add(Ky, likelihood.gaussian_variance(Y_metadata)+1e-8)
---> 56 Wi, LW, LWi, W_logdet = pdinv(Ky)
57
58 alpha, _ = dpotrs(LW, YYT_factor, lower=1)
/Users/mjg/anaconda/lib/python2.7/site-packages/GPy/util/linalg.pyc in
pdinv(A,
*args) 206 207 """ --> 208 L = jitchol(A, args) 209 logdet = 2.
np.sum(np.log(np.diag(L)))
210 Li = dtrtri(L)
/Users/mjg/anaconda/lib/python2.7/site-packages/GPy/util/linalg.pyc in
jitchol(A, maxtries)
63 diagA = np.diag(A)
64 if np.any(diagA <= 0.):
---> 65 raise linalg.LinAlgError("not pd: non-positive diagonal elements")
66 jitter = diagA.mean() * 1e-6
67 num_tries = 1
LinAlgError: not pd: non-positive diagonal elements
—
Reply to this email directly or view it on GitHub
https://github.com/SheffieldML/GPy/issues/300#issuecomment-182170021.

OK, so what is the equivalent call to:
m['.*het_Gauss.variance'] = variance[:, None]
for the GPCoregionalizedRegression?

The noise parameters are named as :
mixed_noise.Gaussian_noise_j.variance, with j = 0, 1, ...
m['.*noise.variance'] should work.
2016-02-10 0:31 GMT-08:00 Matthew Graham<EMAIL_ADDRESS>
OK, so what is the equivalent call to:
m['.*het_Gauss.variance'] = variance[:, None]
for the GPCoregionalizedRegression?
—
Reply to this email directly or view it on GitHub
https://github.com/SheffieldML/GPy/issues/300#issuecomment-182251764.

Unfortunately that's giving an attribute error:
K = GPy.kern.OU(1)
icm = GPy.util.multioutput.ICM(input_dim = 1, num_outputs = nSurvey, kernel = K)
m = GPy.models.GPCoregionalizedRegression(input_list, output_list, kernel = icm)
---> 59 m['.*noise.variance'] = variance[:, None]
60 m.OU.variance.fix()
61 m.optimize()
/Users/mjg/anaconda/lib/python2.7/site-packages/GPy/core/parameterization/parameterized.pyc in setitem(self, name, value, paramlist)
303             self.trigger_update()
304         else:
--> 305             try: param = self.getitem(name, paramlist)
306             except: raise
307             param[:] = value
/Users/mjg/anaconda/lib/python2.7/site-packages/GPy/core/parameterization/parameterized.pyc in getitem(self, name, paramlist)
284             if paramlist is None:
285                 paramlist = self.grep_param_names(name)
--> 286             if len(paramlist) < 1: raise AttributeError(name)
287             if len(paramlist) == 1:
288                 if isinstance(paramlist[-1], Parameterized):
AttributeError: .*noise.variance

Please have a look at the notebook basic model handling in the Notebook repository from SheffieldML there it will tell you how to set and get parameters. Maybe you do not want to use regular expressions as it is an old feature. You can set and get parameters directly, they behave like numpy arrays :) So I think, not tested, you can go to m.likelihood.variances[:] =  and set them all like setting a numpy array. It might be, that the variances are called variance as well, so just have a play with it. It is actually quite intuative, once you get the hang of it :)

Am 10.02.2016 um 21:45 schrieb Matthew Graham<EMAIL_ADDRESS>Unfortunately that's giving an attribute error:
K = GPy.kern.OU(1)
icm = GPy.util.multioutput.ICM(input_dim = 1, num_outputs = nSurvey, kernel = K)
m = GPy.models.GPCoregionalizedRegression(input_list, output_list, kernel = icm)
---> 59 m['.*noise.variance'] = variance[:, None]
60 m.OU.variance.fix()
61 m.optimize()
/Users/mjg/anaconda/lib/python2.7/site-packages/GPy/core/parameterization/parameterized.pyc in setitem(self, name, value, paramlist)
303 self.trigger_update()
304 else:
--> 305 try: param = self.getitem(name, paramlist)
306 except: raise
307 param[:] = value
/Users/mjg/anaconda/lib/python2.7/site-packages/GPy/core/parameterization/parameterized.pyc in getitem(self, name, paramlist)
284 if paramlist is None:
285 paramlist = self.grep_param_names(name)
--> 286 if len(paramlist) < 1: raise AttributeError(name)
287 if len(paramlist) == 1:
288 if isinstance(paramlist[-1], Parameterized):
AttributeError: .*noise.variance
—
Reply to this email directly or view it on GitHub.

It does not look like you can set the variances for GPCoregionalizedRegression to reflect heteroscedasticity:
m.mixed_noise.Gaussian_noise_0.variance = variance0[:, None]
m.mixed_noise.Gaussian_noise_1.variance = variance1[:, None]
gives
ValueError: could not broadcast input array from shape (165,1) into shape (1)
The same holds for m.likelihood.Gaussian_noise_*.variance

Please have a look at the coregionalized regression tutorial.
http://nbviewer.jupyter.org/github/SheffieldML/notebook/blob/master/GPy/coregionalized_regression_tutorial.ipynb
You can use a different noise term (but just one) per output.
If what you need is heteroscedasticity within the ouputs, the
coregionalized regression model won't be useful to you. Such a model does
not have a friendly wrapper already coded in GPy. The tools are there to
define this model, but it is not straightforward. You would need to follow
the instructions I sent before and also have a look at this tutorial:
http://nbviewer.jupyter.org/github/SheffieldML/notebook/blob/master/GPy/heteroscedastic_regression.ipynb
Also, please be aware that at the moment GPy does not have a routine to
learn a functional form of the noise. You can only learn noise parameters
at the training locations. Noise prediction at new points is not
implemented.
2016-02-16 13:43 GMT-08:00 Matthew Graham<EMAIL_ADDRESS>
It does not look like you can set the variances for
GPCoregionalizedRegression to reflect heteroscedasticity:
m.mixed_noise.Gaussian_noise_0.variance = variance0[:, None]
m.mixed_noise.Gaussian_noise_1.variance = variance1[:, None]
gives
ValueError: could not broadcast input array from shape (165,1) into shape
(1)
The same holds for m.likelihood.Gaussian_noise_*.variance
—
Reply to this email directly or view it on GitHub
https://github.com/SheffieldML/GPy/issues/300#issuecomment-184886029.

OK, the confusion was because you had implied above from my sample code that the CoregionalizedRegression model was the one I wanted to be using. To be specific, the current coregionalized regression is only heteroscedastic in assuming a different single variance per output (which should possibly be stated in some high level description of it).

Yes, that's it.
On Feb 16, 2016 2:22 PM, "Matthew Graham"<EMAIL_ADDRESS>wrote:

OK, the confusion was because you had implied above from my sample code
that the CoregionalizedRegression model was the one I wanted to be using.
To be specific, the current coregionalized regression is only
heteroscedastic in assuming a different single variance per output
(which should possibly be stated in some high level description of it).
—
Reply to this email directly or view it on GitHub
https://github.com/SheffieldML/GPy/issues/300#issuecomment-184899148.

OK, I've reimplemented as the code above but no matter what the sizes of the random test data inputs,
I always get the error that the covariance matrix is not positive definite. Digging into the code, the problem lines are (in ExactGaussianInference.inference)
K = kern.K(X)
which results in a matrix with some machine error negative values (which should be zero). np.diag(K) then is not all positive.
For interest's sake, I tried:
t, obs, err = getTestData(10)
X, Y, idx = GPy.util.multioutput.build_XY([t[:, None]], [obs[:, None]])
K = GPy.kern.OU(1)
B = GPy.kern.Coregionalize(input_dim = 1, output_dim = 1)
coreg_kernel = K.prod(B, name = "B.K")
K0 = K.K(X)
K1 = coreg_kernel.K(X)
print np.diag(K0), np.diag(K1)
[ 1.  1.  1.  1.  1.  1.  1.  1.  1.  1.]
[  0.00000000e+000   2.15827320e-314   0.00000000e+000   0.00000000e+000
2.17763814e-314   0.00000000e+000   0.00000000e+000  -9.18719151e-172
0.00000000e+000   0.00000000e+000]
This does not seem right - surely the result should be the same here in both cases?

Indeed, that doesn't seem right.
There was a time when those lines of code would do what you were expecting,
but not any more. I wonder how long this bug has been hidden in my
tutorial... it has been for a long time, though.
Anyway, the problem is this line:
B = GPy.kern.Coregionalize(input_dim = 1, output_dim = 1)
We have to add the parameter: active_dims = [1], so you write
B = GPy.kern.Coregionalize(input_dim = 1, output_dim = 1, active_dims = [1])
Here you are telling GPy that whenever you pass an input X to the kernel B,
it will only use its second dimension (i.e., column number one). By
default, whenever you define a kernel, the active dimensions used start
form column 0 up to the number of input dimensions (input_dim).
So that will fix the problem. Thanks for your patience :-)
2016-02-17 16:20 GMT-08:00 Matthew Graham<EMAIL_ADDRESS>
OK, I've reimplemented as the code above but no matter what the sizes of
the random test data inputs,
I always get the error that the covariance matrix is not positive
definite. Digging into the code, the problem lines are (in
ExactGaussianInference.inference)
K = kern.K(X)
which results in a matrix with some machine error negative values (which
should be zero). np.diag(K) then is not all positive.
For interest's sake, I tried:
t, obs, err = getTestData(10)
X, Y, idx = GPy.util.multioutput.build_XY([t[:, None]], [obs[:, None]])
K = GPy.kern.OU(1)
B = GPy.kern.Coregionalize(input_dim = 1, output_dim = 1)
coreg_kernel = K.prod(B, name = "B.K")
K0 = K.K(X)
K1 = coreg_kernel.K(X)
print np.diag(K0), np.diag(K1)
[ 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.]
[ 0.00000000e+000 2.15827320e-314 0.00000000e+000 0.00000000e+000
2.17763814e-314 0.00000000e+000 0.00000000e+000 -9.18719151e-172
0.00000000e+000 0.00000000e+000]
This does not seem right - surely the result should be the same here in
both cases?
—
Reply to this email directly or view it on GitHub
https://github.com/SheffieldML/GPy/issues/300#issuecomment-185474749.


Fix stackdriver metrics for node memory using wrong metric type

Because stackdriver is accessing metric with wrong field (IntValue instead of FloatValue) metric is reported as 0.
This PR fixes this behaviour
Tested on gcp
/cc @kawych

/ok-to-test
/lgtm


Fails with no output

I've followed the instructions, and while it works, at times builds fail with no output for the get cache.

Hi!,
Do you have the problem with the examples or using the resource in your pipeline ?  Could you share me more info about it please ?
Thanks!

source: {repository: ymedlop/npm-cache-resource, tag: 6} - after removing the tag it works

Which Node.js version are you using ?

6.2.1

Hi,
I updated the example
source: {repository: ymedlop/npm-cache-resource, tag: "6"}
The tag has to use doble quotes.


Adds Server command for Lua scripts.

Based on #411 server command is not available in Lua scripts. This PR add server command in Lua scripts.

Thanks for the fix!


updates to community page

Updates to Community page

numbers
UGC
Events
New images have gone through tiny.png

Something weird is happening.  The dates are showing up a day earlier than how I entered them in.  @shcheklein @rogermparent @julieg18

Something weird is happening. The dates are showing up a day earlier than how I entered them in.

It's working for me locally, but not the deployed site... Is it working correclty on your local computer @jendefig?

It's working for me locally, but not the deployed site... Is it working correctly on your local computer @jendefig?

Yes.  It's working for me locally too, except the October Heartbeat image isn't coming through.  But I guess it's okay.  🤷♀

Looks like it's an issue on how we convert our strings into dates. Though I'm not sure why it's only showing while deployed. I'll open a pull request for fixing it.

I'll open a pull request for fixing it.

So does that mean this can't be merged until it's fixed?  Or it's just a weird thing with Heroku and this will be fine when merged? @julieg18

I'll open a pull request for fixing it.

So does that mean this can't be merged until it's fixed?  Or it's just a weird thing with Heroku and this will be fine when merged? @julieg18

The dates will stay incorrect until my fix pull request is merged into master.


B 18318 add seed data for multiple moves INT

Agility ticket
Summary
This work is to populate the dev db with multiple move seed data. Open to suggestions on what to add/modify to make the scenarios realistic.
How to test

Run make db_dev_fresh to populate the dev db with data.

Poke around in the database, making sure you can see the moves codes below. You can use this query to see all of the moves: "locator" LIKE 'MMOVE%'

Make sure that the below move groupings are associated with one service member, each having three moves.

Login as a customer and verify you can see the following accounts to log in as:

<EMAIL_ADDRESS>
Moves: MMOVEA, MMOVEB, MMOVEC
Service member affiliation: Army
Should have one service member ID for all three moves

<EMAIL_ADDRESS>
Moves: MMOVED, MMOVEE, MMOVEF
Service member affiliation: Air force
Should have one service member ID for all three moves

<EMAIL_ADDRESS>
Moves: MMOVEG, MMOVEH, MMOVEI
Service member affiliation: Navy
Should have one service member ID for all three moves

<EMAIL_ADDRESS>
Moves: MMOVEJ, MMOVEK, MMOVEL
Service member affiliation: Coast Guard
Should have one service member ID for all three moves

Login as a TOO and click on any of the moves you see with multiple move codes. Ensure you can see the expected shipments for that move code.

Login as a service counselor and click on any of the moves you see with multiple move codes. Ensure you can see the expected information for that move code.

Note: I spent quite a bit of time trying to pass in a custom origin duty location and was unable to get it working. These moves are all in KKFA for now. If we want to expand that to other GBLOCs, I will likely need some assistance on figuring that out.

Fails

:no_entry_sign:

One of these files have been edited:

pkg/testdatagen/scenario/devseed.go
pkg/testdatagen/scenario/subscenarios.go
cmd/generate-test-data/main.go

Please undo changes to these files as we have deprecated
devseed data functions from MilMove entirely and will be
deleting this code on the 8th of November 2023.
View the ADR 0083 for more information

  Generated by :no_entry_sign: dangerJS against 5c4a8097b3422c1dca517c65a82851b7ef77f6d2

Fails
🚫
One of these files have been edited:

pkg/testdatagen/scenario/devseed.go
pkg/testdatagen/scenario/subscenarios.go
cmd/generate-test-data/main.go

Please undo changes to these files as we have deprecated devseed data functions from MilMove entirely and will be deleting this code on the 8th of November 2023.
View the ADR 0083 for more information
Generated by 🚫 dangerJS against 5c4a809

HA this error is hilarious

Fails
🚫
One of these files have been edited:

pkg/testdatagen/scenario/devseed.go
pkg/testdatagen/scenario/subscenarios.go
cmd/generate-test-data/main.go

Please undo changes to these files as we have deprecated devseed data functions from MilMove entirely and will be deleting this code on the 8th of November 2023.
View the ADR 0083 for more information
Generated by 🚫 dangerJS against 5c4a809

HA this error is hilarious

lol- yea-
i remember asking truss about this, they said it was some arbitrary date to start using another method for populating data- its really irrelevant and i will remved this depriecation warning

HA this error is hilarious

lol- yea- i remember asking truss about this, they said it was some arbitrary date to start using another method for populating data- its really irrelevant and i will remved this depriecation warning

Sounds good!

This branch just wants to fight today. Passing server tests locally, haven't figured out why pre_test is failing

Fails

:no_entry_sign:

One of these files have been edited:

pkg/testdatagen/scenario/devseed.go
pkg/testdatagen/scenario/subscenarios.go
cmd/generate-test-data/main.go

Please undo changes to these files as we have deprecated
devseed data functions from MilMove entirely and will be
deleting this code on the 8th of November 2023.
View the ADR 0083 for more information

  Generated by :no_entry_sign: dangerJS against bcc876c06596e5ccc16924e381b7ff66411d9f59

Fails
🚫
One of these files have been edited:

pkg/testdatagen/scenario/devseed.go
pkg/testdatagen/scenario/subscenarios.go
cmd/generate-test-data/main.go

Please undo changes to these files as we have deprecated devseed data functions from MilMove entirely and will be deleting this code on the 8th of November 2023.
View the ADR 0083 for more information
Generated by 🚫 dangerJS against bcc876c

@deandreJones why is this still showing despite you removing it with this PR?: #11968

Ohhh it is cause you merged it into main and not integrationTesting

@deandreJones should @traskowskycaci remove these lines of code in dangerfile.ts in this PR?
  const deprecatedDevSeedFiles = danger.git.fileMatch(
    'pkg/testdatagen/scenario/devseed.go',
    'pkg/testdatagen/scenario/subscenarios.go',
    'cmd/generate-test-data/main.go',
  );

  if (deprecatedDevSeedFiles.modified) {
    fail(`One of these files have been edited:
* ${'`'}pkg/testdatagen/scenario/devseed.go${'`'}
* ${'`'}pkg/testdatagen/scenario/subscenarios.go${'`'}
* ${'`'}cmd/generate-test-data/main.go${'`'}
Please undo changes to these files as we have deprecated
${'`'}devseed${'`'} data functions from MilMove entirely and will be
deleting this code on the 8th of November 2023.
View the [ADR 0083](https://transcom.github.io/mymove-docs/docs/adrs/deprecating-devseed-scenarios) for more information`);
  }

need to merge main back into this

need to merge main back into this

but it's a PR into integrationTesting?

need to merge main back into this

but it's a PR into integrationTesting?

main is the source of truth- branches should always stay up with main

..shoudlnt

unless there are other changes in that file that conflict

unless there are other changes in that file that conflict

I'll merge main into my B-18318-Add-seed-data-for-multiple-moves main branch, then merge that main branch into this one.

unless there are other changes in that file that conflict

Merging main in now


Articulation Points

Issue will be closed if:

You mention more than one algorithm. You can create a separate issue for each algorithm once the current one is completed.
You propose an algorithm that is already present or has been mentioned in a previous issue.
You create a new issue without completing your previous issue.

Note: These actions will be taken seriously. Failure to follow the guidelines may result in the immediate closure of your issue.

Name:
[NEW ALGORITHM]
Articulation Points
About:
Propose a new algorithm to be added to the repository
An articulation point (or cut vertex) in a graph is a vertex whose removal increases the number of connected components of the graph

Labels:
new algorithm, gssoc-ext, hacktoberfest, level1

Assignees:

[x] Contributor in GSSoC-ext
[x] Want to work on it

I will solve it.


⚠ AVS (kty) has degraded performance

In a296154, AVS (kty) (https://avus.com.br) experienced degraded performance:

HTTP code: 200
Response time: 9077 ms

Resolved: AVS (kty) performance has improved in 532dfd0.


update label to identify enterprise-contract pipeline run

Fixes

https://issues.redhat.com/browse/HAC-3644
Description

update the label that is used to identify the enterprise contract pipelinerun
Type of change

[ ] Feature
[x] Bugfix
[ ] Code style update (formatting, renaming)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] Documentation content changes
[ ] Other (please describe):

Screen shots / Gifs for design review

How to test or reproduce?

Browser conformance:

[ ] Chrome
[ ] Firefox
[ ] Safari
[ ] Edge

/retest


Performance info is not an error, change it to debug info

Performance info is not an error, change it to debug info.

When you actual release your app, the debug level log is stripped, but the Performance is important and we'd like to print it even if it is a release app.
I will close PR. Thansk


Resources

This is the initial work to provide more flexibility and extendability to the project parsing system by introducing the concept of the Xcode::Resource which is a fancy wrapper for the dictionary of properties. I attempted to document and test a large number of these changes.
A large number of the classes, like Target and Configuration, are now gone and instead are modules. These modules are mixed into the generic Resource when the isa property value string matches a module defined in the Registry.
Ultimately, at this stage of the code, you can:

Traverse groups
Add sub-groups
Add files
Add frameworks
Add configuration values
Save the project

I have some further work that I am exploring that adds the ability to create targets, configuration lists, configurations, and the other requirements. It's all still rough, but I'm pretty excited about the potential to manipulate Xcode project files to generate legitimate installers for code.
You are welcome to ignore the pull request and not merge it in if it is not what you are looking to accomplish with the gem.

First, let me say: Wow! I didn't know about vendorkit. That is exactly where I was heading with the code that I was writing into Xcoder. From first glance it seems that the solution that I arrived at is similar.
Second, and more importantly, all the original functionality has still been preserved. You are still able to enumerate existing projects, targets, and configs. All the work that I have done is how that they are arranged and for the most part that is not too drastically different.
The real key to understanding the implementation is the Registry module and the Resource module. Every object within the project is a Resource and every property in the hash of properties generates methods equal to the key names. In most cases the values are simply returned, but in the cases of the special identifiers, 24 Length hexadecimal strings, if any of those are every returned instead the system does a look up in the Registry to return the Resource object.
The second fun thing is that each Resource, based on the isa property will have additional module mixed in them. This makes the code flexible to changes by Apple, supporting new formats, and divides the functionality into their specific types without any strangeness that comes from sub-classing.
I have write functionality completed in commits on the following branch: https://github.com/burtlo/xcoder/tree/create-targets

That's pretty impressive stuff!
I'll try and get my head around it tonight and give merging a go.
Thanks for the detail, I think it sounds like a much better approach
On 10/02/2012, at 17:29, Franklin Webber
<EMAIL_ADDRESS>wrote:

First, let me say: Wow! I didn't know about vendorkit. That is exactly where I was heading with the code that I was writing into Xcoder. From first glance it seems that the solution that I arrived at is similar.
Second, and more importantly, all the original functionality has still been preserved. You are still able to enumerate existing projects, targets, and configs. All the work that I have done is how that they are arranged and for the most part that is not too drastically different.
The real key to understanding the implementation is the Registry module and the Resource module. Every object within the project is a Resource and every property in the hash of properties generates methods equal to the key names. In most cases the values are simply returned, but in the cases of the special identifiers, 24 Length hexadecimal strings, if any of those are every returned instead the system does a look up in the Registry to return the Resource object.
The second fun thing is that each Resource, based on the isa property will have additional module mixed in them. This makes the code flexible to changes by Apple, supporting new formats, and divides the functionality into their specific types without any strangeness that comes from sub-classing.
I have write functionality completed in commits on the following branch: https://github.com/burtlo/xcoder/tree/create-targets

Reply to this email directly or view it on GitHub:
https://github.com/rayh/xcoder/pull/2#issuecomment-3901834

Found that the impressive code around the resources should have been using extend with the found module and not class_evaling it all into Resource. I fixed that and the other errors that appeared because the other build phases were missing from the Registry list.

I think a good way to see the awesomeness of what is going on here is to use IRB and traverse the objects within the project:

> project.targets.first

 => PBXNativeTarget 7165D44F146B4EA100DE2F0E {"buildConfigurationList"=>"7165D47D146B4EA100DE2F0E", "productReference"=>"7165D450146B4EA100DE2F0E", "productType"=>"com.apple.product-type.application", "productName"=>"TestProject", "isa"=>"PBXNativeTarget", "buildPhases"=>["7165D44C146B4EA100DE2F0E", "7165D44D146B4EA100DE2F0E", "7165D44E146B4EA100DE2F0E"], "dependencies"=>[], "name"=>"TestProject", "buildRules"=>[]}

> project.targets.first.buildPhases

=> [PBXSourcesBuildPhase 7165D44C146B4EA100DE2F0E {"isa"=>"PBXSourcesBuildPhase", "buildActionMask"=>"2147483647", "files"=>["7165D461146B4EA100DE2F0E", "7165D465146B4EA100DE2F0E", "97C80BE5BF3C7AE5C68D7F71"], "runOnlyForDeploymentPostprocessing"=>"0"}, PBXFrameworksBuildPhase 7165D44D146B4EA100DE2F0E {"isa"=>"PBXFrameworksBuildPhase", "buildActionMask"=>"2147483647", "files"=>["7165D455146B4EA100DE2F0E", "7165D457146B4EA100DE2F0E", "7165D459146B4EA100DE2F0E"], "runOnlyForDeploymentPostprocessing"=>"0"}, PBXResourcesBuildPhase 7165D44E146B4EA100DE2F0E {"isa"=>"PBXResourcesBuildPhase", "buildActionMask"=>"2147483647", "files"=>["7165D45F146B4EA100DE2F0E"], "runOnlyForDeploymentPostprocessing"=>"0"}]

 > project.targets.first.buildPhases.first.files.first

=> PBXBuildFile 7165D461146B4EA100DE2F0E {"isa"=>"PBXBuildFile", "fileRef"=>"7165D460146B4EA100DE2F0E"}

> project.targets.first.buildPhases.first.files.first.fileRef

=> PBXFileReference 7165D460146B4EA100DE2F0E {"path"=>"main.m", "isa"=>"PBXFileReference", "lastKnownFileType"=>"sourcecode.c.objc", "sourceTree"=>"<group>"}

This is using none of the convenience methods that I added, similar to the target(name) or config(name) methods that you created and what you get is objects all the way through without any messy code that needs to traverse the hash to build an object tree. Here I walked from the project all the way to a file, through the target's build phase reference to it through the build file.

I've tried to merge, but I'm getting an error:
undefined method `project=' for #<Xcode::Resource:0x007f9b5338db00>

Now, I can see whats /supposed/ to happen, the Xcode::Target module
gets included and that defines an attr_accessor for :project.  If I
debug the initialize() method for Resource, I can see that it picks up
the correct isa and maps it to Xcode::Target.  It then calls
self.extend, and that doesn't cause any errors, yet I cannot set that
property.
Any ideas?
On 10 February 2012 20:07, Franklin Webber
<EMAIL_ADDRESS>wrote:

I think a good way to see the awesomeness of what is going on here is to use IRB and traverse the objects within the project:

> project.targets.first

 => PBXNativeTarget 7165D44F146B4EA100DE2F0E {"buildConfigurationList"=>"7165D47D146B4EA100DE2F0E", "productReference"=>"7165D450146B4EA100DE2F0E", "productType"=>"com.apple.product-type.application", "productName"=>"TestProject", "isa"=>"PBXNativeTarget", "buildPhases"=>["7165D44C146B4EA100DE2F0E", "7165D44D146B4EA100DE2F0E", "7165D44E146B4EA100DE2F0E"], "dependencies"=>[], "name"=>"TestProject", "buildRules"=>[]}

> project.targets.first.buildPhases

=> [PBXSourcesBuildPhase 7165D44C146B4EA100DE2F0E {"isa"=>"PBXSourcesBuildPhase", "buildActionMask"=>"2147483647", "files"=>["7165D461146B4EA100DE2F0E", "7165D465146B4EA100DE2F0E", "97C80BE5BF3C7AE5C68D7F71"], "runOnlyForDeploymentPostprocessing"=>"0"}, PBXFrameworksBuildPhase 7165D44D146B4EA100DE2F0E {"isa"=>"PBXFrameworksBuildPhase", "buildActionMask"=>"2147483647", "files"=>["7165D455146B4EA100DE2F0E", "7165D457146B4EA100DE2F0E", "7165D459146B4EA100DE2F0E"], "runOnlyForDeploymentPostprocessing"=>"0"}, PBXResourcesBuildPhase 7165D44E146B4EA100DE2F0E {"isa"=>"PBXResourcesBuildPhase", "buildActionMask"=>"2147483647", "files"=>["7165D45F146B4EA100DE2F0E"], "runOnlyForDeploymentPostprocessing"=>"0"}]

 > project.targets.first.buildPhases.first.files.first

=> PBXBuildFile 7165D461146B4EA100DE2F0E {"isa"=>"PBXBuildFile", "fileRef"=>"7165D460146B4EA100DE2F0E"}

> project.targets.first.buildPhases.first.files.first.fileRef

=> PBXFileReference 7165D460146B4EA100DE2F0E {"path"=>"main.m", "isa"=>"PBXFileReference", "lastKnownFileType"=>"sourcecode.c.objc", "sourceTree"=>"<group>"}

This is using none of the convenience methods that I added, similar to the target(name) or config(name) methods that you created and what you get is objects all the way through without any messy code that needs to traverse the hash to build an object tree. Here I walked from the project all the way to a file, through the target's build phase reference to it through the build file.

Reply to this email directly or view it on GitHub:
https://github.com/rayh/xcoder/pull/2#issuecomment-3903220

--
Ray Hilton
a: 13/243 Collins Street, Melbourne VIC 3000
m: 0430 484 708
w: http://rayh.com.au

I merged it with some conflicts, but it seems to be alright: https://github.com/burtlo/xcoder/commit/131fd602636621046c1fbeb81dc701062c2e788a
There were conflicts around the project file related to the changes to the parsing. I had moved all of that out to a Parser class, to separate the concerns for future situations on how we might change the parsing of the Xcode project file.

Seems to work ok with the example project, but I have another project
that its failing with.  I'll try and see if I can recreate it in a
test.
On 11 February 2012 04:19, Franklin Webber
<EMAIL_ADDRESS>wrote:

I merged it with some conflicts, but it seems to be alright: https://github.com/burtlo/xcoder/commit/131fd602636621046c1fbeb81dc701062c2e788a
There were conflicts around the project file related to the changes to the parsing. I had moved all of that out to a Parser class, to separate the concerns for future situations on how we might change the parsing of the Xcode project file.

Reply to this email directly or view it on GitHub:
https://github.com/rayh/xcoder/pull/2#issuecomment-3909941

--
Ray Hilton
a: 13/243 Collins Street, Melbourne VIC 3000
m: 0430 484 708
w: http://rayh.com.au

Seems to be PBXAggregateTarget gets added to @project.targets, and
then it fails when we try to assign project to it, as it has no
module.  I've added a mapping to Target, and it seems to work.  Do you
foresee any problems with that?
On 11 February 2012 13:53, Ray Hilton<EMAIL_ADDRESS>wrote:

Seems to work ok with the example project, but I have another project
that its failing with.  I'll try and see if I can recreate it in a
test.
On 11 February 2012 04:19, Franklin Webber
<EMAIL_ADDRESS>wrote:

I merged it with some conflicts, but it seems to be alright: https://github.com/burtlo/xcoder/commit/131fd602636621046c1fbeb81dc701062c2e788a
There were conflicts around the project file related to the changes to the parsing. I had moved all of that out to a Parser class, to separate the concerns for future situations on how we might change the parsing of the Xcode project file.

Reply to this email directly or view it on GitHub:
https://github.com/rayh/xcoder/pull/2#issuecomment-3909941

--
Ray Hilton
a: 13/243 Collins Street, Melbourne VIC 3000
m: 0430 484 708
w: http://rayh.com.au

--
Ray Hilton
a: 13/243 Collins Street, Melbourne VIC 3000
m: 0430 484 708
w: http://rayh.com.au

Ray, that is perfect. That's the exact purpose of the mappings. Map Xcode project types to the Ruby Functionality.
Awesome!


Meeting minutes

Improvements based on group discussion.

Replace the line chart with cards showing the country's GDP and population.
Map projection should be changes.
Need better color scheme, title styling.

Things have been revised accordingly.


#109: Gallery deletion crash

#109

Substituted with https://github.com/ARK-Builders/ARK-Navigator/pull/172


Identify Payment Method

Hi,
I'm looking for a way to identify the payment method used for the invoice or receipt.
This is typically indicated with the words "VISA", "CASH" or "MCARD" in the receipt (see attached)
but I don't think I can simply add a field to the template with REGEX that is searching for these terms as I need to return a standard ENUM for the different payment types.
Can anyone think of a way to do this in REGEX or suggest an approach for adding this to the invoice_template module?
Thanks,
Michael

can you help, @duskybomb ?

Hi all,
I've added a new option to the template as follows:
find_<field_name>
- <option_static> <option_regex>
- <option_static> <option_regex>
- <option_static> <option_regex>

It will look for all of the options and return field_name with value option_static if found.
Note that the list is in priority order i.e. will loop through the options in order and only return the first match.
If no match is found, 'not_found' is returned as the value.
e.g.
  find_payment_method:
  - visa VISA
  - mastercard MCARD
  - amex AMEX
  - cash CASH

The code is as follows and is added to invoice_template.py
      for k, v in self['fields'].items():
            if k.startswith('static_'):
                logger.debug("field=%s | static value=%s", k, v)
                output[k.replace('static_', '')] = v
            # New code for <find_> option starts here
            # check if the field name begins with 'find_' and we have a list of options
            elif k.startswith('find_') and type(v) is list:
                logger.debug("field=%s | find value=%s", k, v)

                # Loop through options
                for v_option in v:

                    # Break the options into the name and regex
                    find_type,find_regex = v_option.split(' ')
                    logger.debug("find_type=%s | find_regex=%s", find_type, find_regex)

                    if re.findall(find_regex, optimized_str):
                        output[k.replace('find_', '')] = find_type
                        logger.debug("Found=%s", find_type)
                        find_match = True
                        break

                if not(find_match):
                    output[k.replace('find_', '')] = 'not_ found'

            # New code for <find_> option ends here

            else:
                logger.debug("field=%s | regexp=%s", k, v)

Feel free to add this to the code base if you think it is useful.
Cheers,
Michael

Sometimes a bill is split into different payment methods, like half in cash and half through card.
We can add a new option payment_method inside fields in template.
payment_method:
	CASH: CASH:(\d+\.\d+)
	VISA: VISA:(\d+\.\d+)
	MCARD: MCARD:(\d+\.\d+)

Then there is slight change in invoice_template. In extract(). The additions have been marked as # *
for k, v in self['fields'].items():
    if k.startswith('static_'):
        logger.debug("field=%s | static value=%s", k, v)
        output[k.replace('static_', '')] = v
    else:
        logger.debug("field=%s | regexp=%s", k, v)

        sum_field = False
        if k.startswith('sum_amount') and type(v) is list:
            k = k[4:]  # remove 'sum_' prefix
            sum_field = True
        # Fields can have multiple expressions
        if type(v) is list:
            res_find = []
            for v_option in v:
                res_val = re.findall(v_option, optimized_str)
                if res_val:
                    if sum_field:
                        res_find += res_val
                    else:
                        res_find = res_val
                        break
        elif k == 'payment_method':  # *
        	res_find = []  # *
		for payment_type, reg_ex in v.items:  # *
			res_val = re.findall(reg_ex, optimized_str)  # *
			if len(res_val) > 0:  # *
				res_find.append(payment_type)  # *
        else:
            res_find = re.findall(v, optimized_str)
        if res_find:
            logger.debug("res_find=%s", res_find)
            if k.startswith('date') or k.endswith('date'):
                output[k] = self.parse_date(res_find[0])
                if not output[k]:
                    logger.error(
                        "Date parsing failed on date '%s'", res_find[0])
                    return None
            elif k.startswith('amount'):
                if sum_field:
                    output[k] = 0
                    for amount_to_parse in res_find:
                        output[k] += self.parse_number(amount_to_parse)
                else:
                    output[k] = self.parse_number(res_find[0])
    		elif type(res_find) is list:  # *
    			output[k] = res_find  # *
            else:
                output[k] = res_find[0]
        else:
            logger.warning("regexp for field %s didn't match", k)

What do you think @m3nu?

It's already possible to define custom fields. There is no need to hardcode it, like you did. If you do it like this, all your field settings will be in Python code eventually. Field settings need to be in the vendor template only.

payment_method:
	- (CASH:\d+\.\d+)
	- (VISA:\d+\.\d+)
	- (MCARD:\d+\.\d+)

I don't think we were handling lists properly
# + : refers to addition
# - : refers to deletion
for k, v in self['fields'].items():
    if k.startswith('static_'):
        logger.debug("field=%s | static value=%s", k, v)
        output[k.replace('static_', '')] = v
    else:
        logger.debug("field=%s | regexp=%s", k, v)

        sum_field = False
        if k.startswith('sum_amount') and type(v) is list:
            k = k[4:]  # remove 'sum_' prefix
            sum_field = True
        # Fields can have multiple expressions
        if type(v) is list:
            res_find = []
            for v_option in v:
                res_val = re.findall(v_option, optimized_str)
                if res_val:
                    if sum_field:
                        res_find += res_val
                    else:
                        res_find.extend(res_val)  # +
                        # - break
        else:
            res_find = re.findall(v, optimized_str)
        if res_find:
            logger.debug("res_find=%s", res_find)
            if k.startswith('date') or k.endswith('date'):
                output[k] = self.parse_date(res_find[0])
                if not output[k]:
                    logger.error(
                        "Date parsing failed on date '%s'", res_find[0])
                    return None
            elif k.startswith('amount'):
                if sum_field:
                    output[k] = 0
                    for amount_to_parse in res_find:
                        output[k] += self.parse_number(amount_to_parse)
                else:
                    output[k] = self.parse_number(res_find[0])
            else:
                if len(red_find) == 1:  # +
                   output[k] = res_find[0]   # +
                else:  # +
                    output[k] = res_find  # +
        else:
            logger.warning("regexp for field %s didn't match", k)

I think OP is only interested in the name of the payment method. Like "VISA", etc. So the amount doesn't need to be extracted. (VISA):\d+\.\d+ We have that elsewhere.
Can you make a PR and test case for this addition?

Working on PR, but for test case we would need a sample invoice (preferably in PDF format).

Merged. It would be interesting to see if Tesseract is good enough to process the invoice posted by OP.

I was able to get this file work with tesseract but I needed to edit the image (Croped it and adjusted brightness and contract). Plus I used tesseract 4.0 (alpha)

Here is the tesseract-ocr output text:

Outlet: Pepper Lunch - Jurong Point
Stall ID: 01 Pepper Lunch Restaurant
Machine No: 01

Cashier: Arpan

Date: 6/6/2018 12:24:44 PM

Receipt No: 000035 Tag No: 24

1, (8M) Salmon & Chicken 1X 7.90 7.90
Pepper Rice

~ (SM) Salmon & 1
Chicken Pepper Rice
- Iced Lemon Tea |

2. a BBQ Beef Pepper 1X 7,90 7.90
ce
- (SM) BBQ Beef Pepper 1
Rice
- Pepsi j
3. (WOL) Beef Pepper Rice 1X 11.90 11.90
- (WDL) Beef Pepper 1

Rice

- Iced Lemon Tea 1

- Miso Soup 1
“Total: ttStS~S 27.10
Sub Total : 27,70
Rounded Amt : 0.00
Net Total : Tl
Inclusive GST 7% : 1.81
~ VISA: 27.10

*xThank Your*
GST RegNo: 200408968W
Papper Lunch Jurong Point Telephone:
6265 7425
63 Jurong Hest Central 3
#B1-62/63 Jurong Point Shopping Centre
Singapore 648331
Pepper Lunch Website:
http://www. pepper lunch. com. so/

How i use it for extracting the retailer name ,date,total and the items with price from receipts.i want to         preprocess lot of supermarket receipts before giving the tesseract 4.0 .
can we use  invoicetodata  for receipts images only for extracting items and other fields .any suggestion ...
thanks in advance

You can define your own fields and add new templates. The process is roughly:

Find the template folder in this repo or locally.
Choose a similar template as base and look into the template docs.
Make your own new template based on debugging and looking at OCR output.
Add your own template folder to the templates to be loaded.
Done.

Thanks a lot Manu,
On Thu 5 Jul, 2018, 6:43 AM Manu<EMAIL_ADDRESS>wrote:

You can define your own fields and add new templates. The process is
roughly:

Find the template folder in this repo or locally.
Choose a similar template as base and look into the template docs
https://github.com/invoice-x/invoice2data/blob/master/TUTORIAL.rst.
Make your own new template based on debugging and looking at OCR
output.
Add your own template folder to the templates to be loaded.
Done.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/invoice-x/invoice2data/issues/141#issuecomment-402579754,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AcLe49tDl9j4O6HzpcNduiW-aSq4hPMiks5uDWg3gaJpZM4UfAHy
.


[$1000] Create new library and native module react-native-wallet

To support tokenization in NewDot which is adding a virtual card to Apple / Google Pay, we will need access to some native methods.
Problem:
Apple Pay and Google Pay are table stakes in the card market today. We support both of them for physical and virtual cards. Going forward, members will be able to access their virtual card details in NewDot. However, it isn't intuitive to know that you can copy your virtual card details, and manually key them into the Apple/Google wallet apps to tokenize your virtual card.
We're going to create a new package that will return the necessary parameters and call native methods for adding a card to the wallet
The endpoint we want to call is CreateDigitalWallet.
We will need to call native modules to fetch specific native variables to call CreateDigitalWallet

deviceID
certificates
nonce
nonceSignature

Then once we have the return values, we will call the native modules to add these to apple / google wallets
CreateDigitalWallet Summary
The parameters required for the new CreateDigitalWallet API/auth command are:

Mutual parameters
Android specific parameters
iOS specific parameters

1.appVersion
1.walletAccountID 2.deviceID
1.certificates 2.nonce 3.nonceSignature

The return values for the new CreateDigitalWallet API/auth command are:

Mutual return values
Android return values
iOS return values

1.cardToken
1.opaquePaymentCard 2.userAddress 3.network 4.tokenServiceProvider 5.displayName 6.lastDigits
1.encryptedPassData 2.activationData 3.ephemeralPublicKey

Upwork Automation - Do Not Edit

Upwork Job URL: https://www.upwork.com/jobs/~01e91e840e65f1cbba
Upwork Job ID:<PHONE_NUMBER>361259008
Last Price Increase: 2024-03-12

Hey @suneox, I copied what was relevant from the issue into this description

Hey @thienlnam - are we just waiting for proposals on this one?

Updated the bounty since this is a more intensive - let's give it another week and then we can hand it to our expert contributors? cc @lschurr

Proposal
Please re-state the problem that we are trying to solve in this issue.
Create new library and native module react-native-wallet
What is the root cause of that problem?
New Feature
What changes do you think we should make in order to solve the problem?
Develop a new package with a CreateDigitalWallet API/auth command to handle the tokenization process seamlessly. This package will call native methods to fetch essential parameters and execute the necessary steps for adding a card to the wallet.
Changes to Implement:

Create a new package with the CreateDigitalWallet API/auth command.
Fetch required parameters using native modules:

Mutual parameters: appVersion
Android specific parameters: walletAccountID, deviceID
iOS specific parameters: certificates, nonce, nonceSignature

Obtain return values from the CreateDigitalWallet command:

Mutual return values: cardToken
Android return values: opaquePaymentCard, userAddress, network, tokenServiceProvider, displayName, lastDigits
iOS return values: encryptedPassData, activationData, ephemeralPublicKey

Implement the necessary logic to call native modules for fetching parameters and executing the tokenization process for both Apple and Google Pay.

What alternative solutions did you explore? (Optional)

Could you review the proposal @akinwale?

@brunovjk Can you add some details such like what native methods you are going to use to fetch the items we'll need to pass to CreateDigitalWallet?

@brunovjk Can you add some details such like what native methods you are going to use to fetch the items we'll need to pass to CreateDigitalWallet?

Of course, first thing in the morning.

Do we already have the entitlements necessary to add cards to apple like mentioned here: https://developer.apple.com/documentation/passkit_apple_pay_and_wallet/pkaddpaymentpassrequest?language=objc ?

@brunovjk Can you add some details such like what native methods you are going to use to fetch the items we'll need to pass to CreateDigitalWallet?

Proposal
Updated
Added detailed steps for fetching parameters and executing tokenization process.
cc: @thienlnam

@brunovjk's proposal looks good to me. However, the relevant API to handle adding cards to to Google Pay is the Android Push Provisioning API, not the Google Pay API which is used for checkout.
@thienlnam Looks like special access is required to be able to view the documentation (for the Android API), while the iOS view controller requires a special entitlement.
Android

Android Push Provisioning API
https://developers.google.com/pay/issuers/request-access?api=true

iOS

PKAddPaymentPassViewController reference

🎀👀🎀 C+ reviewed.

Proposal
Updated
Updated the Android section to reflect the correct API usage. Thanks, @akinwale, for the review and correction.

Are we good to assigne this to @brunovjk? @akinwale @thienlnam?

let's get started on this

I'm diving straight into. Setting up the branch and will keep you posted here throughout the week. When I run into permissions and/or need further information, I'll be sure to ask. Thank you.

Task Checklist for Creating react-native-wallet Library and Native Module:

[ ] Create a new branch in the Expensify app repository.
[ ] Create react-native-wallet folder and run npx create-react-native-library to initialize the library.
[ ] Familiarize with Google Pay and Apple Pay integration documentation.
[ ] Identify necessary native modules for Android and iOS platforms.
[ ] Add react-native-device-info to fetch the app version.
[ ] Integrate dependencies for Android Push Provisioning API.
[ ] Integrate dependencies for iOS PassKit framework.
[ ] Implement native methods to fetch parameters for Android (walletAccountID, deviceID).
[ ] Implement native methods to fetch parameters for iOS (certificates, nonce, nonceSignature).
[ ] Implement code to interact with Google Pay services for Android.
[ ] Implement code to interact with Apple Wallet services for iOS.
[ ] Define JavaScript functions to expose native module methods.
[ ] Implement CreateDigitalWallet functionality in JavaScript.
[ ] Test functionality on Android and iOS devices.
[ ] Ensure compatibility with different React Native versions and device configurations.
[ ] Document usage instructions for react-native-wallet library.
[ ] Provide examples and code snippets for integration.

This checklist will be updated as we progress and if necessary. I will provide daily updates until we have a PR ready for review.

Update:

Created new branch.
Initialized react-native-wallet.

Next: Review payment integration docs.

@thienlnam Do you think we can get a "company account" to access Android Push Provisioning API documentation?

To access restricted sections of our Device Tokenization documentation, you must be signed into a Google Account that's been authorized to view that content.
You must provide a corporate email address for access to the issuer developer's site. Personal email accounts (i.e. gmail, yahoo, msn, etc.) cannot be associated with an NDA and therefore will not be given access.

I should be able to use my account<EMAIL_ADDRESS>but it seems not.

Update:
@thienlnam I've confirmed the need for permissions to access both Android Push Provisioning API and the PKAddPaymentPassViewController. These APIs require special permissions typically granted to card issuers. I've looked into the steps required to request access, but it seems we'll need to navigate some bureaucratic processes with Google and Apple.
Google Requriment:

You must provide a corporate email address for access to the issuer developer's site

Apple Requriment:

You need special permission from Apple to submit apps with this key enabled. For more information, contact<EMAIL_ADDRESS>
I'm wondering if anyone on the team has experience with obtaining these permissions or knows someone who does.
Next: To proceed with the next steps, particularly on Android, authorization is required to access the documentation.

Update:
@thienlnam I've confirmed the need for permissions to access both Android Push Provisioning API and the PKAddPaymentPassViewController. These APIs require special permissions typically granted to card issuers. I've looked into the steps required to request access, but it seems we'll need to navigate some bureaucratic processes with Google and Apple.
Google Requriment:

You must provide a corporate email address for access to the issuer developer's site

Apple Requriment:

You need special permission from Apple to submit apps with this key enabled. For more information, contact<EMAIL_ADDRESS>
I'm wondering if anyone on the team has experience with obtaining these permissions or knows someone who does.
Next: To proceed with the next steps, particularly on Android, authorization is required to access the documentation.

@AndrewGable Probably knows best from our OldApp - do you remember going through this process / do we need to again with NewDot?

We have the permissions, but you probably will have to work with @joekaufmanexpensify to get it added to the NewApp ID until we can add this to HybridApp.

Hey @joekaufmanexpensify! 😊 Do you mind lending a hand here?

Sure, happy to help if I can. @AndrewGable mind clarifying a bit more what this means:

to get it added to the NewApp ID until we can add this to HybridApp.

I'm not super sure based on this what I'm supposed to be helping with.

Hey @joekaufmanexpensify, I hope this message finds you well. I'm working on integrating the React Native Wallet but need guidance on permissions.
Accessing the Android Push Provisioning API and PKAddPaymentPassViewController requires special permissions. I'm unsure how to proceed with Expensify's existing ones. Could you provide direction on obtaining these permissions?
Thanks for your help!

Got it. So this issue is adding support for in-app tokenization of virtual cards from NewDot to Apple and Google Pay? If so, that sounds great, and happy to get the app whitelisted with Apple and Google via Marqeta.
If Marqeta just needs to work with Apple and Google to whitelist the new app so we can access the Apple/Google Pay APIs for it, can you share the NewApp ID or point me in the direction of where I can find it so I can pass that along to Marqeta?

iOS: com.chat.expensify.chat
Android: com.expensify.chat

What's the latest here @joekaufmanexpensify @AndrewGable @brunovjk?

@joekaufmanexpensify is currently handling the process of whitelisting the New App with Apple and Google. Meanwhile, I'm in the process of identifying the necessary native modules for Android and iOS platforms. While iOS seems straightforward, I'm encountering difficulties accessing the documentation for Android:
"To access restricted sections of our Device Tokenization documentation, you must be signed into a Google Account that's been authorized to view that content. Partners must use a Google Account associated with their corporate email address rather than a personal account."
Currently, I don't have such an account configured. Should we obtain it from Expensify, or should I procure it independently? Thank you.

Yep. I just asked Marqeta to whitelist the new app IDs here!

Update
In process of whitelisting the New App with Apple and Google. In process of identifying the necessary native modules for Android and iOS platforms.

Currently, I don't have such an account configured. @lschurr @thienlnam Should we obtain it from Expensify, or should I procure it independently? Thank you.

Ah good question, I can sign up for access with my account and give you third party access - however it seems like you'll need to sign an NDA with Google to do so. Can you provide me the email you intend to use?

Can you provide me the email you intend to use?

<EMAIL_ADDRESS>
you'll  need to sign an NDA with Google

Great, I'll research further. Do you know if there is a cost for this? Thank you very much @thienlnam.

Nope there shouldn't be a cost associated with having to sign their NDA, but let us know if there is

Update:
Start implementing 'PKAddPaymentPassViewController'. @thienlnam We use Swift and Java, right?

Yep. I just asked Marqeta to whitelist the new app IDs here!

Did we have any updates? I don't have access to the link above to check.

I can sign up for access with my account and give you third party access

@thienlnam Sorry to bother, but did you have the chance to invite me? Thank you.

They got back to me and provided access - I had you listed as a third party developer. Did you get sent an NDA for you to sign?

Great! I just registered, google will send the NDA within one business day, they said.
Thank you very much @thienlnam

Update:
In the process of signing the NDA Google to begin implementing Android Push Provisioning API.

Hey @thienlnam - Can you help me associate this feature with a wave or VIP project? Is this part of wave-collect or vip-split? Or something else?

Hi @thienlnam, Could you please share the corporate email address you used to grant me access? I need it to proceed with signing the NDA requested by Google Support. Thank you.

<EMAIL_ADDRESS>
Hey @thienlnam - Can you help me associate this feature with a wave or VIP project? Is this part of wave-collect or vip-split? Or something else?

This would be part of wave-collect - cc @joekaufmanexpensify to confirm here, here was the original issue

Update:
Android: Waiting for response from Google support.
Apple: Implementing 'PKAddPaymentPassViewController'  native modules.

Update:

Android: Waiting for response from Google support. Apple: Implementing 'PKAddPaymentPassViewController' native modules.

Same as above. If I don't hear from Google Support by the beginning of next week, I will create a Google Workspace in my name to continue, we at least need to access the documentation now.

Update:

Android: Waiting for response from Google support.
Apple: Implementing 'PKAddPaymentPassViewController' native modules.

Same as above. If I don't hear from Google Support by the beginning of next week, I will create a Google Workspace in my name to continue, we at least need to access the documentation now.

Same as above

Any update @brunovjk?

@lschurr I continue working on iOS Native Modules to get the information we need. My iOS build is a bit slow these days, but moving forward.
I can't answer @joekaufmanexpensify's question above.

@thienlnam
They got back to me and provided access - I had you listed as a third party developer. Did you get sent an NDA for you to sign?

I still haven't been able to access the necessary Android documentation:

R: The Google Pay Financial Institutions Support Team
Unfortunately, we cannot grant your request. Given the confidential nature of the onboarding materials, users can only view full content after associating a Corporate email address to a Google Account. The process for associating a corporate email address with a Google account begins here.

I'm trying to create a Google Workspace account in my name, I'd appreciate any help on these issues. I can't do much about permissions.

@lschurr I continue working on iOS Native Modules to get the information we need. My iOS build is a bit slow these days, but moving forward.
I can't answer @joekaufmanexpensify's question above.

@thienlnam
They got back to me and provided access - I had you listed as a third party developer. Did you get sent an NDA for you to sign?

I still haven't been able to access the necessary Android documentation:

R: The Google Pay Financial Institutions Support Team
Unfortunately, we cannot grant your request. Given the confidential nature of the onboarding materials, users can only view full content after associating a Corporate email address to a Google Account. The process for associating a corporate email address with a Google account begins here.

I'm trying to create a Google Workspace account in my name, I'd appreciate any help on these issues. I can't do much about permissions.

Continuing to gather iOS native calls this week. Build fixed, more progress expected. Not sure about permissions yet, will update when needed. Thanks.

Update: Same as above.

I'll have to work with you to get the documentation required for Android - in the meantime, please continue working on iOS implementation

Update: Same as above.
Thank you very much @thienlnam

Update: Keep working on iOS implementation

Update: Same as above.

Update: I've implemented the canAddPaymentPass():Returns a Boolean value that indicates whether the app can add cards to Apple Pay functionality, but it's returning false. I included 'com.apple.developer.payment-pass-provisioning': true in Chat.entitlements, but I'm not sure if it's enough yet. Will revisit it first thing Monday morning.

Same as above :D

Update: I've implemented the canAddPaymentPass method in this branch, but I'm facing an issue with the entitlement configuration, resulting in a false response. I'd appreciate it if someone could test the branch :D
Steps to test:

Run the app in the iOS simulator.
Go to Account Settings.
Navigate to the Wallet page.
Scroll down to the 'Add Card to ApplePay' section.
Click on the "Check" button.
Confirm if you get a canAddCard: true response.

While I continue to work on PKAddPaymentPassViewController to obtain certificates, nonce, and nonceSignature, it's essential to resolve the entitlement configuration.

Update: Continue to work on certificates, nonceandnonceSignature`.

@thienlnam @brunovjk do we still need to whitelist the new App ID with Apple/Google? If so, curious if y'all have any ideas for these questions?
That's the info Marqeta needs to do this.

Sorry, I can't answer your question. Unfortunately when it comes to permissions I cannot do much.

Update: Attempted to test canAddPaymentPass() method for adding cards to Apple Pay, but discovered that we need a production build to fully test it:
"Adding a card to Apple Wallet can be tested only via TestFlight, which requires a production build with a specific entitlement in both provisioning profile and in Entitlements-Release.plist."
So I can't really make it work here. Next week I will keep focusing on the PKAddPaymentPassRequestConfiguration to get the native info we need to call PKAddPaymentPassViewController. I will try to create some kind of prototype so we can work on.
I will keep it informed.

Same as above :D

Update: Attempted to test canAddPaymentPass() method for adding cards to Apple Pay, but discovered that we need a production build to fully test it:

"Adding a card to Apple Wallet can be tested only via TestFlight, which requires a production build with a specific entitlement in both provisioning profile and in Entitlements-Release.plist."
I can generate an adhoc build on your PR which should allow this to get tested on an iOS device.

@thienlnam @brunovjk do we still need to whitelist the new App ID with Apple/Google? If so, curious if y'all have any ideas for https://github.com/Expensify/App/issues/36957#issuecomment-2047540384?

I can get the answers to these

Update: Continue working on PKAddPaymentPassViewController on modules/react-native-wallet/ios

Update:
Completed: Implemented basic iOS native module (CreateDigitalWallet) to:

Check if Apple Wallet pass can be added (canAddPaymentPass)
Fetch device ID and app version (fetchWalletParameters)

Next Steps:

Obtain necessary certificates from the server for the iOS module.
Work with the backend team to:

Implement secure nonce and nonce signature generation.
Create the CreateDigitalWallet endpoint.

I'll be OOO until May 6th

Done:

Initiated iOS Setup: Began iOS implementation by setting up local library structure and integrating the canAddPaymentPass method.

Week TODO:

Fetch iOS Parameters: Retrieve required iOS parameters (appVersion, deviceID) to progress with PKAddPaymentPassViewController integration.

Update: Continuing to gather iOS native calls this week

Same as above :D

Same as above :D

Update: Keep working on those iOS modules

Update: Started structuring files for native Android modules

Update: Keep working on native Android modules

Update: Continue working on the local branch

Update: Same as above.

Update: Continue working on the local branch: https://github.com/brunovjk/Expensify-App/tree/feat/36957/modules/react-native-wallet

Update: Continue working on the local branch: https://github.com/brunovjk/Expensify-App/tree/feat/36957/modules/react-native-wallet

Same as above

Same as above

Same as above

is the ios part done? i'd like to test it if it is

@Darren120 I believe you can't really test it, we still need to work out some details, akinwale and thienlnam are doing a first review, then when we can test it properly, I'll let you know :D

@Darren120 I believe you can't really test it now, we still need to work out some details, then when we can test it properly, I'll let you know :D

Update: Making progress on error handling, documentation, and unit tests.
Next: Build the library locally

Update: Same as above

Update: Same as above

Update: Continue working on test build. It's going well :D

Update: Same as above

Update: Build successfully. Working on Unit tests
Next: Improve handling of errors and documentation.

Update: iOS Build successfully. Working on android

Update: Adjusting/testing calls to the module on Android.

Update: Same as above

Update: Working on Unit tests

Update: Reviewing library/PR

Working on component/page to test the library.
Next: Complete unit tests.

I’ve been working on the example app but ran into some challenges due to permission limitations as an external contributor. I’m gathering all the necessary information and will provide a more detailed update later today. Apologies for the delay—I've been focused on getting everything just right, which has taken more time than anticipated, but I’ll work to accelerate progress. Thank you for your patience!

Hey @thienlnam 👋  I wanted to let you know that I’ve created a fully mocked example app, which allows us to test the functionality without worrying about permissions for now. In hindsight, I realize I should have done this weeks ago, and I apologize for the delay, but for now, I’ve got everything set for the mocked example app. I’m currently refining the linking process and will be running more tests before circling back to you ASAP this week.
After that, we can test the app with the necessary permissions. Alternatively, we have the C+ email, we could explore that option too. In the meantime, please check out the example app README for more details. I’ll keep you updated! Looking foward to move this feature forward as quickly as possible.

We should have some availability from our agencies - asked for some assistance there

Thanks :D Update: I'm still fixing the library link with the example app.

Hello, I'm Michał from SWM, and I will help finish this feature

Hey @Skalakid! Thanks so much for jumping in. I’m here if you need anything from me, just let me know.

@brunovjk @thienlnam do we have any react-native-wallet related channel on Slack for communication? If no, how about creating one? It will be easier for use to plan the work and split the tasks :D

Working on getting it created (need a slack admin to add Bruno), in the meantime I would just use #expensify-open-source for any real-time communication

I sent a report on Michael's Slack DM about our task, along with next steps.

Hello, there is not much update from my side on this issue.
Recently, I focused on reviewing the current library code and launching the example app. I wasn't able to do it because of some library linking problems that I started investigating. I will try to prioritize this tomorrow and give you more info

Curious for the latest on this issue? We ran into this issue trying to set up Expensify Cards for our digital wallets.

Curious for the latest on this issue? We ran into this issue trying to set up Expensify Cards for our digital wallets.

Last update: https://expensify.slack.com/archives/C07PBN9MVV3/p1728902736774509?thread_ts=1728901923.086159&cid=C07PBN9MVV3

Update: I'm reviewing and testing the code from @brunovjk's PR. I have some problems with native module linking and SDK. I was able to create a blank new library and run the SDK there. I connected it to the NewDot app, and functions are being executed; however they fail because it seems like dev app isn't whitelisted for the Push Provisioning API (com.expensify.chat.dev). Can we check that, please?

Still reviewing and testing the library. Latest updates on the channel #react-native-wallet

Relabelling as Lauren is OOO now.

Update: We're currently blocked on Apple's provisioning profiles but making progress. The Android POC has started.
#react-native-wallet

Removing Isa since I'm back :)

Update: We are still working on Android POC. We are facing some issues with listToken functions and we are waiting for the verification if it's not a problem with backend communication with TSP

Update: POC it's ready - Slack thread. The only thing that we caught during the implementation was the problem with the listToken function - Slack thread.

@brunovjk Good news! However, I am unable to access the linked Slack threads.

Thanks @akinwale, I will post here if any important information comes up.


Unable to access lobbies

Whenever I start, I do the tutorial and I look at the leader board and it says it can't authenticate my Oculus account, and I can't access any lobbies at all, the mod menu also doesn't work.

the mod hasnt been maintained since 2022, also thats probably not an issue with the mod


Remove picwriter dependency for coupler_adiabatic component

I rewrote the coupler_adiabatic component using phidl and gdsfactory components. I also ran tidy3d mode solver on the original and the new components, and the effective indices match pretty well ([1.4393703 1.4260598] for the picwriter component, and [1.4332894, 1.4260652] for the new component).
Comparison between the two in KLayout:

There are two big advantages to rewriting this component:

We can now import this faster
We can visualize the 3d representation, which you couldn't do with the picwriter component

I will try to refactor for better code quality.

i will send a PR with the fixed version, based of this one
thank you Skandan!
fixes 1 issue from https://github.com/gdsfactory/gdsfactory/issues/471
https://github.com/gdsfactory/gdsfactory/issues/471