Byte dribble delay support

Allow setting configurable delay after each byte WireMock is sending in the response body.
This resolves #472
Example:
stubFor(get(urlEqualTo("/delayedDribble")).willReturn(
    .aResponse()
        .withStatus(200)
        .withBody(new byte[] {1, 2, 3})
        .withByteDribbleDelay(10)));

Removed some spurious white spaces as a separate commit because they were inconsistent within the same class file (WireMockHandlerDispatchingServlet) we were working on.

Hey, is there anything else that needs to be done for this to be merged? It'd be be a great feature to have!

There are two things about this I think might improve it:

Dribble the whole response, not just the body (more realistic IMO)
Be able to specify it in terms of the total response time + number of chunks, rather than delay between individual bytes (thus effectively supporting varying chunk sizes). Alternatively this could be framed as chunk size + inter-chunk delay.

I could live without 1) I think, as it's probably a bit harder to buffer the entire response and have everything still work correctly.

@tomakehurst Hey, I've changed it to now use total response time and number of chunks, let me know what you think.
I'm not sure with the current setup if it's possible to dribble the whole response. Do you have any ideas for how to do this? Cheers

Thanks, I'll try and take a look at this over the next day or two.
The solution I had in mind for dribbling the whole response was to grab the TCP socket (like the fault implementations do), buffer the response then dribble it into the socket.

OK, I've had a little play with this. Generally looks good, but I've managed to make it blow up with a particular combination of parameters.
Here's the test case that demonstrates this: https://github.com/tomakehurst/wiremock/blob/sky-uk-byte-dribble-delay/src/test/java/com/github/tomakehurst/wiremock/ResponseDribbleAcceptanceTest.java#L73
I see this error when I run it:
java.lang.InterruptedException: sleep interrupted
	at java.lang.Thread.sleep(Native Method)
	at com.github.tomakehurst.wiremock.servlet.WireMockHandlerDispatchingServlet.writeAndTranslateExceptionsWithChunkedDribbleDelay(WireMockHandlerDispatchingServlet.java:195)
	at com.github.tomakehurst.wiremock.servlet.WireMockHandlerDispatchingServlet.applyResponse(WireMockHandlerDispatchingServlet.java:159)
	at com.github.tomakehurst.wiremock.servlet.WireMockHandlerDispatchingServlet$ServletHttpResponder.respond(WireMockHandlerDispatchingServlet.java:125)
	at com.github.tomakehurst.wiremock.http.AbstractRequestHandler.handle(AbstractRequestHandler.java:63)
	at com.github.tomakehurst.wiremock.servlet.WireMockHandlerDispatchingServlet.service(WireMockHandlerDispatchingServlet.java:102)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:365)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
	at org.eclipse.jetty.server.Server.handle(Server.java:499)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:258)
	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
	at java.lang.Thread.run(Thread.java:745)

Hey, @tomakehurst sorry for not getting a chance to look at this sooner.
I think what's happening is you're actually triggering a socket timeout as there are 2 chunks and the total duration is 2 x the socket timeout.
It's not that nice that an interrupted exception is printed/returned from the Wiremock server though. I don't think there's much we could do other then suppress it/assume that it's been timeout out and log a meaningful message/throw a meaningful exception?

OK, I've tweaked my branch to ignore the InterruptedException rather than re-throwing it. I can't see any value in it being surfaced as an error in this particular case.
I've also found another oddity, which is that having tweaked the test case I added so that it comes in under the timeout, the total duration is coming out at half what it should be. I've pushed to my branch again if you wouldn't mind taking a look?

Hmm it looks like the first test ResponseDribbleAcceptanceTest.requestIsSuccessfulButTakesLongerThanSocketTimeoutWhenDribbleIsEnabled is actually not really testing the duration correctly as it's only checking that it was longer then the socket timeout rather then the specified total duration.
Moving the start time to before we make the request and changing the assertion in that test to be greater then or equal to the total duration fixes it. Should I make that change or are you okay to do it in your branch?

Ah, got it. Quite a trivial change, so I'll make it on the branch.
After that I think we're all set to merge it.


Select HumanTimeLocale depending on language

This makes possible to override HumanTimeLocale fields, which could be necessary for some languages, e.g. Russian (related #18).
@cies don't merge this yet because I have a question.  Now lambdaCmsTimeLocale seems to be redundant.  Look, we can remove this function and instead of that define functions returning HumanTimeLocales in each language module, e.g.:
-- module LambdaCms.I18n.Dutch
dutchHumanTimeLocale :: (CoreMessage -> String) -> HumanTimeLocale
dutchHumanTimeLocale rm = (defaultHumanTimeLocale rm) { timeLocale = dutchTimeLocale }

-- module LambdaCms.I18n.Italian
italianHumanTimeLocale rm = (defaultHumanTimeLocale rm) { timeLocale = italianTimeLocale }

-- module LambdaCms.I18n
lambdaCmsSelectHumanTimeLocale ("nl":_) = dutchHumanTimeLocale
lambdaCmsSelectHumanTimeLocale ("it":_) = italianHumanTimeLocale
lambdaCmsSelectHumanTimeLocale ("ru":_) = russianHumanTimeLocale
lambdaCmsSelectHumanTimeLocale (_:rest) = lambdaCmsSelectHumanTimeLocale rest
lambdaCmsSelectHumanTimeLocale _        = defaultHumanTimeLocale

-- module LambdaCms.Core.Foundation
lambdaCmsHumanTimeLocale = do
    langs <- languages
    y <- getYesod
    let rm = unpack . renderMessage y langs
        htl = lambdaCmsSelectHumanTimeLocale langs rm
    return htl

What will you say?

we can remove this function and instead of that define functions returning HumanTimeLocales in each language module

indeed. looks good to me. you want to include that in the PR?

@cies, yep.

I've pushed final commits, let's wait for Travis.

Can you upload this on Hackage after merge?


Make sure we handle unicode file names

This should only really be a matter of passing a byte array to the rust code instead of a c string and operating under the assumption that this is Utf8-encoded (or similar), but I want to double-check that this will work properly in this project before I commit to others.

works from Qt -> rust, now to get it to work the other way :)

WOOOO

(notice the title at the top haha, proper unicode ^^ that data has round-tripped from qt -> rust -> qt again, yay!)


Update stale workflow to 90 days

Change Overview

Noticed that a lot of issues were closed in the last few weeks. Even though 60 days is long enough, it is hard to keep long-term issues active before the work is planned.
Making the default time 90 days to allow an extra time buffer to triage issues.

Pull request type
Please check the type of change your PR introduces:

[ ] :construction: Work in Progress
[ ] :rainbow: Refactoring (no functional changes, no api changes)
[ ] :hamster: Trivial/Minor
[x] :bug: Bugfix
[ ] :sunflower: Feature
[ ] :world_map: Documentation
[ ] :robot: Test

Looks like this PR needs to be merged manually.


fix relative path recursive put

There seems to be an issue with recursive put that can cause issues like:
FS.put("relative/path/dir", "/remote", recursive=True)
FS.find("/remote") == ["/remote/e/path/dir/file1.txt", "/remote/e/path/dir/file2.txt"]

This is due follow lines (637-640) from fsspec.spec:
rpaths = [
    posixpath.join(rpath2, path[len(lpath) :].lstrip("/"))
    for path in lpaths
]

lpaths are absolute posix paths like /absolute/relative/path/dir/file1.txt. While lpath is the unmodified input lpath which can be like relative/path/dir. This results in a corresponding rpath of /remote/e/path/dir/file1.txt.
This would also affect certain absolute paths in windows.
lpath = r"C:\\some\\windows\\dir"
lpaths = ["C:/some/windows/dir/file1.txt", "C:/some/windows/dir/file2.txt"]  # is changed to posix via 'make_path_posix'
rpath2 = "/remote"
rpaths = [
    posixpath.join(rpath2, path[len(lpath) :].lstrip("/"))
    for path in lpaths
]
rpaths == ["/remote/le1.txt", "/remote/le2.txt"]

The following "one-line" change should fix both these scenarios and also lpaths like ../parent/path.
Also added a couple of tests for this. One that tests FS.put("dirbasename", "/remote", recursive=True), and another that tests recursive put on current directory: FS.put(".", "/remote", recursive=True).

For some reason, the changes in memory and mapping are showing up here - could you check your git base/merging? Otherwise, the PR looks good.

i did the PR before those changes while the tests were failing so I merged the subsequent commits to get the working tests. Could you suggest how to remedy this with git? Is there a way to rebase it so that only my commit appears and occurs after yours?

hope that fixed it

It seems to be solved - perhaps it was github figuring things out for a while.
Thanks for this!


es6 backends and recline

Hey folks - Over the last couple weeks I have been working on porting some of the recline data backends to es6!
The projects are here
https://www.npmjs.com/~nucivic
Some es6 language features including Promise and fetch mean that these backends can be largely run as standalone modules, without dependencies (an exception being the es2sql library for the cartodb backend).
Given all of this, and some other development initiatives that are underway, including a react-based dashboard which leverages these backends to fetch data and create data visualizations using nvd3, I'm curious if there is any interest in building a next generation of the recline module which could leverage the data backends.
I'm imagining something implemented in es6 that could function as a standalone library (with or without the data backends) and handle common data modeling needs, querying, etc, in the spirit of the recline library, but with a smaller footprint (no backbone, jquery, underscore deps for instance), that could be used in react apps (our primary use case) as well as across other javascript platforms.

@starsinmypockets yes definitely interest in a next generation recline backend.
In general, I think there are 2 things:
A. Would really welcome one or more other persons becoming maintainers of recline - if you were interested (I know nucivic are active here) that would be very welcome.
B. I think we should take a moment to think about overall architecture of recline v2.
On B I have some existing thoughts e.g.:

Recline should become a "core" with just main models with Views and Backends in their own repos (it might make sense to have a github.com/recline org for this (I already reserved it ...)
In terms of the core i started some thoughts here: https://github.com/rgrp/dataframe.js (i put it here just because it was very early stage thoughts)

I would be quite happy to drop backbone etc - and definitely interested in using react (though core may need very few deps)

In terms of views I think we just want to document this better and get these in one place and establish a pattern.

I have also been thinking about the view "descriptor" language (i.e. the json lang for describing views) - see https://github.com/dataprotocols/dataprotocols/issues/77

/cc @pwalsh @amercader @brew

Just a comment on Backbone/React/whatever.
My personal preference is definitely for React, but aside from that, I'm very happy with the approach we've recently taken in OpenSpending, where a little effort in design and implementation has ensured that component logic is not tightly coupled to any framework, and we then have a thin binding for the framework (Angular in that case).

https://github.com/openspending/babbage.ui/tree/feature/modern/src/components
https://github.com/openspending/babbage.ui/tree/feature/modern/src/bindings

This pattern enables the possibility of shipping bindings for several js frameworks, and not tying the core lib to any one in particular.

@starsinmypockets any thoughts?

I haven't had a change to review the code @pwalsh posted but I like the sound and the approach.
We're discussing next steps internally (and have heads down on a data visualization product!)
I am not familiar with the entire recline feature set, but we are using something internally to normalize  data from disparate data sources (backends) and performing queries against them (either against the remote data source itself or a against loaded memoryStore). It's largely modeled off of recline. The philosophy is to leverage es6 and to eliminate dependencies, and to remain very framework agnostic:
framework to load and query backends:
https://github.com/NuCivic/react-dashboard/tree/master/src/models
backends:
https://github.com/NuCivic/csv-es6-data-backend
https://github.com/NuCivic/csv-cartodb-data-backend
https://github.com/NuCivic/dkan-es6-data-backend

Hey @starsinmypockets
Your points on data source backends are well heard here. See my comments on on a related codebase for some recent discussion on this. Looking at your code you are going in a similar direction there.
Side note: I'm a little confused by your terminology around "es6-*" backends. The fact it the libs are written in es6 does not make it an "es6 backend" as I'd understand it: it all gets compiled to es5 anyway in the browser, right? I'm bringing this up not to engage in bikeshedding, but because I was genuinely trying to understand what it means.
In any event, I've been writing quite a lot of ES6 and it is always a pleasure to read other ES6 code - I'd love to collaborate specifically on the "query interface + multiple backends" stuff quite directly, if possible.

Agreed that the naming is a bit obtuse. I'm open to suggestions!


Remove Lens from production guides

Description
This PR aims to remove as much references to Lens as possible. The main job is to just translate whatever action was performed in Lens into a normal shell command, not to adjust the guides themselves (see also additional info below).
I did not remove Lens from past upgrade guides, pages marked as draft (k3d installation) and pages that will be deprecated (see also additional info below).
Related issues
Close https://github.com/united-manufacturing-hub/MgmtIssues/issues/1322
Checklist
You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.

[x] I have read the contributing guidelines and the code of conduct.
[x] I have followed the documentation style.

Additional information
There are many guides that may need a complete revision or even a definitive deprecation. Here's a non-comprehensive list that I wrote while working on this PR:
To revise:

administration/access-services-from-outside-cluster.md
administration/delete-assets.md
administration/optimize-database-datatypes.md
administration/optimize-time-consuming-queries.md
administration/reduce-database-size.md

To deprecate:

administration/access-kafka-outside-cluster.md
security/change-vernemq-acl-configuration.md

In general though, I would give a deeper look into all the guides, and test if they still work with the latest changes.
Also, in the historian  page of the new data model there are instruction on how to access the database. Personally, I would suggest to just update the guide on how to access the database and reference it from the data model page

Added tests for most of the commands in this PR. helm upgrade commands are quite tricky to test, thus I left them out of the PR. Same goes for accessing the psql shell in the TimescaleDB pod. Everything else is tested


[BUG] Int8 finetuning throwing a type error

🐛 Bug
Int8 finetuning throwing a type error

I'm trying to finetune EleutherAI/pythia-2.8b-deduped model with oasst dataset on a machine with 8 V100 GPUs.
Only the following parameters are changed with others being default:

Backbone DType: int8
Batch Size: 3
Epochs: 3

The error I see is:
RuntimeError: expected scalar type Half but found Float
Stack trace:
2023-06-26 10:11:09,750 - INFO: Added key: store_based_barrier_key:1 to store for rank: 3
2023-06-26 10:11:09,757 - INFO: Added key: store_based_barrier_key:1 to store for rank: 0
2023-06-26 10:11:09,788 - INFO: Added key: store_based_barrier_key:1 to store for rank: 5
2023-06-26 10:11:09,831 - INFO: Added key: store_based_barrier_key:1 to store for rank: 1
2023-06-26 10:11:09,835 - INFO: Added key: store_based_barrier_key:1 to store for rank: 7
2023-06-26 10:11:09,837 - INFO: Added key: store_based_barrier_key:1 to store for rank: 6
2023-06-26 10:11:09,861 - INFO: Added key: store_based_barrier_key:1 to store for rank: 2
2023-06-26 10:11:09,904 - INFO: Added key: store_based_barrier_key:1 to store for rank: 4
2023-06-26 10:11:09,904 - INFO: Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-06-26 10:11:09,905 - INFO: Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-06-26 10:11:09,907 - INFO: Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-06-26 10:11:09,909 - INFO: Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-06-26 10:11:09,911 - INFO: Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-06-26 10:11:09,911 - INFO: Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-06-26 10:11:09,912 - INFO: Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-06-26 10:11:09,913 - INFO: Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-06-26 10:11:09,997 - INFO: Added key: store_based_barrier_key:2 to store for rank: 0
2023-06-26 10:11:10,007 - INFO: Added key: store_based_barrier_key:2 to store for rank: 1
2023-06-26 10:11:10,017 - INFO: Added key: store_based_barrier_key:2 to store for rank: 3
2023-06-26 10:11:10,017 - INFO: Added key: store_based_barrier_key:2 to store for rank: 5
2023-06-26 10:11:10,017 - INFO: Added key: store_based_barrier_key:2 to store for rank: 2
2023-06-26 10:11:10,018 - INFO: Added key: store_based_barrier_key:2 to store for rank: 4
2023-06-26 10:11:10,027 - INFO: Added key: store_based_barrier_key:2 to store for rank: 7
2023-06-26 10:11:10,028 - INFO: Added key: store_based_barrier_key:2 to store for rank: 6
2023-06-26 10:11:10,028 - INFO: Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-06-26 10:11:10,028 - INFO: Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-06-26 10:11:10,028 - INFO: Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-06-26 10:11:10,028 - INFO: Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-06-26 10:11:10,028 - INFO: Training in distributed mode with multiple processes, 1 GPU per process. Process 6, total: 8 local rank: 6.
2023-06-26 10:11:10,028 - INFO: Training in distributed mode with multiple processes, 1 GPU per process. Process 0, total: 8 local rank: 0.
2023-06-26 10:11:10,028 - INFO: Training in distributed mode with multiple processes, 1 GPU per process. Process 2, total: 8 local rank: 2.
2023-06-26 10:11:10,028 - INFO: Training in distributed mode with multiple processes, 1 GPU per process. Process 4, total: 8 local rank: 4.
2023-06-26 10:11:10,037 - INFO: Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-06-26 10:11:10,037 - INFO: Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-06-26 10:11:10,038 - INFO: Training in distributed mode with multiple processes, 1 GPU per process. Process 3, total: 8 local rank: 3.
2023-06-26 10:11:10,038 - INFO: Training in distributed mode with multiple processes, 1 GPU per process. Process 5, total: 8 local rank: 5.
2023-06-26 10:11:10,038 - INFO: Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-06-26 10:11:10,038 - INFO: Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-06-26 10:11:10,038 - INFO: Training in distributed mode with multiple processes, 1 GPU per process. Process 1, total: 8 local rank: 1.
2023-06-26 10:11:10,038 - INFO: Training in distributed mode with multiple processes, 1 GPU per process. Process 7, total: 8 local rank: 7.
2023-06-26 10:11:13,500 - WARNING: No OpenAI API Key set. Setting metric to BLEU.
2023-06-26 10:11:13,500 - INFO: Global random seed: 689550
2023-06-26 10:11:13,500 - WARNING: No OpenAI API Key set. Setting metric to BLEU.
2023-06-26 10:11:13,500 - WARNING: No OpenAI API Key set. Setting metric to BLEU.
2023-06-26 10:11:13,500 - WARNING: No OpenAI API Key set. Setting metric to BLEU.
2023-06-26 10:11:13,500 - WARNING: No OpenAI API Key set. Setting metric to BLEU.
2023-06-26 10:11:13,500 - WARNING: No OpenAI API Key set. Setting metric to BLEU.
2023-06-26 10:11:13,500 - WARNING: No OpenAI API Key set. Setting metric to BLEU.
2023-06-26 10:11:13,501 - WARNING: No OpenAI API Key set. Setting metric to BLEU.
2023-06-26 10:11:13,501 - INFO: Preparing the data...
2023-06-26 10:11:13,501 - INFO: Setting up automatic validation split...
2023-06-26 10:11:13,598 - INFO: Preparing train and validation data
2023-06-26 10:11:13,599 - INFO: Loading train dataset...
Using pad_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using sep_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using sep_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using sep_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using sep_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using sep_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using sep_token, but it is not set yet.
2023-06-26 10:11:13,733 - INFO: Stop token ids: [tensor([   29,    93, 31984, 49651]), tensor([   29,    93, 43274, 49651])]
Using pad_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using sep_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using sep_token, but it is not set yet.
2023-06-26 10:11:13,786 - INFO: Sample prompt: <|prompt|>As an AI what are your personal feelings on Sarah Connor?<|endoftext|><|answer|>
2023-06-26 10:11:13,787 - INFO: Loading validation dataset...
Using pad_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using sep_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using sep_token, but it is not set yet.
Using sep_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using sep_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using sep_token, but it is not set yet.
2023-06-26 10:11:13,874 - INFO: Using int8 for backbone
2023-06-26 10:11:13,876 - INFO: Using int8 for backbone
2023-06-26 10:11:13,877 - INFO: Using int8 for backbone
Using pad_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using sep_token, but it is not set yet.
2023-06-26 10:11:13,879 - INFO: Stop token ids: [tensor([   29,    93, 31984, 49651]), tensor([   29,    93, 43274, 49651])]
2023-06-26 10:11:13,882 - INFO: Sample prompt: <|prompt|>What types of tests do we have in software development?<|endoftext|><|answer|>
2023-06-26 10:11:13,882 - INFO: Number of observations in train dataset: 8191
2023-06-26 10:11:13,883 - INFO: Number of observations in validation dataset: 83
Using pad_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using sep_token, but it is not set yet.
2023-06-26 10:11:13,889 - INFO: Using int8 for backbone
Using pad_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using sep_token, but it is not set yet.
2023-06-26 10:11:13,895 - INFO: Using int8 for backbone
2023-06-26 10:11:13,900 - INFO: Using int8 for backbone
2023-06-26 10:11:13,904 - INFO: Using int8 for backbone
2023-06-26 10:11:13,987 - INFO: Using int8 for backbone
trainable params: 1,310,720 || all params: 2,776,519,680 || trainable%: 0.04720730090413045
trainable params: 1,310,720 || all params: 2,776,519,680 || trainable%: 0.04720730090413045
trainable params: 1,310,720 || all params: 2,776,519,680 || trainable%: 0.04720730090413045
trainable params: 1,310,720 || all params: 2,776,519,680 || trainable%: 0.04720730090413045
trainable params: 1,310,720 || all params: 2,776,519,680 || trainable%: 0.04720730090413045
trainable params: 1,310,720 || all params: 2,776,519,680 || trainable%: 0.04720730090413045
trainable params: 1,310,720 || all params: 2,776,519,680 || trainable%: 0.04720730090413045
trainable params: 1,310,720 || all params: 2,776,519,680 || trainable%: 0.04720730090413045
2023-06-26 10:11:24,967 - INFO: Training Epoch: 1 / 3
2023-06-26 10:11:24,968 - INFO: train loss:   0%|          | 0/341 [00:00<?, ?it/s]
Using pad_token, but it is not set yet.
Using cls_token, but it is not set yet.
Using sep_token, but it is not set yet.
2023-06-26 10:11:25,750 - INFO: Stop token ids: [tensor([   29,    93, 31984, 49651]), tensor([   29,    93, 43274, 49651])]
/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:321: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:321: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:321: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:321: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:321: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:321: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:321: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py:321: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization
warnings.warn(f"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization")
/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:230: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
attn_scores = torch.where(causal_mask, attn_scores, mask_value)
/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:230: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
attn_scores = torch.where(causal_mask, attn_scores, mask_value)
/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:230: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
attn_scores = torch.where(causal_mask, attn_scores, mask_value)
/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:230: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
attn_scores = torch.where(causal_mask, attn_scores, mask_value)
/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:230: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
attn_scores = torch.where(causal_mask, attn_scores, mask_value)
/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:230: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
attn_scores = torch.where(causal_mask, attn_scores, mask_value)
/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:230: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
attn_scores = torch.where(causal_mask, attn_scores, mask_value)
/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py:230: UserWarning: where received a uint8 condition tensor. This behavior is deprecated and will be removed in a future version of PyTorch. Use a boolean condition instead. (Triggered internally at ../aten/src/ATen/native/TensorCompare.cpp:493.)
attn_scores = torch.where(causal_mask, attn_scores, mask_value)
2023-06-26 10:11:26,183 - ERROR: Exception occurred during H2O LLM Studio run:
Traceback (most recent call last):
File "/home/ubuntu/h2o-llmstudio/train_wave.py", line 106, in
run(cfg=cfg)
File "/home/ubuntu/h2o-llmstudio/train.py", line 672, in run
val_loss, val_metric = run_train(
File "/home/ubuntu/h2o-llmstudio/train.py", line 365, in run_train
scaler.scale(loss).backward()  # type: ignore
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward
torch.autograd.backward(
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/autograd/init.py", line 200, in backward
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/autograd/function.py", line 274, in apply
return user_fn(self, *args)
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py", line 479, in backward
grad_A = torch.matmul(grad_output, CB).view(ctx.grad_shape).to(ctx.dtype_A)
RuntimeError: expected scalar type Half but found Float
2023-06-26 10:11:26,190 - ERROR: Exception occurred during H2O LLM Studio run:
Traceback (most recent call last):
File "/home/ubuntu/h2o-llmstudio/train_wave.py", line 106, in
run(cfg=cfg)
File "/home/ubuntu/h2o-llmstudio/train.py", line 672, in run
val_loss, val_metric = run_train(
File "/home/ubuntu/h2o-llmstudio/train.py", line 365, in run_train
scaler.scale(loss).backward()  # type: ignore
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward
torch.autograd.backward(
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/autograd/init.py", line 200, in backward
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/autograd/function.py", line 274, in apply
return user_fn(self, *args)
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py", line 479, in backward
grad_A = torch.matmul(grad_output, CB).view(ctx.grad_shape).to(ctx.dtype_A)
RuntimeError: expected scalar type Half but found Float
2023-06-26 10:11:26,190 - ERROR: Exception occurred during H2O LLM Studio run:
Traceback (most recent call last):
File "/home/ubuntu/h2o-llmstudio/train_wave.py", line 106, in
run(cfg=cfg)
File "/home/ubuntu/h2o-llmstudio/train.py", line 672, in run
val_loss, val_metric = run_train(
File "/home/ubuntu/h2o-llmstudio/train.py", line 365, in run_train
scaler.scale(loss).backward()  # type: ignore
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward
torch.autograd.backward(
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/autograd/init.py", line 200, in backward
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/autograd/function.py", line 274, in apply
return user_fn(self, *args)
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py", line 479, in backward
grad_A = torch.matmul(grad_output, CB).view(ctx.grad_shape).to(ctx.dtype_A)
RuntimeError: expected scalar type Half but found Float
2023-06-26 10:11:26,195 - ERROR: Exception occurred during H2O LLM Studio run:
Traceback (most recent call last):
File "/home/ubuntu/h2o-llmstudio/train_wave.py", line 106, in
run(cfg=cfg)
File "/home/ubuntu/h2o-llmstudio/train.py", line 672, in run
val_loss, val_metric = run_train(
File "/home/ubuntu/h2o-llmstudio/train.py", line 365, in run_train
scaler.scale(loss).backward()  # type: ignore
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward
torch.autograd.backward(
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/autograd/init.py", line 200, in backward
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/autograd/function.py", line 274, in apply
return user_fn(self, *args)
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py", line 479, in backward
grad_A = torch.matmul(grad_output, CB).view(ctx.grad_shape).to(ctx.dtype_A)
RuntimeError: expected scalar type Half but found Float
2023-06-26 10:11:26,204 - ERROR: Exception occurred during H2O LLM Studio run:
Traceback (most recent call last):
File "/home/ubuntu/h2o-llmstudio/train_wave.py", line 106, in
run(cfg=cfg)
File "/home/ubuntu/h2o-llmstudio/train.py", line 672, in run
val_loss, val_metric = run_train(
File "/home/ubuntu/h2o-llmstudio/train.py", line 365, in run_train
scaler.scale(loss).backward()  # type: ignore
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward
torch.autograd.backward(
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/autograd/init.py", line 200, in backward
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/autograd/function.py", line 274, in apply
return user_fn(self, *args)
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py", line 479, in backward
grad_A = torch.matmul(grad_output, CB).view(ctx.grad_shape).to(ctx.dtype_A)
RuntimeError: expected scalar type Half but found Float
2023-06-26 10:11:26,212 - ERROR: Exception occurred during H2O LLM Studio run:
Traceback (most recent call last):
File "/home/ubuntu/h2o-llmstudio/train_wave.py", line 106, in
run(cfg=cfg)
File "/home/ubuntu/h2o-llmstudio/train.py", line 672, in run
val_loss, val_metric = run_train(
File "/home/ubuntu/h2o-llmstudio/train.py", line 365, in run_train
scaler.scale(loss).backward()  # type: ignore
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward
torch.autograd.backward(
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/autograd/init.py", line 200, in backward
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/torch/autograd/function.py", line 274, in apply
return user_fn(self, *args)
File "/home/ubuntu/.local/share/virtualenvs/h2o-llmstudio-B1eqiLCk/lib/python3.10/site-packages/bitsandbytes/autograd/_functions.py", line 479, in backward
grad_A = torch.matmul(grad_output, CB).view(ctx.grad_shape).to(ctx.dtype_A)
RuntimeError: expected scalar type Half but found Float
Any assistance on this is appreciated! Thank you.

Thanks for opening the issue @binga, I'll have a look. Could you maybe share cfg.yaml file of the failed experiment?
I tried int8 on latest main (while keeping default params for eveything else), training starts without errors.

Sure, here you go.
architecture:
backbone_dtype: int8
force_embedding_gradients: false
gradient_checkpointing: false
intermediate_dropout: 0.0
pretrained: true
pretrained_weights: ''
augmentation:
random_parent_probability: 0.0
skip_parent_probability: 0.0
token_mask_probability: 0.0
dataset:
add_eos_token_to_answer: true
add_eos_token_to_prompt: true
answer_column: output
chatbot_author: H2O.ai
chatbot_name: h2oGPT
data_sample: 1.0
data_sample_choice:
- Train
- Validation
limit_chained_samples: false
mask_prompt_labels: true
parent_id_column: None
personalize: false
prompt_column:
- instruction
text_answer_separator: <|answer|>
text_prompt_start: <|prompt|>
train_dataframe: data/user/oasst/train_full.pq
validation_dataframe: None
validation_size: 0.01
validation_strategy: automatic
environment:
compile_model: false
find_unused_parameters: false
gpus:
- '0'
- '1'
- '2'
- '3'
- '4'
- '5'
- '6'
- '7'
huggingface_branch: main
mixed_precision: true
number_of_workers: 8
seed: -1
trust_remote_code: true
use_fsdp: false
experiment_name: green-kudu
llm_backbone: EleutherAI/pythia-2.8b-deduped
logging:
logger: None
neptune_project: ''
number_of_texts: 10
output_directory: output/user/green-kudu/
prediction:
batch_size_inference: 0
do_sample: false
max_length_inference: 256
metric: BLEU
min_length_inference: 2
num_beams: 1
num_history: 2
repetition_penalty: 1.2
stop_tokens: ''
temperature: 0.3
top_k: 0
top_p: 1.0
tokenizer:
add_prefix_space: false
add_prompt_answer_tokens: false
max_length: 512
max_length_answer: 256
max_length_prompt: 256
padding_quantile: 1.0
use_fast: true
training:
adaptive_kl_control: true
advantages_gamma: 0.99
advantages_lambda: 0.95
batch_size: 3
differential_learning_rate: 1.0e-05
differential_learning_rate_layers: []
drop_last_batch: true
epochs: 3
evaluate_before_training: false
evaluation_epochs: 1.0
grad_accumulation: 1
gradient_clip: 0.0
initial_kl_coefficient: 0.2
kl_horizon: 10000
kl_target: 6.0
learning_rate: 0.0001
lora: true
lora_alpha: 16
lora_dropout: 0.05
lora_r: 4
lora_target_modules: ''
loss_function: TokenAveragedCrossEntropy
offload_reward_model: false
optimizer: AdamW
ppo_batch_size: 1
ppo_clip_policy: 0.2
ppo_clip_value: 0.2
ppo_epochs: 4
ppo_generate_temperature: 1.0
reward_model: OpenAssistant/reward-model-deberta-v3-large-v2
save_best_checkpoint: false
scaling_factor_value_loss: 0.1
schedule: Cosine
train_validation_data: false
use_rlhf: false
warmup_epochs: 0.0
weight_decay: 0.0

Here are the steps I followed before setting up h2o-llmstudio.

I used this AMI from AWS - Deep Learning AMI GPU PyTorch 2.0.1 (Ubuntu 20.04) 20230620
Installed Python using these steps - https://github.com/h2oai/h2o-llmstudio#system-installs-python-310
Installed VNC using these steps - https://ubuntu.com/tutorials/ubuntu-desktop-aws#1-overview
Run make setup.

Seems to be an open issue with bitsandbytes library, occurring on Tesla V100 GPUs.
I also found this stackoverflow post that has a concise code example to reproduce the error.
I will monitor the issue linked above and push a fix (probably an updated bitsandbytes version) once the issue has been resolved.

@binga could you please try setting and check if it changes something?
gradient_checkpointing = True

Understood, thanks Max.
@psinger, I tried this yesterday and I see the same failure.
On Tue, Jun 27, 2023 at 13:15 Philipp Singer @.***>
wrote:

@binga https://github.com/binga could you please try setting and check
if it changes something?
gradient_checkpointing = True
—
Reply to this email directly, view it on GitHub
https://github.com/h2oai/h2o-llmstudio/issues/185#issuecomment-1608969955,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABRIQ3RM6H5YFBJVYOWZNNLXNKFRTANCNFSM6AAAAAAZT6DFZI
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
Best,
Phani.

Thanks @binga - sorry but cannot reproduce your issue with same settings.
How are you running LLM Studio, via GUI and make wave? I am just trying to make sure that it is not a package mismatch issue you are seeing there.

Seems to be a V100 issue indeed though, also discussed here:
https://github.com/tloen/alpaca-lora/issues/485

@psinger -- I am running it via GUI. Happy to do a quick call and debug together if you're available!

@binga if V100 is the reason, as it seems to be, we will unfortunately not find a solution to this
And did you try int4 instead of int8?

Understood. Trying it now.

@psinger - int4 finetuning works on V100 (32GB) machine. Thank you.
int8 shows the same error described in #185 .
This is with gradient_checkpointing=True.

Closing this as int4 is a workaround for V100 GPUs.


Tile match parent

You can now pass match_parent to tileSize, tileWidth and tileHeight
This helps overcoming an unexpected behaviour where setting only one of the dimensions forces the other one to the default value, which is 44dp.
Closes #349

Previous PR was https://github.com/prolificinteractive/material-calendarview/pull/353, I've created this one targeting future-1.x

Is this gonna be merged any soon? It is needed indeed.

+1
@Maragues I initially created issue #132 and agree that it was incorrectly closed. I just tried out your fork and it seems to do exactly what I was hoping for when I posted my issue, so nice work!
@ekchang Is there something preventing this from being merged?

@dsn5ft I am no longer the primary maintainer of this repo, pinging @quentin41500 ;)

I'm going to keep my branch https://github.com/Maragues/material-calendarview/tree/tile-match-parent up to date with master, you can build your own .aar file and use it in your projects.
Once this PR is merged I'll move back to this repository.

👍 LGTM


Create wiki

Create wiki for:

How to install Arch
How to install Ubuntu
How to use a configuration

Consider creating wiki for gallery, keybinds, install process, and FAQ once project is larger.


Guzzle 3.0 extra path is removed from base_url when using Client and ServiceDescription

There is no way i can specify extra path in base url when using ServiceDescription
Example:
File CustomClientConfig.php
<?php return array(
    'operations' => array(
        'test' => array(
            'httpMethod' => 'GET',
            'uri'        => '/test-cmd/{var}',
            'parameters' => array(
                'var' => array('location' => 'uri'),
            )
        )
    )
);

Code:
<?php

$client = Client::factory(array(
    'base_url'  => 'http://api.test.com/some/required/path/'
));

$description = ServiceDescription::factory(__DIR__.'/CustomClientConfig.php');
$client->setDescription($description);

$cmd = $client->getCommand('test', array('var' => 456));
$cmd->prepare();
echo $cmd->getRequest()->getUrl();

Output is: http://api.test.com/test-cmd/456
Instead of http://api.test.com/some/required/path/test-cmd/456
Why there is required "/some/required/path/" missing?

I've been stuck with the same issue. But apparently the documentation (v6) is clean...

How does this apply for https, also you have an https in the first example @mtvbrianking

@zardilior it works inconsiderate of the protocol (http || https)


ci: add workflow for rustdoc

GitHub action workflow to build documentation on push event to main
branch.

Codecov Report

Merging #4 (26fafc4) into main (6deaaa0) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main       #4   +/-   ##
=======================================
  Coverage   89.66%   89.66%
=======================================
  Files          17       17
  Lines        7545     7545
=======================================
  Hits         6765     6765
  Misses        780      780

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6deaaa0...26fafc4. Read the comment docs.


Add an IPAM backend: external

External IPAM

Add an IPAMD backend: external. Which enable ipamd run as a sidecar container with cilium-agent. When allocate/release IP, the socket call chain like: kubelet(cni) -> cilium-agent -> ipamd.
cilium-agent use ipam API Client to allocate/release IP.

What's Next

Decouple ipamd with cilium-agent, which means migrate the kubernetes, crd, cluster-pool, eni, azure and etc IPAM backend to external IPAM backend, external IPAM backend(ipamd) run as a sidecar container or a separate goroutine. And change Daemon.ipam hold a ipam API Client  instance.
Simplify socket call chain to: kubelet(cni) -> ipamd, and cilium-agent -> ipamd.
If --enable-endpoint-routes disabled, we may set cilium-agent config --k8s-require-ipv4-pod-cidr to be true, if we must set the Pod CIDR route to the cilium-host device. And ipamd is responsible for setting the Node annotation io.cilium.network.ipv4-pod-cidr. And Node annotation io.cilium.network.ipv4-pod-cidr value support multi CIDR as a string combine with ',' will be better.

In particular I would be interested in a discussion of the contrast between this approach vs. CRD IPAM, which is intended to allow external components to help control IPAM.
Link: https://docs.cilium.io/en/latest/concepts/networking/ipam/crd/

given the extension of this change a design document where this can be discussed

Thanks @aanm . Should I post a google doc to discuss about it ?

In particular I would be interested in a discussion of the contrast between this approach vs. CRD IPAM, which is intended to allow external components to help control IPAM.

@joestringer Thanks for the reply. I know that CRD IPAM is used to extend the custom IPAM. However, I think it also has its limitations. For examples:

Different cloud provider must modify the CiliumNode spec to implement their function, which makes CiliumNode spec more and more complexity and obscure.
CRD IPAM is based on Node's Pod CIDRs, However, some CNI's IPAM is based on higher device topo's CIDRs. For example, macvlan's CIDR may based on TOR, Nodes on the same TOR are shared the same CIDR.

So, I think Cilium should define the IPAM API(already do) and cilium-agent use this IPAM API client to call an external IPAM implement daemon. Azure, AWS, Alibaba and etc should provide an IPAM daemon based on their self CRD or other any other implement we do not care, instead of modify  CiliumNode spec and do some hack.

given the extension of this change a design document where this can be discussed

Thanks @aanm . Should I post a google doc to discuss about it ?

@answer1991 yes, that would be perfect!

@aanm Is there any example docs? And shall I post the doc into the specified workspace or post to my personal docs?

given the extension of this change a design document where this can be discussed

Thanks @aanm . Should I post a google doc to discuss about it ?

@answer1991 yes, that would be perfect!

@aanm Is there any example docs? And shall I post the doc into the specified workspace or post to my personal docs?

@answer1991 something like https://docs.google.com/document/d/1DCDrgydIrLHID6k0sP1MMiOjRu5Q03nAjwpAsRVfkQc/edit?usp=sharing

@answer1991 something like https://docs.google.com/document/d/1DCDrgydIrLHID6k0sP1MMiOjRu5Q03nAjwpAsRVfkQc/edit?usp=sharing

I have no rights to visit this page, and I have request the permission. Please approve it, thanks @aanm

Converrted this to draft as design discussion is to be had before merging.

Just a couple of notes that might be helpful for further exploring this topic:-

Different cloud provider must modify the CiliumNode spec to implement their functions, which makes CiliumNode spec more and more complexity and obscure.

For what it's worth, there is a benefit tradeoff here: It can be easier to debug what's going on if there's a resource that contains some intended state of the system, rather than for instance relying on the cilium-agent and IPAM agent to sufficiently log its activities. Pinpointing things like "I expect that the CiliumNode should have ____ state, and it [does/does not] include that information" can point towards either cilium-agent or cilium-operator being the source of a problem.

CRD IPAM is based on Node's Pod CIDRs, However, some CNI's IPAM is based on higher device topo's CIDRs. For example, macvlan's CIDR may based on TOR, Nodes on the same TOR share the same CIDR.

I believe this is the case for AWS ENI IPAM as well, it acquires individual IPs from the cloud API and populates the pool. Do you see some restriction in the current mechanism that forces node-by-node CIDRs?

@answer1991 something like https://docs.google.com/document/d/1DCDrgydIrLHID6k0sP1MMiOjRu5Q03nAjwpAsRVfkQc/edit?usp=sharing

I have no rights to visit this page, and I have request the permission. Please approve it, thanks @aanm

I've contacted the original author. Ping me if you still haven't access in 2 days.


WIP Dark mode enhancements

Since login-forms announces support for dark mode, we should have at least minimum support for it in these modules too.
This is a WIP pull request, there's some code here that belongs in TOTP but I didn't have WiFi at the time to build it there.
See #139

Just had another thought about this - I haven't tested what it does in the CMS right, but I think we'll need to explicitly disable the Dark Mode styles there.

I think @Cheddam's work is superseding this


Kubeflow Deployment with kfctl_k8s_istio

https://www.kubeflow.org/docs/started/k8s/kfctl-k8s-istio/
Note, the following (quoted docs below)  is completely useless in getting access to Kubeflow after a deployment:
What is the default dns name inside the cluster?
What is the dns name for access outside this cluster?
How do we check if the dns name for the service has been deployed?
You need to consider your audience.  It's not K8S operators.  I don't or should not need to know istio in order to use Kubeflow - wasn't that kind of the idea behind Kubeflow?
There's not even enough information to begin to troubleshoot any particular problem that might occur.
Don't get me wrong, I'm happy this open source effort is here - But, you really gotta give us some clue as to how to proceed.

Access the Kubeflow user interface (UI)
After Kubeflow is deployed, the Kubeflow Dashboard can be accessed via istio-ingressgateway service. If loadbalancer is not available in your environment, NodePort or Port forwarding can be used to access the Kubeflow Dashboard. Refer to Ingress Gateway guide.

cc @swiftdiaries @yanniszark @Bobgy @joeliedtke @jlewi @RFMVasconcelos


Just Cause 3 invisible water and low detail terrain over high detail terrain

Low detail terrain over high detail terrain

but the weird thing is that it only happens in certain areas of the map

invisible water:

Only tessellated water is affected because the low resolution water on the horizon is visible.

Software information
Steam Version of game
Reproducible with new and old nvidia compiler.
Settings.json:
{
    "DisplayFullscreen": 1,
    "DisplayFullscreenWidth": 1920,
    "DisplayFullscreenHeight": 1080,
    "DisplayWindowedWidth": 1920,
    "DisplayWindowedHeight": 1080,
    "DisplayRefreshRate": 0,
    "DisplayVSync": 0,
    "DisplayGamma": 50,
    "GraphicsMotionBlur": 0,
    "GraphicsAniso": 16,
    "GraphicsLodFactor": 3,
    "GraphicsShadowsQuality": 3,
    "GraphicsPostEffects": 1,
    "GraphicsTextureQuality": 3,
    "GraphicsWaterDetail": 0,
    "GraphicsShadowsResolution": 1,
    "GraphicsSSAO": 1,
    "GraphicsEdgeFade": 1,
    "GraphicsGI": 0,
    "GraphicsAA": 3,
    "GraphicsSSReflection": 0,
    "GraphicsBokehDOF": 0,
    "GraphicsWaterTesselation": 0,
    "GraphicsAOITQuality": 0
}

water tessellation on -> invisible water
water tessellation off -> invisible water
System information

GPU: GeForce GTX 970
Driver: 396.24
Wine version: 3.8-staging
DXVK version: 0.51

Apitrace file(s)
apitrace: loaded into C:\Program Files (x86)\Steam\steamapps\common\Just Cause 3\JustCause3.exe 00d9:fixme:debug_buffer:RtlCreateQueryDebugBuffer (0, 0): stub 00d9:fixme:debug_buffer:RtlCreateQueryDebugBuffer (168, 0): returning 0x20f30 00d9:fixme:debug_buffer:RtlQueryProcessDebugInformation (217, 14, 0x20f30): stub 00d9:fixme:debug_buffer:RtlDestroyQueryDebugBuffer (0x20f30): stub 00d9:fixme:thread:NtSetInformationThread info class 7 not supported yet 00d9:fixme:ver:GetCurrentPackageId (0x22fe30 (nil)): stub apitrace: unloaded from C:\Program Files (x86)\Steam\steamapps\common\Just Cause 3\JustCause3.exe
It didn't even create a .trace file.
Log files

d3d11.log:
info:  D3D11CoreCreateDevice: Probing D3D_FEATURE_LEVEL_11_0 info:  D3D11CoreCreateDevice: Using feature level D3D_FEATURE_LEVEL_11_0 info:  D3D11CoreCreateDevice: Probing D3D_FEATURE_LEVEL_11_0 info:  D3D11CoreCreateDevice: Using feature level D3D_FEATURE_LEVEL_11_0 info:  D3D11CoreCreateDevice: Probing D3D_FEATURE_LEVEL_11_0 info:  D3D11CoreCreateDevice: Using feature level D3D_FEATURE_LEVEL_11_0 warn:  D3D11RasterizerState: Depth clamp not properly supported
dxgi.log:
info:  Enabled instance layers: info:  Enabled instance extensions: info:    VK_KHR_surface info:    VK_KHR_win32_surface info:  GeForce GTX 970: info:    Driver: 396.24.0 info:    Vulkan: 1.1.70 info:    Memory Heap[0]:  info:      Size: 4096 MiB info:      Flags: 0x1 info:      Memory Type[7]: Property Flags = 0x1 info:      Memory Type[8]: Property Flags = 0x1 info:    Memory Heap[1]:  info:      Size: 11994 MiB info:      Flags: 0x0 info:      Memory Type[0]: Property Flags = 0x0 info:      Memory Type[1]: Property Flags = 0x0 info:      Memory Type[2]: Property Flags = 0x0 info:      Memory Type[3]: Property Flags = 0x0 info:      Memory Type[4]: Property Flags = 0x0 info:      Memory Type[5]: Property Flags = 0x0 info:      Memory Type[6]: Property Flags = 0x0 info:      Memory Type[9]: Property Flags = 0x6 info:      Memory Type[10]: Property Flags = 0xe info:  Enabled instance layers: info:  Enabled instance extensions: info:    VK_KHR_surface info:    VK_KHR_win32_surface info:  GeForce GTX 970: info:    Driver: 396.24.0 info:    Vulkan: 1.1.70 info:    Memory Heap[0]:  info:      Size: 4096 MiB info:      Flags: 0x1 info:      Memory Type[7]: Property Flags = 0x1 info:      Memory Type[8]: Property Flags = 0x1 info:    Memory Heap[1]:  info:      Size: 11994 MiB info:      Flags: 0x0 info:      Memory Type[0]: Property Flags = 0x0 info:      Memory Type[1]: Property Flags = 0x0 info:      Memory Type[2]: Property Flags = 0x0 info:      Memory Type[3]: Property Flags = 0x0 info:      Memory Type[4]: Property Flags = 0x0 info:      Memory Type[5]: Property Flags = 0x0 info:      Memory Type[6]: Property Flags = 0x0 info:      Memory Type[9]: Property Flags = 0x6 info:      Memory Type[10]: Property Flags = 0xe warn:  Vulkan extension VK_EXT_vertex_attribute_divisor not supported warn:  Vulkan extension VK_KHR_maintenance2 not supported info:  Enabled device extensions: info:    VK_KHR_descriptor_update_template info:    VK_KHR_sampler_mirror_clamp_to_edge info:    VK_KHR_maintenance1 info:    VK_KHR_shader_draw_parameters info:    VK_KHR_swapchain warn:  DxgiDevice::SetMaximumFrameLatency: Stub warn:  DXGI: MakeWindowAssociation: Ignoring flags info:  DxgiVkPresenter: Recreating swap chain:  Format:       VK_FORMAT_B8G8R8A8_UNORM Present mode: VK_PRESENT_MODE_FIFO_KHR Buffer size:  1920x1080 info:  Enabled instance layers: info:  Enabled instance extensions: info:    VK_KHR_surface info:    VK_KHR_win32_surface info:  GeForce GTX 970: info:    Driver: 396.24.0 info:    Vulkan: 1.1.70 info:    Memory Heap[0]:  info:      Size: 4096 MiB info:      Flags: 0x1 info:      Memory Type[7]: Property Flags = 0x1 info:      Memory Type[8]: Property Flags = 0x1 info:    Memory Heap[1]:  info:      Size: 11994 MiB info:      Flags: 0x0 info:      Memory Type[0]: Property Flags = 0x0 info:      Memory Type[1]: Property Flags = 0x0 info:      Memory Type[2]: Property Flags = 0x0 info:      Memory Type[3]: Property Flags = 0x0 info:      Memory Type[4]: Property Flags = 0x0 info:      Memory Type[5]: Property Flags = 0x0 info:      Memory Type[6]: Property Flags = 0x0 info:      Memory Type[9]: Property Flags = 0x6 info:      Memory Type[10]: Property Flags = 0xe warn:  Vulkan extension VK_EXT_vertex_attribute_divisor not supported warn:  Vulkan extension VK_KHR_maintenance2 not supported info:  Enabled device extensions: info:    VK_KHR_descriptor_update_template info:    VK_KHR_sampler_mirror_clamp_to_edge info:    VK_KHR_maintenance1 info:    VK_KHR_shader_draw_parameters info:    VK_KHR_swapchain warn:  Vulkan extension VK_EXT_vertex_attribute_divisor not supported warn:  Vulkan extension VK_KHR_maintenance2 not supported info:  Enabled device extensions: info:    VK_KHR_descriptor_update_template info:    VK_KHR_sampler_mirror_clamp_to_edge info:    VK_KHR_maintenance1 info:    VK_KHR_shader_draw_parameters info:    VK_KHR_swapchain info:  DxgiVkPresenter: Recreating swap chain:  Format:       VK_FORMAT_B8G8R8A8_UNORM Present mode: VK_PRESENT_MODE_IMMEDIATE_KHR Buffer size:  1920x1080

@FuchtelJockel, probably the Nvidia driver bug. Try the latest Nvidia-beta 396.18.11+
See the DXVK minimal driver requirements: https://github.com/doitsujin/dxvk/wiki/Driver-support

@FuchtelJockel It's possible to apitrace Just Cause 3, you need to download a crack. (Did that on my Steam copy and it apitraces just fine)

@sambow23 I found a working crack but the game crashes with apitrace otherwise it works. Could you send me a link to your crack?

I hope it's obvious why I deleted the last two comments.
Please tets whether these issues still occur with the latest 396.18.11 driver (wich, is more recent than 396.24), especially since that driver contains some tessellation-related fixes. If that doesn't help, there's not much I can do without an apitrace.

In that case there's nothing I can do.

  GPU: NVIDIA GTX 750 Ti 2GB
  Driver: 396.18.11
  Wine version: wine-staging 3.8
  DXVK version: 0.52

Invisible water and swimming/sailing are fixed for me.

I can't test it because after reverting to the default driver that ships with my distro the xorg server dies if I update to 396.18.11 (maybe not ready for Xorg 1.20 ).  But with the 396.24 driver I still get the same render issues.

added trace file

I'm now on 396.24.02 and DXVK 0.53 but I still get the same issues.
@pchome could I get your setting, any workarounds used?

wow after disabling them everything works thanks

Sounds like this is fixed.


Bug - Minor - Testimonial - Testimonial with overlap is slightly less overlapped than the design specified

On the Testimonial with overlap - the media is less overlapped than as displayed in the figma design
Steps to reproduce:

Go to https://www.figma.com/file/oNwe2jHIICPMIe551EY62l/Mack-Reskin---Full-Project?type=design&node-id=2220-92911&mode=design&t=2P4DrqmyDIq0fIpl-0
Go to https://11-testimonial-block--vg-macktrucks-com-rd--netcentric.hlx.page/drafts/tdziezyk/v2/testimonial
Notice how much the media part overlaps the image in the design vs on the page

Expected result
Media segment should slightly more overlap on the image

Actual result
Media segment overlaps the image slightly less than on the design

#11


Add check for filtering the artifacts through proxy

The issue is that there are always two records with both UPLOAD and DOWNLOAD for the artifacts through the proxy.
Ref: http://pastebin.test.redhat.com/1105798
Previously, we checked the transferoperation to filter this out, but looks we have not defined it in event mode, or is it worth doing that ?  anyway let's fix the issue with this workaround to unblock tests.

let's ref another PR. Closing .


AGIC Ingress controller: Liveness probe failed

Describe the bug
After creating new chart of  application-gateway-kubernetes-ingress/ingress-azure --version 1.0.0
pods created get stuck with Readiness probe failed and "new-pod-ingress-azure" gets in in 0/1 pods running.
After running a "describe pod" of that pod we are getting:
Warning  Unhealthy  13s (x10 over 113s)  kubelet, aks-agentpool-76661633-1  Readiness probe failed: Get http://<IP_ADDRESS>:8123/health/ready: dial tcp <IP_ADDRESS>:8123: connect: connection refused
  Warning  Unhealthy  7s (x6 over 107s)    kubelet, aks-agentpool-76661633-1  Liveness probe failed: Get http://<IP_ADDRESS>:8123/health/alive: dial tcp <IP_ADDRESS>:8123: connect: connection refused

To Reproduce
We already created 5 o 6 cluster using this procedure and it´s first time we have this issue:
steps:
1- kubectl create -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml
2-Create azure identity in RG where nodes are running (MC_xxxxxx) and adding WAF as reader in RG and as contributor in azure identity component
3- kubectl create -f aadpodidentity.yml
4- helm install -f dev-chart.yml application-gateway-kubernetes-ingress/ingress-azure --version 1.0.0
5- create binding : yml values: AzureIdentity: name of identity created in step 3.
selector: [nameofstep4]-ingress-azure
Ingress Controller details
Name:           bunking-lizard-ingress-azure-7b54978988-rs8h2
Namespace:      default
Priority:       0
Node:           aks-agentpool-76661633-1/<IP_ADDRESS>
Start Time:     Mon, 18 Nov 2019 20:42:53 -0300
Labels:         aadpodidbinding=bunking-lizard-ingress-azure
                app=ingress-azure
                pod-template-hash=7b54978988
                release=bunking-lizard
Annotations:    prometheus.io/port: 8123
                prometheus.io/scrape: true
Status:         Running
IP:             <IP_ADDRESS>
IPs:            <none>
Controlled By:  ReplicaSet/bunking-lizard-ingress-azure-7b54978988
Containers:
  ingress-azure:
    Container ID:   docker://bd46f4833b77b806132fd7cd7729aca5ffbf117bbee3a2e7c011ffb109894db0
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.0.0
    Image ID:       docker-pullable://mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:c295f99ae66443c5a392fd894620fcd1fc313b9efdec96d13f166fefb29780a9
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Mon, 18 Nov 2019 20:44:58 -0300
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Mon, 18 Nov 2019 20:43:59 -0300
      Finished:     Mon, 18 Nov 2019 20:44:58 -0300
    Ready:          False
    Restart Count:  2
    Liveness:       http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:      http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      bunking-lizard-cm-ingress-azure  ConfigMap  Optional: false
    Environment:
      AZURE_CONTEXT_LOCATION:  /etc/appgw/azure.json
      AGIC_POD_NAME:           bunking-lizard-ingress-azure-7b54978988-rs8h2 (v1:metadata.name)
      AGIC_POD_NAMESPACE:      default (v1:metadata.namespace)
    Mounts:
      /etc/appgw/azure.json from azure (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from bunking-lizard-sa-ingress-azure-token-z9hvt (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  azure:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:  File
  bunking-lizard-sa-ingress-azure-token-z9hvt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  bunking-lizard-sa-ingress-azure-token-z9hvt
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                  From                               Message
  ----     ------     ----                 ----                               -------
  Normal   Scheduled  2m12s                default-scheduler                  Successfully assigned default/bunking-lizard-ingress-azure-7b54978988-rs8h2 to aks-agentpool-76661633-1
  Normal   Pulling    67s (x2 over 2m11s)  kubelet, aks-agentpool-76661633-1  Pulling image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.0.0"
  Normal   Pulled     67s (x2 over 2m9s)   kubelet, aks-agentpool-76661633-1  Successfully pulled image "mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.0.0"
  Normal   Created    67s (x2 over 2m6s)   kubelet, aks-agentpool-76661633-1  Created container ingress-azure
  Normal   Killing    67s                  kubelet, aks-agentpool-76661633-1  Container ingress-azure failed liveness probe, will be restarted
  Normal   Started    66s (x2 over 2m6s)   kubelet, aks-agentpool-76661633-1  Started container ingress-azure
  Warning  Unhealthy  13s (x10 over 113s)  kubelet, aks-agentpool-76661633-1  Readiness probe failed: Get http://<IP_ADDRESS>:8123/health/ready: dial tcp <IP_ADDRESS>:8123: connect: connection refused
  Warning  Unhealthy  7s (x6 over 107s)    kubelet, aks-agentpool-76661633-1  Liveness probe failed: Get http://<IP_ADDRESS>:8123/health/alive: dial tcp <IP_ADDRESS>:8123: connec

Output of `kubectl logs .

kubectl logs bunking-lizard-ingress-azure-7b54978988-rs8h2
ERROR: logging before flag.Parse: I1118 23:42:59.206842       1 main.go:302] Using verbosity level 5 from environment variable APPGW_VERBOSITY_LEVEL
I1118 23:42:59.256720       1 environment.go:168] KUBERNETES_WATCHNAMESPACE is not set. Watching all available namespaces.
I1118 23:42:59.256746       1 main.go:132] App Gateway Details: Subscription: mysubsxxxxxxxxxxxxxxxxxxxxxx, Resource Group: MYRESOURCEG, Name: MYAKSCLUSTER
I1118 23:42:59.256752       1 auth.go:90] Creating authorizer from Azure Managed Service Identity
* Any Azure support tickets associated with this issue.
Note. We are using kubernetes version 1.15.5 (preview in Azure) Could this be related?

Any Azure support tickets associated with this issue.
119111924000415

I think we might find something in the AAD Pod's logs. Would you be able to share those as well?

Here is for: mic-6fdb84dfbb-59nrl (created automatically after STEP 1 )
kubectl logs mic-6fdb84dfbb-59nrl
I1119 01:25:48.282781       1 main.go:69] Starting mic process. Version: 1.5.3. Build date: 2019-10-12-00:24
W1119 01:25:48.282827       1 main.go:74] --kubeconfig not passed will use InClusterConfig
I1119 01:25:48.282831       1 main.go:88] kubeconfig () cloudconfig (/etc/kubernetes/azure.json)
I1119 01:25:48.283008       1 main.go:99] Client QPS set to: 5. Burst to: 5
I1119 01:25:48.283028       1 mic.go:89] Starting to create the pod identity client. Version: 1.5.3. Build date: 2019-10-12-00:24
I1119 01:25:48.366257       1 mic.go:95] Kubernetes server version: v1.14.8
**_I1119 01:25:48.366814       1 log.go:16] Initialized health probe on port &[8080]
I1119 01:25:48.366835       1 log.go:11] Started health probe_**
I1119 01:25:48.366842       1 mic.go:144] Initiating MIC Leader election
I1119 01:25:48.366849       1 leaderelection.go:175] attempting to acquire leader lease  default/aad-pod-identity-mic...

And other 3 where created after step 1. Starting with nmi-xxxxxx
kubectl get pods
NAME                                           READY   STATUS             RESTARTS   AGE
hopping-robin-ingress-azure-6756bb8464-zq8dv   0/1     CrashLoopBackOff   172        10h
mic-6fdb84dfbb-59nrl                           1/1     Running            0          12h
mic-6fdb84dfbb-rtfxg                           1/1     Running            0          12h
nmi-d7bsq                                      1/1     Running            0          12h
nmi-tdzbq                                      1/1     Running            0          12h
nmi-zjkpw                                      1/1     Running            0          12h

kubectl logs nmi-d7bsq
time="2019-11-19T01:25:46Z" level=info msg="Starting nmi process. Version: 1.5.3. Build date: 2019-10-12-00:23. Log level: info."
time="2019-11-19T01:25:46Z" level=info msg="CRD lite informers started "
**_time="2019-11-19T01:25:46Z" level=info msg="Initialized health probe on port &[8080]"
time="2019-11-19T01:25:46Z" level=info msg="Started health probe"_**
time="2019-11-19T01:25:46Z" level=info msg="Listening on port 2579"
ERROR: logging before flag.Parse: E1119 01:32:46.237265       1 client.go:161] pod list empty
time="2019-11-19T01:32:46Z" level=warning msg="List pod error: pod list empty. Retrying, attempt number: 1"
ERROR: logging before flag.Parse: E1119 01:32:46.537448       1 client.go:161] pod list empty
time="2019-11-19T01:32:46Z" level=warning msg="List pod error: pod list empty. Retrying, attempt number: 2"
ERROR: logging before flag.Parse: E1119 01:32:46.837756       1 client.go:161] pod list empty
time="2019-11-19T01:32:46Z" level=warning msg="List pod error: pod list empty. Retrying, attempt number: 3"
ERROR: logging before flag.Parse: E1119 01:32:47.138060       1 client.go:161] pod list empty
time="2019-11-19T01:32:47Z" level=warning msg="List pod error: pod list empty. Retrying, attempt number: 4"
time="2019-11-19T01:32:52Z" level=warning msg="failed to get assigned ids for pod:default/hopping-robin-ingress-azure-6756bb8464-bx5rw in ASSIGNED state, retrying attempt: 1" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
time="2019-11-19T01:32:57Z" level=warning msg="failed to get assigned ids for pod:default/hopping-robin-ingress-azure-6756bb8464-bx5rw in ASSIGNED state, retrying attempt: 2" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
time="2019-11-19T01:33:02Z" level=warning msg="failed to get assigned ids for pod:default/hopping-robin-ingress-azure-6756bb8464-bx5rw in ASSIGNED state, retrying attempt: 3" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
time="2019-11-19T01:33:07Z" level=warning msg="failed to get assigned ids for pod:default/hopping-robin-ingress-azure-6756bb8464-bx5rw in ASSIGNED state, retrying attempt: 4" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
time="2019-11-19T01:33:12Z" level=warning msg="failed to get assigned ids for pod:default/hopping-robin-ingress-azure-6756bb8464-bx5rw in ASSIGNED state, retrying attempt: 5" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
time="2019-11-19T01:33:17Z" level=warning msg="failed to get assigned ids for pod:default/hopping-robin-ingress-azure-6756bb8464-bx5rw in ASSIGNED state, retrying attempt: 6" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
time="2019-11-19T01:33:22Z" level=warning msg="failed to get assigned ids for pod:default/hopping-robin-ingress-azure-6756bb8464-bx5rw in ASSIGNED state, retrying attempt: 7" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
time="2019-11-19T01:33:27Z" level=warning msg="failed to get assigned ids for pod:default/hopping-robin-ingress-azure-6756bb8464-bx5rw in ASSIGNED state, retrying attempt: 8" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
time="2019-11-19T01:33:32Z" level=warning msg="failed to get assigned ids for pod:default/hopping-robin-ingress-azure-6756bb8464-bx5rw in ASSIGNED state, retrying attempt: 9" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
time="2019-11-19T01:33:37Z" level=warning msg="failed to get assigned ids for pod:default/hopping-robin-ingress-azure-6756bb8464-bx5rw in ASSIGNED state, retrying attempt: 10" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
time="2019-11-19T01:33:42Z" level=warning msg="failed to get assigned ids for pod:default/hopping-robin-ingress-azure-6756bb8464-bx5rw in ASSIGNED state, retrying attempt: 11" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
time="2019-11-19T01:33:47Z" level=warning msg="failed to get assigned ids for pod:default/hopping-robin-ingress-azure-6756bb8464-bx5rw in ASSIGNED state, retrying attempt: 12" req.method=GET req.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
time="2019-11-19T01:33:52Z" level=warning msg="failed to get assigned ids for pod:default/hopping-robin-ingress-azure-6756bb8464-bx5rw in ASSIGNED state, retrying attempt

So those pods they are able to start health probe successfully, but ingres is not. What can this issue be related to?. Was recently a change made of AGIC code? We built more than 10 clusters on the past weeks and it´s the first time we are dealing with this issue.-
Thanks!

We are facing the same issue.
To Reproduce
We are installing the Azure App Gateway Ingress Controller as following:
# install azure ad pod identities
./kubernetes-d/linux-amd64/helm repo add aad-pod-identity https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts
./kubernetes-d/linux-amd64/helm repo update
./kubernetes-d/linux-amd64/helm upgrade --install aad-pod-identity aad-pod-identity/aad-pod-identity --atomic
#kubectl apply -f https://raw.githubusercontent.com/Azure/aad-pod-identity/master/deploy/infra/deployment-rbac.yaml
# install azure app gateway ingress controller
./kubernetes-d/linux-amd64/helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
./kubernetes-d/linux-amd64/helm repo update
# manipulating config.yaml
./kubernetes-d/linux-amd64/helm upgrade -f=./app-gw-ingress/helm-config.yaml --install ingress-azure application-gateway-kubernetes-ingress/ingress-azure --version=$appgwChartVersion --namespace=default --atomic

The applied helm-chart values:
# Specify aks cluster related information. THIS IS BEING DEPRECATED.
# aksClusterConfiguration:
#   apiServerAddress: AKS_API_SERVER_ADDRESS

# The values.yaml file is important to templates.
# This file contains the default values for a chart.
# These values may be overridden during helm install or helm upgrade.
replicaCount: 1

# Verbosity level of the App Gateway Ingress Controller
verbosityLevel: 3

image:
  repository: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress
  tag: 0.7.1
  pullPolicy: Always

# kubernetes:
#     # Namespace(s) AGIC watches; Leaving this blank watches all namespaces;
#     # Accepts one or many comma-separated values
#     watchNamespace:
#     # Port for AGIC's HTTP API endpoint
#     httpServicePort: 8124

################################################################################
# Specify which application gateway the ingress controller will manage
#
appgw:
  subscriptionId: SUBSCRIPTION_ID
  resourceGroup: RESOURCE_GROUP_NAME
  name: APP_GW_NAME

################################################################################
# Specify the authentication with Azure Resource Manager
#
# Two authentication methods are available:
# - Option 1: AAD-Pod-Identity (https://github.com/Azure/aad-pod-identity)
armAuth:
  type: aadPodIdentity
  identityResourceID: ARM_AUTH_RESOURCE_ID
  identityClientID: ARM_AUTH_CLIENT_ID
# - Option 2: ServicePrincipal as a kubernetes secret
# armAuth:
#   type: servicePrincipal
#
#   # Generate this value with:
#   #   az ad sp create-for-rbac --subscription <subscription-uuid> --sdk-auth | base64 -w0
#   secretJSON: <base64-encoded-JSON-blob>

################################################################################
# Specify if the cluster is RBAC enabled or not
rbac:
  enabled: true # true/false

nodeSelector: {}

Describe the bug

azure-app-gateway-ingress-controller pod stuck in crashLoopBackOff
kubectl describe of the pod shows the Liveness and Readiness probe failing in the same manner as described by the author in the first post

Current workaround
We currently worked around by manually editing the deployment of the azure-app-gateway-ingress-controller and removing the Readiness and Liveness probes there. Afterwards the azure-app-gateway-ingress-controller pod is able to spin up correctly and stays stable in the Running status.
´kubectl edit deployment azure-app-gateway-ingress-controller´ -> remove the complete Readiness and Liveness section, so that it looks somehow like this:
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "3"
  creationTimestamp: "2019-11-18T16:34:42Z"
  generation: 3
  labels:
    app: ingress-azure
    chart: ingress-azure-1.0.0
    heritage: Tiller
    release: ingress-azure
  name: ingress-azure
  namespace: default
  resourceVersion: "425992"
  selfLink: /apis/apps/v1/namespaces/default/deployments/ingress-azure
  uid: f357ad89-67ac-49f4-8f12-3041b3d10081
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: ingress-azure
      release: ingress-azure
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/port: "8123"
        prometheus.io/scrape: "true"
      creationTimestamp: null
      labels:
        aadpodidbinding: ingress-azure
        app: ingress-azure
        release: ingress-azure
    spec:
      containers:
      - env:
        - name: AZURE_CONTEXT_LOCATION
          value: /etc/appgw/azure.json
        - name: AGIC_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: AGIC_POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        envFrom:
        - configMapRef:
            name: ingress-azure
        image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:0.7.1
        imagePullPolicy: Always
        name: ingress-azure
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/appgw/azure.json
          name: azure
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: ingress-azure
      serviceAccountName: ingress-azure
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /etc/kubernetes/azure.json
          type: File
        name: azure
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2019-11-20T08:27:25Z"
    lastUpdateTime: "2019-11-20T08:27:25Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2019-11-20T08:27:21Z"
    lastUpdateTime: "2019-11-20T08:27:25Z"
    message: ReplicaSet "ingress-azure-78d4c6c4d6" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 3
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

Further Question
Is it somehow possible to archive a running azure-app-gateway-ingress-controller without the need of manually editing the deployment of it after it was deployed with the helm-chart?

Same issue here, because the Liveness probe failed

this is new issue, as we already deployed in last weeks at least 4 o 5 AKS clusters with AGIC  lattest version 1.0.0, and pods started running successfully.-

@delqn any chances to help us out here? 😃

@ahojman (I'm also @delqn) Thanks for your patience with this.  It is KubeCon weeks and a few of us are here in San Diego at the conference, so responses could be delayed.
@Yiyito11 I was able to reproduce the issue w/ Kubernetes v 1.15.x.  Thanks for offering the workaround.
It seems that the ingress pod is waiting on obtaining authorization from AAD Pod Identity and that never succeeds.  I need to dig into that to understand what and why is happening.
@ahojman would you be able to use the workaround @Yiyito11 proposed above while we are troubleshooting this?

The workaround to remove Liveness Probe and Readiness Probe from the Deployment, as suggested by @gitflo1, seems to be doing the trick.
Blocks removed from AGIC's Deployment:

Causes the running AGIC pod to be replaced by another new one, as expected.
Checking the new pod logs:
$ kubectl log deadly-ibex-ingress-azure-c97797d69-btp8f

log is DEPRECATED and will be removed in a future version. Use logs instead.
ERROR: logging before flag.Parse: I1121 16:20:29.990455       1 main.go:302] Using verbosity level 3 from environment variable APPGW_VERBOSITY_LEVEL
I1121 16:20:30.028429       1 environment.go:168] KUBERNETES_WATCHNAMESPACE is not set. Watching all available namespaces.
I1121 16:20:30.028470       1 main.go:132] App Gateway Details: Subscription: <SubID>, Resource Group: <RG-Name>, Name: <AppGw-Name>
I1121 16:20:30.028494       1 auth.go:90] Creating authorizer from Azure Managed Service Identity
I1121 16:20:30.499006       1 main.go:179] Ingress Controller will observe all namespaces.
I1121 16:20:30.531956       1 context.go:129] k8s context run started
I1121 16:20:30.532008       1 context.go:168] Waiting for initial cache sync
I1121 16:20:30.632318       1 context.go:176] Initial cache sync done
I1121 16:20:30.632344       1 context.go:177] k8s context run finished
I1121 16:20:30.632487       1 worker.go:35] Worker started
I1121 16:20:30.632747       1 httpserver.go:57] Starting API Server on :8123
I1121 16:20:30.791415       1 mutate_app_gateway.go:154] BEGIN AppGateway deployment
I1121 16:20:31.198186       1 mutate_app_gateway.go:182] Applied App Gateway config in 406.740784ms
I1121 16:20:31.198218       1 mutate_app_gateway.go:198] cache: Updated with latest applied config.
I1121 16:20:31.198685       1 mutate_app_gateway.go:203] END AppGateway deployment

@draychev - just deployed a couple more AGICs; they seem be working fine w/o having to apply this workaround....

Hi all, I've got two questions:

Are there any known side-effects of the recently merged PR #675 affecting (or even fixing) this bug?
As soon as #674 is merged, will there be a patch-version of the helm-chart available?

Thank you for your help!

@gitflo1 Thanks for asking.

No, not yet. I am yet to test it myself before I can be fully confident.
Yes, we will publish a release candidate after this PR is merged.

@akshaysngupta thank you for resolving this issue with the RC 1.0.1-rc1. I just tested the RC helm-chart version on our prior failing AKS clusters and the issue was resolved.

@akshaysngupta Can you please confirm if 1.0.1-rc1 is the current.
I tried upgrading to 1.16.7 but the chart wont get deployed.
helm install ingress-azure -f agic-helm-config.yaml application-gateway-kubernetes-ingress/ingress-azure
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "Deployment" in version "apps/v1beta2"

Use helm repo update and try installing 1.2.0-rc1

cool, that worked. Any idea why this version isnt showing up on search " helm search repo application-gateway-kubernetes-ingress -l"
Can you also please update the document to use this version?
Thanks
Gopinath T

Akshay
I am getting the below error on NMI pod
E0426 11:01:05.884078       1 client.go:185] pod list empty
W0426 11:01:05.884106       1 client.go:208] List pod error: pod list empty. Retrying, attempt number: 1
E0426 11:01:06.184319       1 client.go:185] pod list empty
W0426 11:01:06.184354       1 client.go:208] List pod error: pod list empty. Retrying, attempt number: 2
E0426 11:01:06.484518       1 client.go:185] pod list empty
W0426 11:01:06.484556       1 client.go:208] List pod error: pod list empty. Retrying, attempt number: 3
E0426 11:01:06.784753       1 client.go:185] pod list empty
W0426 11:01:06.784798       1 client.go:208] List pod error: pod list empty. Retrying, attempt number: 4
E0426 11:02:27.088122       1 server.go:366] failed to get matching identities for pod: default/ingress-azure-6d874bfd5d-6spj2, error: getting assigned identities for pod default/ingress-azure-6d874bfd5d-6spj2 in CREATED state failed after 16 attempts, retry duration [5]s. Error:
I0426 11:02:27.088211       1 server.go:190] Status (403) took<PHONE_NUMBER>2 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0426 11:03:57.092436       1 server.go:366] failed to get matching identities for pod: default/ingress-azure-6d874bfd5d-6spj2, error: getting assigned identities for pod default/ingress-azure-6d874bfd5d-6spj2 in CREATED state failed after 16 attempts, retry duration [5]s. Error:
I0426 11:03:57.092554       1 server.go:190] Status (403) took<PHONE_NUMBER>6 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0426 11:04:20.970623       1 server.go:366] failed to get matching identities for pod: default/ingress-azure-6d874bfd5d-6spj2, error: context canceled
I0426 11:04:20.970648       1 server.go:190] Status (404) took<PHONE_NUMBER>1 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0426 11:04:23.619286       1 client.go:185] pod list empty
W0426 11:04:23.619310       1 client.go:208] List pod error: pod list empty. Retrying, attempt number: 1
E0426 11:04:23.919510       1 client.go:185] pod list empty
W0426 11:04:23.919672       1 client.go:208] List pod error: pod list empty. Retrying, attempt number: 2
E0426 11:05:44.223140       1 server.go:366] failed to get matching identities for pod: default/ingress-azure-6d874bfd5d-5pmjm, error: getting assigned identities for pod default/ingress-azure-6d874bfd5d-5pmjm in CREATED state failed after 16 attempts, retry duration [5]s. Error:
I0426 11:05:44.223270       1 server.go:190] Status (403) took<PHONE_NUMBER>1 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
I followed the steps inhttps://github.com/Azure/aad-pod-identity#deploy-the-azure-aad-identity-infra and ensure that kubectl logs demo in step 6 is returning results without error
time="2020-04-26T11:07:21Z" level=info msg="starting demo pod demo/demo <IP_ADDRESS>"
time="2020-04-26T11:08:03Z" level=info msg="succesfull doARMOperations vm count 0" podip=<IP_ADDRESS> podname=demo podnamespace=demo
time="2020-04-26T11:08:03Z" level=info msg="succesfully acquired a token using the MSI, msiEndpoint(http://<IP_ADDRESS>/metadata/identity/oauth2/token)" podip=<IP_ADDRESS> podname=demo podnamespace=demo
time="2020-04-26T11:08:03Z" level=info msg="succesfully acquired a token, userAssignedID MSI, msiEndpoint(http://<IP_ADDRESS>/metadata/identity/oauth2/token) clientID(5ec5d303-f605-46dc-880c-d1c01307bdfd)" podip=<IP_ADDRESS> podname=demo podnamespace=demo
time="2020-04-26T11:08:03Z" level=info msg="succesfully made GET on instance metadata, {"compute":{"location":"australiaeast","name":"aks-nodepool1-37221196-vmss_0","offer":"aks","osType":"Linux","placementGroupId":"7efc1cf2-96e5-4a97-a44c-3eb80031ffb8","platformFaultDomain":"0","platformUpdateDomain":"0","publisher":"microsoft-aks","resourceGroupName":"MC_az-aks-dev_az-aks-dev-cl_australiaeast","sku":"aks-ubuntu-1604-202004","subscriptionId":"60961e5e-39f3-46f1-9f11-16309a937340","tags":"aksEngineVersion:aks-release-v0.47.0-1-aks;creationSource:aks-aks-nodepool1-37221196-vmss;orchestrator:Kubernetes:1.15.10;poolName:nodepool1;resourceNameSuffix:37221196","version":"2020.04.16","vmId":"f317eeaf-926f-4438-b0f1-7dddb4a0f505","vmSize":"Standard_DS2_v2"},"network":{"interface":[{"ipv4":{"ipAddress":[{"privateIpAddress":"<IP_ADDRESS>","publicIpAddress":""}],"subnet":[{"address":"<IP_ADDRESS>","prefix":"16"}]},"ipv6":{"ipAddress":[]},"macAddress":"000D3ACB42A6"}]}}" podip=<IP_ADDRESS> podname=demo podnamespace=demo
time="2020-04-26T11:08:23Z" level=info msg="succesfull doARMOperations vm count 0" podip=<IP_ADDRESS> podname=demo podnamespace=demo
time="2020-04-26T11:08:23Z" level=info msg="succesfully acquired a token using the MSI, msiEndpoint(http://<IP_ADDRESS>/metadata/identity/oauth2/token)" podip=<IP_ADDRESS> podname=demo podnamespace=demo
time="2020-04-26T11:08:23Z" level=info msg="succesfully acquired a token, userAssignedID MSI, msiEndpoint(http://<IP_ADDRESS>/metadata/identity/oauth2/token) clientID(5ec5d303-f605-46dc-880c-d1c01307bdfd)" podip=<IP_ADDRESS> podname=demo podnamespace=demo
time="2020-04-26T11:08:23Z" level=info msg="succesfully made GET on instance metadata, {"compute":{"location":"australiaeast","name":"aks-nodepool1-37221196-vmss_0","offer":"aks","osType":"Linux","placementGroupId":"7efc1cf2-96e5-4a97-a44c-3eb80031ffb8","platformFaultDomain":"0","platformUpdateDomain":"0","publisher":"microsoft-aks","resourceGroupName":"MC_az-aks-dev_az-aks-dev-cl_australiaeast","sku":"aks-ubuntu-1604-202004","subscriptionId":"60961e5e-39f3-46f1-9f11-16309a937340","tags":"aksEngineVersion:aks-release-v0.47.0-1-aks;creationSource:aks-aks-nodepool1-37221196-vmss;orchestrator:Kubernetes:1.15.10;poolName:nodepool1;resourceNameSuffix:37221196","version":"2020.04.16","vmId":"f317eeaf-926f-4438-b0f1-7dddb4a0f505","vmSize":"Standard_DS2_v2"},"network":{"interface":[{"ipv4":{"ipAddress":[{"privateIpAddress":"<IP_ADDRESS>","publicIpAddress":""}],"subnet":[{"address":"<IP_ADDRESS>","prefix":"16"}]},"ipv6":{"ipAddress":[]},"macAddress":"000D3ACB42A6"}]}}" podip=<IP_ADDRESS> podname=demo podnamespace=demo
my helm-config is as below and the MI and the SPN that the cluster is configured with has been granted contributor access and Managed Identity provider role at subscription level.
verbosityLevel: 3
appgw:
subscriptionId: 60961e5e-39f3-46f1-9f11-16309a937340
resourceGroup: az-aks-dev
name: az-aks-dev-agic
usePrivateIP: false
shared: false
- Option 1: AAD-Pod-Identity (https://github.com/Azure/aad-pod-identity)
armAuth:
type: aadPodIdentity
identityResourceID: /subscriptions/60961e5e-39f3-46f1-9f11-16309a937340/resourcegroups/az-aks-dev/providers/Microsoft.ManagedIdentity/userAssignedIdentities/az-aks-dev-agic-mi
identityClientID:  5ec5d303-f605-46dc-880c-d1c01307bdfd
rbac:
enabled: true # true/false
Specify aks cluster related information. THIS IS BEING DEPRECATED.
aksClusterConfiguration:
apiServerAddress: az-aks-dev-az-aks-dev-60961e-fe73df2e.hcp.australiaeast.azmk8s.io
below is helm version used.
helm install ingress-azure -f helm-config.yaml application-gateway-kubernetes-ingress/ingress-azure --version 1.2.0-rc1

if I point demo pod to use the aadidentity and its binding created during the release ie ingress-azure, the demo pod also fails. 0.18 is ingress pod and 0.19 is demo pod
E0426 11:33:49.942556       1 server.go:366] failed to get matching identities for pod: default/ingress-azure-db7567987-6f7h9, error: context canceled
I0426 11:33:49.942583       1 server.go:190] Status (404) took<PHONE_NUMBER>4 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0426 11:34:44.897257       1 server.go:366] failed to get matching identities for pod: default/demo, error: getting assigned identities for pod default/demo in CREATED state failed after 16 attempts, retry duration [5]s. Error:
I0426 11:34:44.897317       1 server.go:190] Status (403) took<PHONE_NUMBER>0 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0426 11:36:04.901363       1 server.go:366] failed to get matching identities for pod: default/demo, error: getting assigned identities for pod default/demo in CREATED state failed after 16 attempts, retry duration [5]s. Error:
I0426 11:36:04.901398       1 server.go:190] Status (403) took<PHONE_NUMBER>7 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0426 11:37:24.905651       1 server.go:366] failed to get matching identities for pod: default/demo, error: getting assigned identities for pod default/demo in CREATED state failed after 16 attempts, retry duration [5]s. Error:
I0426 11:37:24.905685       1 server.go:190] Status (403) took<PHONE_NUMBER>7 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0426 11:38:44.910002       1 server.go:366] failed to get matching identities for pod: default/demo, error: getting assigned identities for pod default/demo in CREATED state failed after 16 attempts, retry duration [5]s. Error:
I0426 11:38:44.910057       1 server.go:190] Status (403) took<PHONE_NUMBER>9 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0426 11:39:49.946541       1 server.go:366] failed to get matching identities for pod: default/ingress-azure-db7567987-6f7h9, error: context canceled
I0426 11:39:49.946591       1 server.go:190] Status (404) took<PHONE_NUMBER>2 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>
E0426 11:40:04.914209       1 server.go:366] failed to get matching identities for pod: default/demo, error: getting assigned identities for pod default/demo in CREATED state failed after 16 attempts, retry duration [5]s. Error:
I0426 11:40:04.914295       1 server.go:190] Status (403) took<PHONE_NUMBER>2 ns for req.method=GET reg.path=/metadata/identity/oauth2/token req.remote=<IP_ADDRESS>

Why is it that helm repo update doesn't show any new packages like 1.2.0-rc1?

Check your AzureIdentity and AzureIdentityBinding. There has been a change on lattest version of aad-pod-identity. check it out here.

@vaishali-prophecy @Yiyito11 We have released 1.2.0-rc2 to address the issue related to AAD Pod Identity breaking change.
@vaishali-prophecy
To list all available versions using helm 3:
helm repo add \
    application-gateway-kubernetes-ingress \
    https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
helm repo update

# this will list release candidates as well
helm search repo application-gateway-kubernetes-ingress --devel -l


Show attribute information text in tooltip on consent screen

The attribute release policy (ARP) in the metadata of a service
provider contains a 'motivation' field explaining why the service
provider requires this attribute.
This commit addes a tooltip on the consent screen presenting the
motivation to the user.
Manage version 2.0.18 or higher is required for the motivations to
show up.
See: https://www.pivotaltracker.com/story/show/158146257

I've updated the pivotal story with browser test results.

Comments mentioned above were fixed five days ago. I've now improved browser support for IE10 and UC.