Update the attribute mappings

In Step 7, the admin is asked to map the email address, first name, and last name. However, when creating a user in Azure AD, the first name and last name are optional attributes. Many of our customers don't fill these values out. However, displayname is a required attribute. Therefore, we would like to update this section to ask the admin to add the user.displayname attribute in step 7b. In Tableau Cloud, we'd like for the screenshot to ask the admin to map the display name to the "full name" option (aka displayname):

Note: If the admin has configured users that do not have a first name and last name then during SCIM provisioning, then the display name will not contain the user's name. Therefore, we need to map a required attribute (aka displayname/fullname) from Azure AD.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: cbfa13a7-4d2d-b203-d7fa-760833a745cb
Version Independent ID: 72ff67dc-732d-478e-1595-9d4bd6737714
Content: Tutorial: Azure AD SSO integration with Tableau Cloud - Microsoft Entra
Content Source: articles/active-directory/saas-apps/tableauonline-tutorial.md
Service: active-directory
Sub-service: saas-app-tutorial
GitHub Login: @jeevansd
Microsoft Alias: jeedes

@sananoorani
Thanks for your feedback! We will investigate and update as appropriate.

@sananoorani, we're updating the article as mentioned, the changes will reflect within 2-3 working days.
#please-close

Hello - This issue was closed, but I reviewed the documentation and it is still not fixed. Can someone please take a look? Customers are still having issues with this.@MarileeTurscak-MSFT @v-hgampala @jeevansd

Sorry for that, you will get the updates into the document with in 48 hours.


Search highlighting bug

Doesn't work for the following case:
Author name: J.K. Rowling
Search queries:

"row" --> Fail
"J.K." --> Success
"J.K. Rowling" --> Success
". Row" --> Fail

I think it has to do with the "." in the author field.

Live examples as of 18-03-09
Search queries that should be highlighted but arent':

Maid's (Handmaid's tale)
? (Var det bra så?)
så? (Var det bra så?) [ only highlights the "s"]

Related code can be found here.


Syncing print Github token.

The Github token should not be displayed in console.
CODE SETTINGS SYNC UPLOAD SUMMARY
Version: 3.4.0
--------------------
GitHub Token: [!!!!!!!]
GitHub Gist:<PHONE_NUMBER>
GitHub Gist Type: Secret

Restarting Visual Studio Code may be required to apply color and file icon theme.
--------------------

...

Consider encrypting the token in setting too.

Since now Settings Sync automatically retrieves token from Github Application. I think its safe to remove from the summary.

Came here to add a vote for this -- even on a non-autoretrieved token maybe truncate it to the first 4 characters if you need it to display at all? A readwrite gist token is pretty powerful.

maybe truncate it to the first 4 characters

Nice idea, will be fixed in upcoming version.

fixed,released


Add support for usage with vanilla JS

[x] allow users to provide custom metadata provider
[x] provide TS and JS metadata providers

WDYT @stalniy?

Pull Request Test Coverage Report for Build 139

61 of 61 (100.0%)  changed or added relevant lines in 14 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build 137:
0.0%

Covered Lines:
1331

Relevant Lines:
1331

💛  - Coveralls

At the first sight looks cool! :)
I need to spend a bit more time to understand MetadataProvider API. Will give feedback tomorrow in the morning.

Sorry for the delay with feedback a bit busy... My wife has birthday on this Saturday, so can't spend enough time on Open Source :)
I'll look into this till the end of the week

I am merging this now, feel free to add comments anyway, any feed back appreciated :]

Looks good.
As far as I see currently MetadataStorsge is restricted to load metadata from files/directories (its discover) do something like this.
What would be awesome for my project is an ability to implement custom logic for discovery process. I want to load metadata from database (this would allow to implement metadata part of EAV design on top of existing ORM).
To do so discover needs to be an async function which delegates everything to MetadataProvider.
WDYT?

Load metadata from another ORM? :] Sounds strange, but whatever :]
So what you are proposing, if I understand, is to be able to use vanilla JS entities and make them behave based on schema stored in database? Why do you want this? What about having common base entity instead?

As far as I see currently MetadataStorsge is restricted to load metadata from files/directories (its discover).

MetadataStorage is not an extension point, it is the wrapper around the whole metadata thing. It reads entities (with #18) or entitiesDirs options to find existing entities (which are always files).
The extension point is MetadataProvider with basically only one method:
abstract discoverEntity(meta: EntityMetadata, name: string): void

name is the name of discovered entity, and meta is the place where to store the metadata. So it is not really like this, it is not restricted to files/directories. So just by changing the signature to async method, you could provide something like this:
import { Cascade, EntityMetadata, EntityProperty, MetadataProvider, ReferenceType } from 'mikro-orm';

export class DatabaseMetadataProvider extends MetadataProvider {

  async discoverEntity(meta: EntityMetadata, name: string): void {
    const db = await this.getMetadataDatabase();
    const properties = await db.find({ entity: name });
    Object.entries(properties).forEach(prop => {
      if (typeof prop === 'string') {
        schema.properties[name] = { type: prop };
      }
    });

    Utils.merge(meta.properties, properties.map(prop => this.initProperty(prop)));
  }

  private async getMetadataDatabase(): MongoClient { ... }
  private async initProperty(data: any): EntityProperty { ... }

}

I should probably also export Utils helpers from index.ts so Utils.merge can be used when extending internal parts of ORM like this.

Thinking about this now, I will probably rename the method to MetadataProvider.loadEntityMetadata() as the entity is already discovered (ORM knows about it), and the responsibility of this method is to load the metadata (schema), not to discover it.

Btw there is metadata caching in place, you would need to disable that or provide custom cache adapter that would handle cache invalidation somehow. Now it checks modified time of entity file, which wouldn't work for this case.

I did change it MetadataProvider.loadEntityMetadata() and made it async.

Cool! Thank you very much!
So, the reasons behind DatabaseMetadataProvider is to be able to make ORM work with EAV (Entity Attribute Value) in the similar way as with regular database structure.
In EAV, you need to store metadata of all entities in database (so the user can change them later).
Probably you heard that EAV is an anti-pattern. Why do I want to use it?
Because this is a very good way to design data-driven applications, EAV allows fast write and easy schema migrations.
What else will I need to customize?
UnitOfWork and EntityMapper (I think you name it EntityLoader). But I didn't have time to dive deeper into your source code :) Probably will start investigate it more in the begining of April


typeof type guard doesn't narrow the type of array elements

const l = ["hello",
           () => "world"]

// Compiles OK:
const o = l[0]
const s1 : string = typeof o === 'string' ? o : o()  // OK

// ERROR compiling:
const s2 : string = typeof l[0] === 'string' ? l[0] : l[0]()
// TS2349: Cannot invoke an expression whose type lacks a call signature.
// Type 'string | (() => string)' has no compatible call signatures.

As you see, after the typeof type guard, o type is correctly narrowed from string | (() => string) to string or to (() => string), but in the next line the l[0] type is not narrowed so a compiler error is raised.
TypeScript Version:  2.5.3

Duplicate of https://github.com/Microsoft/TypeScript/issues/10530

Automatically closing this issue for housekeeping purposes. The issue labels indicate that it is unactionable at the moment or has already been addressed.


Kubectl describe prefix match feature

In kubectl describe we actually have this prefix match feature:
kubectl describe secret default
Name:         default-token-d5dff
Namespace:    default

Which is not in kubectl get:
kubectl get secret default
Error from server (NotFound): secrets "default" not found

The logic is written in https://github.com/kubernetes/kubernetes/blob/180af4240c42906f072112e7ac7818f4b3574c5a/staging/src/k8s.io/kubectl/pkg/cmd/describe/describe.go#L240-L253
But for this feature I have couple of questions:

Should kubectl get , kubectl logs, kubect edit...behave the same?
This feature doesn't work for resource tuple like kubectl describe secrets/default
This feature only works for one resource as the logic is kind of hack.
Most strange thing is the original PR seems not designed to solve the prefix match problem.
For prefix use cases like deployment name, replicas name(nginx-deployment to match nginx-deployment-6f6cf6d4db-9dxxb), wa can use label selector to get the resources(kubectl describe po -l name=myLabel).

It seems not a bug, just I think the feature we have here is not complete and not strong enough. For example, move the prefix mathing logic to builder as other commands can use it.
PS: similiar issue found:https://github.com/kubernetes/kubernetes/issues/10674

@zhouya0  it seems in the original PR was focused in the describe command not for global commands.
Not sure about others in the development group but I would just deprecate such generic match for describe. As you mentioned  it can be achieved by labels or even with kubectl autocomplete.

Yes, I'd be fairly opposed to seing kubectl get with that behavior. describe is probably fine because it's meant for humans rather than machines, but we can't have kubectl get return arbitrary objects in scripts, it has to do exactly what it is asked. Just my two cents.


Why do push constants get their name differently from other blocks?

Push constants use get_name(var.self) which is inconsistent with other buffer blocks. Why does it not use get_remapped_declared_block_name(var.self, ssbo_instance_name) to be consistent?
Thanks
Luke

This is because how GL drivers interact with the uniform. When you query the uniform from API you query based on instance name, e.g.
layout(push_constant) uniform PushConstants
{
    float a, b;
} push;

becomes
uniform PushConstants push;

and API-side you do glGetUniformLocation("push.a");.
With UBOs you use the block name, so that was the logical thing to return.


Experimental

Hi @Gopal518,
I'm a bit confused, what is this PR? Perhaps you've created it by accident?
If you'd like to propose to add/improve something, please remove all the build and local files from the PR, so it's clearer to see what changes you propose.
Thanks,
Patrik

I'm going to close this, there doesn't seem to be any actual code changes in this PR.


Flyte node timeout is an usererror

TL;DR
Node timeout error should be tagged as an user error
Type

[x] Bug Fix
[ ] Feature
[ ] Plugin

Are all requirements met?

[x] Code completed
[ ] Smoke tested
[ ] Unit tests added
[ ] Code documentation added
[ ] Any pending items have an associated Issue

Tracking Issue
https://github.com/lyft/flyte/issues/277

Codecov Report

Merging #114 into master will increase coverage by 1.07%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master     #114      +/-   ##
==========================================
+ Coverage   50.01%   51.08%   +1.07%
==========================================
  Files         128      129       +1
  Lines        7966     8218     +252
==========================================
+ Hits         3984     4198     +214
- Misses       3594     3616      +22
- Partials      388      404      +16

Impacted Files
Coverage Δ

pkg/controller/nodes/executor.go
59.95% <100.00%> (+0.58%)
:arrow_up:

...kg/controller/nodes/subworkflow/launchplan/noop.go
83.33% <0.00%> (-1.29%)
:arrow_down:

pkg/controller/controller.go
0.00% <0.00%> (ø)

pkg/compiler/admin.go
100.00% <0.00%> (ø)

pkg/controller/nodes/task/transformer.go
97.36% <0.00%> (+0.70%)
:arrow_up:

pkg/controller/handler.go
91.42% <0.00%> (+1.89%)
:arrow_up:

...g/controller/nodes/subworkflow/launchplan/admin.go
79.83% <0.00%> (+3.49%)
:arrow_up:

pkg/controller/nodes/dynamic/handler.go
68.56% <0.00%> (+4.77%)
:arrow_up:

pkg/controller/nodes/task/backoff/handler.go
79.46% <0.00%> (+5.55%)
:arrow_up:

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fd3571d...8efff64. Read the comment docs.

How did this improve coverage?

Unit test?

@kumare3 probably because this line was already being covered in a test, and since we increased the size of the block, it looked like more code (percentage) is covered

How did this improve coverage?

the line was already covered, and has been split from 1 into 2 :)


subscriber agreement error - "WARN lKnownLetsEncryptChallengeEndpointConfig : Please review carefully and accept TOS https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf"

I'm testing on a server with an already existing certificate from letsencrypt. However, the sample program cannot be run. I am attaching the log.
``:: Spring Boot ::        (v2.3.4.RELEASE)
2022-07-30 09:25:36.481  INFO 895693 --- [           main] example.SpringBootApp                    : Starting SpringBootApp on **** with PID 895693 (/home/alex/letsencrypt-helper-example-0.1.3-SNAPSHOT.jar started by root in /home/)
2022-07-30 09:25:36.490  INFO 895693 --- [           main] example.SpringBootApp                    : No active profile set, falling back to default profiles: default
2022-07-30 09:25:39.248  INFO 895693 --- [           main] lKnownLetsEncryptChallengeEndpointConfig : KeyStore exists: /etc/letsencrypt/live/*********/keystore.p12
2022-07-30 09:25:39.571  INFO 895693 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 443 (https) 80 (http)
2022-07-30 09:25:39.659  INFO 895693 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2022-07-30 09:25:39.660  INFO 895693 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.38]
2022-07-30 09:25:39.960  INFO 895693 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2022-07-30 09:25:39.964  INFO 895693 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 3338 ms
2022-07-30 09:25:40.556  WARN 895693 --- [ificate Watcher] lKnownLetsEncryptChallengeEndpointConfig : Please review carefully and accept TOS https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
2022-07-30 09:25:40.608  WARN 895693 --- [ificate Watcher] lKnownLetsEncryptChallengeEndpointConfig : Failed updating KeyStore
java.lang.NullPointerException: null
at com.github.valb3r.letsencrypthelper.tomcat.TomcatWellKnownLetsEncryptChallengeEndpointConfig.updateCertificateAndKeystore(TomcatWellKnownLetsEncryptChallengeEndpointConfig.java:395) ~[letsencrypt-helper-tomcat-0.2.4.jar!/:na]
at com.github.valb3r.letsencrypthelper.tomcat.TomcatWellKnownLetsEncryptChallengeEndpointConfig.executeCheckCertValidityAndRotateIfNeeded(TomcatWellKnownLetsEncryptChallengeEndpointConfig.java:336) ~[letsencrypt-helper-tomcat-0.2.4.jar!/:na]
at com.github.valb3r.letsencrypthelper.tomcat.TomcatWellKnownLetsEncryptChallengeEndpointConfig.letsEncryptCheckCertValidityAndRotateIfNeeded(TomcatWellKnownLetsEncryptChallengeEndpointConfig.java:308) ~[letsencrypt-helper-tomcat-0.2.4.jar!/:na]
at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]`
The library and its functionality is what I need. I would love to resolve any issues.

@yarrou It heavily depends on the configuration of the existing KeyStore - the exception you provide indicates that the Keystore is different. This library in general is not aimed on supporting already existing KeyStores out of the box.
Probably this might be seen as enhancement


HIVE-22329 Create DriverContext

The current DriverContext class is not really a context, rather a queue, thus it should be renamed to DriverQueue. At the same time Driver has a a lot of fields, they should be in a real context class called DriverContext, which would also facilitate the extraction of independent parts of the Driver class.

@kgyrtkirk I've modified the patch according to your comments. Also modified the logic of passing the arguments to a test, now the TaskQueue and the context are passed at the initialize call, and execute has no argument, which is much cleaner. TaskQueue now doesn't exposes the context.


Confusing messages with await_only_futures, unnecessary_parenthesis and futures with ternary operator

linter:
  rules:
    - await_only_futures
    - unnecessary_parenthesis

import 'dart:async';

main() async {
  bool cond = true;
  await (cond ? Future<void>.value(true) : Future<void>.value(true));
  await cond ? Future<void>.value(true) : Future<void>.value(true);
}

I'm sure one of those await lines should be valid, but the first one warns Unnecessary parenthesis can be removed. and the second one warns Await only futures.. I think one of them may be wrong?

The one that is definitely wrong is the unnecessary parentheses warning.

awaitExpression ::= await unaryExpression

and conditionalExpression is not a unary expression.
The analyzer is correct that you can remove the parentheses, it's still syntactically valid, but it won't mean the same thing any more (it'll likely be (await cond) ? ... or even awaiting some initial sub-expression of cond). In that case you are awaiting a non-future, so that warning is correct.
Ob-nitpick: It should probably say "Unnecessary parentheses can be removed" - removing just one isn't going to work :) Or "Unnecessary parenthesization"

I cannot reproduce a report on the first await line. unnecessary_parenthesis was fixed perhaps?

I also can't repro on SDK v2.1 either. There have been changed to unnecessary_parenthesis since I opened this, so maybe it was one of those.

It was fixed by 51b8fc7


Implementation Documentation

Hi,
Is there documentation on how top implement these scripts?
Thanks

Hi,
We have just added an example of dataset and an command submition to help you to run this script.
Thanks,
Lucas Ponce


Some thumbnails are too big

Note: Building (the thumbnail seems to be taken from inside the building, looking at the floor).

Building:

The problem is that you are using GridMap. GridMap doesn't have a built-in method to calculate the AABB needed to calculate the camera position.
I'm not sure if I should implement it myself. The addon is primarily used to place "subscenes" as objects, whereas GridMap is a kind of container.

Makes sense, I was able do a work around by adding a transparent mesh to encompass the building.
Feel free to close this issue.


⚠ ETF has degraded performance

In d6c5e1d, ETF (https://markets.hankyung.com/equity-product/etf) experienced degraded performance:

HTTP code: 200
Response time: 8396 ms

Resolved: 비트코인 performance has improved in 2348716.


Video segmentation crashes with more than 100 frames

I have a video I want to try to run inference on, it's a 50s video at 60 fps, so 3000 frames in total.
I try to run inference with my RTX3090 but I can't get past the 100 frames, otherwise I automatically get a CUDA OOM message.
I was wondering if there's anything I can do to complete a full video inference on an RTX3090 or whether I'm limited to running it at a 100 frames batches.
Thanks

You can split the video into multiple clips and merge predictions afterwards. However, this is not supported in this code.


Missing "host" header in ASGI scope

Tried granian today with --interface asgi and a starlette application but I'm missing the "host" header in the asgi scope.

Headers are sent by the client, Granian is not removing nor adding anything to request headers.
Tested with the following code and can't reproduce your issue:
from starlette.applications import Starlette
from starlette.responses import PlainTextResponse
from starlette.routing import Route

def homepage(request):
    print('scope', request.scope)
    print('headers', request.headers)
    print('host', request.headers.get('host'))
    return PlainTextResponse('Hello, world!')

routes = [
    Route('/', homepage),
]

app = Starlette(debug=True, routes=routes)

Request:
 ❯ curl -v http://localhost:8000
*   Trying <IP_ADDRESS>:8000...
* Connected to localhost (<IP_ADDRESS>) port 8000 (#0)
> GET / HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/1.1 200 OK
< server: granian
< content-length: 13
< content-type: text/plain; charset=utf-8
< date: Thu, 25 Jan 2024 09:48:51 GMT
<
* Connection #0 to host localhost left intact
Hello, world!

Server logs:
 ❯ granian --interface asgi t:app
[INFO] Starting granian (main PID: 71370)
[INFO] Listening at: <IP_ADDRESS>:8000
[INFO] Spawning worker-1 with pid: 71372
[INFO] Started worker-1
[INFO] Started worker-1 runtime-1
scope {'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'extensions': {'http.response.pathsend': {}}, 'type': 'http', 'http_version': '1.1', 'server': ('<IP_ADDRESS>', 8000), 'client': ('<IP_ADDRESS>', 55309), 'scheme': 'http', 'method': 'GET', 'root_path': '', 'path': '/', 'raw_path': b'/', 'query_string': b'', 'headers': [(b'host', b'localhost:8000'), (b'user-agent', b'curl/8.1.2'), (b'accept', b'*/*')], 'state': {}, 'app': <starlette.applications.Starlette object at 0x1026237d0>, 'starlette.exception_handlers': ({<class 'starlette.exceptions.HTTPException'>: <bound method ExceptionMiddleware.http_exception of <starlette.middleware.exceptions.ExceptionMiddleware object at 0x10273b920>>, <class 'starlette.exceptions.WebSocketException'>: <bound method ExceptionMiddleware.websocket_exception of <starlette.middleware.exceptions.ExceptionMiddleware object at 0x10273b920>>}, {}), 'router': <starlette.routing.Router object at 0x10273b6b0>, 'endpoint': <function homepage at 0x10262ca40>, 'path_params': {}}
headers Headers({'host': 'localhost:8000', 'user-agent': 'curl/8.1.2', 'accept': '*/*'})
host localhost:8000

Thanks for your extra fast reply!
I did some further investigation, and your example works with http, but as soon as i'm using https, the host header disappears from the scope.
> granian --interface asgi --ssl-certificate ssl-cert.pem --ssl-keyfile ssl-key.pem t:app
...
scope {'asgi': {'version': '3.0', 'spec_version': '2.3'}, 'extensions': {'http.response.pathsend': {}}, 'type': 'http', 'http_version': '2', 'server': ('<IP_ADDRESS>', 8000), 'client': ('<IP_ADDRESS>', 21342), 'scheme': 'https', 'method': 'GET', 'root_path': '', 'path': '/', 'raw_path': b'/', 'query_string': b'', 'headers': [(b'user-agent', b'curl/7.76.1'), (b'accept', b'*/*')], 'state': {}, 'app': <starlette.applications.Starlette object at 0x7fd984583bd0>, 'router': <starlette.routing.Router object at 0x7fd984116110>, 'endpoint': <function homepage at 0x7fd984562480>, 'path_params': {}}
headers Headers({'user-agent': 'curl/7.76.1', 'accept': '*/*'})
host None

Am I doing sth wrong? With hypercorn I get the host header as expected.

@kramar11 thank you for the additional details, I'll investigate this asap.

@kramar11 after re-checking your comment, I noticed this is HTTP/2.
Host header is not allowed on HTTP/2, thus is not present in the scope's headers.
Now the actual issue here is that Granian doesn't allow access to pseudo-header :authority which would contain the relative information.
I see Hypercorn automatically translate such pseudo-header content into the Host header on HTTP/2, as per ASGI spec.
Now I don't like this, but now Granian is out of spec for ASGI, so I gonna do the change. But a the same time I need to figure out something for RSGI and WSGI which can incur in the same issue.


Neo-light theme partly breaks build monitor style

In order to get a modern (read flat) looking UI for our Jenkins installation I activated the "simple theme" plugin and the neo-light stylesheet (http://jenkins-contrib-themes.github.io/jenkins-neo-theme/dist/neo-light.css).
This partly breaks the appearance of build monitor, because there's now dark gray text on green or red background, which is hardly readable. "Back in the green", "n builds have failed", "last build" time.

Hey @luebbe and thanks for reporting this! @khipis has also reported this problem in #291 and was kind enough to fix it in #295.
A new release of Build Monitor will be available early next week, so please update your installation as soon as convenient.
Best,
Jan

@jan-molak
For this specific theme we are having other problems:
Footer breaks:

Also font size and other styling such as bold breaks.
Maybe would be a good idea to figure out how a more general way of fixing errors instead of making everything important?

Am I missing something? I was hoping to see a new release of build monitor in the past months, but the latest release is still: https://github.com/jan-molak/jenkins-build-monitor-plugin/tree/v1.11%2Bbuild.201701152243
Is everything ok? Have the releases moved to a different location?

Hey @luebbe, I was hoping to get a couple more PRs merged in before the release, but it seems to take longer than expected. I've just pushed the latest build to the Jenkins Update Centre and will push the next one whenever the PRs are ready.
Hope that helps!
J

Thanks a million! That helped a lot


metrics (ticdc): remove p99 lag distribution

What problem does this PR solve?

Issue Number: close #8971 #8871
What is changed and how it works?
Check List
Tests

Manual test (add detailed scripts or steps below)

Questions
Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?
Release note
None

/test cdc-integration-mysql-test


unable to load and save model

Unable to save and load model
disc_model = LogisticRegression()
%time disc_model.train((train_cands_person[0], F_train_cands_person[0]), train_marginals, n_epochs=50,batch_size=256, lr=0.0001)

disc_model.save(model_name="disc_model", save_dir='checkpoints', verbose=True, global_step=0)
new_model = LogisticRegression()
# loading model throws error
new_model.load(model_name="disc_model", save_dir='checkpoints', verbose=True, global_step=0)

-------------error encountered--------------------------------------
RuntimeError: Error(s) in loading state_dict for LogisticRegression:
	Unexpected key(s) in state_dict: "linear.weight".

------------------------also tried-----------------------------------------
torch.save(disc_model.state_dict(),path)
torch.save(disc_model,path)-----#not working
disc_model.load_state_dict(torch.load(path))-------#not working

Whenever we try to load the model it throws the above error.
Please demonstrate a solution or tell the steps to save and load the model. A working snippet would be fine.

As another workaround, you could also do
# Define the new model
new_model = LogisticRegression()

# Manually load settings and cardinality from a saved trained model.
checkpoint = torch.load("./best_model.pt")
new_model.settings = checkpoint["config"]
new_model.cardinality = checkpoint["cardinality"]

# Build a model using the loaded settings and cardinality.
new_model._build_model()

# Load the trained model
new_model.load(model_file="best_model.pt", save_dir="./")


Test PR for 4762

Test PR - do not merge

Nighlty Run https://github.com/liquibase/liquibase-pro-tests/actions/runs/6090817944


srcOnly: Some improvements

Description of changes
Continuation of https://github.com/NixOS/nixpkgs/pull/344699
Take 2: Hopefully I don't rebase on the wrong branch this time!

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[x] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/prs-already-reviewed/2617/1992

Bisect says 0b77f4ad4f3e5b3ed90cd95fc62d88c95910b414 srcOnly: some readability improvements broke my nixos system build as:
error: builder for '/nix/store/i5yqj8ii5rv166ns67iz0xbdn4frz7gm-glibc-2.40-36-source-unsecvars.drv' failed to produce output path for output 'bin' at '/nix/store/i5yqj8ii5rv166ns67iz0xbdn4frz7gm-glibc-2.40-36-source-unsecvars.drv.chroot/root/nix/store/irxl72hizfw8zz3wqyyj961ggrq6ivai-glibc-2.40-36-source-unsecvars-bin'
error: 1 dependencies of derivation '/nix/store/r5mk1mld0k6yghk3b0l4hs13halz3bqi-security-wrapper-chsh-x86_64-unknown-linux-musl.drv' failed to build
error: 1 dependencies of derivation '/nix/store/2pnq4zhv3hyns5agni7ycdva7fz3cmpp-unit-script-suid-sgid-wrappers-start.drv' failed to build
error: 1 dependencies of derivation '/nix/store/g9g3z2jgd7xddjyixsxg87c566wld86z-unit-suid-sgid-wrappers.service.drv' failed to build
error: 1 dependencies of derivation '/nix/store/vqs9wb7vfn5d2rqj5grcaxqjf681aq6p-system-units.drv' failed to build
error: 1 dependencies of derivation '/nix/store/irfa0dpz26al8dmckq2x8q58x4kqy6in-etc.drv' failed to build
error: 1 dependencies of derivation '/nix/store/mc5m0qhs0iip9i63vy7q1lbwrh58pb8q-nixos-system-nixos-24.11.20240929.0b77f4a.drv' failed to build

Probably because bin output is now still declared and expects some output.

Looks like outputs = [ "out" ]; was just lost. Probably because bin output is now still declared and expects some output.

Yup, clearly forgot to re-include that when rewriting. I'll submit a fix now.

Have any derivations I can include as nixos tests?

As all wrappers failed in nixos/modules/security/wrappers/default.nix I would expect any existing test that exercises it to fail. Otherwise merely wrapping glibc should be enough:
# master: ok
$ nix build --impure --expr 'with import ./. {}; srcOnly glibc'

# staging (unfixed): fail
$ nix build --impure --expr 'with import ./. {}; srcOnly glibc'
error: builder for '/nix/store/vzhwaql5m22555lp017p289ybaiz5nns-glibc-2.40-36-source.drv' failed to produce output path for output 'bin' at '/nix/store/vzhwaql5m22555lp017p289ybaiz5nns-glibc-2.40-36-source.drv.chroot/root/nix/store/jw57377nvnn124wabps40kiqamnri1b2-glibc-2.40-36-source-bin'


Replace average response times with percentiles & min/max

Web performance has high outliers making average response times misleading. Can we replace mean with percentiles? I propose p50 (median), p95, p99.  I'd also add min and max.

exported stats already contain the following percentiles:
"50%","66%","75%","80%","90%","95%","98%","99%","100%"

Hi @cgoldberg Sorry, I wasn't really clear here. I was thinking about the WebUI. The charts plot avergage response times. Would different coloured lines representing percentiles be possible? Also in the stats we have average. My suggestion would be to a least replace that with p99. WDYT?

I think mean could still be useful. Rather than replace, add on the missing desired stats. And/or perhaps make it configurable what to show, in case someone doesn't want to see the mean for example.

I think the Web UI is supposed to display an average over a moving window, not the overall average... see: https://github.com/locustio/locust/issues/667
I'm not sure if a moving average is preferable to an overall percentile?

Closing due to lack of activity. Some improvements have been made in this area though...

Is it now possible to configure percentiles and the maximum value of response time on the graph? I want to see this on a chart so that I can understand at what point the short-term delays occur. I couldn't find this in the documentation, unfortunately...


not working in windows server 2019

Describe the bug
giving me error
 Running in mode: Interactive, Simple
 (COMException): Filename: \\?\C:\Windows\system32\inetsrv\config\applicationHost.config
Error:

To Reproduce

download build v2.1.8 from website
extract it to c:\winacme
open cmd and do cd c:\winacme
wacs.exe
Press N or M

will give error
 Running in mode: Interactive, Simple
 (COMException): Filename: \\?\C:\Windows\system32\inetsrv\config\applicationHost.config
Error:

Hello @darkworks do you give admin permissions to the application in the Properties -> Compatibility tab, o execute the application as Administrator? Maybe this is the reason because cannot access to this file.

@Hardcorian i run it as Administrator , right click and run as Administrator ,

Do you have IIS installed?

no i have apache

One doesn't really exclude the other though. Does the program detect IIS? It should be listed in the output. It bases this detection on the registry: specifically the version information listed under HKLM\Software\Microsoft\InetStp. I will add some better error handling in the next release, but if you remove those registry keys you can stop the program from detecting IIS.

A fix for these conditions has been release in 2.1.9


OLAP support in Doradus

hi all, I see OLAP mention a lot in the documentation , but most of the document talk and example  about columnar db .
I wander (maybe I missed it ) if you've support  for OLAP Cube!.  I'm especially interested in the "roll-up" capabilities.
I would also like also to discuss  on offline (my mail is agonenil at gmail...)  threat  a use case that we are considering to use doradus.

OK, I'll email with you offline.


Add reason to deny user

If we have this command and it will be used it is nice to send out an automatic email of why somebody is denied.
And that the denied participant gets an automatic email of this reason. We could maybe have the options;

"out of area"
"has other interested sizes or gender preferences"
"other"

Add a goatcounter action to track that as well (see 5544a1ab9267e176aefba7c5d29184ecaf01596d )

@paloeka Do I add a list of reasons predefined reasons? what are they?

Yes, I think you could label them as predefined reasons.
Hosts will need to select one of the options and in the email sent to the denied participant, they will get to know their reason of rejection

@paloeka Would you like to create an example of the new email with each option?

Hi,
Unfortunately, your request to join The  Loop has been denied, because you live outside the area where that Loop is active. Maybe there is another Loop where you live, or you can start one yourself! Go to the website: www.clothingloop.org and start a new Loop.
Nichon, on behalf of the Clothing Loop team
Hi,
Unfortunately, your request to join The  Loop has been denied, because your size and/or gender preferences regarding clothing differ from the Loop you tried to become a part of. Maybe there is another Loop where that matches your preferences, or you can start one yourself! Go to the website: www.clothingloop.org and start a new Loop.
Nichon, on behalf of the Clothing Loop team
Hi,
Unfortunately, your request to join The  Loop has been denied for another reason. However, you can start one yourself! Go to the website: www.clothingloop.org and start a new Loop.
Nichon, on behalf of the Clothing Loop team


KG series support

https://github.com/Neural-Systems-at-UIO/WebAlign/commit/103e96999390c8949cdcfc59fb21adb5b28583da for series in collab buckets. Gets both linear and nonlinear alignment data.

https://github.com/Neural-Systems-at-UIO/WebAlign/commit/322e72784dbf8d85ea892c6bd21e2ba7c3711c34 supports the old image service too.

Validated 21.06.2023


Revive the Beast

Make MiniBeast run on latest Clojure, Overtone, Quil, and add some niceties like CLI args.

To try this out
clojure -Sdeps '{:deps {overtone/mini-beast {:git/url "https://github.com/overtone/mini-beast" :git/sha "b315addc1fc1b3125fce778a953500089c908781"}}}' -M -m minibeast.core --help


Extract For Brackets Error — Creative Cloud Sign In Error

Hello,
Brackets has the following issue in the attachment.

Sorry @cwjensen,
the Extract for Brackets service has been discontinued for Brackets since 1.6. For more information, see http://blog.brackets.io/2016/04/14/update-about-extract-for-brackets-preview/?lang=en.
The extract feature is available in Dreamweaver CC, which is now based on Brackets if that helps.
Sorry again.
(Duplicate of https://github.com/adobe/brackets/issues/13341)


Create a distribute feature

like https://docs.opentrons.com/v2/new_protocol_api.html#opentrons.protocol_api.InstrumentContext.distribute
Can lift from https://github.com/rickwierenga/pylabrobot-art-studio, where this is already implemented. Just need to make it nice and general

@BioCam is the existing transfer sufficient for your needs?

I am just reading through the liquid_handler.transfer() function. It is quite a neat, small function:
  async def transfer(
    self,
    source: Well,
    targets: List[Well],
    source_vol: Optional[float] = None,
    ratios: Optional[List[float]] = None,
    target_vols: Optional[List[float]] = None,
    aspiration_flow_rate: Optional[float] = None,
    dispense_flow_rates: Optional[Union[float, List[Optional[float]]]] = None,
    **backend_kwargs
  ):
    """Transfer liquid from one well to another.

    Examples:

      Transfer 50 uL of liquid from the first well to the second well:

      >>> lh.transfer(plate["A1"], plate["B1"], source_vol=50)

      Transfer 80 uL of liquid from the first well equally to the first column:

      >>> lh.transfer(plate["A1"], plate["A1:H1"], source_vol=80)

      Transfer 60 uL of liquid from the first well in a 1:2 ratio to 2 other wells:

      >>> lh.transfer(plate["A1"], plate["B1:C1"], source_vol=60, ratios=[2, 1])

      Transfer arbitrary volumes to the first column:

      >>> lh.transfer(plate["A1"], plate["A1:H1"], target_vols=[3, 1, 4, 1, 5, 9, 6, 2])

    Args:
      source: The source well.
      targets: The target wells.
      source_vol: The volume to transfer from the source well.
      ratios: The ratios to use when transferring liquid to the target wells. If not specified, then
        the volumes will be distributed equally.
      target_vols: The volumes to transfer to the target wells. If specified, `source_vols` and
        `ratios` must be `None`.
      aspiration_flow_rate: The flow rate to use when aspirating, in ul/s. If `None`, the backend
        default will be used.
      dispense_flow_rates: The flow rates to use when dispensing, in ul/s. If `None`, the backend
        default will be used. Either a single flow rate for all channels, or a list of flow rates,
        one for each target well.

    Raises:
      RuntimeError: If the setup has not been run. See :meth:`~LiquidHandler.setup`.
    """

    # Deprecation check for single values
    if isinstance(targets, Well):
      raise NotImplementedError("Single target is deprecated, use a list of targets.")
    if isinstance(dispense_flow_rates, numbers.Rational):
      raise NotImplementedError("Single dispense flow rate is deprecated, use a list of flow rates")

    if target_vols is not None:
      if ratios is not None:
        raise TypeError("Cannot specify ratios and target_vols at the same time")
      if source_vol is not None:
        raise TypeError("Cannot specify source_vol and target_vols at the same time")
    else:
      if source_vol is None:
        raise TypeError("Must specify either source_vol or target_vols")

      if ratios is None:
        ratios = [1] * len(targets)

      target_vols = [source_vol * r / sum(ratios) for r in ratios]

    await self.aspirate(
      resources=[source],
      vols=[sum(target_vols)],
      flow_rates=aspiration_flow_rate,
      **backend_kwargs)
    for target, vol in zip(targets, target_vols):
      await self.dispense(
        resources=[target],
        vols=[vol],
        flow_rates=dispense_flow_rates,
        use_channels=[0],
        **backend_kwargs)

But for complex functionality I think we should be inspired by Opentrons approach and build some Hamilton wisdom into it:
Opentrons cleanly distinguishes between their transfer() (simple liquid transfers from 1 well to another 1 well) and distribute() functions (smart function that distributes liquid from one source well into multiple destination wells, autonomously identifying the best way to do so, i.e. it goes back to the source well when it has reached its current tip's max volume).
If I understand the above correctly it kind of merges these two concepts into one function with the name liquid_handler.transfer().
I believe Opentrons is right to have transfer() be simple transfers of liquid from 1 source well to 1 destination well, and in analogy for multi-channel systems like the STAR would be from 1 list of source wells to 1 list of destination wells, something like this:
lh.transfer(source_plate["A1:H1"], destination_plate["A1:H1"],vols=[80, 40, 50, 100, 30, 50, 80, 75])

This by itself is already adding functionality to Hamilton machines that the hardware and firmware would have no problems with, but that I could not find in GUI-based proprietary software, no matter how much I searched.
But, this is very different from a distribute() function, we might also want to discuss whether PLR should call this feature "distribute" because some people might be more familiar with the term "aliquot":
lh.distribute(water_trough, dest_matrix = [destination_plate["A1:H1"]], vols_matrix=[[80, 40, 50, 100, 30, 50, 80, 75]], use_channels=[0])

I am proposing a couple of things simultaneously in this little line here:

Adapt the "smartness" we have seen from Opentrons, i.e. the function has to identify what tip it carries on the targeted channel, what that tip's max volume is, how to chunk the channel's distribute_list into aspiration_cycles that ensure there is never too much liquid in the tip, ensures a certain retention_volume in the tip (important for accurate aspiration_cycles repeats, and performs the dispense and aspirations autonomously.
Super-power far beyond the capability's of any system I have encountered: make full use of the multi-channel system using a destination_matrix and volume_matrix argument:
Instead of giving the function a transfer volume list a transfer volume matrix is, as the name implies, a 3-dimensional array: 0-7 rows representing each channel (in an 8-channel system, more if more channels exist), but it has no real limit in the number columns.

A simple showcase to highlight the power I imagine this to have:
We all perform normalizations, no secret here. Let's say you have a 96-well plate of protein extractions one needs to normalise.
We calculate that we need to add the following buffer to a new well, and then add x volume from the extraction plate to it:

There is no reason to transfer all these volumes of the same buffer to an empty well in separate transfers, distributing / aliquotting is way faster, particularly if when one scales to multiple plates.
But at the same time using a single tip (like withe OT-2) is painfully slow.
This is where PLR's new distribute function can come in: use all channels available on the robot one has; the function is simply given the matrix of destination resources, and a matrix of volumes, and then identifies how to chunk the matrix - column by column - into aspiration cycles to execute.
It would be vastly faster than anything currently available and totally in the realm of y-independent channel systems ... the only thing that's missing is a flexible-enough, detailed-enough, computational design-empowered software = PLR :)
Normalisation is just one application, I can see use cases like inoculations, experimental setup construction and more.
This function should also have arguments like pre_aspirate_dispense=[5,5,5,None] (i.e. the volume to dispense straight back to the source well to avoid the known issue with the first aliquot), and  retention_vols=[5,5,5,None] (i.e. the volume that should stay back in the tip to ensure the last aliquot is identical to all the previous ones).
This is a big vision for a PLR method imo but I am proposing it because I believe it to be incredibly powerful, and a way to set PLR instantly apart and showcase what programmationally-designed liquid handling can do.

To be clear, I don't believe the information regarding channel parameter settings to make such a function accurate and reliable is given with the current clinging to "liquid classes", but I'm sure we can figure out a better way using the full set of aspiration and dispensation arguments ;)

I will make this

hahaha 🤣
I thought you might like it

I will make this

I know how much fun it will be to actually build this function, so I won't take this from you @rickwierenga (plus, I've got a ton of work in my day job and other PLR dev tasks) but how about we have a meeting solely discussing how to implement this function in the next couple of day?
For example, we must absolutely polish STAR.aspirate() and STAR.dispense() before heading into this (unit standardisation in particular) and we must ensure we have access to all arguments. I see these tasks as urgent precedence constraints to the implementation of lh.distribute() / lh.aliquot().
PS. I did warn you that once PLR is more open to complex, more "user-focused" features I will let a lot of ideas loose :P

a meeting

Sure, what time?

PS. I did warn you that once PLR is more open to complex, more "user-focused" features I will let a lot of ideas loose :P

This has always been the goal: front ends exist to compose complex functions out of easy-to-implement atomic commands. (obviously while maintaining granular control.) From the paper: "Composition of the unit operations, such as the discard operation (drop tip in trash) and the transfer operations (combined aspirate and dispense), are performed at the LiquidHandler level and above". So excited we are nearing a stage where we can productively implement composite commands more than simple transfer and discard 🎉

@rickwierenga, I'm available today until 15:00 UCT+1 or on Wednesday? :)

sorry to delay, but Wednesday is a lot better for me

I know, let's do Wednesday during the day at 11:00 UCT+1?
This meeting shouldn't be longer than 45 min because we just want to create a requirements list for the distribute function to begin the initial design; there are quite a lot of design considerations tbh.

Perfect!


Content assist/intellisense box opens with scrolled to some random location.

Content assist/intellisense box opens with scrolled to some random location.
issue is observed with Typescript and javascript.

Please follow this guide to submit a proper issue report. This should include:

What are you trying to do?
What went wrong?
Steps to reproduce the problem

Screenshots would also be helpful

issue resolved after updating VSC


Allow to run background jobs with Sidekiq

I did some performance tests comparing resque with delayed_job. While resque was a little faster I don't think it would be worth supporting it since it requires multiple processes to gain parallelism.
Sidekiq has two major advantages over resque, with version 4.2 sinatra is not a dependency anymore and it offers concurrency as long as the jobs are waiting for IO (or doing and other task that is lifting the GIL)
Running Agents in threads needs a lot of manual testing and since Sidekiq requires redis I would not recommend or document it in our guides for now.

Looks like a good addition! How much speepup are you seeing with Sidekiq?

That depends on the workload. My first test was with a CommanerAgent which updates the options of another Agent. 10k runs took DelayedJob 290 seconds on Sidekiq it was 215 seconds. Noticeably faster, but not by magnitudes because ruby was working most of the time, only releasing the GIL for the SQL queries.
To simulate a slow external service I used a SleepAgent which sleeps for 0.5 seconds when checked. For 1k runs Sidekiq took 22 (1000 * 0.5 / 25 + overhead) seconds, with one DelayedJob worker 518 (1000 * 0.5 + overhead) seconds. As long as ruby is waiting for IO (or sleeping in this case) a different thread can execute, which speeds up the processing.

Any updates on this? How can I help to get this merged? We just need to fix the conflicts or is there any other issues regarding Sidekiq support?

@agaelebe My main concern is that we do not know if all the gems we are using are thread safe. It ran fine on my instance when I tested it, but I am not using that many Agents.
It would probably also make sense to include https://github.com/huginn/huginn/pull/1892 because AFAIK sidekiq does not time out jobs per default.

So, I run a rubucop extension to check for code patterns that are not thread safe:
424 files inspected, 88 offenses detected
Of these offenses 34 are related to Mutable constants (so we can just freeze the constants). There are also some related to the use of instance class variables, instance variables in class methods and the use of global variables.
Do you think it is worth to make these changes?
I can also run this cop for all the gems that Huginn depends on.
Probably this wouldn't guarantee that all the code is thread safe but we would be closer to that.

Do you think it is worth to make these changes?

That sounds like a great start, I am not sure how many of those offenses matter when using MRI with its GVL, but it can't hut to fix the offenses.

I can also run this cop for all the gems that Huginn depends on.

That sounds like a lot of work, we have quite a few dependencies 😄 Getting a few people with big Huginn instances to test the branch would probably work as well.

@virtadpt The PR is now on top of master. To test it on your instance you need to check out either my branch or this PR: git fetch origin pull/1731/head:sidekiq &&  git checkout sidekiq
To enable sidekiq set BACKGROUND_JOB_PROCESSOR=sidekiq in your .env file, then bundle install like you normally would after updates. If your redis server isn't running on localhost or a non standard port you can specify the url with REDIS_URL=redis://somehost:6379/1.
Then replace your DJ systemd scripts with sidekiq: bundle exec sidekiq. The amount of threads can be specified with the -c command line flag, but the default of 10 should be a good start.

So I've got this running on my install of huginn and it definitely seems faster, although I do have a problem where events are getting stuck in the sidekiq queue and getting stuck as "busy".  Maybe someone else has bumped into the issue?


mesos/docker automated local cluster deployment

Adds "mesos/docker" cluster provider (KUBERNETES_PROVIDER=mesos/docker).
See the mesos/docker getting started guide for the commands to run the e2e tests.
Integration Points:

Added KUBE_SERVER in hack/ginkgo-e2e.sh to allow overriding scheme (uses https by default, overridden by detect-master in cluster/mesos/docker/util.sh)
Added validate-cluster function to be provided by cluster util.sh with a default impl in cluster/kube-util.sh.
Made all echos in cluster/kube-util.sh go to STDERR to avoid breaking usage in functions that use STDOUT.
Added ./cluster/test-e2e.sh to delegate to ./hack/ginkgo-e2e.sh. (This is not technically required for this PR, but seems to add legitimacy to the e2e scripts in ./hack. They should perhaps be moved instead, but I didn't want to tackle that in this PR.)
exec ginkgo in ./hack/ginkgo-e2e.sh to improve ctrl-c responsiveness
Updated docs/getting-started-guides/README.md (affected every line in the table)

Added mesos/docker
Updated maintainers link for mesos/GCE
Made guide links relative
Grouped "community" supported clusters below the "project" supported ones.

Progress:

[X] Build kubernetes-mesos binaries (./docker/build/build.sh in kubernetes-mesos:docker-compose)

[X] Use binaries from ./build/release.sh or make instead of kubernetes-mesos-build

[X] Build kubernetes-mesos docker image (./docker/km/build.sh in kubernetes-mesos:docker-compose)

[x] Migrate image builder to kubernetes repo

[X] Build mesos-slave docker image w/ iptables (req by k8s) that supports docker-in-docker

[x] Migrate image builder to kubernetes repo

[X] Build kubernetes-mesos-test docker image (./docker/test/build.sh in kubernetes-mesos:docker-compose)

[X] Support running unit tests in kubernetes-mesos-test image
[X] Support running integration tests in kubernetes-mesos-test image
[X] Support running e2e tests in kubernetes-mesos-test image
[x] Migrate image builder to kubernetes repo

[X] Support ./cluster/kube-up.sh
[X] Support ./cluster/kube-down.sh
[x] Support make test_e2e
[x] Support go run hack/e2e.go -test
[x] Resolve networking issues (find way to expose docker container IPs on mac)
[x] Support boot2docker
[x] Support disk mounting with docker-in-docker
[x] Cleanup usage docs & migrate to docs dir
[x] Switch from v1beta3 to v1 API usage
[x] Support kubectl logs

Deferred Features:

Add to CI (mesosphere internal and/or google's)
Support N mesos-slave containers (hard-coded to 2 for now)
Version docker images (with k8s version, git sha, and dirty state) to avoid re-building every kube-up

:+1:

Who would be a good reviewer for this PR?

I'll be having my team at mesosphere review it before I ask for external review.

e2e test log: https://gist.githubusercontent.com/karlkfi/4f4a5b0519c9fc77e93c/raw/kubernetes-docker-mesos-e2e.log
Summarizing 26 Failures:

[Fail] Services [It] should be able to change the type and nodeport settings of a service
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:395

[Timeout...] Services [It] should serve a basic endpoint from pods
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:141

[Fail] Events [It] should be sent by kubelets and the scheduler about pods scheduling and running
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/events.go:79

[Fail] Networking [It] should function for intra-pod communication
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/networking.go:144

[Timeout...] Services [It] should serve multiport endpoints from pods
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:242

[Fail] Nodes Resize [BeforeEach] should be able to delete nodes.
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/resize_nodes.go:252

[Fail] Services [It] should be able to create a functioning NodePort service
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:332

[Fail] Proxy version v1beta3 [It] should proxy through a service and a pod
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/proxy.go:138

[Fail] Pods [It] should be updated
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pods.go:340

[Fail] Nodes Network [BeforeEach] should survive network partition.
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/resize_nodes.go:340

[Fail] ServiceAccounts [It] should mount an API token into pods
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service_accounts.go:58

[Fail] Kubectl client Update Demo [It] should create and stop a replication controller
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/util.go:677

[Fail] Pods [It] should contain environment variables for services
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pods.go:402

[Fail] Kubectl client Update Demo [It] should scale a replication controller
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/util.go:677

[Fail] Addon update [BeforeEach] should propagate add-on file changes
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/addon_update.go:204

[Fail] PreStop [It] should call prestop when killing a pod
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/pre_stop.go:64

[Fail] Nodes Resize [BeforeEach] should be able to add nodes.
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/resize_nodes.go:252

[Fail] Services [It] should release NodePorts on delete
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:739

[Fail] DNS [It] should provide DNS for services
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/dns.go:302

[Fail] Etcd failure [BeforeEach] should recover from SIGKILL
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/etcd_failure.go:46

[Fail] Services [It] should check NodePort out-of-range
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:674

[Fail] Etcd failure [BeforeEach] should recover from network partition with master
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/etcd_failure.go:46

[Fail] DNS [It] should provide DNS for the cluster
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/dns.go:138

[Fail] Services [It] should release the load balancer when Type goes from LoadBalancer -> NodePort
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service.go:542

[Fail] ReplicationController [It] should serve a basic image on each replica with a public image
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/rc.go:137

[Fail] Service endpoints latency [It] should not be very high
/go/src/github.com/GoogleCloudPlatform/kubernetes/_output/dockerized/go/src/github.com/GoogleCloudPlatform/kubernetes/test/e2e/service_latency.go:97

Ran 72 of 86 Specs in 7192.542 seconds
FAIL! -- 46 Passed | 26 Failed | 2 Pending | 12 Skipped --- FAIL: TestE2E (7197.62s)
FAIL

Ginkgo ran 1 suite in 1h59m58.898051402s
Test Suite Failed

Rebase to fix the dns.go e2e tests which failed before due to lower-case "tcp" protocol in the service definition.

Getting closer. Added instructions to expose boot2docker container IPs to the host. YAY!
This fixes make test_e2e which no longer needs to be run in a container.
DNS tests also pass now:
go run hack/e2e.go -test -check_cluster_size=false "-test_args=-ginkgo.focus=DNS"

This PR is ready for review. @davidopp who would be a good reviewer?

TODO: Support KUBECONFIG

@davidopp was out sick, unfortunately - I'm sorry I didn't see this early. I hope he'll be able to get to this tomorrow, since the rest of the (US) week is basically a merge freeze.

Just ran the full e2e suite in our CI and it hung for 20+ hours on "Services - should serve a basic endpoint from pods"

With my pr?

@sttts I assume you mean #10390
If so, then no, I didn't try that patch. It's kind of a PITA to programmatically test changes in multiple branches...
That said, I don't remember that test looping endlessly and not timing out... Failure should at least exit eventually.

Yes, #10390 will fix the endpoints and the endless waiting. You are right, a proper timeout should be applied.

I'm unclear whether this is ready to review. I am happy to do it but please have someone from Mesosphere give it an LGTM as a signal that it is ready for me to review.

@davidopp The mesos/docker cluster scripts seem to be working as expected, but the e2e tests themselves are failing for several reasons. The e2e fixes are being addressed in other PRs.
It would be good to have someone at Google go through the integration points listed in the OP where the changes might interact with other cluster scripts.
Getting Started Guide Validation:

@sttts and I have validated that the guide works on Mac OS X Yosemite.
@spacejam is validating on Ubuntu.
@spacejam previously found that kube-up fails on Arch Linux (likely due to volume mounting with docker-in-docker).

We don't have any other bash experts on the team familiar with the rest of the kubernetes bash code. So the most valuable review would come from someone who has been involved with the other cluster scripts.

Rebased.

@k8s-bot ok to test

Shippable failed with a data race in a piece of code that hasn't been changed in this PR.
E0702 00:51:59.719896 13788 reflector.go:165] Failed to watch *api.Pod: resource version in the future not supported by this fake
E0702 00:52:00.310996 13788 reflector.go:165] Failed to watch *api.Pod: resource version in the future not supported by this fake
E0702 00:52:01.313600 13788 reflector.go:165] Failed to watch *api.Pod: resource version in the future not supported by this fake
E0702 00:52:02.323958 13788 reflector.go:165] Failed to watch *api.Pod: resource version in the future not supported by this fake
E0702 00:52:02.337016 13788 reflector.go:165] Failed to watch *api.Pod: resource version in the future not supported by this fake
E0702 00:52:03.394049 13788 reflector.go:165] Failed to watch *api.Pod: resource version in the future not supported by this fake
==================
WARNING: DATA RACE
Write by goroutine 43:
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework.(*FakeControllerSource).Change()
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework/_test/_obj_test/fake_controller_source.go:85 +0x1a0
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework.(*FakeControllerSource).Delete()
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework/_test/_obj_test/fake_controller_source.go:49 +0x9e
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework_test.func·002()
/root/workspace/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework/controller_test.go:136 +0x7e
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework.ResourceEventHandlerFuncs.OnAdd()
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework/_test/_obj_test/controller.go:108 +0x8a
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework.(*ResourceEventHandlerFuncs).OnAdd()
<autogenerated>:21 +0xca
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework.func·001()
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework/_test/_obj_test/controller.go:180 +0x594
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework.(*Controller).processLoop()
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework/_test/_obj_test/controller.go:80 +0xe0
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework.*Controller.(github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework.processLoop)·fm()
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework/_test/_obj_test/controller.go:60 +0x33
github.com/GoogleCloudPlatform/kubernetes/pkg/util.func·004()
/root/workspace/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/util/util.go:112 +0x5e
github.com/GoogleCloudPlatform/kubernetes/pkg/util.Until()
/root/workspace/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/util/util.go:113 +0xbb
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework.(*Controller).Run()
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework/_test/_obj_test/controller.go:60 +0x243
Previous read by goroutine 44:
github.com/GoogleCloudPlatform/kubernetes/pkg/api/meta.(*genericAccessor).ResourceVersion()
<autogenerated>:13 +0x101
github.com/GoogleCloudPlatform/kubernetes/pkg/client/cache.(*Reflector).watchHandler()
/root/workspace/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/client/cache/reflector.go:245 +0x94c
github.com/GoogleCloudPlatform/kubernetes/pkg/client/cache.(*Reflector).listAndWatch()
/root/workspace/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/client/cache/reflector.go:181 +0xc0f
github.com/GoogleCloudPlatform/kubernetes/pkg/client/cache.func·005()
/root/workspace/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/client/cache/reflector.go:100 +0x6d
github.com/GoogleCloudPlatform/kubernetes/pkg/util.func·004()
/root/workspace/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/util/util.go:112 +0x5e
github.com/GoogleCloudPlatform/kubernetes/pkg/util.Until()
/root/workspace/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/util/util.go:113 +0xbb
Goroutine 43 (running) created at:
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework_test.ExampleInformer()
/root/workspace/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework/controller_test.go:153 +0x381
testing.runExample()
/root/.gvm/gos/go1.3/src/pkg/testing/example.go:98 +0x4b8
testing.RunExamples()
/root/.gvm/gos/go1.3/src/pkg/testing/example.go:36 +0x2ea
testing.Main()
/root/.gvm/gos/go1.3/src/pkg/testing/testing.go:436 +0xe4
main.main()
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework/_test/_testmain.go:107 +0x18b
Goroutine 44 (running) created at:
github.com/GoogleCloudPlatform/kubernetes/pkg/client/cache.(*Reflector).RunUntil()
/root/workspace/src/github.com/GoogleCloudPlatform/kubernetes/_output/local/go/src/github.com/GoogleCloudPlatform/kubernetes/pkg/client/cache/reflector.go:100 +0x130
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework.(*Controller).Run()
github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework/_test/_obj_test/controller.go:58 +0x1e8
==================
E0702 00:52:04.526067 13788 reflector.go:165] Failed to watch *api.Pod: resource version in the future not supported by this fake
E0702 00:52:05.527365 13788 reflector.go:165] Failed to watch *api.Pod: resource version in the future not supported by this fake
PASS
coverage: 75.6% of statements
Found 1 data race(s)
FAIL	github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework	6.907s

@k8s-bot: ok to test

We don't have any other bash experts on the team familiar with the rest of the kubernetes bash code. So the most valuable review would come from someone who has been involved with the other cluster scripts.

Sigh, you happened to catch an on-call with a ton of experience in exactly that. Reviewing.

(The sigh is not because I'm not happy for the contribution. The sigh is because bash.)

@k8s-bot: add to whitelist

@davidopp: I'll hand it back to you for approval after the cluster scripts look reasonable.

@zmerlynn I went with a variation on option A: moved the sourcing of provider's util.sh into the bottom of kube-util.sh
So now all clusters will get the default function implementations in kube-util.sh and can override them as desired.

Looks like that fixed the tests :)
Too bad we don't have CI for the other clusters yet.

I hate to do this, but would you mind splitting the PR up into just the pieces that refactor the cluster/ bits and the rest? It should be fairly straightforward, but then it'll be easy to review that bit and hand the rest off. (Sorry to not ask before.)
I like the refactor now, though it's coming in a little late. It's pretty binary, though, and is either going to break things or not - the problem really is whether it breaks something in one of the providers we're not testing. :(

@zmerlynn This PR is almost all cluster/ code. Did you mean pull out the refactor I just did or did you want more in there?

Sorry, yes, pull out the refactor and leave the cluster/mesos code for a separate PR that relies on the refactor.

I'm happy to give a final LGTM once @zmerlynn is happy, but it makes sense for him to be the primary Google reviewer on this PR.

I pulled out that last commit and put it in #10686
I'll rebase this PR once that is merged.

DNS seems to be broken now for this cluster. It looks like it was changed when I last rebased.
kube2sky thrashes, unable to talk to the apiserver.
$ ./cluster/kubectl.sh get pods
NAME                READY     REASON    RESTARTS   AGE
kube-dns-v4-gthzr   3/3       Running   0          28s
Karls-MBP:kubernetes karl$ ./cluster/kubectl.sh logs kube-dns-v4-gthzr kube2sky
I0703 23:48:04.427049       1 kube2sky.go:389] Etcd server found: http://<IP_ADDRESS>:4001
I0703 23:48:05.429831       1 kube2sky.go:455] Using http://localhost:8080 for kubernetes master
I0703 23:48:05.430655       1 kube2sky.go:456] Using kubernetes API
E0703 23:48:05.431718       1 reflector.go:136] Failed to list *api.Service: Get http://localhost:8080/api/v1/services: dial tcp <IP_ADDRESS>:8080: connection refused
E0703 23:48:05.431815       1 reflector.go:136] Failed to list *api.Endpoints: Get http://localhost:8080/api/v1/endpoints: dial tcp <IP_ADDRESS>:8080: connection refused

Looks like the TokensController (manages auto-creating service account secrets) is only created when --service-account-private-key-file is provided to the ControllerManager.
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/contrib/mesos/pkg/controllermanager/controllermanager.go#L153-L166
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/cmd/kube-controller-manager/app/controllermanager.go#L238-L251
However, I can't find any usage of --service-account-private-key-file in any other cluster scripts...

Ok, I think the issue is that the mesos/docker cluster is configured to use port 8888 instead of 8080, but kube2sky assumes that the port is 8080 unless you use --kube_master_url or --kubecfg_file. It never even gets to requiring an SSL cert.

Ok, it looks like the easiest way to resolve the issue was to add --kube_master_url to the kube-dns ReplicationController yaml that was defining the kube2sky parameters.
Unfortunately, the common yaml files are salt templates. Previously, we were doing some sed magic to manually render them, but that becomes nearly impossible when you throw a variable url with slashes (that need to be escaped at runtime) into the mix (or at least my sed-foo isn't that awesome). So I had to convert the templates into shell scripts that print the rendered yaml. This works great for us, but will make it hard to maintain with two types of templates. For now it solves the immediate problem, but requiring all clusters to use salt is not going to be sustainable. So we'll have to figure out a long term solution.

--kube_master_url and --kubecfg_file are legacy artifacts at this point,
and are the wrong way to fix this.
I can't say for sure how to make it generate (just not familiar with all
these flags), but I know @eparis was looking at this this week, too.
On Fri, Jul 3, 2015 at 6:12 PM, Karl Isenberg<EMAIL_ADDRESS>wrote:

Looks like the TokensController (manages auto-creating service account
secrets) is only created when --service-account-private-key-file is
provided to the ControllerManager.
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/contrib/mesos/pkg/controllermanager/controllermanager.go#L153-L166
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/cmd/kube-controller-manager/app/controllermanager.go#L238-L251
However, I can't find any usage of --service-account-private-key-file in
any other cluster scripts...
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/10049#issuecomment-118447518
.

@thockin it's our fault no one can find it. Many salt scripts seems to like to use _ instead of -. Post 1.0 we should normalize our code some more. grep for service_account_private_key_file
In any case both ansible and salt reuse the server.key. Although I believe this could be any key signed by the ca.
For salt:
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/cluster/saltbase/salt/kube-controller-manager/kube-controller-manager.manifest#L22
For ansible:
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/contrib/ansible/roles/master/templates/controller-manager.j2
CoreOS seems to be doing something in their cloud_config, but it seems rather nonsensical to me.

@eparis Thanks for the references.
I'm still confused how kube-dns knows what ip/port to find the apiserver on. It seems to default to localhost:8080 which would never work in a pod. How are you supposed to configure that if not with the command line parameters?
It looks like we're base64-ing the whole kubectl config in the Secret. Is it supposed to read it from there? I haven't reverse engineered that bit yet. Not clear how it works.

If it finds a service account token it uses
$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT
On Sat, Jul 4, 2015 at 12:36 AM, Karl Isenberg<EMAIL_ADDRESS>wrote:

@eparis https://github.com/eparis Thanks for the references.
I'm still confused how kube-dns knows what ip/port to find the apiserver
on. It seems to default to localhost:8080 which would never work in a pod.
How are you supposed to configure that if not with the command line
parameters?
It looks like we're base64-ing the whole kubectl config in the Secret. Is
it supposed to read it from there? I haven't reverse engineered that bit
yet. Not clear how it works.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/10049#issuecomment-118479541
.

But the service account only gets created if TokensController is running, which it's not, because we're running in http mode without ssl, and without --service-account-private-key-file specified...
We were planning on enabling https eventually, but until now it's at least worked with ssl disabled.

Right, well, in that case you are way outside the "mainstream" and maybe
those "legacy artifacts" are really your escape valves.
On Sat, Jul 4, 2015 at 11:26 PM, Karl Isenberg<EMAIL_ADDRESS>wrote:

But the service account only gets created if TokensController is running,
which it's not, because we're running in http mode without ssl, and without
--service-account-private-key-file specified...
We were planning on enabling https eventually, but until now it's at least
worked with ssl disabled.
—
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/10049#issuecomment-118585696
.

Doing things differently than the rest of the providers isn't going to do us any favors, it's true. But it seems a little questionable for the presence of an ssl cert to be required for and trigger a service discovery mechanism. I would expect the API service to be discoverable and then the fact that it uses an https scheme would trigger looking for an ssl cert.

Rebased to latest master.

Rebase to master (including the v1 merge) and fixed some doc issues.Rebase to master (including the v1 merge) and fixed some doc issues.

Rebased again due to broken master.

rebased again. looks like mungedocs went for a wild ride today.

We'll take a look at this after 1.0 (i.e. next week).

Reabsed, now that #10686 is merged.

For those keeping score at home:

SSL is now enabled using --service-account-private-key-file
KubeDNS is being deployed
KubeUI is being deployed

I just tested guestbook-go and it works without modifications, yay!

Now that the guide suggests to make the docker IPs routable, we can probably back out the validate-cluster changes. Unfortunately, it doesn't look like they currently work.
$ ./cluster/kubectl.sh get cs
NAME                 STATUS      MESSAGE              ERROR
controller-manager   Unhealthy                        Get http://<IP_ADDRESS>:10252/healthz: dial tcp <IP_ADDRESS>:10252: connection refused
scheduler            Unhealthy                        Get http://<IP_ADDRESS>:10251/healthz: dial tcp <IP_ADDRESS>:10251: connection refused
etcd-0               Healthy     {"health": "true"}   nil

I'm not exactly sure why that isn't working, but it causes ./cluster/validate-cluster.sh to fail.

Am I totally off base, or is kubectl get cs using hardcoded IPs and Ports to guess where the controller-manager and scheduler are?
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/pkg/master/master.go#L700
That seems a tad crazypants...

I believe the whole point of the componentStatuses endpoint is to get health of the controller manager and scheduler on the local machine, so I think what we're doing makes sense?

So it seems that componentStatuses assumes that the core components are always on the same IP localhost, and always use the default ports, but I'm pretty sure that's just convenience and coincidence, because we've just shown that it's possible to deploy the core components into their own docker containers, with their own IPs. And I'm pretty sure it'd be possible to run them on separate machines all together, because the apiserver endpoint is configurable. This is the first time I've seen the apiserver need to know where the other components are, which would create a triangular configuration dependency, if they weren't hardcoded.
How does the vanilla docker cluster work? Are they all in the same container?

As "beeps" pointed out in IRC, the salt config runs the core components with host networking: https://github.com/GoogleCloudPlatform/kubernetes/blob/master/cluster/saltbase/salt/kube-scheduler/kube-scheduler.manifest#L6
And the docker guide also uses host networking:
https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/docker.md
So it looks like everyone is either deploying the core components with host networking or as a static pod on a kublet, giving them shared networking.
So it may not be crazy, given the existing deployment patterns, but it does tightly couples all three components, which is a bad idea. Pods were meant to make it easier to network, not enable circular dependencies.
This also means that going forward the HA story requires replicating all three components equally.
Plus, I really think that requiring host networking is a crutch. It forces the user to throw away a layer of isolation without giving them an option.
The only two ways I see out of this:

Configure the ip/port of the core components on the apiserver
Have the core components register their ip/port in etcd and use it for discovery

Unfortunately, the first option still don't help this cluster deployment much. It requires configuring with hostnames and using a proxy for reverse linking, because docker won't allow circular links.

Rebased to latest master.

Rebased to the branch of #12130 in order to unblock this PR with changes outside of cluster/mesos/docker.

The only modifications outside of Mesos code left:

new cluster/test-e2e.sh script
little change in kube-up.sh with its separate PR: https://github.com/GoogleCloudPlatform/kubernetes/pull/12130.

The previously included change to skip ValidateClusterSize in hack/e2e.go is not needed anymore with token based https authentication. If still of value in another context, it is moved to https://github.com/mesosphere/kubernetes/tree/skip-clustersize-validation-go.

Squashed all the commits. Have been way too many over the time.

/cc @jdef @tsenart for internal review

@jdef addressed your comments. ptal.

TODO:

[ ] Switch to docker-compose kill or just use rm -f (until we add support for re-upping with state persistance at some future time)
[ ] Avoid committing secrets (e.g. certs and keys). Lazily generate certs/keys and store them on the user's machine, outside of the repo (ex: ~/.kube/cluster/mesos/docker/)

Blocked by #12130, which allows us to override the cluster-validation method.
This is sub-optimal long-term, but the long term fix requires #11908 or similar alternative to make componentstatuses to play nicely with multiple-ip core component deployment.

Latest force push changes:

Add waiting in kube-up for the addons to be "Ready"
Extract addon status retrieval to its own function for use by validate-cluster
Change kube-down to use docker-compose kill. Since recovery is not a currently supported option, we don't need to wait for graceful termination.

@zmerlynn You wanna take another pass at reviewing this? We've reached a pretty stable build and want to get it merged pretty soon. We're just blocked on #12130, but when it's merged we're good to go.

I ran out of steam about halfway through. I'll pick it up in the AM.

I have no idea what you mean by your deploy-addon.sh comment, @zmerlynn. Can you give more context?
Is there some readme describing how addons are suppossed to be deployed? The addon (esp kube-ui) docs were in shambles when we started. There didn't seem to be any obvious way to launch the addons without "provisioning a machine" with salt or ansible, which is pretty anachronistic when using docker-compose.  So we reverse engineered the existing salt deployments to figure out what was being deployed. Now it's being deployed as part of the provider-specific mesos/docker kube-up, after docker-compose up.

We eventually just want to launch addons from something like a master component that has knowledge of the current known set of master addons that will eventually consume its cluster addon configuration (i.e. whether to deploy or not) from some central configuration (that doesn't exist right now), with reconciliation logic for whats currently running and the version its running at (to handle upgrade).
The reason there was no obvious way to launch addons without provisioning a machine is because we've already started moving the world there, but it's in a hodge-podge of Salt config and messy bash scripts right now and needs to work it's way back out into something more proper, in part because there's no central cluster configuration authority.
The style you have right now is fine but brittle to upgrades and config changes, which we invested in pretty heavily for the Salt config, and will continue to invest in with further kube-addon component work (and make it non-Salt specific). See e.g. #9469 and #9470, which aren't specific enough about what they're doing, but I would like to see them land in a non-Salt specific way.

@zmerlynn addressed all your comments, if not state otherwise above.

Ok, so I took a stab at making the ssl stuff a bit more extracted/readable/maintainable. It's probably not what you were asking for, @zmerlynn, but it should allow us to maintain it a little more easily. If it's not doing what everyone else is doing, at least it's explicit about it.
Please resume your review. We'll squash again when you're satisfied.

Reviewing soon. Got sidetracked.

I just finished a first pass and with the exception of the cluster/mesos/docker/mesos-slave/bin/wrapdocker provenance issue and a pile of nits, it overall LGTM.

I've extracted the mesos-slave image to our mesosphere/docker-containers repo: https://github.com/mesosphere/docker-containers/pull/21
Image is public here: https://registry.hub.docker.com/u/mesosphere/mesos-slave-dind/

Squashed commits.
Just blocked on #12130 now (marked ok-to merge). Then we'll have to rebase.

#12130 is merged.
Rebased & squashed again.
Thanks @zmerlynn, for the reviews!
Based on your above LGTM, I'm going to tag this baby and hand it off to on-call.

Sorry, give me one sec to re-review.

Just did a quick pass for obvious stuff. I think we're good.

:+1:
P.S. Comment 200!
P.P.S. Longest PR ever...


why there is no unity project ?

error!!!!!!!!!!!!!!!!!!

good idea

now there is https://github.com/mobfox/MobFox-Android-SDK-Core-Lib/tree/master/plugins/UnityDemo
enjoy.


Type of return value of get header(s) utils is wrong

For example, getRequestHeader returns string | string[] while it should return string | string[] | undefined, because HTTP headers might be not sent from the client.
To be honest, I added these kinds of utilities to h3, so I will fix them. 🙏
I will check other related utilities as well.
Related issue: https://github.com/NozomuIkuta/h3-cors/issues/1

I found that type of return value of get header(s) utils is correct on h3 side.
Note that its return type is string | string[] | undefined as expected and as defined in http.d.ts.

On the other hand, its return type is string | string[] in my h3-cors code base.

The difference between these 2 code bases is whether strict: true is set in tsconfig.json or not.
With strict set true, string | string[] | undefined is suggested in h3-cors as in h3.

I used unjs/template for scaffolding of h3-cors. So, for compatibility and consistency in terms of type safety and developer experience, strict: true should be added to the template.
@pi0 @danielroe
What do you think?

I would agree that strict: true should be the default. @pi0?

Nice idea for enabling strict :)


x/exp/shiny: multi-second lag on tile example on Mac

Reproduce: run exp/shiny/example/tile using gldriver on OS X, drag the image around with mouse.
Expected outcome: The tiles smoothly move around the window.
Actual outcome: The animation is smooth, but there is a delay (up to several seconds) before they begin moving.
This is on OS X 10.10, on a MacBook Pro (Retina, 13-inch, Early 2015).

/cc @crawshaw

I had the same issue with a proof-of-concept game engine I'm working on. On linux and android, it works fine, but on mac input is delayed several seconds (in addition to the keyboard on working)

@boomshroom are you using x/mobile/app or shiny on OS X? I believe I have this issue under control in the app package but I have not ported the fix to shiny yet.

@crawshaw shiny. mobile/app works fine on both linux and mac. (I have a shiny backend and an mobile/app backend with most code being shared between the 2)

Ok great, then when I finish up porting my changes next week this will be fixed.

This is been fixed.

There still is some input lag (unlike linux where it's practically instant), but it's much better. maybe a half-second of input lag.


Move all 'product feature ideas' away from this repo's issues into ProductBoard

Context
As part of https://github.com/2i2c-org/infrastructure/issues/3702, we're trying to clean up the list of open issues in this repository. One important category of issues is 'product ideas' or 'feature requests', which should now live in Product Board.
Proposal
Go through all the open issues here in this repository, and move them into productboard. This would probably entail having to reformat them into a story form or similar. The open issue can then be closed, with a link to the appropriate product board item.
Since we're moving stuff away from a public GitHub repository, it's important that the appropriate productboard location is also public. https://github.com/2i2c-org/meta/issues/900 tracks the issues in dealing with that.
Updates and actions
No response

Here's a collected list of issues (sorting from oldest first):

https://github.com/2i2c-org/infrastructure/issues/51
https://github.com/2i2c-org/infrastructure/issues/55
https://github.com/2i2c-org/infrastructure/issues/85
https://github.com/2i2c-org/infrastructure/issues/262
https://github.com/2i2c-org/infrastructure/issues/277
https://github.com/2i2c-org/infrastructure/issues/620
https://github.com/2i2c-org/infrastructure/issues/928
https://github.com/2i2c-org/infrastructure/issues/1083
https://github.com/2i2c-org/infrastructure/issues/1230
https://github.com/2i2c-org/infrastructure/issues/1240
https://github.com/2i2c-org/infrastructure/issues/1311
https://github.com/2i2c-org/infrastructure/issues/1414
https://github.com/2i2c-org/infrastructure/issues/1669
https://github.com/2i2c-org/infrastructure/issues/1712
https://github.com/2i2c-org/infrastructure/issues/1758
https://github.com/2i2c-org/infrastructure/issues/1771
https://github.com/2i2c-org/infrastructure/issues/2052
https://github.com/2i2c-org/infrastructure/issues/2096
https://github.com/2i2c-org/infrastructure/issues/2106
https://github.com/2i2c-org/infrastructure/issues/2165
https://github.com/2i2c-org/infrastructure/issues/2208
https://github.com/2i2c-org/infrastructure/issues/2257
https://github.com/2i2c-org/infrastructure/issues/2306
https://github.com/2i2c-org/infrastructure/issues/2379
https://github.com/2i2c-org/infrastructure/issues/2368
https://github.com/2i2c-org/infrastructure/issues/2564
https://github.com/2i2c-org/infrastructure/issues/2563
https://github.com/2i2c-org/infrastructure/issues/2803
https://github.com/2i2c-org/infrastructure/issues/2737
https://github.com/2i2c-org/infrastructure/issues/3017
https://github.com/2i2c-org/infrastructure/issues/2956
https://github.com/2i2c-org/infrastructure/issues/3367
https://github.com/2i2c-org/infrastructure/issues/3636
https://github.com/2i2c-org/infrastructure/issues/3565
https://github.com/2i2c-org/infrastructure/issues/3532
https://github.com/2i2c-org/infrastructure/issues/3521

These will be coalesced into a number of productboard items

I'm going to comment on the appropriate issue with a productboard link for each of them, but not actually close it until we have a way to make this publicly visible

Thanks @yuvipanda. As you port this list to PB,  I would recommend being as strict as you can when considering how many of these are "nice to have" versus "must have", and determine whether some of these might warrant being culled. Moving ideas across systems and into a new process is a golden opportunity to do some house clearing.

I don't want to close any of the issues until we figure out https://github.com/2i2c-org/meta/issues/900. So I'm going to create a temporary 'feature-requests' repo, and move all these issues there so it can clean up the infrastructure repo.

I've moved all the issues, and @Gman0909 had earlier created appropriate productboard items. I am going to link them as appropriate before calling this one done.

Does this issue still need work or can we call it done?


Access control new structure!

now whenever you create a new command you have full control how the command should be like and how it will be processed

Pull Request Test Coverage Report for Build 130

60 of 66 (90.91%) changed or added relevant lines in 15 files are covered.
2 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.2%) to 91.976%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

src/main/java/seedu/ptman/logic/commands/DeleteCommand.java
4
5
80.0%

src/main/java/seedu/ptman/logic/commands/EditCommand.java
4
5
80.0%

src/main/java/seedu/ptman/logic/commands/RedoCommand.java
4
5
80.0%

src/main/java/seedu/ptman/logic/commands/UndoCommand.java
4
5
80.0%

src/main/java/seedu/ptman/logic/parser/RedoCommandParser.java
6
7
85.71%

src/main/java/seedu/ptman/logic/parser/UndoCommandParser.java
6
7
85.71%

Files with Coverage Reduction
New Missed Lines
%

src/main/java/seedu/ptman/logic/parser/DeleteCommandParser.java
1
93.33%

src/main/java/seedu/ptman/logic/commands/Command.java
1
80.0%

Totals

Change from base Build 128:
-0.2%

Covered Lines:
1811

Relevant Lines:
1969

💛  - Coveralls


Refactor: move remove expired IAM util to pkg

We will have a global cleanup job in guardian that can reuse this code.

not required anymore.


Add ibmcloud provider

Add provider to support Fixed provision services. These provision faster (40 to 60 minutes) then other packages.
Nodes a added without hourly billing.

Thanks @derekhiggins the new changes looks fine to me (no anymore active waits). Keeping an hold to allow @tdomnesc having his own round of review. Nice work!
/lgtm
/hold

Minor things I missed the first time around: the locations could have been in the config so we can change them in vault secret instead of code and a sample config in config/samples/ofcir_v1_cipool.yaml . But we can do this after, let's test it first.
/unhold


Lack of license

I think it's better to add a license to your project, just my 2 cents.

the apache version 2 added.


Fix width of third local operand of multifmt slice

Hi,
this is a proposal to fix #86 by using the width of FP format dst_fmt_i for assigning the third local operand in the fpnew_opgroup_multifmt_slice.
With this patch the operands of the individual fpnew_fma_multi instances are assigned correctly, as shown in the waveform below.

@lucabertaccini do you want to merge or should I? Not sure how it is handled here 😅


Organization as subject headings not visible

Eg. "Främlingslegionen"
Several instances has "[Frankrike. Armée. Légion étrangère]" as subject heading. This is not visible in Queerlit

https://github.com/gu-gridh/queerlit-gui/blob/47251f853b953b2a7025787af2646afcab199cff/src/services/libris.service.ts#L374
Check out properties for people with noble titles and nested terms. Example for nested: https://libris.kb.se/find?%40id=https:%2F%2Flibris.kb.se%2Fp60mbdk138b9vf3%23it


Update the docs - $routeChangeError vs $stateChangeError (in ui-router)

https://www.firebase.com/docs/web/libraries/angular/guide/user-auth.html

Both of these methods work well with the resolve() methods of ngRoute and ui-router.

Apparently, when using ui-router the $routeChangeError should be $stateChangeError
app.run(["$rootScope", "$location", function($rootScope, $location) {
$rootScope.$on("$routeChangeError", function(event, next, previous, error) {
  // We can catch the error thrown when the $requireAuth promise is rejected
  // and redirect the user back to the home page
  if (error === "AUTH_REQUIRED") {
    $location.path("/home");
  }
});
}]);

I'm not the routing expert and it took me a while to figure out why the code wasn't executed - it was name of the event that was wrong.
Example plunker: https://plnkr.co/edit/JSRCuerEpr2JKvQtmaQ2?p=preview

The docs are correct in that the sentence you reference (and the paragraph that precedes it) are referring to the $waitForAuth() and $requireAuth methods, not $routeChangeError and $stateChangeError. Note that we have full code samples for both ngRoute and ui-router right below that sentence. It correctly uses $routeChangeError for ngRoute and $stateChangeError for ui-router. You can switch between them by clicking on the buttons at the top of the code sample. We will see if we can make things less confusing though.

Magic!
Didn't notice that:

Many WOW.


Dollar ($) char followed by a single quote makes incorrect output

Hi,
I noticed that if I write a block code which contains a single quote next to a $ sign, the highlighted output will be uncorrect.
Example:

<?php echo 'Test variable$'; ?>

The resulting output will be:
<?php echo 'Test variable
; ?>

Also, when there's other text: the output it's even worse:
Example:

<?php echo 'Test variable$$'; ?>
<?php echo 'Test 2'; >

Output:
<?php echo 'Test variable
<?php echo 'Test 2'; >
; ?>
<?php echo 'Test 2'; >

To solve this I tried to add another dollar sign so it becomes:
Example:

<?php echo 'Test variable$'; ?>

The resulting output will be correct:
<?php echo 'Test variable'; ?>

Anyone noticed this behavior?
Have a good day!

Wow! That was a nasty bug and it also impacted Handlebars and Smarty. Thank you so much for reporting this. It should be fixed now.


TypeError: Typed property DarkGhostHunter\Preloader\Opcache::$status must be array, bool used

Hello @DarkGhostHunter
I have troubles with using getList() method on my develop server:
TypeError: Typed property DarkGhostHunter\Preloader\Opcache::$status must be array, bool used in /projectName/vendor/darkghosthunter/preloader/src/Opcache.php:29
Stack trace:
#0 /projectName/vendor/darkghosthunter/preloader/src/Opcache.php(49): DarkGhostHunter\Preloader\Opcache->getStatus()
#1 /projectName/vendor/darkghosthunter/preloader/src/Preloader.php(87): DarkGhostHunter\Preloader\Opcache->getScripts()
#2 /projectName/vendor/darkghosthunter/preloader/src/Preloader.php(77): DarkGhostHunter\Preloader\Preloader->prepareLister()
#3 /projectName/PreloadGenerator.php(12): DarkGhostHunter\Preloader\Preloader->getList()

This happens when opcache is disabled and opcache_get_status() returns false (bool) instead of an array. That's why i have an exception here:
    public function getScripts() : array
    {
        return $this->getStatus()['scripts'];
    }

My suggestion is to check wether it bool (false) or array and throw more intuitive exception when it is false.
Made a PR with this changes - https://github.com/DarkGhostHunter/Preloader/pull/39

Read the PR, should fix the problem that enabling Preloader on non-opcache-enabled environments (it should throw the error).
Still, the package is not meant to be activated if Opcache is disabled, wherever it is.

Thank you for answer @DarkGhostHunter,
My point is to clarify the exception message without changing the main behavior. There still will be an exception when opcache is disabled, cause opcache_get_status() will return false. But this exception will be clear to end user, cause it took me some time to understand what is going on when i got TypeError.

Fixed in v2.0.4.


Admins Should Be Able To See Initial Account Balance and Maximum Debt Allowed

Usability issue, SocialPredict setup doesn't show the initial values in setup.yaml...maybe it should?
Osnat to add a note about this to the readmes for local and staging setup

READMEs updated