getPage seems to fail on the second run

🐛 Bug Report
When i run one test with getPage() it works fine, but if i duplicate that test and run it again inside another it() i get an error.
Im trying to write multiple tests to test different state/data being sent but i can only use one getPage per test-suite
    TypeError: Cannot destructure property 'getInitialProps' of 'AppComponent' as it is undefined.

      at Object.fetchAppData (node_modules/next-page-tester/dist/_app/fetchAppData.js:12:13)
      at Object.fetchRouteData (node_modules/next-page-tester/dist/fetchData/fetchRouteData.js:9:42)
      at Object.getPageInfo (node_modules/next-page-tester/dist/makePageElement.js:24:40)
      at getPage (node_modules/next-page-tester/dist/getPage.js:84:38)

PS. Are you calling something like jest.resetModules() between tests?

No im not calling that, there's nothing being called between tests (not explicitly by me anyway)
Yeah im trying to see if i can isolate it.
Manually setting nextRoot doesn't help either

In getAppFile.js it seems to return an object pointing to undefined values.
I've checked the absolute path to the _app.tsx and it's correct. But for some reason require([path]) returns {default: undefined}

This happens when import fails. It can for example mean that you have an error at module initialisation time or a cyclic import somewhere.
The DX from our side is not ideal and should probably be improved. We should throw an error indicating that import failed even though the file exists. I've been hit by this before. cc @toomuchdesign

Yep, https://github.com/toomuchdesign/next-page-tester/blob/master/src/loadPage.ts#L15 is throwing because my _app.tsx tries to import a .css file. Looks like its not ignoring it for some reason.
Maybe i need to look into my babel configuration?

Yep that fixed it!
I just needed to sort my statc assets out.
Sadly this framework swallowed the error just like you said @Meemaw
The fix should be to have a separate try to the requires() and log out if they fail, then the user has the information to fix it

Hi @jasonwilliams, hi @Meemaw!
do you have any guess about how/where we're swallowing errors? If so, I'd love to set this information in the form of a failing test so that we can better consider improvements in this sense.

@toomuchdesign We're not swallowing any errors per se. Unfortunately require mechanism is a bit flawed and worn throw an error if the import failed. It will just resolve to undefined.
We can easily detect that case if the file exists but the require resolved to undefined. In that we should throw an error with some useful information when this might happen:

cyclic dependncies
module initialization errors
...

I'm also noticing errors when _app imports external .tsx files.
I guess this might be caused to the fact that we use require. Is it somehow related to this is issue?

I think this happens due to some Typescript errors. Unfortunately you wont see those because tsconfig.json does not include test files. Perhaps this can be improved. If test files were removed to avoid compiling them in build, this is usually handled with a tsconfig.build.json which ignores them, but the default tsconfig.json should include them (this way errors will show in VSCode as well)

Yep @Meemaw, this was actually my issue! 👏 👏👏


Timeouts

implement optional parameter for https://requests.readthedocs.io/en/master/user/quickstart/#timeouts

You can tell Requests to stop waiting for a response after a given number of seconds with the timeout parameter. Nearly all production code should use this parameter in nearly all requests. Failure to do so can cause your program to hang indefinitely.

at https://github.com/davidhuser/dhis2.py/blob/master/dhis2/api.py#L202

Implemented in v2.2.0 with a default of 5 seconds.


Allow passing in db interfaces for insert operations

Currently Kysely relies on the user passing in db.generated when inserting not null columns with default values since it only knows these columns are not null. If we could pass in separate interfaces for insert operations this process would be more type-safe.
Here's an idea for the api
interface Person {
  id: number
  first_name: string
  last_name: string
  gender: 'male' | 'female' | 'other'
}

interface Database {
  person: Person
}

interface InsertablePerson {
  // id is generated as identity
  id?: number
  first_name: string
  last_name: string
  gender: 'male' | 'female' | 'other'
}

interface InsertableDatabase {
  person: InsertablePerson
}

// Insertable interfaces are passed to the second type param.
const db = new Kysely<Database, InsertableDatabase>({
  dialect: new PostgresDialect({
    host: 'localhost',
    database: 'kysely_test',
  })
})

Some libraries that generate db table interfaces, for example kanel, also generate interfaces for insert operations, which accounts for generated values. It'd be very much useful if we could pass those interfaces to Kysely.

I've considered that, but this would make the types much more complex and things would have one more generic argument everywhere. The tradeoff is not good.

There's some discussion about this idea here

Reopened this as I'm working on a POC for this after multiple requests.

Here's what I've currently thinking:
import {
  Kysely,
  PostgresDialect,
  Generated,
  ColumnType,
  Selectable,
  Insertable,
  Updateable,
} from 'kysely'

interface PersonTable {
  // Columns that are generated by the database should be marked
  // using the `Generated` type. This way they are automatically
  // made optional in inserts.
  id: Generated<number>

  first_name: string
  gender: 'male' | 'female' | 'other'

  // If the column is nullable in the database, make its type nullable.
  // Don't use optional properties. Optionality is always determined
  // automatically by Kysely.
  last_name: string | null

  // You can specity the type for each operation (select, insert and update)
  // using the `ColumnType<SelectType, InsertType, UpdateType>` wrapper.
  // Here we define a column `modified_at` that is selected as a `Date`,
  // can optionally be provided as a `string` in inserts and can never
  // be updated:
  modified_at: ColumnType<Date, string | undefined, never>
}

// You can extract the select, insert and update interfaces like this
// if you want (you don't need to):
type Person = Selectable<PersonTable>
type InsertablePerson = Insertable<PersonTable>
type UpdateablePerson = Updateable<PersonTable>

interface PetTable {
  id: Generated<number>
  name: string
  owner_id: number
  species: 'dog' | 'cat'
}

interface MovieTable {
  id: Generated<string>
  stars: number
}

// Keys of this interface are table names.
interface Database {
  person: PersonTable
  pet: PetTable
  movie: MovieTable
}

Generated is just a shortcut for ColumnType
export type Generated<S> = ColumnType<S, S | undefined, S>

Here's some more documentation
/**
 * This type can be used to specify a different type for
 * select, insert and update operations.
 *
 * Also see the {@link Generated} type.
 *
 * ### Examples
 *
 * The next example defines a number column that is optional
 * in inserts and updates. All columns are always optional
 * in updates so therefore we don't need to specify `undefined`
 * for the update type. The type below is useful for all kinds of
 * database generated columns like identifiers. The `Generated`
 * type is actually just a shortcut for the type in this example:
 *
 * ```ts
 * ColumnType<number, number | undefined, number>
 * ```
 *
 * The above example makes the column optional in inserts
 * and updates, but you can still choose to provide the
 * column. If you want to prevent insertion/update you
 * can se the type as `never`:
 *
 * ```ts
 * ColumnType<number, never, never>
 * ```
 *
 * Here's one more example where the type is different
 * for each different operation:
 *
 * ```ts
 * ColumnType<Date, string, never>
 * ```
 */
export type ColumnType<
  SelectType,
  InsertType = SelectType,
  UpdateType = SelectType
> = {
  readonly __select__: SelectType
  readonly __insert__: InsertType
  readonly __update__: UpdateType
}

/**
 * A shortcut for defining database-generated columns. The type
 * is the same for all three but when inserted, the value can
 * be omitted.
 *
 * This is implemented using {@link ColumnType} like this:
 *
 * ```ts
 * type Generated<S> = ColumnType<S, S | undefined, S>
 * ```
 */
export type Generated<S> = ColumnType<S, S | undefined, S>

/**
 * Evaluates to `K` if `T` can be `null` or `undefined`.
 */
type IfNullable<T, K> = undefined extends T
  ? K
  : null extends T
  ? K
  : never

/**
 * Evaluates to `K` if `T` can't be `null` or `undefined`.
 */
type IfNotNullable<T, K> = undefined extends T
  ? never
  : null extends T
  ? never
  : T extends never
  ? never
  : K

/**
 * Evaluates to `K` if `T` isn't `never`.
 */
type IfNotNever<T, K> = T extends never ? never : K

export type SelectType<T> = T extends ColumnType<infer S, any, any> ? S : T
export type InsertType<T> = T extends ColumnType<any, infer I, any> ? I : T
export type UpdateType<T> = T extends ColumnType<any, any, infer U> ? U : T

/**
 * Keys of `R` whose `InsertType` values can be `null` or `undefined`.
 */
export type NullableInsertKeys<R> = {
  [K in keyof R]: IfNullable<InsertType<R[K]>, K>
}[keyof R]

/**
 * Keys of `R` whose `InsertType` values can't be `null` or `undefined`.
 */
export type NonNullableInsertKeys<R> = {
  [K in keyof R]: IfNotNullable<InsertType<R[K]>, K>
}[keyof R]

/**
 * Keys of `R` whose `SelectType` values are not `never`
 */
type NonNeverSelectKeys<R> = {
  [K in keyof R]: IfNotNever<SelectType<R[K]>, K>
}[keyof R]

/**
 * Keys of `R` whose `UpdateType` values are not `never`
 */
export type NonNeverUpdateKeys<R> = {
  [K in keyof R]: IfNotNever<UpdateType<R[K]>, K>
}[keyof R]

/**
 * Given a table interface, extracts the select type from all
 * {@link ColumnType} types.
 *
 * ### Examples
 *
 * ```ts
 * interface PersonTable {
 *   id: Generated<number>
 *   first_name: string
 *   modified_at: ColumnType<Date, string, never>
 * }
 *
 * type Person = Selectable<PersonTable>
 * // {
 * //   id: number,
 * //   first_name: string
 * //   modified_at: Date
 * // }
 * ```
 */
export type Selectable<R> = {
  [K in NonNeverSelectKeys<R>]: SelectType<R[K]>
}

/**
 * Given a table interface, extracts the insert type from all
 * {@link ColumnType} types.
 *
 * ### Examples
 *
 * ```ts
 * interface PersonTable {
 *   id: Generated<number>
 *   first_name: string
 *   modified_at: ColumnType<Date, string, never>
 * }
 *
 * type InsertablePerson = Insertable<PersonTable>
 * // {
 * //   id?: number,
 * //   first_name: string
 * //   modified_at: string
 * // }
 * ```
 */
export type Insertable<R> = {
  [K in NonNullableInsertKeys<R>]: InsertType<R[K]>
} & {
  [K in NullableInsertKeys<R>]?: InsertType<R[K]>
}

/**
 * Given a table interface, extracts the update type from all
 * {@link ColumnType} types.
 *
 * ### Examples
 *
 * ```ts
 * interface PersonTable {
 *   id: Generated<number>
 *   first_name: string
 *   modified_at: ColumnType<Date, string, never>
 * }
 *
 * type UpdateablePerson = Updateable<PersonTable>
 * // {
 * //   id?: number,
 * //   first_name?: string
 * // }
 * ```
 */
export type Updateable<R> = {
  [K in NonNeverUpdateKeys<R>]?: UpdateType<R[K]>
}

Now implemented in 0.15.0


Clarify container external connectivity in multi-network scenario

It is not yet explained in the docs.
Isuse raised in #21741
Signed-off-by: Alessandro Boch<EMAIL_ADDRESS>

Please clarify...

Lexicographical order as in the Oxford English Dictionary
Lexicographical order as in ASCII/UTF
Lexicographical order as in an OrderedDict

a) Lexicographical order in the networks section
b) Lexicographical order under each service
Thanks.

@darkermatter
The ordering is achieved by applying the golang comparison operator < between the network names strings. As per the go docs, it is a byte-wise lexical ordering. My knowledge about the ordering ends here ;-)
Endpoint name (if this is what you were referring to with "service") is not taken into account.

LGTM

LGTM 🐶

Hi @aboch,
Allow me to clarify my request for clarification :-S
In docker-compose.yml v2 files there are networks and services (among others) sections. I define the networks in the networks section and use them in the services section under each service I define.
networks:
  network1: ...
  network2: ...

services:
  service1:
    networks:
      - network1
  service2:
    networks:
      - network2
      - network1

Where is the order for the container/service derived from?
Cheers.

@darkermatter networks in this case are sorted by name, so network1 comes before network2

So, for service2, eth0 will be network1?

@darkermatter Yes.

Thanks!
It makes sense for rapid development (of docker) but now means we have to think of things like
networks:
  aaa_external_routable_me_first_definitely_me_first: ...
  backend: ...

@darkermatter Actually what I said is not correct.
The default gateway will be the network1's gateway address.
Which network eth0 connects the container to depends on the order with which the daemon will connect the networks to the container. So it may be eth0 or eth1 and it is independent on the default gateway choosing logic.

For all intents and purposes, the gateway is the important factor. I just used eth0 as an example as it's typically the interface used for external routable traffic.
If there are various external networks and they are attached to by the daemon independently of how things are specified in docker-compose.yml then the default gateway won't necessarily be aaa_external_routable_me_first_definitely_me_first due to the order of the <IP_ADDRESS> entry in the routing table. If that's the case then the documentation is misleading at best.
WRT #21741, from my understanding, I don't think this should be closed as I think there is a deeper solution required than simply changing the documentation.

@darkermatter
No, the default gateway will be via aaa_external... network.
Regardless of the order with which the container gets connected to the multiple networks, Docker takes care of programming the default gw in your container based on the lexical order logic, at any connect time. Whenever the container connects to aaa_external... network, the default gw for the container is re-programmed via that network.

Thanks ever so much @aboch  - that's really useful information.
Cheers!


Urban Sound Classification using CNN.ipynb test accuracy 0.149?

I run the Urban Sound Classification using CNN.ipynb and got a test accuracy of 0.149.  and the accuracy in paper  Environmental sound classification with convolutional neural networks by Karol J. Piczak. is more than 0.5. I wonder whether any code goes wrong? thanks

The model discussed in blogpost is not proposed by Karol J. Piczak. I mentioned clearly Karol model/idea was used as inspiration. Also, I would like to mention this is not simple problem to solve.


Clean up some typos, styling in README

Part of JOSS review https://github.com/openjournals/joss-reviews/issues/6575

Thank you for your review. I think I modified the paper throughly


Fail to deploy an addon using addon template

Describe the bug
I am trying to use the addon template feature to rewrite the managed-serviceaccount API, which needs to create these permissions on the hub cluster:
but I got error when deploying the addon:
#managedClusterAddon
    message: |-
      Failed to set permission for hub agent: rolebindings.rbac.authorization.k8s.io "open-cluster-management:managed-serviceaccount:clusterrole:agent" is forbidden: user "system:serviceaccount:open-cluster-management-hub:addon-manager-controller-sa" (groups=["system:serviceaccounts" "system:serviceaccounts:open-cluster-management-hub" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:
      {APIGroups:[""], Resources:["configmaps"], Verbs:["*"]}
      {APIGroups:[""], Resources:["secrets"], Verbs:["*"]}
      {APIGroups:["authentication.open-cluster-management.io"], Resources:["managedserviceaccounts"], Verbs:["get" "list" "watch"]}
      {APIGroups:["authentication.open-cluster-management.io"], Resources:["managedserviceaccounts/status"], Verbs:["get" "update" "patch"]}
      {APIGroups:["coordination.k8s.io"], Resources:["leases"], Verbs:["*"]}
    reason: SetPermissionFailed
    status: "False"
    type: RegistrationApplied

To Reproduce
Steps to reproduce the behavior:
Expected behavior
The addon can be deployed successfully. OR some docs explain how to hand this.
Environment ie: OCM version, Kubernetes version and provider:
Additional context
Add any other context about the problem here.

Add some doc to explain how to grant permissions for the addon manager to address this issue: https://github.com/open-cluster-management-io/open-cluster-management-io.github.io/pull/373


attributes support

Is your feature request related to a problem? Please describe.
If you have a category, you always want to know its attributes.
Describe the solution you'd like
How about an optional parameter for getCategoryDefinition?
Well, it's already implemented in category.
Maybe you should add a hint about the calling convention
... categories
foo->bar
... category bar

Hi @polluks
as you already found out it is possible to query a single category with all its attributes. Unluckily there is no single endpoint available for that yet, so there is this workaround to load all categories and look for the single searched one.
This is not working currently because of issue #3 ...
We will change the behaviour and will give a ping here when it is working again!

After fixing #3 this now works as expected.


fix debugging on windows

Because debugger is running within vscode without using a the powershell shell we need to bypass the drive letter fix while debugging.
This will fix the issue mentiond in #15

Great. It works


hitting a creature's body part hitbox doesn't make tamed creatures attack it

using setLastAttackedEntity on the parent of the hitbox fails, seems to be because any use of it gets overriden by the use of same method in attackTargetEntityWithCurrentItem, and because setLastAttackedEntity turns any entity that is not an instance of EntityLivingBase into null. must find a way to make setLastAttackedEntity execute afterwards

ye looks like a mixin should do i hope


[aws-sdk-cpp] Fix failure to call curl[c-ares]

Fixes https://github.com/microsoft/vcpkg/issues/36661
Fix calling curl using config mode.

[X] Changes comply with the maintainer guide.
[ ] SHA512s are updated for each updated download.
[ ] The "supports" clause reflects platforms that may be fixed by this new version.
[ ] Any fixed CI baseline entries are removed from that file.
[ ] Any patches that are no longer applied are deleted from the port's directory.
[ ] The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
[ ] Only one version is added to each modified port's versions file.

Compile test pass with following triplets:
x64-windows
arm64-osx

We are getting build errors with the 2024.04.26 release. I think its due to this PR.
Heres the error we get during cmake configure.
 -- Generating done (0.1s)
  The link interface of target "aws-cpp-sdk-core" contains:

    CURL::libcurl

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.

We do not add curl directly as a dependency, neither do we run find_package for it. We just have the following setup in cmake
find_package(
    AWSSDK CONFIG
    COMPONENTS core s3
    REQUIRED
)

and in vcpkg.json we have:
        {
            "name": "aws-sdk-cpp",
            "features": [
                "s3"
            ]
        },

I assume I just need to add curl directly as a dependency, but I would guess that the aws sdk should have this in its port description?

Yeah, switching  imported targets means having to add find_dependency to exported config. https://github.com/microsoft/vcpkg/pull/38215#discussion_r1568346188, but it was merged nevertheless. I have to let Microsoft learn the hard way.

Thanks!


MPC: No proper scheduling when using def_end_timestep

Describe the bug
When scheduling my dishwasher I reduce the def_total_hours by runtime. I reduce it every 0.5 hours (-1 timestep). I set def_end_timestep = def_total_hours * 2. I expect that the result schedules the deferrable load one by one for 6 timesteps (3h to run). Each 30m the hours will be reduced by 0.5 and the timesteps will be reduced by 1. Nevertheless there are gaps in some of the calculations. The gaps do not happen when def_end_timestep is not set OR when its set to a higher value e.g. 12.
Long story short: I like to make EMHASS know that the dishwasher is running and that the energy is needed, therefore I need to use def_end_timestep to force it to this given timeframe otherwise it will schedule the "hours to run" somewhen later which makes no sense in this case.
To Reproduce

iteration def_total_hours = 3, def_end_timestep = 6

curl -i -H "Content-Type: application/json" -X POST -d '{
        "load_cost_forecast": [0.2554, 0.2536, 0.2536, 0.2484, 0.2484, 0.2459, 0.2459, 0.2452, 0.2452, 0.2442, 0.2442, 0.2435, 0.2435, 0.2462, 0.2462, 0.2511, 0.2511, 0.2584, 0.2584, 0.2695, 0.2695, 0.2708, 0.2708, 0.2662, 0.2662, 0.2607, 0.2607, 0.2557, 0.2557, 0.2517, 0.2517, 0.2512, 0.2512, 0.2556, 0.2556, 0.2618, 0.2618, 0.2665, 0.2665, 0.276, 0.276, 0.2759, 0.2759, 0.2624, 0.2624, 0.2563, 0.2563, 0.2474],
        "prediction_horizon": 36,
        "pv_power_forecast": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 408, 798, 1200, 1608, 2330, 4349, 5582, 5938, 5293, 4092, 3819, 3779, 3645, 3471, 3129, 3126, 2708, 1445, 758, 274, 0, 0, 0, 0, 0, 0, 0],
        "alpha": 0.75,
        "beta": 0.25,
        "num_def_loads": 3,
        "def_total_hours": [
            0,
            3,
            0
          ],
        "P_deferrable_nom":  [
            2750,
            2000,
            2200
          ],
        "treat_def_as_semi_cont": [true, true, true],
        "set_def_constant": [true, true, true],
        "def_start_timestep": [
            0,
            0,
            0
          ],
        "def_end_timestep": [
            0,
            6,
            0
          ]
      }' http://localhost:5000/action/naive-mpc-optim

now reduce the def_total_hours and def_end_timestep each 30 minutes:
2. iteration: 2.5 / 5
3. iteration: 2 / 4
4. iteration: 1.5 / 3
5. iteration: 1 / 2
6. iteration: 0.5 / 1
Expected behavior
In each iteration the loads should be scheduled without any gap!
Screenshots
1. iteration (def_total_hours=3, def_end_timestep=6)

=> Not ok, gap between timestep 1 and 3, last timestep is doubled up (4000 W instead of 2000 W)
2. iteration (def_total_hours=2.5, def_end_timestep=5)

=> Not ok, gap and double watts
3. iteration (def_total_hours=2, def_end_timestep=4)

=> Ok
4. iteration (def_total_hours=1.5, def_end_timestep=3)

=> Ok
5. iteration (def_total_hours=1, def_end_timestep=2)

=> Not ok, second timestep is scheduled to the very last element (+18h)
5. iteration (def_total_hours=0.5, def_end_timestep=1)

=> Ok
Home Assistant installation type

Home Assistant OS

Your hardware

OS: HA OS
Architecture: amd64

EMHASS installation type

Add-on

Additional context
When not using def_end_timestep the calculation is done like expected in "one go":
3 hours:

2.5 hours:

1 hour:

When using a greater def_end_timestep it works like expected, too:
2.5 hours, def_end_timestep = 12:

1 hour, def_end_timestep = 12:

@michaelpiron do you have any ideas on this?

@sti0 for each of the iterations, could you tell what was the status of the optimization? Was it “optimal” in each iteration?
can you share the log (DEBUG level) of a bad iteration?
When did you run each iteration? Did you run it at exact at the start of a new half hour? (Xh00 and Xh30)
Last info I’d need: which load cost profile did you feed to the optimizer? Flat tariff?
thx

@michaelpiron will try to reproduce it with the needed information tonight

In fact, the start and end timestep feature was not built to accomodate your use case. Your case would need a logic that tells emhass that the deferrable load has started its program, and hence the remaining hours need to be planned at the beginningof the optimization window (without having to specify the end timestep).
Let’s for now try to debug further and see how you can use start-end timesteps to accomodate your use case as good as possible. In the longer run however, we’ll need a new feature.

But in fact the time endstep should not produce wrong results because only because it's forced to produce in a given timeframe. It should produce the same result if the endtimestep is 6 or 12 as long as the time condition was met.
But I agree a feature to tell EMHASS that the load is running and can't be stopped during the run would be a better solution.

@michaelpiron please find the wanted information next:

which load cost profile did you feed to the optimizer? Flat tariff?
I'm using self-consumption. With dynamic tariff from Tibber feed into when calling MPC.

method_ts_round is set to last.
1. Iteration - NOK
MPC Call @ 22:39 with def_total_hours = 3 and def_end_timestep = 6:

Debug log:
2024-02-15 22:39:05,195 - web_server - INFO - Setting up needed data
2024-02-15 22:39:05,242 - web_server - INFO - Retrieve hass get data method initiated...
2024-02-15 22:39:07,078 - web_server - INFO - Retrieving weather forecast data using method = list
2024-02-15 22:39:07,080 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2024-02-15 22:39:07,080 - web_server - INFO - Retrieve hass get data method initiated...
2024-02-15 22:39:09,444 - web_server - INFO -  >> Performing naive MPC optimization...
2024-02-15 22:39:09,444 - web_server - INFO - Performing naive MPC optimization
2024-02-15 22:39:09,449 - web_server - INFO - Perform an iteration of a naive MPC controller
2024-02-15 22:39:09,458 - web_server - DEBUG - Deferrable load 0: Proposed optimization window: 0 --> 0
2024-02-15 22:39:09,458 - web_server - DEBUG - Deferrable load 0: Validated optimization window: 0 --> 0
2024-02-15 22:39:09,461 - web_server - DEBUG - Deferrable load 1: Proposed optimization window: 0 --> 6
2024-02-15 22:39:09,461 - web_server - DEBUG - Deferrable load 1: Validated optimization window: 0 --> 6
2024-02-15 22:39:09,465 - web_server - DEBUG - Deferrable load 2: Proposed optimization window: 0 --> 0
2024-02-15 22:39:09,465 - web_server - DEBUG - Deferrable load 2: Validated optimization window: 0 --> 0
2024-02-15 22:39:09,532 - web_server - INFO - Status: Optimal
2024-02-15 22:39:09,532 - web_server - INFO - Total value of the Cost function = -2165.71

2. Iteration - NOK
MPC Call @ 22:45 with def_total_hours = 2.5 and def_end_timestep = 5:

2024-02-15 22:45:14,503 - web_server - INFO - Setting up needed data
2024-02-15 22:45:14,511 - web_server - INFO - Retrieve hass get data method initiated...
2024-02-15 22:45:16,249 - web_server - INFO - Retrieving weather forecast data using method = list
2024-02-15 22:45:16,250 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2024-02-15 22:45:16,251 - web_server - INFO - Retrieve hass get data method initiated...
2024-02-15 22:45:18,440 - web_server - INFO -  >> Performing naive MPC optimization...
2024-02-15 22:45:18,440 - web_server - INFO - Performing naive MPC optimization
2024-02-15 22:45:18,446 - web_server - INFO - Perform an iteration of a naive MPC controller
2024-02-15 22:45:18,453 - web_server - DEBUG - Deferrable load 0: Proposed optimization window: 0 --> 0
2024-02-15 22:45:18,453 - web_server - DEBUG - Deferrable load 0: Validated optimization window: 0 --> 0
2024-02-15 22:45:18,456 - web_server - DEBUG - Deferrable load 1: Proposed optimization window: 0 --> 5
2024-02-15 22:45:18,456 - web_server - DEBUG - Deferrable load 1: Validated optimization window: 0 --> 5
2024-02-15 22:45:18,461 - web_server - DEBUG - Deferrable load 2: Proposed optimization window: 0 --> 0
2024-02-15 22:45:18,461 - web_server - DEBUG - Deferrable load 2: Validated optimization window: 0 --> 0
2024-02-15 22:45:18,534 - web_server - INFO - Status: Optimal
2024-02-15 22:45:18,534 - web_server - INFO - Total value of the Cost function = -1917.27
2024-02-15 22:45:27,078 - web_server - INFO - EMHASS server online, serving index.html...

3. Iteration - OK
MPC Call @ 22:47 with def_total_hours = 2 and def_end_timestep = 4:

2024-02-15 22:47:07,647 - web_server - INFO - Setting up needed data
2024-02-15 22:47:07,654 - web_server - INFO - Retrieve hass get data method initiated...
2024-02-15 22:47:08,765 - web_server - INFO - Retrieving weather forecast data using method = list
2024-02-15 22:47:08,766 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2024-02-15 22:47:08,767 - web_server - INFO - Retrieve hass get data method initiated...
2024-02-15 22:47:11,600 - web_server - INFO -  >> Performing naive MPC optimization...
2024-02-15 22:47:11,600 - web_server - INFO - Performing naive MPC optimization
2024-02-15 22:47:11,606 - web_server - INFO - Perform an iteration of a naive MPC controller
2024-02-15 22:47:11,615 - web_server - DEBUG - Deferrable load 0: Proposed optimization window: 0 --> 0
2024-02-15 22:47:11,615 - web_server - DEBUG - Deferrable load 0: Validated optimization window: 0 --> 0
2024-02-15 22:47:11,618 - web_server - DEBUG - Deferrable load 1: Proposed optimization window: 0 --> 4
2024-02-15 22:47:11,618 - web_server - DEBUG - Deferrable load 1: Validated optimization window: 0 --> 4
2024-02-15 22:47:11,621 - web_server - DEBUG - Deferrable load 2: Proposed optimization window: 0 --> 0
2024-02-15 22:47:11,621 - web_server - DEBUG - Deferrable load 2: Validated optimization window: 0 --> 0
2024-02-15 22:47:11,676 - web_server - INFO - Status: Optimal
2024-02-15 22:47:11,677 - web_server - INFO - Total value of the Cost function = -1674.04

4. Iteration - OK
MPC Call @ 22:50 with def_total_hours = 1.5 and def_end_timestep = 3:

2024-02-15 22:50:23,523 - web_server - INFO - Setting up needed data
2024-02-15 22:50:23,533 - web_server - INFO - Retrieve hass get data method initiated...
2024-02-15 22:50:24,563 - web_server - INFO - Retrieving weather forecast data using method = list
2024-02-15 22:50:24,564 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2024-02-15 22:50:24,565 - web_server - INFO - Retrieve hass get data method initiated...
2024-02-15 22:50:27,221 - web_server - INFO -  >> Performing naive MPC optimization...
2024-02-15 22:50:27,221 - web_server - INFO - Performing naive MPC optimization
2024-02-15 22:50:27,230 - web_server - INFO - Perform an iteration of a naive MPC controller
2024-02-15 22:50:27,236 - web_server - DEBUG - Deferrable load 0: Proposed optimization window: 0 --> 0
2024-02-15 22:50:27,236 - web_server - DEBUG - Deferrable load 0: Validated optimization window: 0 --> 0
2024-02-15 22:50:27,239 - web_server - DEBUG - Deferrable load 1: Proposed optimization window: 0 --> 3
2024-02-15 22:50:27,239 - web_server - DEBUG - Deferrable load 1: Validated optimization window: 0 --> 3
2024-02-15 22:50:27,241 - web_server - DEBUG - Deferrable load 2: Proposed optimization window: 0 --> 0
2024-02-15 22:50:27,241 - web_server - DEBUG - Deferrable load 2: Validated optimization window: 0 --> 0
2024-02-15 22:50:27,277 - web_server - INFO - Status: Optimal
2024-02-15 22:50:27,277 - web_server - INFO - Total value of the Cost function = -1425.58
2024-02-15 22:50:29,383 - web_server - INFO - EMHASS server online, serving index.html...

5. Iteration - NOK
MPC Call @ 22:51 with def_total_hours = 1 and def_end_timestep = 2:
=> Scheduled all loads into the first timestep!!

2024-02-15 22:51:48,198 - web_server - INFO - Setting up needed data
2024-02-15 22:51:48,202 - web_server - INFO - Retrieve hass get data method initiated...
2024-02-15 22:51:49,433 - web_server - INFO - Retrieving weather forecast data using method = list
2024-02-15 22:51:49,435 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2024-02-15 22:51:49,435 - web_server - INFO - Retrieve hass get data method initiated...
2024-02-15 22:51:51,598 - web_server - INFO -  >> Performing naive MPC optimization...
2024-02-15 22:51:51,598 - web_server - INFO - Performing naive MPC optimization
2024-02-15 22:51:51,604 - web_server - INFO - Perform an iteration of a naive MPC controller
2024-02-15 22:51:51,610 - web_server - DEBUG - Deferrable load 0: Proposed optimization window: 0 --> 0
2024-02-15 22:51:51,610 - web_server - DEBUG - Deferrable load 0: Validated optimization window: 0 --> 0
2024-02-15 22:51:51,613 - web_server - DEBUG - Deferrable load 1: Proposed optimization window: 0 --> 2
2024-02-15 22:51:51,613 - web_server - DEBUG - Deferrable load 1: Validated optimization window: 0 --> 2
2024-02-15 22:51:51,615 - web_server - DEBUG - Deferrable load 2: Proposed optimization window: 0 --> 0
2024-02-15 22:51:51,615 - web_server - DEBUG - Deferrable load 2: Validated optimization window: 0 --> 0
2024-02-15 22:51:51,678 - web_server - INFO - Status: Optimal
2024-02-15 22:51:51,679 - web_server - INFO - Total value of the Cost function = -1173.76

6. Iteration - OK
MPC Call @ 22:54 with def_total_hours = 0.5 and def_end_timestep = 1:

2024-02-15 22:54:03,792 - web_server - INFO - Setting up needed data
2024-02-15 22:54:03,800 - web_server - INFO - Retrieve hass get data method initiated...
2024-02-15 22:54:05,526 - web_server - INFO - Retrieving weather forecast data using method = list
2024-02-15 22:54:05,527 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2024-02-15 22:54:05,528 - web_server - INFO - Retrieve hass get data method initiated...
2024-02-15 22:54:07,614 - web_server - INFO -  >> Performing naive MPC optimization...
2024-02-15 22:54:07,614 - web_server - INFO - Performing naive MPC optimization
2024-02-15 22:54:07,619 - web_server - INFO - Perform an iteration of a naive MPC controller
2024-02-15 22:54:07,625 - web_server - DEBUG - Deferrable load 0: Proposed optimization window: 0 --> 0
2024-02-15 22:54:07,625 - web_server - DEBUG - Deferrable load 0: Validated optimization window: 0 --> 0
2024-02-15 22:54:07,627 - web_server - DEBUG - Deferrable load 1: Proposed optimization window: 0 --> 1
2024-02-15 22:54:07,627 - web_server - DEBUG - Deferrable load 1: Validated optimization window: 0 --> 1
2024-02-15 22:54:07,630 - web_server - DEBUG - Deferrable load 2: Proposed optimization window: 0 --> 0
2024-02-15 22:54:07,630 - web_server - DEBUG - Deferrable load 2: Validated optimization window: 0 --> 0
2024-02-15 22:54:07,667 - web_server - INFO - Status: Optimal
2024-02-15 22:54:07,667 - web_server - INFO - Total value of the Cost function = -918.33

Doing the same again after full clock brings the right result in most of the iterations. But still produces the wrong result when running with 1h.
MPC Call @ 23:01 with def_total_hours = 3 and def_end_timestep = 6:

MPC Call @ 23:02 with def_total_hours = 2.5 and def_end_timestep = 5:

MPC Call @ 23:03 with def_total_hours = 2 and def_end_timestep = 4:

MPC Call @ 23:04 with def_total_hours = 1.5 and def_end_timestep = 3:

MPC Call @ 23:05 with def_total_hours = 1 and def_end_timestep = 2:

=> WRONG RESULT
2024-02-15 23:05:13,647 - web_server - INFO - Setting up needed data
2024-02-15 23:05:13,654 - web_server - INFO - Retrieve hass get data method initiated...
2024-02-15 23:05:14,832 - web_server - INFO - Retrieving weather forecast data using method = list
2024-02-15 23:05:14,834 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2024-02-15 23:05:14,834 - web_server - INFO - Retrieve hass get data method initiated...
2024-02-15 23:05:17,173 - web_server - INFO -  >> Performing naive MPC optimization...
2024-02-15 23:05:17,173 - web_server - INFO - Performing naive MPC optimization
2024-02-15 23:05:17,178 - web_server - INFO - Perform an iteration of a naive MPC controller
2024-02-15 23:05:17,183 - web_server - DEBUG - Deferrable load 0: Proposed optimization window: 0 --> 0
2024-02-15 23:05:17,183 - web_server - DEBUG - Deferrable load 0: Validated optimization window: 0 --> 0
2024-02-15 23:05:17,185 - web_server - DEBUG - Deferrable load 1: Proposed optimization window: 0 --> 2
2024-02-15 23:05:17,186 - web_server - DEBUG - Deferrable load 1: Validated optimization window: 0 --> 2
2024-02-15 23:05:17,188 - web_server - DEBUG - Deferrable load 2: Proposed optimization window: 0 --> 0
2024-02-15 23:05:17,188 - web_server - DEBUG - Deferrable load 2: Validated optimization window: 0 --> 0
2024-02-15 23:05:17,241 - web_server - INFO - Status: Optimal
2024-02-15 23:05:17,241 - web_server - INFO - Total value of the Cost function = -1192.86
2024-02-15 23:05:21,132 - web_server - INFO - EMHASS server online, serving index.html...

MPC Call @ 23:06 with def_total_hours = 0.5 and def_end_timestep = 1:

Another strange behavior which I noticed right now. While my deferrable load 2 (washing maschine) is running, deferrable load 0 got scheduled but its "deactivated" (running hours are always 0). And the load seems to be a very small value (not typical usage).
In the following screenshot p_deferrable load 2 is supposed to run for 1.5h with end timestep set to 3 (2200W each timestep).

After running it again a few minutes later it comes to deferrable load 1 (which is also defined with def_total_hours = 0 atm).

2024-02-16 12:24:48,198 - web_server - INFO - Setting up needed data
2024-02-16 12:24:48,232 - web_server - INFO - Retrieve hass get data method initiated...
2024-02-16 12:24:49,729 - web_server - INFO - Retrieving weather forecast data using method = list
2024-02-16 12:24:49,731 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2024-02-16 12:24:49,731 - web_server - INFO - Retrieve hass get data method initiated...
2024-02-16 12:24:52,597 - web_server - INFO -  >> Performing naive MPC optimization...
2024-02-16 12:24:52,597 - web_server - INFO - Performing naive MPC optimization
2024-02-16 12:24:52,601 - web_server - INFO - Perform an iteration of a naive MPC controller
2024-02-16 12:24:52,607 - web_server - DEBUG - Deferrable load 0: Proposed optimization window: 0 --> 0
2024-02-16 12:24:52,607 - web_server - DEBUG - Deferrable load 0: Validated optimization window: 0 --> 0
2024-02-16 12:24:52,608 - web_server - DEBUG - Deferrable load 1: Proposed optimization window: 0 --> 0
2024-02-16 12:24:52,609 - web_server - DEBUG - Deferrable load 1: Validated optimization window: 0 --> 0
2024-02-16 12:24:52,610 - web_server - DEBUG - Deferrable load 2: Proposed optimization window: 0 --> 3
2024-02-16 12:24:52,610 - web_server - DEBUG - Deferrable load 2: Validated optimization window: 0 --> 3
2024-02-16 12:24:52,649 - web_server - INFO - Status: Optimal
2024-02-16 12:24:52,649 - web_server - INFO - Total value of the Cost function = -1200.78
2024-02-16 12:24:57,922 - web_server - INFO - EMHASS server online, serving index.html...

Very strange...
When enforcing an end timestep, the only constraint that is added, is the following:

In your case, this means:
Deferrable load 2 needs to operate before timestep 3, so its consumed energy between step 3 & end of prediction horizon needs to equal zero.
I don't immediately grasp how this would affect the operating scheme of deferrable load 0.
I find it also strange that the optimizer delivers a so-called 'optimal' result, whereas not all constraints were respected (in this case: load 0 was not allowed to run, but it got scheduled anyway.)
I'm not sure if the issues you encounter are specifically related to the start_end timestep feature.
Will need to dive deeper. I'll do it as soon as I can.
@davidusb-geek would you have any idea?

No idea. This will need testing to debug under these same conditions. I haven't found the time to look at this.

I have the same problem. I do schedule my heatpump to run x hours with a 12 hour interval. I have a 30m timestep.
For each timestep I reduce the def_end_timestep and reduce x with the remaining hours to run.
My theory is that it has something to do with the nominal power and when in the window the optimisation runs. When the problem occurs it distributes the total power in fewer timesteps. It seem to sum up to the total.
The problem seem to be intermittent. Someone with script skillz may be able to reproduce this.

Let Deferrable load = x and def_end_timestep = y
If y = 2x I get in trouble. But not always.
Deferrable load 0 = 2000 w
Deferrable load 1 = 1200 w
The log say Optimal in all cases
2024-02-25 10:29:15,384 - web_server - INFO -  >> Performing naive MPC optimization...
2024-02-25 10:29:15,385 - web_server - INFO - Performing naive MPC optimization
2024-02-25 10:29:15,390 - web_server - INFO - Perform an iteration of a naive MPC controller
2024-02-25 10:29:15,393 - web_server - DEBUG - Deferrable load 0: Proposed optimization window: 0 --> y
2024-02-25 10:29:15,393 - web_server - DEBUG - Deferrable load 0: Validated optimization window: 0 --> y
2024-02-25 10:42:01,485 - web_server - DEBUG - Deferrable load 1: Proposed optimization window: 0 --> 1
2024-02-25 10:42:01,485 - web_server - DEBUG - Deferrable load 1: Validated optimization window: 0 --> 1
2024-02-25 10:29:15,452 - web_server - INFO - Status: Optimal
Example:
x = 8 and y = 16

So if y = 2x + 1, the optimization runs fine?
I’m trying to reconstruct the issue

So if y = 2x + 1, the optimization runs fine? I’m trying to reconstruct the issue

Yes I think so. I notised this problem this morning so I'm not really sure. But that is my workaround for now.

Any progress on this?

I don't if @michaelpiron was able to reproduce this?

I think on to something. The problem seems to be related to
set_deferrable_load_single_constant: true
I did a quick test now. Will test more tonight.
When I set it to false it behaves as it should it seems.

set_deferrable_load_single_constant: true ---> problem
set_deferrable_load_single_constant: false ---> no problem
To my understanding the villain is set_deferrable_load_single_constant.
@michaelpiron this maybe can be useful if you want to reproduce the issue?

I don't fully understand the functionality of the list_set_deferrable_load_single_constant ('Set each deferrable load as a constant fixed value variable with just one startup for each optimization task (ex. each 24h)').
I've always worked with "false" as I don't need that functionality for my use cases.
So I might not be the best person to deal with this issue.
As written by @davidusb-geek in the HA forum, this is not really a bug, but just the way this type of optimization works. (https://community.home-assistant.io/t/emhass-an-energy-management-for-home-assistant/338126/2098)
Should the optimization code be changed? perhaps, but at this point I'm not able to say how it should change.

Maybe a solution could be to provide a helper to the optimization to indicate that a deferrable load is currently in operation. But this means that the user has to provide this information. Need to figure out the simplest way to achieve this.

Is there any new feature in an upcoming release which fixes this issue?


Fixed auth

Created auth(gituser, gitpass) and testauth() methods.
To solve the issue (https://github.com/jwass/geojsonio.py/issues/11) I had to create an auth method using github3.py.

Hi @medeirosjoaquim,
Thanks for contributing this!
Any chance you can get the tests running so this can be merged?

I dont know much of python or python tests. Could you point me some tutorial to start?
It's some time now but I remember that the test couldn't find the variables I've created and I couldn't figure it out. Soon I'll have more free time and I would like to finish this cause I find very useful this implementation.

Please fix this!

Please fix this!

it's fixed, I just don't have time to get back to python and make it pass the tests (I'm working with other stuff :(   ), . If you need you can use the version in my branch
https://github.com/medeirosjoaquim/geojsonio.py/tree/fixed-auth-pull-request

Please fix this!

Done!

Hi @medeirosjoaquim,
Thanks for contributing this!
Any chance you can get the tests running so this can be merged?

It's done.

Wow! Great! Thank you @medeirosjoaquim ❤

I have AttributeError: module 'geojsonio' has no attribute 'auth' when doing
myAuth = geojsonio.auth('username','password')

I have AttributeError: module 'geojsonio' has no attribute 'auth' when doing
myAuth = geojsonio.auth('username','password')
Also, the documentation on the fork is kind of misleading since it still suggest doing pip install geojsonio... Additionally I got super confused because seems from the original source code, I should put a gitauth.txt?

It's not documentation of the fork. In fact, it's not a fork. I forked for those who wanted to try the fix because it's a PR and it's waiting for the maintainer to merge. I did the fix almos a year ago and only a few months ago I finished the tests. If in some weekend to come I have the time, I will try to put it in order and then I reply here. Thanks for your comment!

I have AttributeError: module 'geojsonio' has no attribute 'auth' when doing
myAuth = geojsonio.auth('username','password')
Also, the documentation on the fork is kind of misleading since it still suggest doing pip install geojsonio... Additionally I got super confused because seems from the original source code, I should put a gitauth.txt?

I just made a little clean up in the read-me file, and tested again the fix. Works ok.
You will not install it.
You must have the gitauth.txt in the same folder as the geojsonio.py and then
just run the script (python geojsonio.py yourdataset.geojson)...
Hope it helps.
you can check my test here : https://gist.github.com/medeirosjoaquim/ba7df592d29f1e358f3f249bb1cbbce4

Can this be merged please? This issue is blocking the adoption of this library

Can this be merged please? This issue is blocking the adoption of this library

in the meanwhile, you can use the fork with the fix. You don't need to pip install, just get the files, put your credentials and run.
https://github.com/medeirosjoaquim/PyGeoJson

Hi @medeirosjoaquim, I can't get your solution to work. I keep getting the following:
Bizs-MacBook-Pro:PyGeoJson-master bizyoder$ python geojsonio.py africagrid20170906existing.geojson
Traceback (most recent call last):
File "geojsonio.py", line 236, in
main()
File "geojsonio.py", line 229, in main
url = make_url(contents, args.domain)
File "geojsonio.py", line 106, in make_url
gist = _make_gist(contents)
File "geojsonio.py", line 186, in _make_gist
gist = gh.create_gist(description, files)
File "/Users/bizyoder/anaconda3/lib/python3.6/site-packages/github3/github.py", line 149, in create_gist
json = self._json(self._post(url, data=new_gist), 201)
File "/Users/bizyoder/anaconda3/lib/python3.6/site-packages/github3/models.py", line 90, in _json
if self._boolean(response, status_code, 404) and response.content:
File "/Users/bizyoder/anaconda3/lib/python3.6/site-packages/github3/models.py", line 108, in _boolean
raise GitHubError(response)
github3.models.GitHubError: 502 Server Error
Any thoughts?

github3.models.GitHubError: 502 Server Error

Hi! How are you? I will take a look. But, since it's a 502 error, it looks like a Github's downtime.
I would be nice if the fix were merged to work on this.

Hi @medeirosjoaquim, I can't get your solution to work. I keep getting the following:
Bizs-MacBook-Pro:PyGeoJson-master bizyoder$ python geojsonio.py africagrid20170906existing.geojson
Traceback (most recent call last):
File "geojsonio.py", line 236, in
main()
File "geojsonio.py", line 229, in main
url = make_url(contents, args.domain)
File "geojsonio.py", line 106, in make_url
gist = _make_gist(contents)
File "geojsonio.py", line 186, in _make_gist
gist = gh.create_gist(description, files)
File "/Users/bizyoder/anaconda3/lib/python3.6/site-packages/github3/github.py", line 149, in create_gist
json = self._json(self._post(url, data=new_gist), 201)
File "/Users/bizyoder/anaconda3/lib/python3.6/site-packages/github3/models.py", line 90, in _json
if self._boolean(response, status_code, 404) and response.content:
File "/Users/bizyoder/anaconda3/lib/python3.6/site-packages/github3/models.py", line 108, in _boolean
raise GitHubError(response)
github3.models.GitHubError: 502 Server Error
Any thoughts?

Hi! I took a look,. it still works fine here. I've noticed that you're using python3. This repo was set first with python 2.7 . Would you mind trying to run it with python 2.7 and then tell me if it works? Perhaps
in the python 3 lib for github api the URLs are different and, because of that, it's receiving this 502 error.

UPDATE:
Now you should use an ACCESS TOKEN. It's cleaner and safer. just go to my fork and follow the instructions:
https://github.com/medeirosjoaquim/PyGeoJson

a suggestion, I don't know if Git updated Auth tokens, but I ran into some trouble last night:
The new path is:
Settings > Developer settings > Personal access tokens
Select scopes:
gist - Create gists (checked)

Hi, still having this issue


Strategy to handle events from third party JS libs

Can you please let me know if you have any plans to consider providing support for publish-listen events from third party libraries ?
I am interested mostly in Plotly.js.. Dash is an obvious choice, but fell in love with the realtime abilities of Wave.

Related to #297


Block: Activate and prepare Azure DevOps

As someone setting up the project infrastructure, I want to have a detailed guide on how to activate and prepare Azure DevOps so that I can get this done fast and without errors.
Detail requisites for cases where Azure DevOps is provided.

Document completed for Practical guide to activate and prepare Azure DevOps, It is under review.

According to guidelines had few changes and created pull request.


s3control put_public_access_block additional permissions update

Describe the issue
s3control put_public_access_block requires additional IAM permissions from S3 service.
The documentation states that:
"For this operation, users must have the s3:PutBucketPublicAccessBlock permission"
When I tried using it - it seems that this is not the correct s3 permission needed.
The correct one seems to be:
s3:PutAccountPublicAccessBlock
I've tested and validated it.
Links
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3control.html#S3Control.Client.put_public_access_block

Hi @severel, thanks for bringing this to our attention!
I was able to reproduce and can confirm that the IAM permission needed for put_public_access_block is s3:PutAccountPublicAccessBlock.
Since this also affects the documentation of other SDKs, I created an issue here in our cross-SDK repository: here. I reached out to the team regarding this and will update that issue accordingly.
As I'm going to close this, please refer to that issue in aws/aws-sdk repo going forward for updates.


SEO & quality improvements for best practices & checklists

OPS Build status updates of commit 7e5206f:
:warning: Validation status: warnings

File
Status
Preview URL
Details

toc.json
:warning:Warning

Details

building-blocks/extending-templates/toc.json
:warning:Warning

Details

bread/toc.json
:warning:Warning

Details

docs/best-practices/api-design.md
:white_check_mark:Succeeded
View

docs/best-practices/api-implementation.md
:white_check_mark:Succeeded
View

docs/best-practices/auto-scaling.md
:white_check_mark:Succeeded
View

docs/best-practices/background-jobs.md
:white_check_mark:Succeeded
View

docs/best-practices/caching.md
:white_check_mark:Succeeded
View

docs/best-practices/cdn.md
:white_check_mark:Succeeded
View

docs/best-practices/data-partitioning-strategies.md
:white_check_mark:Succeeded
View

docs/best-practices/data-partitioning.md
:white_check_mark:Succeeded
View

docs/best-practices/monitoring.md
:white_check_mark:Succeeded
View

docs/best-practices/naming-conventions.md
:white_check_mark:Succeeded
View

docs/best-practices/retry-service-specific.md
:white_check_mark:Succeeded
View

docs/best-practices/transient-faults.md
:white_check_mark:Succeeded
View

docs/checklist/availability.md
:white_check_mark:Succeeded
View

docs/checklist/dev-ops.md
:white_check_mark:Succeeded
View

docs/checklist/index.md
:white_check_mark:Succeeded
View

docs/checklist/resiliency-per-service.md
:white_check_mark:Succeeded
View

docs/checklist/resiliency.md
:white_check_mark:Succeeded
View

docs/checklist/scalability.md
:white_check_mark:Succeeded
View

toc.json

[Warning] Error happen when converting toc.json to Pdf. Details: Could not find file 'T:\3l5z\toc.json'.

building-blocks/extending-templates/toc.json

[Warning] Error happen when converting building-blocks/extending-templates/toc.json to Pdf. Details: Could not find a part of the path 'T:\3l5z\building-blocks\extending-templates\toc.json'.

bread/toc.json

[Warning] Error happen when converting bread/toc.json to Pdf. Details: Could not find a part of the path 'T:\3l5z\bread\toc.json'.

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

OPS Build status updates of commit e33a247:
:warning: Validation status: warnings

File
Status
Preview URL
Details

bread/toc.json
:warning:Warning

Details

building-blocks/extending-templates/toc.json
:warning:Warning

Details

toc.json
:warning:Warning

Details

docs/best-practices/api-design.md
:white_check_mark:Succeeded
View

docs/best-practices/api-implementation.md
:white_check_mark:Succeeded
View

docs/best-practices/auto-scaling.md
:white_check_mark:Succeeded
View

docs/best-practices/background-jobs.md
:white_check_mark:Succeeded
View

docs/best-practices/caching.md
:white_check_mark:Succeeded
View

docs/best-practices/cdn.md
:white_check_mark:Succeeded
View

docs/best-practices/data-partitioning-strategies.md
:white_check_mark:Succeeded
View

docs/best-practices/data-partitioning.md
:white_check_mark:Succeeded
View

docs/best-practices/monitoring.md
:white_check_mark:Succeeded
View

docs/best-practices/naming-conventions.md
:white_check_mark:Succeeded
View

docs/best-practices/retry-service-specific.md
:white_check_mark:Succeeded
View

docs/best-practices/transient-faults.md
:white_check_mark:Succeeded
View

docs/checklist/availability.md
:white_check_mark:Succeeded
View

docs/checklist/dev-ops.md
:white_check_mark:Succeeded
View

docs/checklist/index.md
:white_check_mark:Succeeded
View

docs/checklist/resiliency-per-service.md
:white_check_mark:Succeeded
View

docs/checklist/resiliency.md
:white_check_mark:Succeeded
View

docs/checklist/scalability.md
:white_check_mark:Succeeded
View

bread/toc.json

[Warning] Error happen when converting bread/toc.json to Pdf. Details: Could not find a part of the path 'T:\frt4\bread\toc.json'.

building-blocks/extending-templates/toc.json

[Warning] Error happen when converting building-blocks/extending-templates/toc.json to Pdf. Details: Could not find a part of the path 'T:\frt4\building-blocks\extending-templates\toc.json'.

toc.json

[Warning] Error happen when converting toc.json to Pdf. Details: Could not find file 'T:\frt4\toc.json'.

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


Create part from encrypted object

Test for https://github.com/ceph/ceph/pull/38541
Signed-off-by: Ilsoo Byun<EMAIL_ADDRESS>

@IlsooByun should this test do some final verification at the end?

@mattbenjamin I added the verification code.


BETYdb metadata upload package

@KristinaRiemer I have updated the metadata upload package. Can you check if you can build/install the package when you have time? Let me know of any suggestions that you have to improve the package.

I was able to successfully install this package by running R CMD build betydbMetadata and R CMD INSTALL betydbMetadata_1.0.0.tar.gz from the command line after pulling this repo down locally. I tried doing devtools::install_github('az-digitialag/betydbtools#2') and a few similar variations but wasn't able to get that approach to work.

Try install_github('kimberlyh66/betydbtools@metadata_package', subdir = 'metadata')

BETYdb-YABA now has a metadata Python client. This was a great prototype but is now obsolete


Add support for options.headers in m.request + tests/docs

I also had to edit the mocks accordingly, so I could inspect the headers set.

This is a followup to #1464, targeting the rewrite.


Isolate private ksvc using different envoy listener ports

Add support for multiple private ingresses backed by multiple envoy listeners (each with a unique port).
This is an experimental Kourier-only feature guarded by a flag in config-kourier:
# Control the desired level of incoming traffic isolation.
#
# When set to "none" (the default), all incoming traffic flows through
# a shared ingress and listeners.
#
# When set to "port", incoming traffic is isolated by using different
# listener ports.
#
# NOTE: This flag is in an alpha state.
traffic-isolation: none

The selection of which envoy listener to use is done at the namespace-level. The traffic of all internal services within the same namespace is redirected to the same envoy listener. Here are the two annotations for mapping a namespace to a particular envoy listener:
// ListenerAnnotationKey is the annotation key for grouping/isolating ingresses.
ListenerAnnotationKey = "kourier.knative.dev/listener"

// ListenerPortAnnotationKey is the annotation key for assigning the ingress to a particular
// envoy listener port. Only applicable to internal services.
ListenerPortAnnotationKey = "kourier.knative.dev/listener-port"

The allocation/assignment of a namespace to an envoy listener is an orthogonal concern and not address in this PR. This can be done by manually editing namespace annotations or automatically by a Tenant controller.
Each envoy listener is exposed using a k8s service (e.g kourier-internal). This PR does not automatically create/delete these services since this concern is also related to tenant management.
Additional context:

Feature Track
PoC showing how Network Policies with this PR can be used to isolate tenants.

Changes

Add a new configuration parameter for enabling port-level internal services isolation
Generate an envoy configuration containing multiple internal services

/kind enhancement

Fixes #

Release Note

Add a new configuration parameter for enabling port-level internal services isolation. See documentation for more details.

Docs

Remaining tasks:

[ ] unit tests/e2e tests
[X] Discuss how to enable/disable this feature - Added traffic-isolation field in config-kourier
[X] Specify/check for port availability - Not in this PR
[X] Generate private Kourier services - Not in this PR
[X] Validate annotations -  Not in this PR
[ ] internal service encryption - Can we do this in a separate PR?
[ ] Documentation

/cc @evankanderson @nak3 @rhuss

@lionelvillard hi, could you add a link to the ft doc in the description?

@lionelvillard hi, could you add a link to the ft doc in the description?

done.

/hold

/unhold

@lionelvillard tests seem passing will take a look at the PR and review it. /cc @nak3 pls review too.

@lionelvillard any plan to document policy setup. For example there is an image with all the policies but it would be nice to elaborate on it. Example what is the minimum setup per scenario.  Running the POC locally I see for example in order to make ping-pong work in the same tenant and ns and to be able to access it externally to the cluster I need to set:
$ oc get networkpolicy -n tenant1-ns1
NAME                               POD-SELECTOR   AGE
allow-all-from-to-tenant-ns        <none>         55s
allow-all-in-ns                    <none>         3m3s
allow-serving-to-tenant            <none>         16s
allow-tenant-ingress-from-tenant   <none>         9m13s

$ oc logs ping-00001-deployment-6574dcc4c8-k45vp  -n tenant1-ns1 -c user-container | tail
2022/06/24 17:25:51 ping succeeded
2022/06/24 17:25:53 ping succeeded
2022/06/24 17:25:55 ping succeeded
2022/06/24 17:25:57 ping succeeded
2022/06/24 17:25:59 ping succeeded
2022/06/24 17:26:01 ping succeeded
2022/06/24 17:26:03 ping succeeded
2022/06/24 17:26:05 ping succeeded
2022/06/24 17:26:07 ping succeeded
2022/06/24 17:26:09 ping succeeded
$ oc logs ping-00001-deployment-6574dcc4c8-k45vp  -n tenant1-ns1 -c user-container | tai^C
[stavros@easy mknative]$ oc logs pong-00001-deployment-79d87c75b-7n6t6    -n tenant1-ns1 -c user-container | tail
2022/06/24 17:25:59 pong
2022/06/24 17:26:01 pong
2022/06/24 17:26:03 pong
2022/06/24 17:26:05 pong
2022/06/24 17:26:07 pong
2022/06/24 17:26:09 pong
2022/06/24 17:26:11 pong
2022/06/24 17:26:13 pong
2022/06/24 17:26:15 pong
2022/06/24 17:26:17 pong
$ curl -v  http://ping.tenant1-ns<IP_ADDRESS>.1.sslip.io
*   Trying <IP_ADDRESS>:80...
* Connected to ping.tenant1-ns<IP_ADDRESS>.1.sslip.io (<IP_ADDRESS>) port 80 (#0)
> GET / HTTP/1.1
> Host: ping.tenant1-ns<IP_ADDRESS>.1.sslip.io
> User-Agent: curl/7.71.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-length: 0
< date: Fri, 24 Jun 2022 17:26:23 GMT
< x-envoy-upstream-service-time: 3
< server: envoy
<
* Connection #0 to host ping.tenant1-ns<IP_ADDRESS>.1.sslip.io left intact

@skonto

any plan to document policy setup sooner?

yes I can work on that. To make sure we are on the same page, you are referring to adding a new page in knative.dev describing a minimum network policy setup for multi-tenant environment?

For example there is an image with all the policies but it would be nice to elaborate on it

Sorry I'm not parsing this. What image?

@skonto

For example there is an image with all the policies but it would be nice to elaborate on it

Sorry I'm not parsing this. What image?

Picture, not OCI artifact.

@evankanderson :P

/test integration-tests_net-kourier_main

@skonto @nak3 @evankanderson I improved a bit the PoC readme. It's now quite clear (I hope) what cluster admins need to do create/delete a tenant.
Remaining tasks are tracked by these issues:

Documentation: https://github.com/knative/docs/issues/5079
e2e tests: https://github.com/knative-sandbox/net-kourier/issues/870
Internal encryption: https://github.com/knative-sandbox/net-kourier/issues/871

Are you ok with merging this PR before 1.6 is cut next week?

I have two comments about test and annotations.
1. testing
I see that there is a plan to add e2e test https://github.com/knative-sandbox/net-kourier/issues/870 but we should try to run serving's e2e to make sure if this does not break anything like this https://github.com/knative/serving/pull/13058, should't we?
2. annotations in the user namesspace
I think that we should change the following annotations to a label like adimin.knative.dev/tenant-name to stop exposing the internal information in the user's namespacee.

kourier.knative.dev/listener: the envoy listener suffix to use
kourier.knative.dev/listener-port: the envoy listener port

There are some reasons for that such as:

It increases risks to atack/invade Kourier Gateway by cheating the exposed annotation.
For troubleshooting/admin friendly, we should use label or at least one annotation.
The general label instead the Kourier specific annotation allows other ingress to use the same label even if they apply the different implementation.

I guess it is possible by introducing a ConfigMap or new CRD for it.

Thanks @nak3 for your review!

I have two comments about test and annotations.
1. testing
I see that there is a plan to add e2e test #870 but we should try to run serving's e2e to make sure if this does not break anything like this knative/serving#13058, should't we?

I'm on it!

2. annotations in the user namesspace
I think that we should change the following annotations to a label like adimin.knative.dev/tenant-name to stop exposing the internal information in the user's namespacee.

kourier.knative.dev/listener: the envoy listener suffix to use
kourier.knative.dev/listener-port: the envoy listener port

There are some reasons for that such as:

It makes some risks to attack/invade Kourier Gateway by cheating the exposed annotation.

How using a label different from using an annotation? Is it possible to restrict access to labels and not annotations?

For troubleshooting/admin friendly, we should use label or at least one annotation.
The general label instead of the Kourier specific annotation allows other ingress to use the same label even if they apply the different implementation.

My only concern is that there is no concept of tenant in Knative (AFAIK).

I guess it is possible by introducing a ConfigMap or new CRD for it.

How using a label different from using an annotation? Is it possible to restrict access to labels and not annotations?

Changing annotation to label is trivial thing. I wanted to suggest we should stop exposing the internal information by changing adimin.knative.dev/tenant-name or some general name.
The annotation with adimin.knative.dev/tenant-name is also fine, but admin can list the namespace with kubectl  get ns  -l adimin.knative.dev/tenant-name=my-tenant if we use the label instead of annotation, that's the only thing for annotation vs label.

@nak3 What about kourier.knative.dev/listener-port. Should we keep it the same or rename to admin.knative.dev/tenant-port?

Knative serving test PR: https://github.com/knative/serving/pull/13094

Umm... I was thinking that we can introduce a new CRD or CM like this. Then, we can completely disallow users to change the listener name and port.
               +--------------------+
               | kourier controller |
               +--------------------+
          	         |
                         | read listener name & port.
 new CRD (or CM)         |
+-------------------+    |
|tenant:            | <--+
|  name: my-tenant  |
|    lister: ten-a  |
|    port: 8081     |
|    members:       |                    namespace (Kourier does not need to touch.)
|      - namespaceA |                  +------------+
|      - namespaceB |                  | namespaceA |
+-------------------+                  +------------+

It is not good idea?

Umm... I was thinking that we can introduce a new CRD or CM like this. Then, we can completely disallow users to change the listener name and port.

@rhuss @skonto @evankanderson WDYT?

The most important point for me is that it is unsafe to add the kourier.knative.dev/listener{-port} label, which is editable, in users' namespace.
I think users can invade other tenants, other Envoy listener, etc.
(On the other hand, using a new CRD or CM is safe, one place configuration, admin friendly etc :sunglasses: .)

I think it is safer if we rely on some crd or cm but we dont have to do it now imho. We can get this in for 1.6 and update the design in consequent PR if it is not much work. This feature is hidden behind a setting anyway and it is still Alpha. Wdyth?

Sorry, I've been wrestling with gateway api and an internal re-org. I was 👍 before the latest changes, but I'll give it another look before approving.
Thanks for doing this! I'm hopeful that we might be able to follow the same approach with the Gateway API fairly easily, since that allows explicitly specifying a listener port.

I suspect that most tenants will not have access to edit the labels on their namespace; this would also allow bypassing any number of other properties on the namespace. (Similarly, they will not be able to create new namespaces with labels that they want.)

I'd like to understand what the risk of exposing the listener port assignments is, if NetworkPolicy is set up to disallow access from all but one namespace to that port. The public port numbers of a service are exposed via SRV records which aren't restricted within the cluster:
# dig srv mysql-db-mysql-master.wordpress.svc.cluster.local @<IP_ADDRESS>
...
;; ANSWER SECTION:
mysql-db-mysql-master.wordpress.svc.cluster.local. 30 IN SRV 0 50 3306 mysql-db-mysql-master.wordpress.svc.cluster.local.
mysql-db-mysql-master.wordpress.svc.cluster.local. 30 IN SRV 0 50 8080 mysql-db-mysql-master.wordpress.svc.cluster.local.

;; ADDITIONAL SECTION:
mysql-db-mysql-master.wordpress.svc.cluster.local. 30 IN A <IP_ADDRESS>

If an attacker can enumerate the ports (and there are a max of 32k such ports, they can simply probe them all to determine the correct port. On a local network with 1ms RTT, this would take about 90 seconds without parallelism, so it's not much of a defense on its own without NetworkPolicy to block the packets.

Hmm... Not only other tenant, but also it is not possible to hijack any Envoy configuration like Envoy's admin interface? I guess that it is impossible with the default static configuration and cannot come up with how exactly we can attack. So, can I trust you?

It depends on how envoy is configured, but it doesn't seem like this opens new attack surface there.

Can we run https://github.com/knative/serving/pull/13094 also with the feature enabled? We need to have an e2e test somewhere, could you add one here at least. Also is there a plan for the tenant controller here? Is there a list of the remaining tasks? In the issue description all tasks seem done. Let's add the tenant controller idea and the webhook validation too pls.
Also since we plan to iterate let's add the question label vs annotation there.
@lionelvillard could you do this before merging?

@skonto

Can we run knative/serving#13094 also with the feature enabled?

Yes. https://github.com/knative/serving/pull/13112

We need to have an e2e test somewhere, could you add one here at least.

I'm looking at this now, it seems easy enough but due to the time constraint, I would rather do this in a separate PR.
IMO, as soon as both https://github.com/knative/serving/pull/13094 and https://github.com/knative/serving/pull/13112 are green, it should be safe to merge this PR.

Also is there a plan for the tenant controller here?

No plans. This is a much bigger topic!

Let's create a list of the remaining tasks pls (tracking issue). In this description of this issue all tasks seem done. Let's add the tenant controller idea and the webhook validation too. Also since we plan to iterate on this let's add the question about label vs annotation there. @lionelvillard could you do this before merging?

Yes I can create an Github issue tracking all remaining tasks

/hold I need to verify the PoC is still working.

/unhold PoC is working.

@evankanderson @skonto @nak3 @rhuss Please can you take another look? Since the last round of reviews I:

removed the listener label. The Kourier services exposing the envoy listeners are now called kourier-isolation-<listener-port>
remove "none" in the configmap in favor of the empty string value
opened various issues to track the remaining tasks

I'm tracking https://github.com/knative/serving/pull/13094 and https://github.com/knative/serving/pull/13112. So far it looks good.
I'm working on e2e tests (in a separate PR). I don't know if I will have time to finish before the 1.6 release (which is tomorrow).

/unhold

@evankanderson can you please TAL? All tests are green (including the Serving ones).


Remove inaccurate and non-performant EDS metrics

name                           old time/op    new time/op    delta
EndpointGeneration/100/1-8       94.3µs ± 7%    89.7µs ±17%     ~     (p=0.635 n=3+3)
EndpointGeneration/1000/1-8       855µs ± 2%     822µs ± 3%     ~     (p=0.125 n=3+3)
EndpointGeneration/10000/1-8     7.62ms ± 6%    7.14ms ± 3%     ~     (p=0.162 n=3+3)
EndpointGeneration/100/100-8      714µs ± 3%     665µs ± 7%     ~     (p=0.156 n=3+3)
EndpointGeneration/1000/100-8    5.17ms ± 7%    6.45ms ±20%     ~     (p=0.193 n=3+3)

name                           old alloc/op   new alloc/op   delta
EndpointGeneration/100/1-8       6.59kB ± 0%    4.69kB ± 0%  -28.90%  (p=0.000 n=3+3)
EndpointGeneration/1000/1-8      37.9kB ± 0%    35.9kB ± 0%   -5.29%  (p=0.000 n=3+3)
EndpointGeneration/10000/1-8      312kB ± 5%     304kB ± 3%     ~     (p=0.434 n=3+3)
EndpointGeneration/100/100-8     89.9kB ± 0%    78.5kB ± 0%  -12.76%  (p=0.000 n=3+3)
EndpointGeneration/1000/100-8     569kB ± 4%     533kB ± 9%     ~     (p=0.310 n=3+3)

name                           old allocs/op  new allocs/op  delta
EndpointGeneration/100/1-8         60.7 ± 1%      21.0 ± 0%  -65.38%  (p=0.000 n=3+3)
EndpointGeneration/1000/1-8        60.7 ± 1%      21.0 ± 0%  -65.38%  (p=0.000 n=3+3)
EndpointGeneration/10000/1-8       63.3 ± 3%      29.0 ± 0%  -54.21%  (p=0.001 n=3+3)
EndpointGeneration/100/100-8      1.20k ± 0%     0.96k ± 0%     ~     (zero variance)
EndpointGeneration/1000/100-8     4.38k ± 6%     3.90k ±14%     ~     (p=0.225 n=3+3)

These metrics are both extremely costly and are actually not accurate. The metric tries to give a global view of the number of endpoints in a Service - this is wrong by definition, as each proxy will have a different view of EDS based on sidecar/network.
https://github.com/istio/istio/issues/25154

@howardjohn: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

lint_istio
593bcaad0825ef0271f62642542476d141172223
link
/test lint_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Is this the sort of thing that should have a release notes yaml attached to it now?

@howardjohn: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

lint_istio
593bcaad0825ef0271f62642542476d141172223
link
/test lint_istio

integ-pilot-k8s-tests_istio
593bcaad0825ef0271f62642542476d141172223
link
/test integ-pilot-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Is this the sort of thing that should have a release notes yaml attached to it now?

Yes, good point. Just adde one

@howardjohn: The following test failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

lint_istio
627ce981c057e17f3f2ee040fda6727a022cfb2d
link
/test lint_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@howardjohn: The following tests failed, say /retest to rerun all failed tests:

Test name
Commit
Details
Rerun command

lint_istio
627ce981c057e17f3f2ee040fda6727a022cfb2d
link
/test lint_istio

integ-pilot-k8s-tests_istio
627ce981c057e17f3f2ee040fda6727a022cfb2d
link
/test integ-pilot-k8s-tests_istio

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest


dot command fails if spaces in path

I am running ubuntu in a bash command prompt.
I modified line 139 of index.js by adding double quotes. There may be a more general way to quote the parameters to cover more exotic problems, but for my purposes double-quoting was enough.
const render = shell.exec(`dot -Tsvg -o "${svgFile}" "${dotFile}"`);

I have also experienced this, also had a space in one of the parent folders of the project.
After executing:
webpack-stats-graph

I got:
Writing files to statsgraph
Error: dot: option -  unrecognized

Usage: dot [-Vv?] [-(GNE)name=val] [-(KTlso)<val>] <dot files>
. . .

Changing node_modules/webpack-stats-graph/index.js line 139 from
const render = shell.exec(`dot -Tsvg -o ${svgFile} ${dotFile}`);

to
const render = shell.exec(`dot -Tsvg -o "${svgFile}" "${dotFile}"`);

also fixed the issue for me.


Activate proguard on project because methods limit

Project + dependencies are passing the methods limit of dex

I think your dex file is reached to its maximum method count. Try to use multiDexEnabled true
in your deafault config or try to minmize your playservices from your dependencies. Use only required > packages. Try changing your version or exclude unwanted packages

Already? We need discover which is the dependency that are pushing this limit.

Avoid the 64K limit
Before configuring your app to enable use of 64K or more method references, you should take steps to reduce the total number of references called by your app code, including methods defined by your app code or included libraries. The following strategies can help you avoid hitting the DEX reference limit:
Review your app's direct and transitive dependencies - Ensure any large library dependency you include in your app is used in a manner that outweighs the amount of code being added to the app. A common anti-pattern is to include a very large library because a few utility methods were useful. Reducing your app code dependencies can often help you avoid the DEX reference limit.
Remove unused code with ProGuard - Enable code shrinking to run ProGuard for your release builds. Enabling shrinking ensures you are not shipping unused code with your APKs.

Using these techniques might help you avoid the need to enable multidex in your app while also decreasing the overall size of your APK.

This issue don't need to be implemented on this sample project, only on a production ready one (this issue was be copied to production related project)


[Table] [TableCell] align crash - capitalize(string) expects a string argument

[x] The issue is present in the latest release.
[x] I have searched the issues of this repository and believe that this is not a duplicate.

Current Behavior 😯
If you pass boolean to prop align whole application crashes.
exp:
<TableCell align={false}>Data</TableCell>

Error: Material-UI: capitalize(string) expects a string argument.
Expected Behavior 🤔
Some kind of warning and fallback to default prop would be less aggressive options.
Environment 🌎

Tech
Version

Material-UI
v4.10.2

React
16.13.1

Browser
Chrome 84.0.4147.125 (Official Build) (64-bit)

So far, our policy is to let the component crash as long as there is a console.error in the console that explains how to easily resolve the issue.

Duplicate of #17480


(GH-2376) Warn about unknown configuration options

This updates the config validator to issue warnings for unknown
config options. When the validator encounters an unknown option, it
stores a warning indicating where the unknown option is specified, which
is later issued by the logger. Options are only reported as unknown if
the schema for the option defines the :properties key but not the
:additionalProperties key.
!feature

Warn about unknown configuration options
(#2376)
Bolt now issues a warning when it detects an unknown configuration
option. The warning will indicate where the configuration option is
located.

Would it also make sense to start validating the inventory here? We already validate inventory-config from the defaults file.

Hm, this might also be a separate issue but I noticed the message if users have misconfigured log is a little misleading:
lucy ~/githubs/bolt/project ((f69845b...)) $ be bolt command run whoami -t localhost
Invalid configuration at /home/lucy/githubs/bolt/project/bolt-project.yaml:
  Value at 'log.foo' must be one of disable

We might need to have a special handler for that.

@lucywyman
What config are you using for log? If it's set as a string other than disable, the validator will validate it according to the string definition, not the hash one. That will apply to any config option that has can be more than one type, and is a little harder to provide a concise error message for.
This class can't really be used to validate the inventory file, since the schema would need to be recursive since groups can be nested... which isn't really feasible with how the schemas are set up in Ruby. We can definitely make an inventory validator, though, and was something I was thinking about earlier today! :)

Right I understand how the validator is working, but in the case of log (and anything else that takes multiple types) the message is misleading because it doesn't instruct users that they need to provide a hash. For example it's easy to see someone writing:
log:
  mylog: debug

And being confused by the error message. I think the message should read something like:
Invalid configuration at /home/lucy/githubs/bolt/project/bolt-project.yaml:
  Value at 'log.mylog' must be either the String 'disable' or a Hash

Updated the validator to append alternate types to an error message. Currently this is only used by #validate_string, but I think that should suffice for now since none of the other multi-type options will lead to potential confusion. For example, if you're getting an error for modulepath it's not really helpful for us to say "or it must be a String" since someone using an array probably meant to use an array.


JIT provision users to cloud service providers is not working as expected

Moved from https://wso2.org/jira/browse/IDENTITY-7241
Steps to reproduce

Introduce salesforce as a service provider (salesforce-sp) to the WSO2 Identity Server
Introduce salesforce as a provisioning identity provider (salesforce-idp) to the WSO2 Identity Server, and configure the Salesforce provisioning connector.
Introduce the Facebook as an identity provider to the WSO2 Identity Server running and enable JIT provisioning.
Under the salesforce-sp service provider configuration, under local and outbound authentication configuration, select facebook as a federated identity provider. This means, a user who wants to login salesforce-sp, will be redirected to the Facebook identity provider for authentication.
Under the salesforce-sp service provider configuration, under outbound provisioning configuration, select salesforce-idp as a provisioning identity provider.

Facebook is provision users to wso2 IS(Jit provisioning works fine), but not provisioning to SF.
But if you manually trigger using SCIM user will provision to SF
curl -v -k --header "Content-Type:application/json" --user emailk@gmail.com:dakshika --data '{"schemas": ["urn:scim:schemas:core:1.0"],"userName":"random@wso2.com","password":"test25","name":

{"familyName":"Gnaniah"}
,"emails": ["random@wso2.com"],"entitlements": [

{"value":"00e90000011aV2o","display":"ChatterFreeUser"}
]}' https://localhost:9443/wso2/scim/Users

This issue is being closed due to extended inactivity. Please feel free to reopen it if further attention is needed. Thank you for helping us keep the issue list relevant and focused!


Error in validate function

I was use this lib in version 5.19.6171.28316, for validate json with jsonschema phisical archieve.
My code is:
var schema = NJsonSchema.JsonSchema4.FromJson(schemaData);       var errors = schema.Validate(content);
and it is OK, but I update version for newest and change the code for:
var schema = JsonSchema4.FromData(schemaData);        var errors = schema.Validate(content);
But I receive one error usng the same schema and archieve.

You have to use FromJsonAsync, FromData creates a schema for the given data, it is not read as schema but as sample data...

thanks, but when I try
var schema = await JsonSchema4.FromJsonAsync(schemaData);
schemaData = schema.ToJson();
var errors = schema.Validate();

schema dont have ToJson and Validate methods

My mistakes, its OK