[SPARK-26750]Estimate memory overhead with multi-cores

What changes were proposed in this pull request?
Currently, spark esitmate the memory overhead without taking multi-cores into account, sometimes, it might cause direct memory oom, or container killed by yarn for exceeding requested physical memory.
Since the memory overhead(mainly the direct memory used by spark execution/storage and some related jvm native memory, for instance, the thread stacks, GC data etc.) is related to the executor's core number. so maybe we can improve this estimation by taking the core number into account.
How was this patch tested?
NA
Please review http://spark.apache.org/contributing.html before opening a pull request.

I agree that we may can improve this default overhead, but I was curious about how this default 0.1 is generated, by experimatal tests? is there any docs about how the experiment is done? does it taking the number of cores into account when do these tests?

@tgravescs Thanks, I see now.

You have this SHARED_MEMORY_OVERHEAD_MIN, did that number come from somewhere specific?

No, it's just a rough guess.

The 0.1 is also mostly a guess and has worked OK in practice. It could be adjusted if people were encountering the limit regularly, but we don't want it too high.


[Question] Maintenance page

Hi.
How can I implement a "maintenance" page?
When the page is updated, the "nuxt-auth-sanctum" plugin automatically sends a request to "endpoints.user". The answer comes with a 503 code, which I intercept with "interceptors".
But no matter what I do there, I go to the "login" page all the time. Even if the user was already logged in at that moment. I would like to display the "maintenance" page either at the current url or at a separate url.
export default defineAppConfig({
    sanctum: {
        tokenStorage: localTokenStorage,
        interceptors: {
            onResponse: async (
                app: NuxtApp,
                ctx: FetchContext,
                logger: ConsolaInstance
            ) => {
                if (ctx.response?.status == 503) {
                    // Not work
                    navigateTo('/maintenance')

                    // It also doesn't work
                    throw createError({ statusCode: 404, statusMessage: 'Unavailable' })
                }
            },
        },
    },
})

Hey @webmaxx, onResponse middleware provided by ofetch is not executed when it comes to errors, it has a different onResponseError interceptor but it is not exposed in my module.
So, in your scenario you want to redirect users to maintenance page whenever they get 503 response from the API, right?
My module redirects users only when it comes to 401 errors or when the sanctum:auth middleware is used on the target page (in case of 503 there is no user anymore). So, if you open a page that is not covered with sanctum:auth middleware you should not be redirected to login page even if you get 503 response. Is it correct that you are testing it on the page that uses sanctum:auth middleware?

Hey @webmaxx, onResponse middleware provided by ofetch is not executed when it comes to errors, it has a different onResponseError interceptor but it is not exposed in my module.

This condition is successfully fulfilled in the interceptor:
if (ctx.response?.status == 503) {
    console.log('unavailable')
}

So, in your scenario you want to redirect users to maintenance page whenever they get 503 response from the API, right?

Yes.

My module redirects users only when it comes to 401 errors or when the sanctum:auth middleware is used on the target page (in case of 503 there is no user anymore). So, if you open a page that is not covered with sanctum:auth middleware you should not be redirected to login page even if you get 503 response. Is it correct that you are testing it on the page that uses sanctum:auth middleware?

I use globalMiddleware:
sanctum: {
    ...
    globalMiddleware: {
        enabled: true,
    },
}

pages/maintenance.vue

<template>

        Maintaince mode

</template>

However, it is unclear why there is no user when responding with the code 503. A token is used for authorization, which is stored in the local storage. If maintenance mode is disabled on the server, then everything starts working again and the user exists.
php artisan up

This condition is successfully fulfilled in the interceptor

Oh, that's good then. I'll see what changes you may need to do there.

However, it is unclear why there is no user when responding with the code 503. A token is used for authorization, which is stored in the local storage. If maintenance mode is disabled on the server, then everything starts working again and the user exists.

Token in the storage doesn't mean that user exists until API returns a user as a response to identity endpoint, then the result is stored in user state (which is available as useSanctumUser). So the middleware uses that to determine if the redirect should be applied or not. The token itself might be expired or compromised, so we need a confirmation from the API.
I will try to reproduce the behavior you described to make sure that there are no bugs and I'll get back to you with either a solution or a bug-fix.

Alternatively, can add a hook in which could define any actions before redirecting to the authorization page (for example, redefine the page for redirection, or immediately make a redirect). If don't use the hook, then everything will work by default.

So, I think I have a solution 😄
Here is my maintenance page:

<template>

    Page: maintenance

</template>

Here is the sanctum config for the interceptors in app.config.ts:
sanctum: {
  interceptors: {
    onRequest: async (
      app: NuxtApp,
      ctx: FetchContext,
      logger: ConsolaInstance,
    ) => {
      logger.debug(`custom onRequest interceptor (${ctx.request})`)
    },

    onResponse: async (
      app: NuxtApp,
      ctx: FetchContext,
      logger: ConsolaInstance,
    ) => {
      if (ctx.response?.status === 503) {
        // we use runWithContext to make sure that the operation will be done on the same Nuxt instance
        app.runWithContext(() => {
          const currentRoute = app.$router.currentRoute.value
          const maintenanceRoute = '/maintenance'

          if (currentRoute.path === maintenanceRoute) {
            return
          }

          navigateTo(maintenanceRoute)
        })
      }

      logger.debug(`custom onResponse interceptor (${ctx.request})`)
    },
  },
},

In my case, it worked well when the API was down. Please, try this one and let me know if it helps.

Alternatively, can add a hook in which could define any actions before redirecting to the authorization page (for example, redefine the page for redirection, or immediately make a redirect). If don't use the hook, then everything will work by default.

The problem with hooks is that we will have to pass a lot of context there to make a proper decision and sometimes it's not related to the API response, making the module too complicated and easy to break with custom logic.

Alas. There is only a transition to the "login" page. The "navigateTo(maintenanceRoute)" transition does not work.
sanctum: {
    tokenStorage: localTokenStorage,
    interceptors: {
        onResponse: async (
            app: NuxtApp,
            ctx: FetchContext,
            logger: ConsolaInstance,
        ) => {
            console.log('log: 1')

            if (ctx.response?.status === 503) {
                console.log('log: 2')

                app.runWithContext(() => {
                    const currentRoute = app.$router.currentRoute.value
                    const maintenanceRoute = '/maintenance'

                    console.log('log: 3')

                    if (currentRoute.path === maintenanceRoute) {
                        console.log('log: 4')

                        return
                    }

                    console.log('log: 5')

                    navigateTo(maintenanceRoute)

                    console.log('log: 6')
                })
            }

            console.log('log: 7')
        },
    },
},

The following messages are displayed in the console:
GET http://api.sitename.lcl/api/me 503 (Service Unavailable)
log: 1
log: 2
log: 3
log: 5
log: 6
log: 7
nuxt-auth-sanctum:csr:error Unable to load user identity from API FetchError: [GET] "http://api.sitename.lcl/api/me": 503 Service Unavailable

Hey @webmaxx, could you try to use async approach?
if (ctx.response?.status === 503) {
  // we use runWithContext to make sure that the operation will be done on the same Nuxt instance
  await app.runWithContext(async () => {
    const currentRoute = app.$router.currentRoute.value
    const maintenanceRoute = '/maintenance'

    if (currentRoute.path === maintenanceRoute) {
      return
    }

    await navigateTo(maintenanceRoute)
  })
}

It didn't work. The result is the same.

@webmaxx okay, now this is weird 🤔 in my case both options work but I'm using breeze-nuxt application template, so maybe there is something additional that you have in your app that interferes with this behavior.
Since I can't reproduce the same, can you create a minimal repro to test on? or feel free to use breeze-nuxt as a starting point.

Good. I will try to do this using the breeze-next template. I hope it works.
Thank you.

Hey @webmaxx, have you figured out the problem you had?

Alas. Does not work. I will come back to this issue next week and check everything thoroughly.

Unfortunately, it didn't work.
Redirects in interceptors are ignored.
If add a 503 error interception in the "composables/useApiError.ts" file and make a redirect, then this works if access the api. But if just refresh the page (for example, by pressing F5), then will be redirected to the login page, and not to the maintenance page.

Hey @webmaxx, I see. I will add this feature to breeze-nuxt to test it out again and get back to you in a couple of days.

Hey @webmaxx, the reason why your redirects didn't work is connected to this nuxt issue - https://github.com/nuxt/nuxt/issues/23698, but as a workaround I can suggest using external flag like this
if (ctx.response?.status === 503) {
  app.runWithContext(async () => {
    const currentRoute = app.$router.currentRoute.value.path.replace(/\/$/, '')
    const maintenanceRoute = '/maintenance'

    if (currentRoute === maintenanceRoute) {
      return
    }

    await navigateTo(maintenanceRoute, { external: true })
  })
}

I also opened a PR with this implementation for breeze-nuxt, you can take a look before it is merged - https://github.com/manchenkoff/breeze-nuxt/pull/10

Stunningly. It's working. Thank you very much.


webmail Sent folder should show To instead of From

The From is likely to be the user themself so it would be more useful to show the recipient instead.

indeed this makes sense, i've had similar thoughts before. i decided to implement it as suggested: showing recipients instead of sender for messages in the Sent mailbox. and at the same time showing all correspondents for collapsed threads. however, that lead to some non-intuitive behaviour: for threads, all messages would show the other side of the conversation, never the sender.
i now made a change so the from address is still shown as before, but for messages in the designated Sent mailbox, the addressees are also shown, in italic.
the addresses-column is getting a bit crowded now, may need some more whitespace...

OK


Extract test utility functions to separate module and minor linting issues.

I fixed some linting issues in this PR based on @tkchin suggestions in another PR. I also extracted the test_util module from previous unit tests here. This should make the AnalyticsPage PR a bit smaller and less distracting.

I did separate commits in case we need to rollback one or the other, in which case we don't need to roll them both back.

lgtm

lgtm


Validations model specs

Added the the validations:

For the User model:

Name must not be blank.
PostsCounter must be an integer greater than or equal to zero.

For the Post model:

Title must not be blank.
Title must not exceed 250 characters.
CommentsCounter must be an integer greater than or equal to zero.
LikesCounter must be an integer greater than or equal to zero.

Added unit specs for all of your models' methods and validations.

Thank you for the review and suggesstions. Add new tests as per reviewer's comments


Issue 37818: Issue Test Fix.

Once you talk with Chris and Sweta, you can decide if we should add more test coverage for the multi-page scenario in this PR, do it later, or if we already have sufficient coverage.

Sounds good!

Hi Josh!
I had a chat with Chris. He thought that this PR looked fine to merge right now, but if (and this relates to the story that's back on the board) filtering behavior is changed to update a grid's selection, that will probably need to be accompanied by some test updates.
Within the scope of this PR, though, Chris verified it has passed the scope of tests that it ought to be passing, and should be good to go.


Improve IAM database restore error handling

High-level description
This PR improves the IAM database restore script to handle the case gracefully where CockroachDB state has been wiped out completely. I also reduces the amount of errors shown in the logs when a failure occurs. In addition, it also enables tox linting in the packages/cockroach/extra folder.
Corresponding DC/OS tickets (required)

DCOS-46121 packages/cockroachdb: improve exception handling in iam-database-restore script.

Related tickets (optional)

COPS-4976 Cockroachdb failing with message: "failed node liveness heartbeat: context deadline exceeded" - recovery failed.

@mesosphere-mergebot Ready For Review

@mesosphere-mergebot Ready for Review


Design of interface.py

Design of interface.py
I am currently significantly refactoring GDPC (on the refactor branch). Since
the refactor will break compatibility all over the place anyway, I would like to
take the opportunity to make the design of GDPC as good as possible.
However, there are a few design aspects related to interface.py for which I
find it difficult to make a decision. I would like to hear the opinion of others
on these aspects.
First, I will give some background, and then, I will explain the design
questions.
The current "entities"
On master, there are currently four "entities" that directly interact with
the HTTP interface:

direct_interface (free functions)
interface (free functions)
interface.Interface
WorldSlice

On refactor, interface.Interface is renamed to interface.Editor, but
otherwise the entities are the same.
The responsibilities of the "entities" are as follows:

direct_interface contains only direct Python bindings for the HTTP
interface, and has minimal dependencies. It is not intended to be used
by developers, but may be useful if they want to create their own higher-level
classes.

The free functions in interface perform stateless Minecraft operations, such
as executing commands or getting the build area

Editor contains functions to place and get blocks, contains lots of state
to make this easier or more efficient, and contains other functions that also
require this state. It has a lot of responsibilities, but it cannot be broken
up any further due to the shared state (for example, both getBlock and
setBlock need to access the block cache if caching is enabled).

worldSlice contains logic to call the GET /bytes endpoint and parse the
result into a world slice datastructure.

Currently, the URL of the HTTP interface is hardcoded to
http://localhost:9000. I would like to make this configurable, ideally in the
next update to avoid future compatibility issues. I would rather avoid requiring
developers to pass a URL parameter to every interface-related function if they
use a non-default url, so I would prefer if the URL was stored as state.
I would also like to make the interface-related error reporting of GDPC
configurable. Currently, GDPC prints errors to stderr, but users may
not want this. This setting should also be stored as state.
Note that all interface-related entities would depend on these two bits of
state.
My first thought is to add an Interface class (do not confuse with the old
Interface class on master, which is now called Editor), and to put this
state (url, error logging setting) into it along with the free functions from
interface.py. Both Editor and the WorldSlice constructor would require
Interface to get this url and error logging info.
So, assuming the plan described above, the entities and their state are as follows:

direct_interface has no state
Interface state:

URL
Error logging setting

Editor state:

Interface
Local coordinate transform (translation, rotation, flip)
Block buffer
Block cache
Block placement settings
etc.

WorldSlice constructor has no state

Design questions
There are two related design questions for which I find it difficult to make a
decision:

To what degree should Interface and Editor be global?
Does splitting Interface and Editor make sense? Should they be merged?

I will go in more detail for both.
Question 1 - To what degree should Interface and Editor be global?
Interface and Editor are both "entities" with state. There are (at least)
four ways to implement an entity with state, each with advantages and
disadvantages:

Put the state in global variables and implement the entity with free
functions. Equivalent to a singleton class.
Advantages:

Easiest to use, developers need not even know about the existence of state
unless they want to change it.
No classes to pass around.

Disadvantages:

It becomes impossible to have multiple instances of the entity. This
is the only option that makes something impossible.
It is "magic"-y: it may be hard to developers to understand which function
calls affect others, since state is hidden away.

Implement the entity as a class.
Advantages:

Multiple instances are possible.
Simplest and cleanest design from the perspective of GDPC.
Good discoverability via code completion on a class instance.

Disadvantages:

Developers need to instantiate the classes, making GDPC slightly more
difficult to use.
Class instances need to be passed around everywhere. Developers are forced
to add parameters for the classes in nearly all of their functions, again
making GDPC slightly more difficult to use.

Combination of option 1 and 2: implement the entity as a class, but create
a default instance of it and add free functions that work with this default
instance. This is how Editor (previously called Interface) is implemented
on master.
Advantages:

Both the advantages of options 1 and 2: developers can choose.

Disadvantages:

All functions of the entity need to be duplicated (methods and free
functions). Perhaps multiple duplications are even needed, for example if
Interface has a default Editor and there is also a default Interface.
This is somewhat painful to maintain, and may even cause name collision
issues.
Difficult to understand for both GDPC contributors and generator
developers.

Implement the entity as free functions and require that all state is
managed by the caller and is passed to each function as parameters.
direct_interface and the worldSlice constructor are implemented in this
way, but I find it unacceptable for Interface and Editor.

This is all quite abstract, so I'll give some representative syntax examples of
option 1 and option 2 (for option 3, developers can choose either syntax).
Interface global (option 1), Editor global (option 1):
def buildHouse(position):
	runCommand("say 'building house!'")
	placeCuboid(position, position+10, hollow=True)
	...

setURL("http://example.com:9000")
setBuffering(True)

worldSlice = WorldSlice(Rect((0,0), (64,255,64)))
placeBlock((0,64,0), Block("stone"))
buildHouse((10,64,0))

Interface global (option 1), Editor as class (option 2):
def buildHouse(editor, position):
	runCommand("say 'building house!'")
	placeCuboid(editor, position, position+10, hollow=True)
	...

setURL("http://example.com:9000")

editor = Editor(buffering=True)

worldSlice = WorldSlice(Rect((0,0), (64,255,64)))
editor.placeBlock((0,64,0), Block("stone"))
buildHouse(editor, (10,64,0))

Interface as class (option 2), Editor global (option 1) (this one makes
little sense, but also see design question 2):
def buildHouse(interface, position):
	interface.runCommand("say 'building house!'")
	placeCuboid(position, position+10, hollow=True)
	...

interface = Interface(url="http://example.com:9000")

setInterface(interface)
setBuffering(True)

worldSlice = WorldSlice(interface, Rect((0,0), (64,255,64)))
placeBlock((0,64,0), Block("stone"))
buildHouse(interface, (10,64,0))

Interface as class (option 2), Editor as class (option 2):
def buildHouse(interface, editor, position):
	interface.runCommand("say 'building house!'")
	placeCuboid(editor, position, position+10, hollow=True)
	...

interface = Interface(url="http://example.com:9000")

editor = Editor(interface)
editor.setBuffering(True)

worldSlice = WorldSlice(interface, Rect((0,0), (64,255,64)))
editor.placeBlock((0,64,0), Block("stone"))
buildHouse(interface, editor, (10,64,0))

Question 2 - Should Interface and Editor be merged?
I'm also not sure if it even makes sense to split Interface and Editor.
The reasons for splitting them is that the combined entity would have a lot of
state and responsibilities, which is not desirable from a software design
perspective, and that there might be cases where it is useful to have multiple
Editors for a single Interface. However, having them as one class might be much
easier for developers.
Editor can be merged into Interface by simply flattening it: all state and
methods of Editor become state and methods of Interface.
If Interface and Editor are merged, two syntaxes remain for design
question 1:
Interface global:
def buildHouse(position):
	runCommand("say 'building house!'")
	placeCuboid(position, position+10, hollow=True)
	...

setURL("http://example.com/9000")
setBuffering(True)

worldSlice = WorldSlice(Rect((0,0), (64,255,64)))
placeBlock((0,64,0), Block("stone"))
buildHouse((10,64,0))

Interface as class:
def buildHouse(interface, position):
	interface.runCommand("say 'building house!'")
	placeCuboid(interface, position, position+10, hollow=True)
	...

interface = Interface(url="http://example.com/9000")
interface.setBuffering(True)

worldSlice = WorldSlice(interface, Rect((0,0), (64,255,64)))
interface.placeBlock((0,64,0), Block("stone"))
buildHouse(interface, (10,64,0))

Feedback
Which options do you think are best for both questions? Do you have an entirely
different idea? I would like to hear!
I am currently slightly leaning towards merging Interface and Editor into
just Interface, and implementing Interface as option 3 (as class, but
with a default instance and free functions that work with the default instance).

To reduce complexity/confusion for the user, I would at consider merging direct_interface and Interface into the same file, where some or all of the methods now used for the direct interface are static.
Likewise I would also consider making WorldSlice a part of Interface and have the user interact with the data of WorldSlice via the Interface, since the chunk data is useful for caching. It allows you to quickly look up a block or biome at a given location without having to do a request.
Quick note on the URL: It's non-trivial to implement in [GDMC-HTTP](https://github.com/Niels-NTG/gdmc_http_interface/) a way for the user to change the port, which is now 9000. But if there is a need for this I will look into a way for the user to change this inside of Minecraft.
Having said that, splitting Interface and Editor would make sense. Interface should be concerned managing the state such as the cache and if needed updating the cache and doing requests to direct_interface. Editor meanwhile does not contain state except for an instance of Interface which can be passed in its constructor. Editor is consequently quite a bare-bones class that invites users to create their own versions/extensions.
import Interface
import Editor

interfaceInstance = Interface(url='http://localhost:9000')
mainEditor = Editor(interfaceInstance)

import Interface
import Block

class Editor:

    def __init__ (self, interface: Interface):
        self.interface = interface

    def buildHouse(x, y, z, house):
        for block in house:
            if isinstance(block, Block)
            	self.interface.setBlock(block)
        self.interface.sendBlocks()

    def placeCubiod(x1, y1, z1, x2, y2, z2):
        interface.runCommand('fill %s %s %s %s %s %s' % (x1, y1, z1, x2, y2, z2))

I'm not sure about merging direct_interface and interface. In my eyes, direct_interface contains only low-level python bindings, while Interface is slightly higher level. Users will never need to use direct_interface, so I don't expect much confusion. Static methods are also not that necessary in Python, since free functions exist.
WorldSlice will indeed be better integrated/become part of Interface. I think I'll keep it optional to cache it for block retrieval though, since such caching assumes nothing else is changing the world. GDPC and the HTTP mod are of course intended for the competition, but I would prefer if the system is usable more broadly, including contexts where the world is being changed (e.g. by players).
About the url: this is just a bit of future-proofing. I'd like the GDPC to support an arbitrary host url in case that ever becomes useful.
I don't really understand your suggestion on how to split Interface and Editor. If Editor contains no state, why make it a class at all? Why not just provide free functions? Functions like buildHouse should definitely not be part of Editor, otherwise you end up with an enormous blob class. The only reason Interface and Editor are currently split, is that Editor contains some state that is not relevant for the rest of the interface (like the block cache).

I'll go for the following design:

HTTPInterface becomes a class with the URL as state, and all current direct_interface functions as members. There will be a default instance with the default URL.
Editor becomes what is currently interface.py on master; the Interface and Editor classes of refactor will be merged into Editor. Editor will be constructed with a HTTPInterface instance, the default interface by default. There will be no default Editor: users have to construct one (following "There should be one-- and preferably only one --obvious way to do it.").

This design makes it possible to create a duck interface and feed that to an Editor (e.g. an interface that saves placements to a structure file instead of actually placing them).


Long MarkDown is not displayed

When I use a really long markdown text it doesn't show up at all. It seems that the text is invisible as it still taking a lot of space and I can scroll the view that I've put it into.

Hum that's weird indeed 🤔 … When you say "really long markdown" what do you mean?

The document is 8,906 words and 55,819 characters. It turned out that react-native itself is unable to show much this many words in a single Text component. I'm not sure what the threshold for amount of words or characters is. We ended up using a webview to display the document in HTML form instead.

OK thanks for the input! That could be interesting to handle in the v2 👍


Able to view cameras in HomeKit but not on UI

Hi there,
I am trying to move over to your plugin because I want to be able to record footage
As the title says, I am able to view the two cameras I added in HomeKit. Stream loads fast too.
However, I am unable to see the streams in the camera.ui. There are also no recordings. I will paste the HomeBridge logs below. Can you see what is causing this issue?
[6/7/2021, 7:48:56 AM] [CameraUI] [Interface] camera.ui v4.2.7 is listening on port 8182 (http)
[6/7/2021, 7:48:56 AM] [CameraUI] [Interface] Initializing clear timer
[6/7/2021, 7:48:56 AM] [CameraUI] [Sessions] Initializing camera sessions
[6/7/2021, 7:48:56 AM] [CameraUI] [Streams] Initializing camera streams
[6/7/2021, 7:49:00 AM] [Nest] initing thermostat "Bedroom office Thermostat": deviceId: 09AA01AC30160P07 structureId: f4e82ab0-8fd9-11e6-b4e6-22000b90c396
[6/7/2021, 7:49:00 AM] [Nest] initing thermostat "Living Room Thermostat": deviceId: 09AA01AC1116046M structureId: f4e82ab0-8fd9-11e6-b4e6-22000b90c396
[6/7/2021, 7:49:05 AM] [CameraUI] [Socket] ::ffff:<IP_ADDRESS> connected to server
GET /api/auth/check 401 16.720 ms - 44
POST /api/auth/logout 200 3.363 ms - 2
[6/7/2021, 7:49:17 AM] [CameraUI] [Socket] ::ffff:<IP_ADDRESS> disconnected from server
GET /css/login.b911022f.css 304 18.772 ms - -
GET /js/login.59467c9d.js 304 3.970 ms - -
GET<EMAIL_ADDRESS>304 2.122 ms - -
POST /api/auth/login 201 62.682 ms - 405
GET /api/auth/check 200 6.945 ms - 15
[6/7/2021, 7:49:21 AM] [CameraUI] [Socket] ::ffff:<IP_ADDRESS> connected to server
GET /media/notification.91f71662.mp3 206 7.619 ms - 2
GET /media/notification.91f71662.mp3 206 1.935 ms - 28901
GET<EMAIL_ADDRESS>304 1.212 ms - -
GET /api/cameras 304 45.319 ms - -
GET /api/settings/dashboard 304 26.857 ms - -
GET /api/notifications?cameras=Garage%20Cam&pageSize=5 304 17.515 ms - -
GET /api/notifications?cameras=Bedroom%20Cam&pageSize=5 304 32.416 ms - -
[6/7/2021, 7:49:22 AM] [CameraUI] [Interface] Incoming ping request for: rtsp://<IP_ADDRESS>/HighResolutionVideo - Timeout: 5s
[6/7/2021, 7:49:22 AM] [CameraUI] [Interface] Pinging rtsp://<IP_ADDRESS>
[6/7/2021, 7:49:22 AM] [CameraUI] [Interface] Incoming ping request for: rtsp://<IP_ADDRESS>/HighResolutionVideo - Timeout: 5s
[6/7/2021, 7:49:22 AM] [CameraUI] [Interface] Pinging rtsp://<IP_ADDRESS>
GET /api/cameras/Garage%20Cam/status?timeout=5 304 122.257 ms - -
GET /api/cameras/Bedroom%20Cam/status?timeout=5 304 111.453 ms - -
[6/7/2021, 7:49:22 AM] [CameraUI] [Socket] New WebSocket connection from ::ffff:<IP_ADDRESS> to room: stream/Garage Cam
[6/7/2021, 7:49:22 AM] [CameraUI] [Socket] Active sockets in room (stream/Garage Cam): 1
[6/7/2021, 7:49:22 AM] [CameraUI] [Sessions] Garage Cam added to active sessions
[6/7/2021, 7:49:22 AM] [CameraUI] [Sessions] Currently active streams: {"Garage Cam":{"activeStreams":1,"maxStreams":3},"Bedroom Cam":{"activeStreams":0,"maxStreams":3}}
[6/7/2021, 7:49:22 AM] [CameraUI] [Streams] Garage Cam: Stream command: /usr/lib/node_modules/homebridge-camera-ui/node_modules/ffmpeg-for-homebridge/ffmpeg -probesize 32 -analyzeduration 0 -rtsp_transport tcp -stimeout 10000000 -i rtsp://<IP_ADDRESS>/HighResolutionVideo -f mpegts -codec:v mpeg1video -s 1280x720 -b:v 299k -r 20 -bf 0 -preset:v ultrafast -threads 1 -loglevel quiet -codec:a mp2 -ar 44100 -ac 1 -b:a 128k -
[6/7/2021, 7:49:22 AM] [CameraUI] [Socket] New WebSocket connection from ::ffff:<IP_ADDRESS> to room: stream/Bedroom Cam
[6/7/2021, 7:49:22 AM] [CameraUI] [Socket] Active sockets in room (stream/Bedroom Cam): 1
[6/7/2021, 7:49:22 AM] [CameraUI] [Sessions] Bedroom Cam added to active sessions
[6/7/2021, 7:49:22 AM] [CameraUI] [Sessions] Currently active streams: {"Garage Cam":{"activeStreams":1,"maxStreams":3},"Bedroom Cam":{"activeStreams":1,"maxStreams":3}}
[6/7/2021, 7:49:22 AM] [CameraUI] [Streams] Bedroom Cam: Stream command: /usr/lib/node_modules/homebridge-camera-ui/node_modules/ffmpeg-for-homebridge/ffmpeg -probesize 32 -analyzeduration 0 -rtsp_transport tcp -stimeout 10000000 -i rtsp://<IP_ADDRESS>/HighResolutionVideo -f mpegts -codec:v mpeg1video -s 1280x720 -b:v 299k -r 20 -bf 0 -preset:v ultrafast -threads 1 -loglevel quiet -codec:a mp2 -ar 44100 -ac 1 -b:a 128k -
[6/7/2021, 7:49:31 AM] [CameraUI] [Streams] Garage Cam: RTSP stream exited with error! (null)
[6/7/2021, 7:49:31 AM] [CameraUI] [Sessions] Garage Cam removed from active sessions
[6/7/2021, 7:49:31 AM] [CameraUI] [Sessions] Currently active streams: {"Garage Cam":{"activeStreams":0,"maxStreams":3},"Bedroom Cam":{"activeStreams":1,"maxStreams":3}}
[6/7/2021, 7:49:32 AM] [CameraUI] [Streams] Bedroom Cam: RTSP stream exited with error! (null)
[6/7/2021, 7:49:32 AM] [CameraUI] [Sessions] Bedroom Cam removed from active sessions
[6/7/2021, 7:49:32 AM] [CameraUI] [Sessions] Currently active streams: {"Garage Cam":{"activeStreams":0,"maxStreams":3},"Bedroom Cam":{"activeStreams":0,"maxStreams":3}}

@ACWAKKERMANS
can u post your config pls

  {
        "name": "CameraUI",
        "port": 8182,
        "theme": "auto",
        "language": "en",
        "debug": true,
        "mqtt": {
            "active": false
        },
        "http": {
            "active": false,
            "localhttp": false
        },
        "ssl": {
            "active": false
        },
        "cameras": [
            {
                "name": "Garage Cam",
                "unbridge": true,
                "videoConfig": {
                    "source": "-probesize 32 -analyzeduration 0 -rtsp_transport tcp -i rtsp://<IP_ADDRESS>/HighResolutionVideo",
                    "audio": false
                }
            },
            {
                "name": "Bedroom Cam",
                "manufacturer": "Logitech",
                "model": "Alert e750",
                "motion": true,
                "doorbell": true,
                "switches": true,
                "motionDoorbell": true,
                "unbridge": true,
                "videoConfig": {
                    "source": "-probesize 32 -analyzeduration 0 -rtsp_transport tcp -i rtsp://<IP_ADDRESS>/HighResolutionVideo",
                    "audio": false
                }
            }
        ],
        "platform": "CameraUI"
    }

The HomeBridge installation is on a Raspberry Pi (<IP_ADDRESS>). I am accessing the camera.ui from my MacBook (<IP_ADDRESS>) and/or from my iMac (<IP_ADDRESS>)

Can u try without these flags: -probesize 32, -analyzeduration 0 in the source
like:
"videoConfig": {
  "source": "-rtsp_transport tcp -i rtsp://<IP_ADDRESS>/HighResolutionVideo",
   "audio": false
}

Can confirm it still worked in HomeKit without the flags but still doesn't in UI.
The 12:32:22 PM entry below is HomeKit stopping the stream.
[6/7/2021, 12:31:45 PM] [CameraUI] [Streams] Garage Cam: RTSP stream exited with error! (null)
[6/7/2021, 12:31:45 PM] [CameraUI] [Sessions] Garage Cam removed from active sessions
[6/7/2021, 12:31:45 PM] [CameraUI] [Sessions] Currently active streams: {"Garage Cam":{"activeStreams":0,"maxStreams":3},"Bedroom Cam":{"activeStreams":1,"maxStreams":3}}
[6/7/2021, 12:31:53 PM] [CameraUI] [Streams] Bedroom Cam: RTSP stream exited with error! (null)
[6/7/2021, 12:31:53 PM] [CameraUI] [Sessions] Bedroom Cam removed from active sessions
[6/7/2021, 12:31:53 PM] [CameraUI] [Sessions] Currently active streams: {"Garage Cam":{"activeStreams":0,"maxStreams":3},"Bedroom Cam":{"activeStreams":0,"maxStreams":3}}
[6/7/2021, 12:32:22 PM] [Camera FFmpeg] [Bedroom] FFmpeg exited with code: 0 and signal: null (Error)
[6/7/2021, 12:32:22 PM] [Camera FFmpeg] [Bedroom] Stopped video stream.

Actually.. scrap that... checked wrong camera.. now that I changed the flags as you requested, camera doesn't stream in HK at all. "No Response"

Hi there,
I'm quite keen to get the recording of the stream going, but at the moment, I cannot see the cameras in the UI, nor are recordings created. Do I need to fix the cameras showing in the UI in order to get recordings or is the fact I have no recordings a separate issue? Ultimately, I don't really need to see the cameras in the UI. That's what I have the IOS Home app for.

Im actually having a similar problem.
Noticing a unknown IP address in the debug log...
[CameraUI] [Socket] ::ffff:<IP_ADDRESS> disconnected or closed from room: stream/Hikvision Doorbell
IP address of my HB on the same server which is <IP_ADDRESS>
I'm able to connect to the CameraUI portal however streaming not possible at the moment

Hi, I can second the initial issue, and here is the Browser Dev-Debug Output on when I open the Camera-UI (camera-View). In that case <IP_ADDRESS> is the IP of my raspberry driven Homebridge host incl. the Camera-UI plugin. The port 8181 is fine, and all the HTML UI and config stuff shows up all good w/o errors.
Seems like the Camera-ui backend /api/ responds with 401 not authorized errors (seen same in ACWAKKERMANS logs) before that below err_connection_refused, the stream works fine in camera-ffmpg plugin -> Homekit as well as in VLC as well as in the vendors native "browser-app" (reolink).
GET http://<IP_ADDRESS>:8181/api/cameras/Eingang%20hinten/snapshot?buffer=true 401 (Unauthorized)
...
GET http://<IP_ADDRESS>:8181/socket.io/?EIO=4&transport=polling&t=Nk_XmMl net::ERR_CONNECTION_REFUSED
...
GET http://<IP_ADDRESS>:8181/api/auth/check 401 (Unauthorized)

that is same for every camera I have in cam-view


Question about listeners

Since I didn't find anything in the documentation, I'll open a ticket here.
Could you explain to me how to add a listener? I need to execute some code once a SmartInventory was closed. I know the builder has a listener method, but I don't understand how to use it properly.
A small example would be really helpful.
What I'm trying to do:
The player opens the GUI -> ItemStack from his main hand goes to slot 0 in the GUI -> When the player closes the GUI it should be returned to their main hand. But I don't get how to use the listener method inside the builder.

Ohhh, there is no example of this in the docs, I should add one.
Here is a small example:
SmartInventory inventory = SmartInventory.builder()
    .provider(...)
    .listener(new InventoryListener(InventoryCloseEvent.class, event -> {
        // ...
    }))
    .build();

Thanks, that solves my problem.


Fix link to WebSocket in App API docs

Fixes #316

Codecov Report

Merging #317 into release/docs will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##           release/docs   #317   +/-   ##
===========================================
  Coverage           100%   100%
===========================================
  Files                24     24
  Lines              1093   1093
  Branches            115    115
===========================================
  Hits               1093   1093

Impacted Files
Coverage Δ

bocadillo/applications.py
100% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 32bccd8...6d366cd. Read the comment docs.


Switch to semantic DeepEquals()

reflect.DeepEquals doesn't account for quality as it relates to json output.  A nil slice and empty slice are considered different when in fact they produce the same json.
Related to #6997
This comment on #6997 explains the situation well, https://github.com/kubevirt/kubevirt/pull/6997#issuecomment-1019546079
Switch from reflects.DeepEquals to equality.Semantic.DeepEquals() across the entire project

/hold
I want https://github.com/kubevirt/kubevirt/pull/7131  to go in first. That addresses the initial issue we encountered in handler. Then i want this PR to go in.

/cc @vasiliy-ul @EdDev @rmohr
After @vasiliy-ul 's pr goes in. I think we should switch our usage of deep equals across the board.

@vasiliy-ul thanks for the review.
You're making good observations that the mass search and replace of the equality function likely isn't appropriate. I'll take a closer case by case look.

I've updated this PR addressing @vasiliy-ul's comments and making some changes of my own. I went line by line and verified that each usage of the semantic DeepEquals is accurate now versus usage of reflects.DeepEquals.  There are a few area's where reflect.DeepEquals() is still the more accurate approach, so i left those as they were.

/lgtm
/retest

/retest

Maybe it would make sense to do the same switch to semantic DeepEquals()  also in CDI? Ping @awels .

Yeah, in CDI we have some of these using Semantic.DeepEquals already, but not everywhere.

/approve
@vasiliy-ul would you mind checking if you can restore your lgtm?

/lgtm
/retest

/unhold

/cherry-pick release-0.49


Change UplinkAccounts to use Mind instead of body

Description

Currently the UplinkAccountsSystem uses EntityUid, which means that if the Traitor gets cloned, they aren't registered in the accounts database, and so due to recent access changes are unable to access their uplink.

Aren't this access changes temporal, until #5842?

That PR doesn't change the Uplink system so even when it gets merged, a cloned traitor still won't have access to their account.

That PR doesn't change the Uplink system so even when it gets merged, a cloned traitor still won't have access to their account.

Wdym? Anyone who knows code can access Uplink.

To clarify, AFAIK the idea was to just have uplinks accounts linked to a secret code / login, that any player can access if they know it (and have an entity with an uplink component available), with no entity/mind restrictions. The ringtone PR doesn't actually make any changes to uplink stuff, but is intended to be the how the secret code is entered.
You could instead just add a "phone" button and recycle the nuke key-code entry UI or something if that were easier than finishing the ringtone PR.

To clarify, AFAIK the idea was to just have uplinks accounts linked to a secret code / login, that any player can access if they know it (and have an entity with an uplink component available), with no entity/mind restrictions. The ringtone PR doesn't actually make any changes to uplink stuff, but is intended to be the how the secret code is entered.
You could instead just add a "phone" button and recycle the nuke key-code entry UI or something if that were easier than finishing the ringtone PR.

It's not difficult I've just been busy lately, but I am currently working on finishing up the changes sloth requested and pushing it for review sometime soon.

uplink accounts are getting nuked from orbit so this will not longer be accurate.

uplink accounts are getting nuked from orbit so this will not longer be accurate.

Are the changes going to involve actually linking the account/credits to a login/secret, or will it still be tied to a player's entity somehow?


ARROW-4130: [Go] offset not used when accessing binary array

Closes https://github.com/apache/arrow/issues/3270 .

@stuartcarnie can you review this?

@stuartcarnie @alexandreyc can you review?

LGTM
(Sorry for being rather inactive these last weeks, I hope I will have more time to contribute in a few weeks)


Variable Explorer Accessibility Issues

HC theme support:

Value edit textboxes aren't in compliance with HC themes
Bug 402111: Accessibility : MAS42A : High Contrast: Environment selection from combo box is not in compliance with HC themes
Bug 402112: Accessibility : MAS42A : High Contrast: Select Environment combo box when focused using keyboard doesn't get a highlight

Keyboard navigation

Bug 402114: Accessibility : MAS05A : Keyboard : Focus is not visible on the cells present in the 'Variable Explorer' window
Bug 402115: Accessibility : MAS36 : Keyboard : Table headers is not accessible in Variable Explorer window
Bug 402116: Accessibility : MAS36 : Keyboard : Options in the 'Selected Environment' combo box in Variable Explorer window is not invoked using Space/Enter key
Bug 402117: Accessibility : MAS05 : Keyboard : Focus order is incorrect in Variable Explorer window
Bug 402118: Accessibility : MAS05 : Keyboard : Focus moves to the top of the window on deleting a variable in 'Variable Explorer' window
Bug 402120: Accessibility : MAS36 : Keyboard : Table headers are not accessible in R Data window

Generally speaking, there is one big issue with Variable Explorer that causes all those bugs: it uses DataGrid. While DataGrid is a powerful control, it is really hard to get customized for anything expect showing table-style data. In our case, we have two columns:

first column that isn't editable, but has a tree view logic and custom arrow key handling
second column is partially editable and also contains additional controls on top of its base with custom tab order

Additional question about grid headers. It is uncommon that a grid header is part of a tab order - no existing window in VS allows to user to navigate to the header with tabs or arrows.

I think it is for sorting

Are there any common shortcuts for sorting?

File Explorer lets me tab to the header, then arrows to move between columns and space to sort by the selected column.

I wonder if we can get most or all of this for free if we use ListView rather than DataGrid as a basis for this UI (we'd still have to customize it into a tree, but still).

We can but prob not for 2017 at the moment. We can file for exception and then work to replace VE and grid by something else.

Hm... they have a bit different behavior (though probably the right one):|

tab doesn't switch between columns, rows or cells - it only switches between header and list itself
enter & space change sorting and switch selection back to the last selected row in the grid

To implemen this behavior, we have to get rid of the DataGrid and use something more customizable - in range from ListView to ItemsControl

We can file for exception and then work to replace VE and grid by something else.

That's probably the best option for now. Changing anything in VE is quite risky cause UI is tightly bound to the underlying business logic - we can't simply replace DataGrid with ListView.

HC went #3408, the rest is for exception

Done? Close?


Provide ClientHttpRequestInterceptor to make aws-sig-4 signed request with the RestTemplate

:loudspeaker: Type of change

[ ] Bugfix
[x] New feature
[ ] Enhancement
[ ] Refactoring

:scroll: Description
Provide a ClientHttpRequestInterceptor which signs requests using the AWS Signature V4 which is needed for example to call various AWS services (like API Gateway)
:bulb: Motivation and Context
Currently imho it is tedious and lacking a good documentation on how to make signed request using the RestTemplate to the API Gateway.
Alternative to the RestTemplate you can use generated SDKs or subclassing the AmazonWebServiceClient.
Imho being able to make signed requests using the RestTemplate has the benefit for the spring developers to work with a familiar API.
:green_heart: How did you test it?
There are two test verifying the signatures of requests from the aws-sig-v4-test-suite.
:pencil: Checklist

[ ] I reviewed submitted code
[ ] I added tests to verify changes
[ ] I updated reference documentation to reflect the change
[ ] All tests passing
[ ] No breaking changes

:crystal_ball: Next steps
As this is more or less just a preview of the feature and also lacking support for the WebClient, I am happy to discuss if this is a desirable feature and if yes what further steps are needed to see this feature integrated.

Thanks @joshiste for the PR! Would you mind closing this one and creating new one in https://github.com/awspring/spring-cloud-aws repository? https://github.com/spring-cloud/spring-cloud-aws is in the maintenance mode and everything new moves to awspring.

moved to https://github.com/awspring/spring-cloud-aws/pull/113


Fix broken twitter link

The Twitter link was using Markdown inside of HTML and thus was broken because it's being centered in a <p> tag. I changed it to just a regular ol' <a> tag.

@ajcwebdev The link seems to work fine in the README as-is? This doesn't seem like a big deal, but could you maybe provide a screenshot of what you're seeing?

Oh, I was actually looking at the wrong one. 🤦


workaround_ha_apiserver

Can you explain this 'workaround', why does it work?

Actually i don't really know what's the root cause but i noticed that the problem occurs when multiple apiserver restarts are done.
Furthermore, after a good sleep i don't think it is related to the controller-manager at all.
maybe etcd...i'll keep digging further.


iPad banner overlapping the app

AdMob Pro banner at the bottom on iOS 7.12 iPad initially overlaps the app instead of resizing the app as it does on Android. If I tap on the app, the app content scrolls up and shows the overlapped part, but now the top part is missing and cannot be scrolled back down.
Every time I rotate the device, the banner overlaps the bottom part again

before giving a solution, i need confirm the issue. as i cannot reproduce it, can you send me your code?

before giving a solution, i need confirm the issue. as i cannot reproduce it, can you send me your code?

What would you all like? Just the ww folder?
On Wed Dec 31 2014 at 5:48:53 PM Raymond Xie<EMAIL_ADDRESS>wrote:

before giving a solution, i need confirm the issue. as i cannot reproduce
it, can you send me your code?
—
Reply to this email directly or view it on GitHub
https://github.com/floatinghotpot/cordova-admob-pro/issues/18#issuecomment-68476947
.

What would you all like? Just the ww folder?
On Wed Dec 31 2014 at 5:48:53 PM Raymond Xie<EMAIL_ADDRESS>wrote:

before giving a solution, i need confirm the issue. as i cannot reproduce
it, can you send me your code?
—
Reply to this email directly or view it on GitHub
https://github.com/floatinghotpot/cordova-admob-pro/issues/18#issuecomment-68476947
.

www*
On Thu Jan 01 2015 at 12:49:11 AM Austin Lovell<EMAIL_ADDRESS>wrote:

What would you all like? Just the ww folder?
On Wed Dec 31 2014 at 5:48:53 PM Raymond Xie<EMAIL_ADDRESS>wrote:

before giving a solution, i need confirm the issue. as i cannot reproduce
it, can you send me your code?
—
Reply to this email directly or view it on GitHub
https://github.com/floatinghotpot/cordova-admob-pro/issues/18#issuecomment-68476947
.

www*
On Thu Jan 01 2015 at 12:49:11 AM Austin Lovell<EMAIL_ADDRESS>wrote:

What would you all like? Just the ww folder?
On Wed Dec 31 2014 at 5:48:53 PM Raymond Xie<EMAIL_ADDRESS>wrote:

before giving a solution, i need confirm the issue. as i cannot reproduce
it, can you send me your code?
—
Reply to this email directly or view it on GitHub
https://github.com/floatinghotpot/cordova-admob-pro/issues/18#issuecomment-68476947
.

yes, i need see the content in www folder, and please also list the plugins you used.

I wonder if there was ever a solution for this.  I'm seeing this in a GWT cordova 6.3 app on iOS, but not on Android.

in adbs the ads hides under siderbar, i want to do the same, is there answer for that?


Where is the google-speech-dataset.pt ?

There is not a google-speech-dataset.pt which needed by the server.py in model.zip. I tried to get it by "python -m utils.train --data_folder ./training_data --type train", but I got a "model.pt" . Even I rename it and run "python .", an error will arise as below.
RuntimeError: Error(s) in loading state_dict for SpeechModel:
size mismatch for output.weight: copying a param of torch.Size([12, 26624]) from checkpoihe shape is torch.Size([4, 26624]) in current model.
size mismatch for output.bias: copying a param of torch.Size([12]) from checkpoint, whereis torch.Size([4]) in current model.

Should I add some paras into the config.json ?

We mistakenly deleted google-speech-dataset.pt from https://github.com/castorini/honk-models -- it's fixed now. I believe that error results from not specifying n_labels as 12 (keywords).

Thanks for your help, it's working :D


[example][hal-quad] Destroy shader modules immediately after creating PSO

Actually, it's not an important PR, just for clarification that PSOs copy shader code from ShaderModules instead of borrowing.
It's OK to close it without merging ;)

Not sure about this one, I prefer the old one because I think it's easier to follow the code when clearly separated into construction and destruction parts.
Would appreciate other opinions on this.

I'm slightly more in favor of this PR. Would appreciate more opinions :)

We could move shader module creation and destruction completely into the pipeline creation block. This would be a bit cleaner imo, also easier to reason about the lifetime of the shader modules.

@davll would you want to finish the PR according to the last recommendation?


Kk wdac edits add hta info to select types doc

@jgeurten, implemented the HTA update to the two files, as requested in the email.

Hello, @v-kikl Can you resolve the merge conflicts to proceed with the review? Thanks.


Why do we need alphapose?

Why do we need alphapose since code is just for testing . Pose parameters are used only for loss calculation as I see from the paper.

Hi,
Alphapose and Poseflow are only used to get the video tracklet corresponding to a person. The pose parameters are discarded after the tube is computed.


CallNtPowerInformation, parameter mutation by function?

Hi, thank you for this library. Very useful. Context: First time using Rust, first time using Windows SDKs. Apologies if this is extremely basic, I hope it is.
Attempting to get idleness information via CallNtPowerInformation. This function (I believe) expects to place information into the expected structure based on the supplied POWER_INFORMATION_LEVEL. I'm having difficulty reconciling how the structures/function calls should look for a struct not defined in winapi, going through FFI, and as a parameter being mutated. Parameter in question is teststruct in CallNtPowerInformation call. Example:
    let mut teststruct = SystemPowerInformationStruct {
        MaxIdlenessAllowed: 0,
        Idleness: 0,
        TimeRemaining: 0,
        CoolingMode: 0
    };
    // Fill in struct and get result code
    let result = unsafe {
        CallNtPowerInformation(
            SystemPowerInformation,
            null_mut(),
            0,
            &teststruct,
            size_of::<SystemPowerInformationStruct>() as u32
        )
    };

Errors as types differ in mutability. Tried several permutations passing &teststruct, &mut teststruct, ... struggling:
expected enum `std::ffi::c_void`, found struct `prevent_sleep::SystemPowerInformationStruct`
expected enum `std::ffi::c_void`, found reference
...

If this is not the place for support, please close. I could default back out of Rust for the sake of getting it done, but this is good learning/fun. :)
Repo code:
https://github.com/remingtonc/nosleep/blob/windows/src/main.rs#L10-L43

&mut teststruct as *mut _ as *mut _

Thank you for the support and clarification.


Release azure-cli-acr 0.1.1b4

Reviewers, please verify the following for this PR:

[ ] The PR title has format Release <component-name> <version>.
[ ] setup.py has been modified with the same version as in the PR title.
[ ] HISTORY.rst has been modified with appropriate release notes.
[ ] If releasing azure-cli or azure-cli-core, __version__ defined in __init__.py should also be modified to match.

Release of 'azure-cli-acr' with version '0.1.1b4' successful. View at https://pypi.python.org/pypi/azure-cli-acr/0.1.1b4.

GitHub release created. https://github.com/Azure/azure-cli/releases.


Fixed a minor problem with redudant calls to world.getBlockState(pos)

Fixed a minor problem with redudant calls to world.getBlockState(pos)

Undo the capability cast changes, these are necessary to avoid warnings.
This also doesn't really warrant a changelog entry, because these weren't actual perf issues in the first place ;)

Thank you!

you are welcome


[KED-1467] Added a new Dataset to fetch data from API endpoints

Description
Resolves: #283
Development notes

Checklist

[x] Read the contributing guidelines
[ ] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
[x] Updated the documentation to reflect the code changes
[ ] Added a description of this change and added my name to the list of supporting contributions in the RELEASE.md file
[x] Added tests to cover my changes

Notice

[x] I acknowledge and agree that, by checking this box and clicking "Submit Pull Request":

I submit this contribution under the Apache 2.0 license and represent that I am entitled to do so on behalf of myself, my employer, or relevant third parties, as applicable.

I certify that (a) this contribution is my original creation and / or (b) to the extent it is not my original creation, I am authorised to submit this contribution on behalf of the original creator(s) or their licensees.

I certify that the use of this contribution as authorised by the Apache 2.0 license does not violate the intellectual property rights of anyone else.

Hi @ppsimatikas thank you very much for this. This looks useful, but I'd like to know your opinion on whether some of the datasets that we provide out of the box:

JSONDataSet also allows you to specify an HTTP URL as the filepath. You can encode the query parameters in the URL as well.
If you require plain text from the API, TextDataSet works in a similar way.

Some other considerations:

In general, we prefer to use fsspec to interface with different data sources. I was wondering if their HTTPFileSystem could be of use here?
IMHO, a generic API dataset seems too generic to be useful out of the box to many users without further modification. As an example, in my experience, most APIs are paginated and they are paginated differently. Some also follow a hypermedia format where you have to follow links indefinitely. So users will most certainly need to add further modification to this dataset if they want to use it. I'm just wondering whether it's worth adding a generic, base dataset in this case.

Regardless, this is a very valid use case, so another contribution idea if you are keen is to turn it into a tutorial on how to write a read-only API dataset. I think we would be happy to include it in our docs along side the Creating a new dataset tutoria. What do you think?

@limdauto,
Thank you for your quick reply!
As a data engineer I am many times tasked of getting data from external data providers via an API.
I feel like a generic dataset such the one I have in this PR will be super useful for my current and future engagements.
I have looked at both JSON and Text DataSets. They only access a URL via a GET request. You have no control over the request attributes, method, headers, auth, etc...
Most APIs out there, they at least require some kind of authentication headers.
I took a look at the HTTPFileSystem from fsspec. It seems like it has the exact limitations I mentioned above, similar to JSON / Text DataSets. https://filesystem-spec.readthedocs.io/en/stable/_modules/fsspec/implementations/http.html#HTTPFileSystem
I agree, many APIs have totally different structures and paginations. That is the case for many of the data sources in Kedro. For example the pandas.SQLQueryDataSet, you usually never just blindly run a static SQL query. You will most likely require require to add some dynamic filtering, for example, start and end dates.
I have solved that requirement by creating my own datasource while extending the SQLQueryDataSet. Example:
`
class MySQLDataSet(SQLQueryDataSet):
def init(
self, credentials: Dict[str, Any], load_args: Dict[str, Any] = None
) -> None:
super().init(self._get_sql(), credentials, "ORACLE", load_args)
def _get_sql(self) -> str:
    start_date = datetime.now() - 1 month
    end_date = datetime.now()
    return "select * from my_table where date is between " + start_date + " and " + end_date

`
With similar mind set for more complicated API calls the developer can extend my APIDataSet and as creative as the desire !

@ppsimatikas, please let me know if you would like to go ahead and address the comments above. Otherwise, I'm happy to take over and finish it up.

@ppsimatikas, please let me know if you would like to go ahead and address the comments above. Otherwise, I'm happy to take over and finish it up.

I am about to take care of them :) Thank you for ensuring this goes through !

@DmitriiDeriabinQB All comments have been addressed.

Hey team 😄 ! Sorry for reviving a dead thread, but have we considered extending this dataset to support API 'save' operations: e.g., posting pipeline outputs to another endpoint? I have several systems (e.g., pricing, recommenders) that need to push API requests to various other microservices in our stack.
I was wondering whether I should simply extend the API Dataset, or whether there was another best-practice way of handling these 'live' pipeline outputs?

@96imranahmed Hey, how are you doing? Are you using Kedro at Pathao?
I think what you are looking for the after_pipeline_run hook: https://kedro.readthedocs.io/en/0.17.0/kedro.framework.hooks.specs.PipelineSpecs.html#kedro.framework.hooks.specs.PipelineSpecs.after_pipeline_run -- you get access to the run_result in that hook, so you can just instantiate an HTTP request with that payload. If you want to do it on a per-node basis, you can use the after_node_run hook. It fits better than piggybacking on the APIDataSet

Hey @limdauto - doing well! Yes, trying to get the team to push to using Kedro for data-pipelining of our ML model stack (as opposed to doing everything through SQL).
Fabulous - I didn't know these existed! Will get this hooked-up now :) Thanks!


(readme): update README and changelog

Codecov Report

Merging #109 (7c83893) into release-1.0.0 (c342153) will increase coverage by 0.30%.
The diff coverage is 100.00%.

@@                 Coverage Diff                 @@
##             release-1.0.0     #109      +/-   ##
===================================================
+ Coverage            53.53%   53.83%   +0.30%
  Complexity             385      385
===================================================
  Files                   42       42
  Lines                 3413     3388      -25
  Branches               265      262       -3
===================================================
- Hits                  1827     1824       -3
+ Misses                1400     1378      -22
  Partials               186      186

Impacted Files
Coverage Δ
Complexity Δ

.../webank/wecross/stub/fabric/FabricStubFactory.java
8.86% <ø> (ø)
4.00 <0.00> (ø)

...ecross/stub/fabric/hub/HubChaincodeDeployment.java
22.58% <ø> (+3.13%)
2.00 <0.00> (ø)

...ss/stub/fabric/proxy/ProxyChaincodeDeployment.java
27.45% <ø> (+4.50%)
2.00 <0.00> (ø)

.../wecross/stub/fabric/ChaincodeResourceManager.java
77.00% <100.00%> (ø)
21.00 <3.00> (ø)

...k/wecross/stub/fabric/FabricConnectionFactory.java
93.58% <100.00%> (ø)
11.00 <3.00> (ø)

...a/com/webank/wecross/stub/fabric/FabricDriver.java
58.36% <100.00%> (-0.23%)
55.00 <0.00> (ø)

...m/webank/wecross/stub/fabric/FabricConnection.java
58.33% <0.00%> (+0.11%)
54.00% <0.00%> (ø%)

...ross/stub/fabric/proxy/ProxyChaincodeResource.java
56.04% <0.00%> (+0.60%)
10.00% <0.00%> (ø%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ad74bd5...7c83893. Read the comment docs.


Uncaught Error: Syntax error, unrecognized expression: #

After upgrading to redoc.min.js 1.22.2 w/ jQuery 3.3.1 (to resolve a jQuery vulnerability found during a security scan), and either searching or scrolling to an anchor, we get this error in the browser's JavaScript console:
Uncaught Error: Syntax error, unrecognized expression: #operation/createPolicy
    at Function.Sizzle.error (jquery-3.3.1.js?ver=3.3.1:1541)

This error may have been triggered by a bug fix in jQuery that has been breaking things elsewhere:
https://github.com/jquery/jquery/issues/3779
Apparently selectors must now be surrounded in double quotes (").

ReDoc is not using and even has no jquery in the dependencies.
Why do you think it is ReDoc related?

Closing as inactive
Feel free to continue discussion here though


Make the id space management robust against irregular shutdowns

Does this need a changelog entry?

Does this need a changelog entry?

No, this is a follow up for #908.

This is ready for a re-review.


Memories

Added code to automatically remove messages when we hit the model context limit
Updated to the latest gpt3.5-turbo
Added a save and load feature to the history object that automatically loads the last state of the agent
Added a bio to the history object that we can use to manually add some diversity when using multiple agents
Added a memory to the history object that attempts to summarize what the bot has learned
Moved the system message to the history class because it depends on members of the history class

We could move this to its own file or make a new prompting utils file

I also added a world function for getting the position of a certain block type

@MaxRobinsonTheGreat Can you rereview this with the updates?


Use fastdom to improve performance

https://github.com/wilsonpage/fastdom
https://developers.google.com/web/fundamentals/performance/rendering/avoid-large-complex-layouts-and-layout-thrashing

We'll add it on #36

Closed at 327dc1971e35819f06d450bc7507c20367c82cf0.


Flux resolution calc comfy ui

Hey love this Node for Comfy - I compiled a Ratio html viewer - it exported a PNG of your list -  hoping it helps adding a preview onto the node when switching. :) Cant share the html to Git.

Hi,
Thanks, but I can add a preview output to the node that generates the ratio preview. The preview file you made I cannot direct use that but generate something similar.
I want to understand the use case for a preview before I decide whether it should be coded in the flux resolution calculator. Does it help to see it visually?

It does help to see the ratio, it’s a really great node ! It would be cool if the node had a preview above like blender texture nodes.. but I’m not sure if that’s possible :) cheers 🥂

Okay so let me see what I can do. I was about to push the regional flux control nodes, Will try and put something for the calculator preview. Will show it here first before finalizing it. Give me some time.

It does help to see the ratio, it’s a really great node ! It would be cool if the node had a preview above like blender texture nodes.. but I’m not sure if that’s possible :) cheers 🥂

Hi How does this look?

Looks awesome! Does comfy support this type of config ? If not the preview is very helpful.

Great, I will have it pushed in the next update with region control for flux.
The image you posted is a node-based interface, but I don't know which one. It is too low-resolution for me to read anything.
This looks like a 3d Blender output, maybe I am wrong? If it's that, comfy ui doesn't have that. Developing a system like this is comfy. I have to do a feasibility check, and it would be under a paid project (private release only). However, You can find open-source links where Blender is linked to Comfy UI via a bridge.
If its not 3D let me know what it is or upload a higher resolution image please.

Hope this is better. its just blender nodes and the preview is above the node as a selection.. again not sure if comfy is able to do this - your original preview style is super helpful and understand all the time you put in your own :)

Flux Resolution Calculator Update is pushed.
@Sterling177 Doing something like this requires a lot of time. It would be a paid project for me.


Fix br open type

Codecov Report

Merging #83 into master will decrease coverage by 0.02%.
The diff coverage is 75%.

@@            Coverage Diff            @@
##           master     #83      +/-   ##
=========================================
- Coverage   68.33%   68.3%   -0.03%
=========================================
  Files          39      39
  Lines        2845    2849       +4
  Branches       49      49
=========================================
+ Hits         1944    1946       +2
- Misses        874     876       +2
  Partials       27      27

Impacted Files
Coverage Δ

src/tokentree/utils/TokenTreeCheckUtils.hx
37.12% <75%> (+0.47%)
:arrow_up:

src/tokentree/walk/WalkClass.hx
77.01% <0%> (-2.3%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ea88096...da735b2. Read the comment docs.


Add NLP Sandbox badge to README

[ ] Configure URL redirect for https://nlpsandbox.io/tools/date-annotator/leaderboard

Badge preview:

Badge preview:


Use numpy's import_array1(0) to avoid return type weirdness

This affected alpine linux (a lightweight image commonly used in docker builds) due to its use of muslinux, for which we don't have pre-compiled wheels.
This should fix issue #43

The issue has a link to a description of the issue and why this is a valid fix near the end of the discussion.


Cannot find module 'webpack/lib/MultiEntryPlugin'

"prerender-loader": "^1.3.0",
"webpack": "^5.3.2",
"webpack-cli": "^3.3.12",

when I run webpack it turn out this.
I can not find webpack/lib/MultiEntryPluginin webpack5 lib.
but how to fix this?

It looks like webpack 5 has deprecated the MultiEntryPlugin. The fix isn't too bad, but there's several other changes in webpack 5 that break this plugin. I have a working fork up at https://github.com/nettoolkit/prerender-loader, but I've only tested it for my project.

It looks like webpack 5 has deprecated the MultiEntryPlugin. The fix isn't too bad, but there's several other changes in webpack 5 that break this plugin. I have a working fork up at https://github.com/nettoolkit/prerender-loader, but I've only tested it for my project.

It looks like webpack 5 has deprecated the MultiEntryPlugin. The fix isn't too bad, but there's several other changes in webpack 5 that break this plugin. I have a working fork up at https://github.com/nettoolkit/prerender-loader, but I've only tested it for my project.

thx for your answer

It looks like webpack 5 has deprecated the MultiEntryPlugin. The fix isn't too bad, but there's several other changes in webpack 5 that break this plugin. I have a working fork up at https://github.com/nettoolkit/prerender-loader, but I've only tested it for my project.

thx for your answer

Is there any plan on supporting webpack 5? :thought_balloon:  @developit

Is there any plan on supporting webpack 5? :thought_balloon:  @developit

It looks like webpack 5 has deprecated the MultiEntryPlugin. The fix isn't too bad, but there's several other changes in webpack 5 that break this plugin. I have a working fork up at https://github.com/nettoolkit/prerender-loader, but I've only tested it for my project.

@dwoznicki Any plans to publish this package, since this project seems dead?

This project isn't dead, the fork just needs to be a Pull Request.

@developit I can make a PR for my fork, but I have a number of additional changes that are unrelated to the Webpack 5 upgrade. Is that okay?

That makes it a bit more difficult. I'm assuming these are functional changes?

Well, it's a mixed bag. Off the top of my head, the other changes I've made include:

fix for #51
removed microbundle as a dependency (I can't build the current version due to developit/microbundle#74)
various dependency upgrades

Hi guys, any update here?

Upgrading webpack is so painfull. But awaiting for PR is pure suffering.
Wazup?

@developit It looks like I'm no longer blocked by developit/microbundle#74, so I'll look into making my fork into a PR.
I'd really like to do something about #51 though. I can't incorporate prerender-loader into my project without this fix, so even if I submit a webpack 5 patch, I'll still need to keep using my fork. Is there something I could do to get this fix merged?

@dwoznicki , I'm fine using your fork, but can't find @nettoolkit/prerender-loader at npmjs.
Can you push it into public repo as temporary workaround?

Have you tried referencing the GitHub repo in your package.json? I currently have the following line in my dependencies to keep prerender-loader up-to-date:
"prerender-loader": "github:nettoolkit/prerender-loader#master",

Wtf, I can do this? Lol.
/ () __ () \

@dwoznicki thank you so much for the fix! it seems that prerender-loader is the only working webpack5 tool to generate static files. Others (https://github.com/markdalgleish/static-site-generator-webpack-plugin) didn't work for me.
It's quite sad that most of such tools getting deprecated in favour of heavyweight frameworks like Gatsby or Next.js

Any news on this one?

Sadly seems dead :(


Update jobs.json

added a new job role

Hey @handsomeharsh I am sorry but this is a very minimal contribution kindly add more to it then, I'll happily merge your request
Sorry for the trouble


Initial implementation of Subscription Docs

Adds a advanced topic for Subscriptions in the book and a small example for it without the usage of a web integration. This PR fixes https://github.com/graphql-rust/juniper/issues/588.
PR Checklist
Before submitting a PR, you should follow these steps to prevent redundant churn or CI failures:

[X] Ensure proper formatting
[X] Run all tests
[X] Update the CHANGELOG

These look awesome! @nWacky any interest in making sure it matches what you were thinking?

@nWacky I have added these requested changes into the documentation, thanks for the review.


[WIP] Add ACL functionality to named.conf template

It makes sense, but we're really lacking in documentation and this parameter needs some. I don't it's obvious that it needs to be a hash containing string -> list.

Docs added.

:+1: for the code. Would you mind adding a test to dns_init_spec.rb?

@ekohl will do so once #57 is merged and I can rebase.

please rebase :)

@mmoll rebased.

I believe I've addressed all of the issues raised, and this is in a merge-able state.

Thanks!


WIP: [x64-windows-intel] new Triplet:

Requirements:
Intel C/C++/Fortran compiler via oneAPI
manuall copy of llvm-ar and subsequent rename to llvm-ranlib.
cmake platform seems to be Windows-Clang
opened:
~~https://gitlab.kitware.com/cmake/cmake/-/issues/21588~~

[x] waiting for cmake to actually support those compilers.

Any progress?

Please temporary close this PR if you don't have spare time to finish it.

Please temporary close this PR if you don't have spare time to finish it.

Mainly waiting for other PRs to be merge first and if I should hijack one of the normal triplets for testing ;)

Ping @Neumann-A for response. Is work still being done for this PR?

Ping @Neumann-A for response. Is work still being done for this PR?

waiting for other complicated stuff to getting merged.

Ping @Neumann-A for response. Is work still being done for this PR?

waiting for other complicated stuff to getting merged.

Please link other PR to here, this will make it easier for us to track this PR.

#26617
#26370
#26337
#24327

Closing this PR as there has not been activity in over 4 months. Please feel free to reopen it, or another PR, should you wish to continue this.
Note that we were recently forced to no longer install icc in the test lab, though that doesn't necessarily matter for this given that we wouldn't have made icc an official triplet anytime soon.


Reference mismatch

Hi there,
I was wondering if you'd consider implementing a specific flag for vt normalize.  We're getting an error that looks like this with some of our genomes:
Variant is not consistent: chrY:59034049-59034049 - N(REF) vs A(FASTA)
This is because it's in the PAR region and the reference we're feeding it has that masked.  We don't really want to use the -n flag because we would like to catch other errors if they happen, but we'd like to ignore errors where the REF is N.
Thanks,
Denise

The -n flag when enabled, will still output the message that the reference is not consistent.  Is it possible to make a scan of the logs for such warnings that do not occur in the the pseudoautosomal regions?

Hi there,
It is possible to do that, but we have 7k+ genomes that we're processing, so having an additional process to scan the logs is additional overhead.  Figured I'd inquire here whether it'd be possible to have a flag for ignoring this specific case.
Thanks,
Denise

I'm a bit hesitant about having that additional feature as I'd prefer that errors are captured.   It is definitely possible though but I need to convince myself that it is necessary.
Is it possible for the normalization step to use a reference sequence file which does not mask those bases.  Because I assume that the variants were called using a reference that did not mask those bases in the first place.

Hi there,
I do prefer that it prints the log, but that we can not have vt exit when it runs into a REF N situation.  I'm inquiring about an additional flag.  Example behaviour:
vt normalize -o output.vcf -r hg19.ref.fa input.vcf

dies on all reference mismatches
vt normalize -n -o output.vcf -r hg19.ref.fa input.vcf
warns on all reference mismatches
vt normalize -s -o output.vcf -r hg19.ref.fa input.vcf
warns on reference mismatches where ref is N, dies on other reference mismatches

Thanks,
Denise

Ah, I understand now.  I think that is a good feature.

can you please pull it and try it.  The option is -m
http://genome.sph.umich.edu/wiki/Vt#Normalization

FYI seems to be working well.  Thanks.


Integrate TLC SI Summer Schedule

https://docs.google.com/spreadsheets/d/1MmIKjgLCbtk2Py38a9pP2IkLv6fXod1WeheOfpag7Hg/edit#gid=1091060376

There is an issue with the data provided on google docs because many of the professors are labeled as staff. On the client side we use the name of the professor to distinguish different SI sessions for different students.

All professor names have been updated by TLC
TLC JSON updated by and deployed - thanks @g9singh, @a6wu


Vulnerability roundup 76: imagemagick-6.9.9-34: 19 advisories

search, files

[ ] CVE-2018-16323 (nixos-19.09)
[ ] CVE-2018-16328 (nixos-19.09)
[ ] CVE-2018-16329 (nixos-19.09)
[ ] CVE-2018-20467 (nixos-19.09)
[ ] CVE-2018-6405 (nixos-19.09)
[ ] CVE-2019-10131 (nixos-19.09)
[ ] CVE-2019-10714 (nixos-19.09)
[ ] CVE-2019-13133 (nixos-19.09)
[ ] CVE-2019-13134 (nixos-19.09)
[ ] CVE-2019-13135 (nixos-19.09)
[ ] CVE-2019-13136 (nixos-19.09)
[ ] CVE-2019-13137 (nixos-19.09)
[ ] CVE-2019-14980 (nixos-19.09)
[ ] CVE-2019-14981 (nixos-19.09)
[ ] CVE-2019-7175 (nixos-19.09)
[ ] CVE-2019-7395 (nixos-19.09)
[ ] CVE-2019-7396 (nixos-19.09)
[ ] CVE-2019-7397 (nixos-19.09)
[ ] CVE-2019-7398 (nixos-19.09)

Scanned versions: nixos-19.09: e34ac949d1b. May contain false positives.

The PR bumping imagemagick to 6.9.10-68 got in, and it seems to be a fairly recent release.
However, https://legacy.imagemagick.org/script/changelog.php doesn't state this fixes above CVE's (the same for NVD)
cc @risicle

No that would be far too helpful.

At least CVE-2019-7397 is not fixed in 6.9.10-68: https://github.com/ImageMagick/ImageMagick/commit/306c1f0fa5754ca78efd16ab752f0e981d4f6b82
Are there any incompatibilities between 6.9 and 7.0, or why don't we bump it to the freshest release?

@d-goldin I think that was mostly due to time - if you file a PR bumping it, we can look at the fallout in that PR.

Ah, thanks. I didn't see the comment on the thread. I'll give it a try and see.

Imagemagick 6 and 7 are different enough that upstream maintain them in two separate repositories: https://github.com/ImageMagick/ImageMagick https://github.com/ImageMagick/ImageMagick6

CVE-2019-7397 is indeed fixed in 6.9.10-68: https://github.com/ImageMagick/ImageMagick6/commit/3b28c8d93aa469f6d90c8b3c05fe3d88c2584e32#diff-0dae0ad5001ce7208c2567e31867ff99

@risicle: Thanks for locating the other CVE in the v6 repo. Got confused there for a sec. You are right wrt changes - there is quite a bunch; https://imagemagick.org/script/porting.php - it looks like version 6 still has around 2 years of support in it.
I'd suggest then that any matters imagemagick 7 we should open a separate issue and here just make sure that we can find the appropriate commits for the CVEs in the v6 repo to confirm.

We have now 6.9.10-69 in 19.09. Still confused which CVEs got solved and which not...

That should cover all published CVEs AFAIK.

Yes, looks good. Thanks everyone! :-)


[BUG] Full Screen causes colors to be inverted while using DirectX3D11.

Acknowledgement of preliminary instructions

[X] I have read the preliminary instructions, and I am certain that my problem has not already been addressed.
[X] I have thoroughly looked through the available Wiki articles and could not find a solution to my problem.
[X] I am using the latest version of Bloxstrap.
[ ] I did not answer truthfully to all the above checkboxes.

What problem did you encounter?
Basically, if I use DirectX3D11 when using exclusive fullscreen, it caused my screen colors to be inverted for some reason. I don't what is causing this, and with Vulkan it doesn't do this, but there's a different issue there, I can't close out of True Fullscreen so I can't really use it. at all. My hardware is AMD A6-7310 for my CPU, and my graphics is Radeon R4. Does it have to do with my hardware?
 DirectX3D 11 is the full name for it, but it's called Direct3d 11 in bloxstrap

That's the full name. DirectX3D 11 is the full name for it, but it's called Direct3d 11 in bloxstrap

hey buddy DirectX 11 and Direct3D 11 is different okay
people say DX11 and D3D11 theres nothing as DX3D11

That's the full name. DirectX3D 11 is the full name for it, but it's called Direct3d 11 in bloxstrap

hey buddy DirectX 11 and Direct3D 11 is different okay
people say DX11 and D3D11 theres nothing as DX3D11
now yes i know that Direct3D is a part of DirectX used to render 3D stuff but nobody says DirectX3D
you can call it D3DX[version] tho

oh btw just download DX11/12 libs, they come with D3D libs too.
try that and lmk

Does it have to do with my hardware?

Likely, yes. Well, more with software. I don't really see how this could happen if it wasn't a GPU config problem. Check your graphics settings in the Radeon Software.


Translate in Chinese laguage

Hello, I have learned a lot in this Repo.
I want to share it with more Chinese.
Can I translate Chinese for many people?

Hi @gaoljhy. Sure you can. If you want you can make a PR and add the translation to this repo.  If you want to publish it elsewhere it is also fine.

Wow ,big thanks
if i'm done ,i will make a PR right away


yardoc claims --list-undoc is unknown option

yardoc <IP_ADDRESS> claims --list-undoc is an unknown option, but honors it anyways:

$ yardoc --list-undoc
[warn]: Unrecognized/invalid option: --list-undoc
Files:          28
Modules:         2 (    2 undocumented)
Classes:        27 (   26 undocumented)
Constants:       3 (    0 undocumented)
Methods:       159 (   83 undocumented)
 41.88% documented

Undocumented Objects:

(in file: lib/sslyze/cert_info.rb)
SSLyze::CertInfo#validation
SSLyze::CertInfo#ocsp_stapling
SSLyze::CertInfo#chain
SSLyze::CertInfo
...

This is weird but expected behavior. Since unrecognized options are warnings, not hard stops, YARD keeps running, and eventually YARD passes all args through to yard stat under the covers to print the stats at the end. yard stat knows what --list-undoc is, and processes it normally.

Sounds like yard shouldn't parse the options, but simply pass arguments to yard stat for parsing.

Not everything is done in yard stat, only post processing of object statistics. yardoc (which is actually yard doc) does actual work (most of it, in fact), which requires actual arg parsing, and then calls yard stat unless the yard doc option --no-stats is passed:

yard doc --no-stats option: https://github.com/lsegal/yard/blob/master/lib/yard/cli/yardoc.rb#L699-L701
yard stats sub-command: https://github.com/lsegal/yard/blob/master/lib/yard/cli/yardoc.rb#L259-L264

The issue is that YARD::CLI::Stats needs the same options that were passed to YARD for any shared options. The only solution would be to strip out unrecognized args from YARD prior to passing off to sub-commands, but I'm not sure what the benefit of that is, since --list-undoc should not be passed to yard doc in the first place.

Cleaning out old issues.
I'm marking this as WontFix since, though it is a weird UX, it is an edge case that also doesn't really break anything (it works when it says it doesn't, that's not quite as bad as not working when it says it does).
Although I have no plans to take this on myself, I would accept a PR fix it assuming it didn't add significant complexity to the codebase.


Force app to use https

As a general policy for both security and SEO, we want everything running on cu.edu to use https over http. Heroku supports both the http and https protocols for apps by default. Because of the way SSL is terminated at the load balancer and traffic between the load balancer and the dynos within Heroku's internal network is not encrypted in the common runntime, getting an app to default into https can be tricky. https://www.npmjs.com/package/force-ssl-heroku claims to handle this for apps using Express.
There may be a better way to do this, but force-ssl-heroku is a way.

Heroku seems to redirect by default as http://essential-cu.herokuapp.com/impact-reports/onward always ends up on HTTPS for me. If redirecting the initial request is fine then I think this is already covered. To my understanding, the NPM package mentioned above performs a redirect on specified paths so it looks like similar behavior without the need to add a dependency.

http://essential-stage.cu.edu/ -> http://essential-stage.cu.edu/impact-reports/onward
Going directly to a url like http://essential-stage.cu.edu/impact-reports/onward/druwrasewagituthesleshistosluvanofriladrovushuwrusliuanaspauaphathibrivewrowrude does not redirect

I would try to just use the conditinal in https://github.com/rangle/force-ssl-heroku/blob/master/force-ssl-heroku.js into the server.js file in this project.
  if (process.env.NODE_ENV === 'production' &&
    req.headers['x-forwarded-proto'] !== 'https') {

    sslUrl = ['https://', req.hostname, req.url].join('');
    return res.redirect(sslUrl);
  }

Fastly is forcing the production instance into https, but this is not working as expected. Attempts to access http://essential.cu.edu/impact-reports/onward are redirected to https://essential-cu.herokuapp.com/impact-reports/onward. Going directly to https://essential.cu.edu/impact-reports/onward works as expected with a valid cert.

I tried overriding host header from essential-cu.herokuapp.com to essential.cu.edu in the configuration, that solves the issue in the redirect but results in a Heroku level error.

I turning off the Force TLS and enable HSTS option resulted in Fastly resulted in being able to access a URL like http://essential.cu.edu/impact-reports/onward  with a Not Secure warning. Going to look at the option @alexfinnarn suggested in https://github.com/CUCentralAdvancement/essential-cu/issues/43#issuecomment-702841789

keep in mind https://status.heroku.com/incidents/2110 is not resolved and showed up today.

server.all("*", (req, res) => {
  if (req.port === 3000 || req.headers['x-forwarded-proto'] !== 'https') {
    console.log(req.protocol);
    const host = req.hostname == 'localhost' ?  'localhost:3443' : req.hostname;
    return res.redirect('https://' + host + req.originalUrl);
  }

  return handle(req, res);
});

I tried my hand at this and kept running into redirect loops. Heroku seems to turn HTTPS requests back into HTTP requests causing a loop. console.log(req.protocol) kept logging as "http" which seems impossible with return res.redirect('https://' + host + req.originalUrl) following. That's why I think headers need to be used based on spooky Heroku networking.
With a lot of other issues to work on, I don't think this issue is that important to fix, but it probably does relate to what's happening with Drupal on Heroku and HTTP/HTTPS redirect issues.

https://help.heroku.com/J2R1S4T8/can-heroku-force-an-application-to-use-ssl-tls
Heroku recommends using a different node package...well, they point to a search page and then a different package pops up.
I made https://github.com/CUCentralAdvancement/essential-cu/pull/120 which might work but probably needs to be tested on staging and production as I think review apps are still process.env.NODE_ENV === 'dev'.
The review app gives a 503 but nothing in the logs so maybe this is related to Heroku intermittent failures with the build phase?

// This should be done on the server, but Heroku's internal networking makes it difficult.
if (window.location.host === 'essential.cu.edu' && window.location.protocol !== 'https:') {
  window.location.href = 'https://' + window.location.host + window.location.pathname + window.location.search;
}

Not a great "solution" but it works and doesn't cause a redirect loop. Since Next.js uses a template to generate pages, it only has to be placed in one spot...I could actually run this at a higher level so it doesn't have to be in the actual template files, but I guess it's fine for this project. Next.js might only fire it for the initial page load too.

This was done "well enough" for the project.


update packages

:white_check_mark: Build TheIdServer 7.1.1-fix-dependencies1-0010 completed (commit https://github.com/Aguafrommars/TheIdServer/commit/2baf0af4a6 by @)


Bump golang to v1.21

Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

[ ] Added test case and related test data
[ ] Update the functional test

Checklist:

[ ] I have performed a self-review of my own code
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules
[ ] I ran golangci-lint

@nunnatsa

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 84.786%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
1109

Relevant Lines:
1308

💛  - Coveralls