bot crashes every time someone speaks

ATBotcb9-6b0d-45

<EMAIL_ADDRESS>start /var/lib/pufferd/servers/7a03dcb9-6b0d-4585-8bea-b7f88d1ba4be
node dist

I am ready!
uncaughtException TypeError: Cannot read property 'users' of undefined
at PollWatcher.shouldRun (/var/lib/pufferd/servers/7a03dcb9-6b0d-4585-8bea-b7f88d1ba4be/dist/BaseModule.js:133:40)
at watchers.(anonymous function).forEach.watcher (/var/lib/pufferd/servers/7a03dcb9-6b0d-4585-8bea-b7f88d1ba4be/dist/WatcherBus.js:100:33)
at Array.forEach ()
at Client.bot.on.args (/var/lib/pufferd/servers/7a03dcb9-6b0d-4585-8bea-b7f88d1ba4be/dist/WatcherBus.js:99:39)
at emitOne (events.js:120:20)
at Client.emit (events.js:210:7)
at MessageCreateHandler.handle (/var/lib/pufferd/servers/7a03dcb9-6b0d-4585-8bea-b7f88d1ba4be/node_modules/discord.js/src/client/websocket/packets/handlers/MessageCreate.js:9:34)
at WebSocketPacketManager.handle (/var/lib/pufferd/servers/7a03dcb9-6b0d-4585-8bea-b7f88d1ba4be/node_modules/discord.js/src/client/websocket/packets/WebSocketPacketManager.js:102:65)
at WebSocketConnection.onPacket (/var/lib/pufferd/servers/7a03dcb9-6b0d-4585-8bea-b7f88d1ba4be/node_modules/discord.js/src/client/websocket/WebSocketConnection.js:325:35)
at WebSocketConnection.onMessage (/var/lib/pufferd/servers/7a03dcb9-6b0d-4585-8bea-b7f88d1ba4be/node_modules/discord.js/src/client/websocket/WebSocketConnection.js:288:17)
uncaughtException { Error: RSV2 and RSV3 must be clear
at Receiver.getInfo (/var/lib/pufferd/servers/7a03dcb9-6b0d-4585-8bea-b7f88d1ba4be/node_modules/ws/lib/Receiver.js:184:18)
at Receiver.startLoop (/var/lib/pufferd/servers/7a03dcb9-6b0d-4585-8bea-b7f88d1ba4be/node_modules/ws/lib/Receiver.js:153:16)
at Receiver.add (/var/lib/pufferd/servers/7a03dcb9-6b0d-4585-8bea-b7f88d1ba4be/node_modules/ws/lib/Receiver.js:139:10)
at TLSSocket._ultron.on (/var/lib/pufferd/servers/7a03dcb9-6b0d-4585-8bea-b7f88d1ba4be/node_modules/ws/lib/WebSocket.js:142:22)
at emitOne (events.js:115:13)
at TLSSocket.emit (events.js:210:7)
at addChunk (_stream_readable.js:266:12)
at readableAddChunk (_stream_readable.js:253:11)
at TLSSocket.Readable.push (_stream_readable.js:211:10)
at TLSWrap.onread (net.js:587:20)
type: 'error',
target:
WebSocket {
domain: null,
_events:
{ message: [Object],
open: [Object],
error: [Object],
close: [Object] },
_eventsCount: 4,
_maxListeners: undefined,
readyState: 2,
bytesReceived: 3620,
extensions: {},
protocol: '',
_binaryType: 'nodebuffer',
_finalize: { [Function: bound finalize] __ultron: 0 },
_finalizeCalled: false,
_closeMessage: null,
_closeTimer: null,
_closeCode: null,
_receiver:
Receiver {
binaryType: 'nodebuffer',
extensions: {},
maxPayload: 0,
bufferedBytes: 145,
buffers: [Array],
compressed: false,
payloadLength: 465,
fragmented: 0,
masked: false,
fin: true,
mask: null,
opcode: 1,
totalPayloadLength: 0,
messageLength: 0,
fragments: [],
cleanupCallback: null,
hadError: false,
dead: false,
loop: true,
onmessage: [Function],
onclose: [Function],
onerror: [Function],
onping: [Function],
onpong: [Function],
state: 0 },
_sender:
Sender {
perMessageDeflate: undefined,
_socket: [Object],
firstFragment: true,
compress: false,
bufferedBytes: 0,
deflating: false,
queue: [],
onerror: [Function] },
_socket:
TLSSocket {
_tlsOptions: [Object],
_secureEstablished: true,
_securePending: false,
_newSessionPending: false,
_controlReleased: true,
_SNICallback: null,
servername: null,
npnProtocol: false,
alpnProtocol: false,
authorized: true,
authorizationError: null,
encrypted: true,
_events: [Object],
_eventsCount: 8,
connecting: false,
_hadError: false,
_handle: [Object],
_parent: null,
_host: 'gateway.discord.gg',
_readableState: [Object],
readable: true,
domain: null,
_maxListeners: undefined,
_writableState: [Object],
writable: true,
allowHalfOpen: false,
_bytesDispatched: 709,
_sockname: null,
_pendingData: null,
_pendingEncoding: '',
server: undefined,
_server: null,
ssl: [Object],
_requestCert: true,
_rejectUnauthorized: true,
parser: null,
_httpMessage: [Object],
read: [Function],
_consuming: true,
_idleNext: null,
_idlePrev: null,
_idleTimeout: -1,
[Symbol(asyncId)]: 688,
[Symbol(bytesRead)]: 0 },
_ultron: Ultron { id: 0, ee: [Object] },
protocolVersion: 13,
_isServer: false,
url: 'wss://gateway.discord.gg/?v=6&encoding=json',
_req: null } }
uncaughtException { Error: RSV1 must be clear
at Receiver.getInfo (/var/lib/pufferd/servers/7a03dcb9-6b0d-4585-8bea-b7f88d1ba4be/node_modules/ws/lib/Receiver.js:191:18)
at Receiver.startLoop (/var/lib/pufferd/servers/7a03dcb9-6b0d-4585-8bea-b7f88d1ba4be/node_modules/ws/lib/Receiver.js:153:16)
at Receiver.add (/var/lib/pufferd/servers/7a03dcb9-6b0d-4585-8bea-b7f88d1ba4be/node_modules/ws/lib/Receiver.js:139:10)
at TLSSocket._ultron.on (/var/lib/pufferd/servers/7a03dcb9-6b0d-4585-8bea-b7f88d1ba4be/node_modules/ws/lib/WebSocket.js:142:22)
at emitOne (events.js:115:13)
at TLSSocket.emit (events.js:210:7)
at addChunk (_stream_readable.js:266:12)
at readableAddChunk (_stream_readable.js:253:11)
at TLSSocket.Readable.push (_stream_readable.js:211:10)
at TLSWrap.onread (net.js:587:20)
type: 'error',
target:
WebSocket {
domain: null,
_events:
{ message: [Object],
open: [Object],
error: [Object],
close: [Object] },
_eventsCount: 4,
_maxListeners: undefined,
readyState: 2,
bytesReceived: 3624,
extensions: {},
protocol: '',
_binaryType: 'nodebuffer',
_finalize: { [Function: bound finalize] __ultron: 0 },
_finalizeCalled: false,
_closeMessage: null,
_closeTimer:
Timeout {
_called: false,
_idleTimeout: 30000,
_idlePrev: [Object],
_idleNext: [Object],
_idleStart: 391052,
_onTimeout: [Object],
_timerArgs: [Array],
_repeat: null,
_destroyed: false,
[Symbol(asyncId)]: 783,
[Symbol(triggerAsyncId)]: 779 },
_closeCode: null,
_receiver:
Receiver {
binaryType: 'nodebuffer',
extensions: {},
maxPayload: 0,
bufferedBytes: 147,
buffers: [Array],
compressed: false,
payloadLength: 465,
fragmented: 0,
masked: false,
fin: true,
mask: null,
opcode: 1,
totalPayloadLength: 0,
messageLength: 0,
fragments: [],
cleanupCallback: null,
hadError: false,
dead: false,
loop: true,
onmessage: [Function],
onclose: [Function],
onerror: [Function],
onping: [Function],
onpong: [Function],
state: 0 },
_sender:
Sender {
perMessageDeflate: undefined,
_socket: [Object],
firstFragment: true,
compress: false,
bufferedBytes: 0,
deflating: false,
queue: [],
onerror: [Function] },
_socket:
TLSSocket {
_tlsOptions: [Object],
_secureEstablished: true,
_securePending: false,
_newSessionPending: false,
_controlReleased: true,
_SNICallback: null,
servername: null,
npnProtocol: false,
alpnProtocol: false,
authorized: true,
authorizationError: null,
encrypted: true,
_events: [Object],
_eventsCount: 8,
connecting: false,
_hadError: false,
_handle: [Object],
_parent: null,
_host: 'gateway.discord.gg',
_readableState: [Object],
readable: true,
domain: null,
_maxListeners: undefined,
_writableState: [Object],
writable: true,
allowHalfOpen: false,
_bytesDispatched: 709,
_sockname: null,
_pendingData: null,
_pendingEncoding: '',
server: undefined,
_server: null,
ssl: [Object],
_requestCert: true,
_rejectUnauthorized: true,
parser: null,
_httpMessage: [Object],
read: [Function],
_consuming: true,
_idleNext: null,
_idlePrev: null,
_idleTimeout: -1,
[Symbol(asyncId)]: 688,
[Symbol(bytesRead)]: 0 },
_ultron: Ultron { id: 0, ee: [Object] },
protocolVersion: 13,
_isServer: false,
url: 'wss://gateway.discord.gg/?v=6&encoding=json',
_req: null } }
Also, what is the command to shut down the bot from console? I cant find it in any documentation.

note that is supposed to be (colon)100(colon)30

What's all this pufferd stuff? Might be the cause?
Also there is no shutdown command, just Ctrl+C it

puffered is a free web panel that runs the bot

or rather, puffered is the daemon, for the web panel

wait, i just realised there was more error stuff... i added it

Take a look at my config.json.
The example may be out of date, so look at what you may be missing:
{
    "moderator_channel": "moderation-logs",
    "support_channel": "support",
    "rules_channel": "rules",
    "bot_testing_channel": "bot-testing",
    "non_support_channels": [
        "general",
        "general-chat",
        "minecraft",
        "gaming",
        "entertainment",
        "atlauncher-next",
        "development",
        "website-support",
        "introductions"
    ],
    "moderated_channels": [
        "general",
        "general-chat",
        "minecraft",
        "gaming",
        "entertainment",
        "atlauncher-next",
        "development",
        "website-support",
        "introductions",
        "support",
        "bot-testing"
    ],
    "appeal_url": "https://atl.pw/discord-ban-appeal",
    "bypass": {
        "roles": [
            "ATLauncher Staff",
            "NodeCraft Staff",
            "Moderators",
            "AllmightyBot"
        ],
        "users": []
    },
    "bad_words": [
        "bum",
        "hole"
    ],
    "client_id": "",
    "client_secret": "",
    "client_token": ""
}

Yes, definantly out of date, thankyou


R report termite in the attic

VSCode Version:
OS Version:

Steps to Reproduce:

Does this issue occur when all extensions are disabled?: Yes/No

Looking 4 sugar daddy


What here issues

[x] Whats here rows not showing all activities in box
[x] Sometimes first page shows no rows but subsequent rows have data
[x] Window sometimes renders half off the page
[x] Button shows on current activities page
[ ] Clicking an activity on whats here popup will make the whats here box you drew become that activities geometry
[ ]

some state management code setup for last point but not quite done (whats here whos editing)

going to just hide it on activity page for now.  can open separate issue for 'enable whats here on activity page' if/when we need to prioritize that


Granular compiler tests

Addresses https://github.com/ml6team/fondant-use-cases/issues/46
For each compiler we now have a method get_pipeline_configs() to fetch the essential information about a pipeline (component, arguments, dependencies, hardware, ...) from a compiled specification file. In this way, we no longer face issues with having to modify the specification files after every small modification to the framework. It also makes it easier to call the desired attribute without having to dive deep into the schema of the compiler.

Nice changes, will definitely make testing more easy.
I do wonder if we can't do more with these ComponentConfigs ? Now they are a bit of an afterthought and only used for testing. Maybe we can also use them to configure the components for a runner ?

Nice changes, will definitely make testing more easy.
I do wonder if we can't do more with these ComponentConfigs ? Now they are a bit of an afterthought and only used for testing. Maybe we can also use them to configure the components for a runner ?

Thinking about it now, they seem to very similar to the componentOp main difference is that the componentOp defines static configs whereas the ComponentConifgs include some attributes similar to the ComponentOp with some additional dynamic configs that can only be inferred at compile time (context, volumes, context, ...).
How would you want to use them for a runner?

Not sure yet how to use them, but if we only use them for testing, I would remove them from the core Fondant code. We can either

Move them to our test files
Create a testing.py module if we think it can help users test Fondant pipelines as well. Connexion example

I think we are all feeling the same thing: This is really clean but we only use it for testing right now so either:

use it for more (like configuring a compiler/runner)
or make it test specific only (and then maybe indeed move it from the core

Thanks for the suggestions, moved them over to a separate testing.py


resolves #40, starting on #32

This resolves #40 and I am starting on #32 but that isn't working yet.  I also did some refactoring.  I'm going to leave this open for a bit while I work on this in case anyone has comments.

@zambien any update on this? Why introduce two bug fixes in one PR?  If this PR fixes #40 then it should stand alone so it can be merged.  #32 should have its own PR if it isn't working yet.

@blucas fair points there.  Thanks for the nudge because I forgot about this.  I'll cut a release for this shortly since there shouldn't be any breaking change here.  I had meant to finish this out but my real job and other RL issues interfered. :)

Actually, I'll leave this open and open a different one for the issue that is resolved (#40)


Change example domain to MSFT owned domain

company.com is a valid domain that is not owned by Microsoft.  fabrikam.com is registered by Microsoft and has previously been used in Skype documentation.

:warning: Validation status: warnings

File
Status
Preview URL
Details

msteams-platform/concepts/bots/bots-context.md
:warning:Warning

Details

msteams-platform/concepts/bots/bots-context.md

Line 24: [Warning] Invalid file link:(~/concepts/bots/bots-notifications#bot-or-user-added-to-a-team).
Line 134: [Warning] Invalid file link:(~/concepts/bots/bots-notifications#bot-or-user-added-to-a-team).

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


MINOR: followup to Version Probing improvements

Small follow-up to trunk PR https://github.com/apache/kafka/pull/7423
While debugging the 2.3 VP PR we realized we should remove the leader-tracking from the VP system test altogether. We'd already merged the corresponding trunk PR so I made a quick new PR for trunk (also fixes a missed version bump in one of the log messages)

@guozhangwang @mjsax

Kicked off system tests again just to be safe, passed here: https://jenkins.confluent.io/job/system-test-kafka-branch-builder/3147/


Enable the new model serialization for Avs

Contributing to the Azure SDK
Please see our CONTRIBUTING.md if you are not familiar with contributing to this repository or have questions.
For specific information about pull request etiquette and best practices, see this section.

API change check
API changes are not detected in this pull request.


imlib2 dependent programs cannot be statically linked

https://github.com/bonsai-linux/bonsai/blob/master/ports/shared/README.md

http://bonsai-linux.org/uploads/KaM8W.png
possible answer

@mitchweaver why rewrite?


Google Auth Loop

I am getting stuck in a loop during the step where I select what google account I want to use. I get directed to a page saying that my app isn't verified (post sms verification also), I clicked advanced, see my app name, click it, and redirected back to the select account screen.
I have tried:
Rebuilding Electron
Creating new keys(2 sets)
Using both keys as the credentials.json file
Use multiple browsers
Use different google accounts to try and access the API

try to delete cache of browser, or use another browser(copy auth url and paste it into another)


Is there an option for consistent session?

Use case:
Consistent chat with the model.
Example:
Request (me):  Hi, my favorite color is green.
Response (model): That's cool
Request (me): What is my favorite color?
Response (model): I don't know
--> Model does not "remember" the first text. When using chatGPT via the OpenAI website, the model is able to remember the context of previous conversation.
Is this possible with this package?

You can read it https://help.openai.com/en/articles/6787051-does-chatgpt-remember-what-happened-earlier-in-the-conversation

The package likely can. It's your job to send the whole conversation each time. Explained perfectly here: https://www.youtube.com/watch?v=1KJcSx9UQcM&t=51s
around the 6 minute mark.


Swift Tour- Error Handling

Missing translation of Error Handling section of the SwiftTour.md file

already merged, close please

The translation is missing stuff, I made a pull request with them translated. It's still on review


how to use array in post\put request

Please help me to understand how to do
I sent in request uid[]=1&uid[]=2&groupname=blabla
In django i can get uid with request.POST.getlist('uid[]')
but in restless i receive only last value. uid[] : 2
it must be uid: [1, 2, ]
how can i fix this correctly? (sorry for bad english)
this code work for me but not for all.
       post = {}
       for k,v in a.items():
           if k.find('[')>0:
               k = k[0:k.find('[')]
               n = post.get(k, [])
               n.append(v)
               post.update({k: n })
           else:
               post.update({k:v})

then variable post is
post = {u'uid': [u'1', u'2'], u'groupname': u'blabla'}

my question have not reason. Becouse data packed in json on first sent step. So in server i always receive key-value and cant receive many value for one key. Sory.


Error building ACL on Ubuntu Mate

Platform:
Odroid XU4
Operating System:
Ubuntu Mate
I started to build ACL using:
scons extra_cxx_flags="-fPIC" debug=0 benchmark_tests=0 validation_tests=0 neon=1 opencl=1 embed_kernels=1 build=native -j4
BUT, it failed:
In function ‘void arm_gemm::MergeResults(Tout*, const Tin*, int, int, int, int, int, const Tout*, arm_gemm::Activation, bool) [with unsigned int twidth = 8; unsigned int height = 6; bool sve = false; Tin = float; Tout = float]’:
src/core/NEON/kernels/arm_gemm/merges/a32_merge_float_8x6.hpp:85:33: error: this statement may fall through [-Werror=implicit-fallthrough=]
                         outptr1 = dummyres;
                         ~~~~~~~~^~~~~~~~~~
src/core/NEON/kernels/arm_gemm/merges/a32_merge_float_8x6.hpp:86:21: note: here
                     case 3:
                     ^~~~
src/core/NEON/kernels/arm_gemm/merges/a32_merge_float_8x6.hpp:87:33: error: this statement may fall through [-Werror=implicit-fallthrough=]
                         outptr2 = dummyres;
                         ~~~~~~~~^~~~~~~~~~
src/core/NEON/kernels/arm_gemm/merges/a32_merge_float_8x6.hpp:88:21: note: here
                     case 2:
                     ^~~~
src/core/NEON/kernels/arm_gemm/merges/a32_merge_float_8x6.hpp:89:33: error: this statement may fall through [-Werror=implicit-fallthrough=]
                         outptr3 = dummyres;
                         ~~~~~~~~^~~~~~~~~~
src/core/NEON/kernels/arm_gemm/merges/a32_merge_float_8x6.hpp:90:21: note: here
                     case 1:
                     ^~~~
src/core/NEON/kernels/arm_gemm/merges/a32_merge_float_8x6.hpp:91:33: error: this statement may fall through [-Werror=implicit-fallthrough=]
                         outptr4 = dummyres;
                         ~~~~~~~~^~~~~~~~~~
src/core/NEON/kernels/arm_gemm/merges/a32_merge_float_8x6.hpp:92:21: note: here
                     case 0:
                     ^~~~
cc1plus: all warnings being treated as errors

When I put Werror=0, that error disappears. But, then I get another error when using ARM-NN. I'm not sure if the error is related to ACL or not. I just wanted to repeat building ACL cleanly. Could you please help?

The error this statement may fall through [-Werror=implicit-fallthrough=] is just for missing break in switch-case statements in src/core/NEON/kernels/arm_gemm/merges/a32_merge_float_8x6.hpp file. I put the breaks, and the build finished successfully.


An error occurred calling CreateDXGIFactory1 in windows server 2019

Error: DXGI_ERROR_INVALID_CALL
0x887A0001

I found the solution that needed to be add graphics diagnostics tools.
https://docs.microsoft.com/en-us/windows/uwp/gaming/use-the-directx-runtime-and-visual-studio-graphics-diagnostic-features


infra(make): Add generate_data target for running generator scripts

This target makes it easy to run all generator scripts at once. I want to add this as a check to the release process.

I think it should be possible to have that working on Windows with a copy of tzdata.

By the way, if we want to avoid tzdata entirely, the standard abbreviation names are also in CLDR, like this: https://github.com/unicode-org/cldr/blob/b24157695397c4c3d2d058c201da92d10c9d9022/common/main/en_NZ.xml#L165


SLAVEOF command may block a long time

Search before asking

[X] I had searched in the issues and found no similar issues.

Version
2.2.0
Minimal reproduce step

echo 6 > /proc/sys/net/ipv4/tcp_syn_retries
SLAVEOF a non-routable IP address(such as <IP_ADDRESS>)
SLAVEOF new_mater_ip port

What did you expect to see?
Step 3 should not block for a long time because we are switching to the new master, which should be a quick operation.
What did you see instead?
Step 3 will block a long time(> 2min based on above reproduced scenario)
Anything Else?
This is because we use blocking IO when use socket connect(2). In some cases, connect(2) will be blocked for a long time.
Are you willing to submit a PR?

[X] I'm willing to submit a PR!

@caipengbo I think the root cause is we didn't set a timeout for the connect operation, so it will backoff retransmit the sync packet(1s,2s,4s,8s,16s,32s,64s) until reaching the tcp_syn_retries(default is 6 for most Linux distro).

@caipengbo I think the root cause is we didn't set a timeout for the connect operation, so it will backoff retransmit the sync packet(1s,2s,4s,8s,16s,32s,64s) until reaching the tcp_syn_retries(default is 6 for most Linux distro).

Yes, we should add a timeout.

Cool, got it.

Let's set it a little longer than 3 seconds, so we can tolerate 2 times lost sync packages? @git-hulk

Yes, sounds good to me.


return error on invalid queries

If a node query is done without using the keywords 'query' or 'querynode', the rest of the query string is ignored and all nodes are returned.  The system should just return an error stating that the query type was not specified - or possibly default to assuming 'query' type.

fixed in https://github.com/MG-RAST/Shock/pull/255


Input array is not F_CONTIGUOUS

The issue I am having is that I am getting a warning when creating a regridder (with xesmf.Regridder) that the input is not Fortran contiguous. Independent of what I do with the input the warning always appears.
Digging through the code I can see that the lat and lon variables are always transposed (to make them Fortran contiguous), but because as_2d_mesh simply passes the lat/lon arrays through if they are already 2D then these arrays might already be Fortran contiguous, and so transposing them will make them C contiguous instead. My suggestion would be to in ds_to_ESMFgrid only to transpose if necessary, or to use np.asfortranarray instead to ensure the data is always Fortran contiguous.
I've put a jupyter notebook here to demonstrate the issue: https://gist.github.com/leifdenby/a91b616ce5fb077e44d28556208626b4
If this sounds reasonable I'd be happy to make a pull requests which would include tests to ensure this change doesn't break anything else.
Thanks for a great tool btw! Love how easy it is to use :)

Ps. I can of course do (in the second example in my notebook)
ds_in['lat'] = (('x', 'y'), np.asfortranarray(lat).T)
ds_in['lon'] = (('x', 'y'), np.asfortranarray(lon).T)

so that when the transpose back is done inside ds_to_ESMFgrid we get back a Fortran contiguous array, but it just feels a bit hacky to me, but maybe I'm overthinking this?

Thanks for reporting and sorry for the confusing warning message. In short, simply pass C-ordered array (newly created array without .T, or most data read from files) to xesmf.Regridder(), then the warning will disappear (like the first case in your notebook)
I added this warning because ESMPy prefer F-ordered array: nawendt/esmpy-tutorial#4. But as you notice, all arrays are pre-transposed in xESMF code, so the user API actually expects C-order. I didn't expect users to see this "internal warning" if they always use normal, C-ordered numpy arrays.
The performance impact in this case is quite negligible so I'd consider removing this warning. I'll probably only keep the memory-layout check for input data (not grids), which does have a noticeable impact on performance.
(Glad that you find xESMF helpful!)


Sharing is caring

Just wanted to mention @brentvatne was thinking of supporting audio in his react-native-video component. Here's some work I've done on building out an audio native with StreamingKit if it helps any. Check the src/lib for my audio module.

:+1:

@jhabdas thanks for the information. the lumpen-radio project looks really interesting. I registered for your webcast. looking forward to it.

Closing this as it's not an issue. @andreaskeller I'm going to sub a PR to add pause and stop methods, borrowing from https://github.com/cymen/react-native-audioplayer.


Discrepancy between description and image

Description
I found descriptions doesn't match some images in tutorial. 😃  In Introduction > First steps with the plugin, mlflow.yml file is said to be in conf/local in descriptions, but it is shown in conf/base in images.

Hi, thank you very much for the report, I'll fix it soon! For the record, you can specify the environment you want with kedro mlflow init --env=<your-env> command.


Enhancement suggestion: add "open all modified documents" too.

Hi Markiewb,
thanks for this other very useful plugin!
I have a suggestion for an enhancement.
Could you add also a feature to open all the modified documents?
It would be useful for example to have a last look of the changes before a commit.
I guess that a problem arises when there are many modified files and open them all could be problematic. To avoid this you could add an option to specify a limit so that in case the command execution requires to open more than that number of files it displays an alert to advice the user and opens only the first modified files.

@Gittino: Won't fix.
Please use "Team|Show Changes". This opens a window with all the changed files (make sure that you select the project node or the top most node in the repo before invoking the action). All changed files will be listed. You can press CTRL-A to select them all. In the context menu invoke "Open" to "open all the modified documents".


Docs: The Get Started link in the header remains highlighted even if another section is open

CC @GreenMachine01


Future Feature: Document Redaction

There are a handful of references to using the Tyler Redaction Service (The Tyler Intelligent Services OData Web Service v4 API, TylerOCR, etc.) in the EFSP reference guide, and specific court codes for it, such as redactionfee, redactiontargetconfig, redactiondocumenttype, etc.
There is one reference in the Party Types codes that certain party types might need (exact wording is "should", but I don't trust Tyler to be as specific with their words as RFC 2119) to be redacted, but nothing else in the reference guide or in the certification documents about it being required, so I don't think we need it for MVP product.
This ticket is going to be used to track that feature, parts in the code that are relevant will be tagged with this ticket number.

Also of note: I'm not sure if there are ways that we can handle our own redaction by sending in a separate redacted document? We are generating the PDF ourselves. But the only way to pass redaction info with a filing is only by same Tyler GUID for redaction docs, so this might not be something we can do anyways.


Tap notification to abort send

Too often I send a message from the wrong account. Life would be less stressful if I could quickly abort sending by simply tapping the progress notification instead of diving for the modem. :)

This use case is covered by #5065


Add Documentation from PurpleAir API

Document the required get requests we will use to get our data

I'm going to close this as I added the Sensor B note to Issue 14


[Feature] Support odyssey - Scalable PostgreSQL connection pooler

Hello! Thanks for cloudnative-pg
Please add to roadmap support odyssey - Scalable PostgreSQL connection pooler

Hi, thanks for the request, could you expand a little bit more? As you know, cloudnative-pg currently supports pgbouncer, what would odyssey add in your opinion? Thanks!

https://github.com/yandex/odyssey
Advanced multi-threaded PostgreSQL connection pooler and request router.
https://github.com/yandex/odyssey/releases/tag/1.3
Support for protocol-level prepared statements in transaction pooling mode. You can enable pool_reserve_prepared_statement for a database. For client sessions issuing prepared statements Odyssey will automatically prepare those statements in database processes when necessary.

Hello. Any updates?
We use Yandex Odyssey; it is much better than pgbouncer.


KID

I find that you change fid to kid ，Could you public the source code of kid？Thank you.

Requesting for the realization of KID used in this paper, too.

I calculated the KID values for orange/apple dataset the same way you mentioned in your paper for cycleGAN, the results is around 0.82. If I multiply it by 10 it's similar to what you reported in the paper.
Could you please explain if your results are KID values multiply to 100 or *10?

Requesting KID code, too.

Can you share the KID code?


Cancel resolving URis when unmounting discover page

Play button is now appearing with no delay again @kauffj

See, cancelling was useful :)


Implementing snowy environment

 Hello. I am trying to apply snow effect asset packages bought from unreal marketplace. What I want to do is applying 'accumulation of snow' and 'falling snow' effect from the package.
This video which shows what the package does will be helpful for a better understanding of my question.
https://www.youtube.com/watch?v=_V3dcm9Kbhg
First, I wonder if I should apply the effect to actors which consists map one by one or there is a way to apply the effect to all actors at a time. (especially, applying 'accumulation effect' to all actors which consists ground part of the map)

Second, the package uses particle system in unreal engine to implement 'falling snow'. I wonder if there is an 'automatic' way for the falling snow effect to cover entire map or I should tune the extent manually.
I want to know how you implemeted same effect for rainy environment. Thank you.

Hi, did you get any success with this implementation? I am also interested in testing out snow in Carla.


adding private headers for the proxy request

Sometimes there exists cases where you need to add an additional header to the proxied request.
for example if the backend server needs an Authorization: Bearer xxxxx header or other type of X-Auth: my-token header.
Is it possible to add an option to configure this?

I also have a question.
It seems like the proxy is applied to all routes and you have to filter out which routes that you want to exclude?
Why did you not choose to apply the proxy only to the matching routes that the user sets instead?

Actually you can

I also have a question. It seems like the proxy is applied to all routes and you have to filter out which routes that you want to exclude? Why did you not choose to apply the proxy only to the matching routes that the user sets instead?

You can use SetSkipper for that.
Check default implementation because you will need to implement it in your skipper.

Sometimes there exists cases where you need to add an additional header to the proxied request. for example if the backend server needs an Authorization: Bearer xxxxx header or other type of X-Auth: my-token header.
Is it possible to add an option to configure this?

Good feature to implement, but after fast search did not find a way to do it. Currently plugin uses default Echo proxy functionality and it have no such option by default. Have no time right now to implement it, so if you have time - you can make a PR, probably can be useful to have ability to modify requests to the proxy.

thanks for the info...
I have no time either right now sadly but I'll keep an eye out if you eventually do any updates to the library


Terminal doesn't react to shortcuts

Windows 10
Steps to Reproduce:

Open terminal (Powershell)
CTRL + v

or another example

Open terminal (Powershell)
npm start (run -> all ok)
CTRL + c (to terminate, no reaction)

Everything used to work correctly but for some reason, no settings edited, terminal doesn't react to any CTRL + anything or any kind of shortcuts but if I type it works.
I am focused inside terminal because typing works

Duplicate https://github.com/Microsoft/vscode/issues/28635


Is it possible to translate OpenVino person-detection-retail model to another framework, like tensorflow?

I like pre-trained models provided by Intel OpenVino, but I want to run them using Tensorflow and GPU. For example, I am interested in people detection model written in Caffee. Is it legal to translate it to TesorFlow and use it in a commercial product? And does it makes sense? Maybe it is just better to train YOLO model for one class person?
Thank you for any help provided.

Is it legal to translate it to TesorFlow and use it in a commercial product?

Sorry, I don't think you can find any legal advise here in the open. I suggest to consult the legal department in your organization (that will, supposedly, produce the "commercial product"). The repo is licensed under Apache 2.0.
We start releasing some of the models in the original format (see https://github.com/opencv/training_toolbox_tensorflow and https://github.com/opencv/training_toolbox_caffe for example). It might better fit your needs.

Maybe it is just better to train YOLO model for one class person?

I would strongly advise against training YOLO =) In our experience, SSD-like architectures provide a much better accuracy/peformance trade-off.


Fix error with ambient type when running Universal

Hi everyone,
this PR solves issue #773. Since this PR is a bit of a workaround, I wanted to share a few details on what I found on the issue, in case you want to search for a better solution. So please feel free to ignore the following ;-)
With View Engine and strictMetadataEmit enabled, using ambient types did throw a build-time error (Could not resolve type), even when correctly injecting a provider with @Inject(). Also see issue https://github.com/angular/angular/issues/20351. This seems to have been fixed with Ivy: https://github.com/angular/angular/issues/23395#issuecomment-600355368
Since Ivy seems to be doing something different here with ambient types, I thought it might be a general problem with a mixed (and correct) setup like the following:

View Engine lib uses ambient type for constructor param
App has Ivy enabled
Running with Universal

Unfortunately, I am not able to reproduce the behaviour with a clean Angular workspace and just that setup. So something else seems to be involved in producing the error. And this is as far as I got for now.
P.S. Having strictMetadataEmit enabled did help me fix this issue, because it throws an error during build-time when using an ambient type. Just wanted to let you know, in case enabling it could help you in your workflow.
Thank you for the great library!

Please be aware that I have opened an additional PR (#855) for this issue. There might be tradeoffs to each solution for you, so I leave both open for now.


coffeescript 2 with transpile + inlined source maps?

Please include configuration examples to README.md

I'm get error:
Error: Source map to be applied is missing the "file" property
    at assertProperty (/home/myuser/myproject/node_modules/vinyl-sourcemaps-apply/index.js:36:13)
    at applySourceMap (/home/myuser/myproject/node_modules/vinyl-sourcemaps-apply/index.js:15:3)
    at DestroyableTransform.transform [as _transform] (/home/myuser/myproject/node_modules/gulp-coffee/index.js:41:7)
    at DestroyableTransform.Transform._read (/home/myuser/myproject/node_modules/through2/node_modules/readable-stream/lib/_stream_transform.
js:184:10)

My gulp task:
var gulp    = require('gulp'),
    coffee  = require('gulp-coffee'),
    sourcemaps = require('gulp-sourcemaps');

gulp.task('coffee', function(){
    gulp.src(config.coffeePath + '/**/*.coffee')
        .pipe(sourcemaps.init())
        .pipe(coffee({
          bare: true,
          transpile: {
            presets: [
              // Without any configuration options, babel-preset-env behaves exactly the same as babel-preset-latest (or babel-preset-es2015, babel-preset-es2016, and babel-preset-es2017 together).
              'env',
            ],
            plugins: [
              ["transform-react-jsx"]
            ],
          }
        }))
        .pipe(sourcemaps.write('.'))
        .pipe(gulp.dest('./static/js'));
});

I'm experiencing the same problem. Any progress on this?

Does adding sourceMap: true, inlineMap: true to the options work? I think they changed this recently - if that fixes it I will update the module.

➕ 1⃣

Couple of pointers to help you all:

Don't use gulp-sourcemaps directly anymore, sourcemaps are built into gulp 4 so you can just turn on sourcemaps in gulp.src
Make sure you read the coffee-script docs on how to enable sourcemaps, then it should play nicely with the gulp sourcemap support

i guess my issue was more with just being able to transpile at all... gulp-coffee doesn't support the coffeescript's transpile feature. also, i was unable to get sourcemaps to work with gulp 4... i had to still use the gulp-sourcemaps plugin. i currently don't have my project in a state to test it ATM, but would be good to hear someone confirm it is actually working...

Use gulp-babel instead of coffeescript transpile option. In fact, coffeescript docs says as much:

You can also transpile CoffeeScript’s output without using the transpile option, for example as part of a build chain. This lets you use transpilers other than Babel, and it gives you greater control over the process. There are many great task runners for setting up JavaScript build chains, such as Gulp, Webpack, Grunt and Broccoli.

Example:
var gulp = require('gulp');
var babel = require('gulp-babel');
var coffee = require('gulp-coffee');

gulp.task('coffee', function() {
  gulp.src('./src/*.coffee')
    .pipe(coffee({bare: true}))
    .pipe(babel({presets: ['@babel/preset-env']}))
    .pipe(gulp.dest('./public/'));
});


Dead pmts in displays

The bokeh display has a nice feature to hide 0 gain PMTs. We should add this to the matplotlib display too.
For the bokeh display, it might be nice if we can also hoover over the 0 gain PMTs to get their position and number.

Some other things,
this logic does not work:
https://github.com/XENONnT/straxen/blob/master/straxen/analyses/event_display.py#L93
Neither does int('raw') work:
https://github.com/XENONnT/straxen/blob/master/straxen/analyses/event_display.py#L104
Also add the other posrecs

Some other things,
this logic does not work:
https://github.com/XENONnT/straxen/blob/master/straxen/analyses/event_display.py#L93
Neither does int('raw') work:
https://github.com/XENONnT/straxen/blob/master/straxen/analyses/event_display.py#L104
Also add the other posrecs


Can't remove image created by Buildah

Is this a BUG REPORT or FEATURE REQUEST?:
kind bug
Description
An image created by Buildah is listed by Podman but can not be removed by Podman.  It can be removed by Buildah and then it is no longer shown by Podman.
Steps to reproduce the issue:

buildah from alpine
podman images
podman rmi -f {imageid}

Describe the results you received:
# buildah from alpine
# podman images
REPOSITORY                 TAG      IMAGE ID       CREATED       SIZE
docker.io/library/alpine   latest   b908b128a5f6   4 weeks ago   4.14MB
# podman rmi -f b908b128a5f6
failed to remove image: Image used by 2b1024cf1d55d555af8ce15ed009fd9fb3c691c362d7a1c7c90107a0cf84371e: image is in use by a container
# podman ps --all
# buildah images
IMAGE ID             IMAGE NAME                                               CREATED AT             SIZE
b908b128a5f6         docker.io/library/alpine:latest                          Dec 1, 2017 18:48      3.954 MB
# buildah rmi -f b908b128a5f6
ERRO[0000] reference "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.override_kernel_check=true]docker.io/library/b908b128a5f6:latest" does not resolve to an image ID
b908b128a5f6172525f2fd0fed13ee5812ee168f45b623fc9955033568888153
# podman images

// Note the error from Buildah is a known error being tracked in https://github.com/projectatomic/buildah/issues/190

Describe the results you expected:
The image would be removed via Podman or the image in the list should somehow be marked as a Buildah image or otherwise non-deletable by Podman.  It's more confusing as Podman says there are no containers when queried.
Additional information you deem important (e.g. issue happens only occasionally):
Output of podman version:
# podman version
Version:       0.1
Go Version:    go1.9.2
Git Commit:    "ef4035e208f1431745cd044adb4b789fab3a98b6"
Built:         Tue Jan  2 08:27:22 2018
OS/Arch:       linux/amd64
# podman -v
podman version 0.1
# buildah -v
buildah version 0.9 (image-spec 1.0.0, runtime-spec 1.0.0)
# cat /etc/*release
Fedora release 27 (Twenty Seven)
NAME=Fedora
VERSION="27 (Server Edition)"
ID=fedora
VERSION_ID=27
PRETTY_NAME="Fedora 27 (Server Edition)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:fedoraproject:fedora:27"
HOME_URL="https://fedoraproject.org/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=27
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=27
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Server Edition"
VARIANT_ID=server
Fedora release 27 (Twenty Seven)
Fedora release 27 (Twenty Seven)

Additional environment details (AWS, VirtualBox, physical, etc.):
On a VM running Fedora 27 updated as of 1/2/2018

Fixed with #522


Update Nav bar links

agreed.. but that entire block is capitalized.. so it doesn't matter but I will change it.


Only force authorizationMatrix on Adoptium's jenkins

The changes introduced by #154 #155 are specific
to Adoptium and can cause unwanted behaviour on
other users' Jenkins'. Simple solution to only
run that part of the code on Jenkins' that are
owned by Adopt (have adopt in the URL).
Probably good enough to resolve #156 while we're here.
Signed-off-by: Adam Brousseau<EMAIL_ADDRESS>

https://www.jenkins.io/doc/book/pipeline/jenkinsfile/#using-environment-variables

JENKINS_URL
Full URL of Jenkins, such as https://example.com:port/jenkins/ (NOTE: only available if Jenkins URL set in "System Configuration")

Open to suggestions. I figured adopt would cover if the server ended up moving to adoptium instead of adoptopenjdk.

I'm sure there will be lots of places to change if we move doman, but I will let others weigh in on the safest way to check where this logic is running.

I'm sure there will be lots of places to change if we move doman, but I will let others weigh in on the safest way to check where this logic is running.

FYI This came out of a quick brainstorming thread on slack so I'm 👍🏻 on this to resolve the immediate problem. It's better for the default be to not have such a restriction.

bump

I know that the security of this code is not compromised by allowing this behavior on a mismatched URL, but as stated above it just looks uncomfortable to see code added that doesn't adhere to good practice and I wouldn't want people to use that as an example of how to determine that we are really running on Adoptium infrastructure elsewhere where it does matter. Just my 2c.

@tellison so preference would be something like
JENKINS_URL.contains("ci.adoptopenjdk.net")

?

@tellison so preference would be something like
JENKINS_URL.contains("ci.adoptopenjdk.net")

?

No, that doesn't work. Look at the link referenced above for reasons.
I feel bad. As I say this code is not validating external data that could be problematic in this situation, so I'm not insisting on this change!

Sorry I missed the link the first time.
If I understand correctly, an attacker would need to be in control of Adopt's Jenkins' $JENKINS_URL variable in order for the job generation to "expose" OpenJ9 jobs.
Based on the last part of your last comment, are we ok as-is then?

Sorry I missed the link the first time. If I understand correctly, an attacker would need to be in control of Adopt's Jenkins' $JENKINS_URL variable in order for the job generation to "expose"

Yes. As I wrote, the proposal is not insecure. It just causes me to twitch when I see code like this which in other contexts would be insecure. Like pointing an unloaded gun at people - it is safe, but you just don't do that :-)

OpenJ9 jobs. Based on the last part of your last comment, are we ok as-is then?

I'm not blocking it. It is waiting on reviewers ATM. Apologies for the distraction!

@AdamBrousseau Some linter fixes required

Some linter fixes required

Resolved


ci: mark push true only for prs raised by jina bot

Should the push only be triggered when the PR is merged?

Should the push only be triggered when the PR is merged?

Should the push only be triggered when the PR is merged?

yes, for all our PRs, push should only be triggered when PR is merged. But for all PRs raised by jina-bot, since we don't ever merge it, push should be true.

Should the push only be triggered when the PR is merged?

yes, for all our PRs, push should only be triggered when PR is merged. But for all PRs raised by jina-bot, since we don't ever merge it, push should be true.

Should the push only be triggered when the PR is merged?

yes, for all our PRs, push should only be triggered when PR is merged. But for all PRs raised by jina-bot, since we don't ever merge it, push should be true.

Totally agree, when merged is it activate the push action?

Should the push only be triggered when the PR is merged?

yes, for all our PRs, push should only be triggered when PR is merged. But for all PRs raised by jina-bot, since we don't ever merge it, push should be true.

Totally agree, when merged is it activate the push action?

Should the push only be triggered when the PR is merged?

yes, for all our PRs, push should only be triggered when PR is merged. But for all PRs raised by jina-bot, since we don't ever merge it, push should be true.

Totally agree, when merged is it activate the push action?

Yah. That's a different workflow file.

Should the push only be triggered when the PR is merged?

yes, for all our PRs, push should only be triggered when PR is merged. But for all PRs raised by jina-bot, since we don't ever merge it, push should be true.

Totally agree, when merged is it activate the push action?

Yah. That's a different workflow file.


[FEATURE] Make use of meta config when defining model metadata

Make use of the meta dictionary either in the model config block or properties (.yml) file(s) when defining the model metadata. (e.g. src_pk etc.).
I think this would be cleaner and could improve readability of the models.
Docs for reference.

This is a great request and something we are planning to experiment with.
We want to ensure a smooth transition to this feature and we're working out whether this would be replacement functionality or whether it would work alongside macro parameters (and then whether that would be a good idea!)
We originally suggested this change here so we're excited for the possibilities in this.
Thanks again!

Yes this was the kind of thing I had in mind! Just hadn't had a chance to play around with it yet. Thanks for this :)

Hi @DVAlexHiggs, interesting that this isn't available at compile. I think other config options would be so this could just be behaviour specific to meta.
Have you tried using a custom config header name instead, such as metadata for example?

@DVAlexHiggs interesting, thanks for looking at this. And thanks for raising the ticket in dbt-core.
I suspect this might be an issue related to the timing of when dbt parses the properties file(s). The reason I say this is because I've created macros that behave in a similar way, but where the config options are specified in the config block of the model, they work as expected.
Are you having the same problem if you define meta in the config block?

I think I did try this. Will clarify. It would be better if it would work either way, however.

No unfortunately that does not work either, oddly.
{{ config(materialized='table', meta={'param_1': 'COLUMN_1',
                                     'param_2': 'COLUMN_2',
                                     'param_3': 'COLUMN_3',
                                     'param_4': 'COLUMN_4',
                                     'source_model_name': 'stg_orders'}
) }}

{% do log('meta: ' ~ config.get('meta'), true) %}
{% do log('param_1: ' ~ config.get('meta')['param_1'], true) %}
{% do log('param_2: ' ~ config.get('meta')['param_2'], true) %}
{% do log('param_3: ' ~ config.get('meta')['param_3'], true) %}
{% do log('param_4: ' ~ config.get('meta')['param_4'], true) %}
{% do log('source_model_name: ' ~ config.get('meta')['source_model_name'], true) %}

{% set param_1 = config.get('meta')['param_1'] -%}
{% set param_2 = config.get('meta')['param_2'] -%}
{% set param_3 = config.get('meta')['param_3'] -%}
{% set param_4 = config.get('meta')['param_4'] -%}
{% set source_model_name = config.get('meta')['source_model_name'] -%}

SELECT 1 AS "{{ param_1 }}"
       , 2 AS "{{ param_2 }}"
       , 3 AS "{{ param_3 }}"
       , 4 AS "{{ param_4 }}"

FROM {{ ref(source_model_name) }}

Minor update @alexjreilly
https://github.com/dbt-labs/dbt-core/issues/4776


Feature - Include large repository in e2e test

Is your feature request related to a problem? Please describe.
Include a large repository for the e2e test to validate memory leaks.
Describe the solution you'd like

Docker run with constrained memory --memory=900mb scorecard
Use a large repository like https://github.com/ApolloAuto/apollo
If there is a memory leak it is likely to fail.

Additional context
https://github.com/ossf/scorecard/issues/512

With randomized e2e tests, this should be covered.


Fix potential issue with odo binary being outdated when running tests directly with Ginkgo

What type of PR is this:
/kind tests
/area testing
What does this PR do / why we need it:
#5538 made both integration and E2E tests Makefile targets depend on the install one. This is fine when running the tests with make, but I noticed we could still run into the same issue when running them using the ginkgo CLI (as I often do when iterating on the code): the odo binary might be outdated and not match the current version of the code being tested, if one forgot to run make install before or make sure to add the relevant binary for odo to their PATH.
The approach I'm suggesting here is to:

Let Ginkgo build odo using the make bin target, prior to running the tests specs. To avoid building in parallel because of parallel tests specs, we could leverage Ginkgo's SynchronizedBeforeSuite/SynchronizedAfterSuite to build the project only once. And this should not have an impact on performance thanks to the Go build cache.
Use the path to that built binary everywhere a test tries to run odo
Let developers/testers choose a different binary path if they wish to do so, e.g., via an environment variable (aptly named ODO_BINARY_PATH). If this environment variable is set, we would not build odo, but we would use that path instead everywhere a test runs odo. I'm thinking of @kadel 's comment here: https://github.com/redhat-developer/odo/pull/5538#issuecomment-1064989123

This is somehow inspired by how integration tests are written in Rust CLI apps: https://rust-cli.github.io/book/tutorial/testing.html#testing-cli-applications-by-running-them
PR acceptance criteria:

[ ] Unit test

[ ] Integration test

[ ] Documentation

How to test changes / Special notes to the reviewer:

It is regularly necessary to run integration tests on several versions of odo to debug tests. With this entngled dependency, it will be more difficult to do this.

It is regularly necessary to run integration tests on several versions of odo to debug tests. With this entngled dependency, it will be more difficult to do this.

The ability to specify the path to the odo binary via an environment variable, as proposed here, would solve this, no?
While I understand the need to test different versions of odo, I still don't get how we can be sure the binary tested is actually the right one (in line with the test code), unless you set up your environment accordingly.
Anyways, no problem if this issue impacts only me. I'll try to find a different way to work around this locally.

we could still run into the same issue when running them using the ginkgo CLI

I used to run tests using ginkgo cli in the past when I was using vim to write Go. But since moving to GoLand, I use the FIt/FDescribe constructs of ginkgo and use GoLand to run the tests. The first thing it does when running tests is to build a temporary odo binary from the current code and use it for the tests. I imagine VS Code would do something similar.

we could still run into the same issue when running them using the ginkgo CLI

I used to run tests using ginkgo cli in the past when I was using vim to write Go. But since moving to GoLand, I use the FIt/FDescribe constructs of ginkgo and use GoLand to run the tests.

Are the FIt/FDescribe constructs the same as running ginkgo --focus="test name" ?

The first thing it does when running tests is to build a temporary odo binary from the current code and use it for the tests. I imagine VS Code would do something similar.

I agree with you, but I think this works this way only if you call the project code like a library in the tests.
I was thinking of our integration tests that execute odo using exec.Commmand. The executable gets resolved from the PATH, so I think the issue of ensuring the right binary would still persist regardless of the IDE.
I'm also using GoLand, and for example, if you remove odo from your PATH, you will notice that the tests do not pass at all even if the IDE built the project first:

Closing. Found a workaround locally with a hook into the shell.


OrderBy doesn't work if ThenInclude uses (or how to order nested collection)

Steps to reproduce
There is no docs found about ordering nested collection (ThenInclude() should be used), but found some fresh samples which should work (based on discussion).
Hence using the same approach the following code should sort nested items, but it doesn't (no exception thrown):
var res = _db.Set<ApplicationUser>()
                .OrderBy(u => u.Blogs.OrderBy(b => b.Posts.OrderBy(p => p.PostOrder).First().PostOrder))
                .Include(u => u.Blogs)
                    .ThenInclude(b => b.Posts)
                .Single(u => u.Id == userId);

Further technical details
EF Core version: 1.1.2
Database Provider: Doesn't work on both Microsoft.EntityFrameworkCore.SqlServer and Npgsql.EntityFrameworkCore.PostgreSQL (this is the main one)
Operating system: Windows 7 / Mac OS X 10.12.6
IDE: VS Code / Visual Studio 2017

Include only determines which navigations are going to be loaded in the final projection. For your scenario you can try ordering on the client:
var res = _db.Set<ApplicationUser>()
	.Include(u => u.Blogs)
	.ThenInclude(b => b.Posts)
	.Single(u => u.Id == userId)
	.OrderBy(u => u.Blogs.OrderBy(b => b.Posts.OrderBy(p => p.PostOrder).First().PostOrder));

If you want to sort on the server, then you would have to construct the projection manually (i.e. Include can't be used for this) - and this currently would produce a separate query to fetch Posts for each Blog (which is potentially very slow)

@maumar thanks for your response.
In other words, it means that for hierarchical domain model structure it's much better to have separate DbSet for each layer (i.e. DbSet, DbSet and DbSet) and merge date in memory.
Is this efficient and idiomatic for EF?

Given the ResultOperator Single doing ordering on client is really bad perf. Need a better query which achieves ordering on server side. Also ordering could be translatable on server side (at least in the absence of Includes.
@neooleg - Can you share ApplicationUser, Blog & Post classes?

@neooleg - Also

the following code should sort nested items, but it doesn't

Include does not sort records in collection at present. If you want to sort records in a collection navigation property, you need to do it in memory. Given you have retrieved all the related data from server already, the sorting in memory is not huge perf hit.

@neooleg - Can you share ApplicationUser, Blog & Post classes?

@smitpatel, sure. The main idea is that domain model is hierarchical.
public class ApplicationUser : IdentityUser<int>
{
  public ICollection<Blog> Blogs { get; private set; }
}

public class Board
{
    public int Id { get; private set; }
    public ICollection<Post> Posts { get; private set; }
    // Blog fields ...
}

public class Post
{
    public int Id { get; private set; }
    public ICollection<Comment> Comments { get; private set; }
    // Post fields ...
}

// Comment looks similar

Hence, for hierarchical structure -- is it make sense to have such context which should allow to query and sort on each level on server? Then combine data on transport/protocol layer.
public class ApplicationDbContext : IdentityDbContext<ApplicationUser, ApplicationRole, int>
{
    public ApplicationDbContext(DbContextOptions dbOptions)
        : base(dbOptions)
    {
    }

    public DbSet<Blog> Blogs { get; private set; }
    public DbSet<Post> Posts { get; private set; }
    public DbSet<Comment> Comments { get; private set; }
}

Thanks!

@neooleg - Thanks for info on classes. After running variety of queries with this domain,

Include/ThenInclude does not order the items. The navigation property may or may not have ordering (like HashSet) so at present there is no guarantee of ordering while loading. See https://github.com/aspnet/EntityFrameworkCore/issues/9067 & https://github.com/aspnet/EntityFrameworkCore/issues/7450 which talks more about ordering of entities and may help out in what you are trying to achieve.

The first query you posted is calling OrderBy but based on the lambda inside of it, it seems like the ordering is just for include. Since you are trying to order users by Blogs, it will be evaluated on server. In this case, assuming UserId is PK, it would be fine but using Single without OrderBy may give different result based on database. Also First is always evaluated in memory because it throws exception for empty sequence and we need to evaluate in memory to throw exception. If you want to translate it to server fully, consider using FirstOrDefault

I had re-opened this issue because I was expecting to see if the order by is getting translated to server. But as ordering is by blog it does not work. If blogs are ordered by blogorder then we indeed translate orderby to server.

// Query
                var query = db.Set<ApplicationUser>()
                    .OrderBy(u => u.Blogs.OrderBy(b => b.Posts.OrderBy(p => p.PostOrder).FirstOrDefault().PostOrder).FirstOrDefault().BlogOrder)
                    .SingleOrDefault(u => u.Id == userId);

// SQL
      SELECT TOP(2) [u].[Id]
      FROM [Users] AS [u]
      WHERE [u].[Id] = @__userId_0
      ORDER BY (
          SELECT TOP(1) [b].[BlogOrder]
          FROM [Blogs] AS [b]
          WHERE [u].[Id] = [b].[ApplicationUserId]
          ORDER BY (
              SELECT TOP(1) [p].[PostOrder]
              FROM [Posts] AS [p]
              WHERE [b].[Id] = [p].[BlogId]
              ORDER BY [p].[PostOrder]
          )
      )

closing as
Duplicate of #9067

@smitpatel thanks for detailed answer!


provide web searches without leaving the powertoys interface

I think it would be amazing if it were possible to make web searches without leaving the powertoys interface

Have you ever tried researching a topic only to find out 6 minutes later that you have 30 tabs open (most are google search tabs), you can't remember which of them was important
or perhaps you wanted to quickly look something up only to make sure that it is what you think it is, for instance, you need to look at the Wikipedia article

it could be done by implementing a phone version of the webpage in the powertoys interface

#3245


ci: use deterministic allocator for benchmarks

An attempt to reduce variance in benchmarks. Just testing at this stage.

#4483  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @overlookmotel and the rest of your teammates on  Graphite

This does seem to work in that it removes almost all variance from the benchmarks.
Results for 5 dummy commits on #4485 which is on top of this PR:

Remaining problem is that changing the allocator is showing as quite a significant speed-up on many benchmarks. Probably this allocator is faster than system allocator! Need to slow it down somehow so it matches system allocator better.

Need to slow it down somehow so it matches system allocator better.

Probably doesn't matter, we are interested in relative change, not the absolute value from codspeed.

Probably doesn't matter, we are interested in relative change, not the absolute value from codspeed.

Problem is that I think it will give us unrealistic relative results too.
Let's say we introduce a change that removes a bunch of allocations, but requires some extra work to do that (caching structs, bookkeeping etc). That is very likely to be a performance gain in real world, but if the allocator we use for benchmarks makes allocation unrealistically cheap (as this one does), benchmarks will lie to us and tell us it's a perf regression. For example, with this allocator, benchmarks probably would have told us #4213 was a bad change.
Ditto for using an arena allocator - benchmarks may tell us that using bumpalo is hurting perf (because its not much cheaper than our unrealistically cheap "global" allocator), when in fact it's a large perf gain.
NB: Allocations are a small part of the code overall. So if we're seeing 10% perf boost on some benchmarks, probably that means this allocator is ~double the speed of the system one. That's a very big discrepancy.
We might have more luck with https://crates.io/crates/dlmalloc which it sounds like is a closer analogue to the default system allocator (from libc), and so may reduce this discrepancy. I'll look into that.

Have asked for help on CodSpeed Discord: https://discord.com/channels/1065233827569598464/1065686090452828251/threads/1267428778691399690

Merge or close?

Close!


Second and third-order derivatives of test functions

Hello,
I need to compute a coefficient that is function of the first, second and third-order derivative of the test functions. Getting the first-order derivative is achieved with CalcDShape() but I could not find a way to compute the second- or third-order derivatives (assuming that the test functions are of a sufficient order).
Is there a way to get the high-order derivatives of test function?
Thanks,
Marco

One approach is to use the gradient matrix returned by the method FiniteElement::ProjectGrad and apply it multiple times. Basically, if you have a local vector of size dofs mutiplying it with the gradient matrix gives you a local vector with dim*dofs entries -- the first dofs entries are the x-derivatives, then y-derivatives, etc. At the end you can interpolate the coefficients to any quadrature point by contracting with the vector returned by FiniteElement::CalcShape. If the element mapping is not linear, this is not going to be the exact -- essentially, the gradient is projected back in the same space and, in general, this is just an approximation.

Thanks for the help. I will give it a try.
An approximation of the derivative values should be enough as it will be
used to compute a smoothness indicator.
Marco
On Wed, Oct 16, 2019 at 1:09 AM Veselin Dobrev<EMAIL_ADDRESS>wrote:

One approach is to use the gradient matrix returned by the method
FiniteElement::ProjectGrad and apply it multiple times. Basically, if you
have a local vector of size dofs mutiplying it with the gradient matrix
gives you a local vector with dim*dofs entries -- the first dofs entries
are the x-derivatives, then y-derivatives, etc. At the end you can
interpolate the coefficients to any quadrature point by contracting with
the vector returned by FiniteElement::CalcShape. If the element mapping
is not linear, this is not going to be the exact -- essentially, the
gradient is projected back in the same space and, in general, this is just
an approximation.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/mfem/mfem/issues/1118?email_source=notifications&email_token=ABD4GCW6VPVBKSQ4CV6DXQ3QO2OZVA5CNFSM4JA76NV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBLCURA#issuecomment-542517828,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABD4GCVYI5QYNOGQEDMSGOTQO2OZVANCNFSM4JA76NVQ
.

--
Marc-Olivier Delchini

Hello,
I tried to use FiniteElement::ProjectGrad and it gives me the following error message:
FiniteElement::ProjectGrad(...) is not implemented for this element!
My implementation is as follows:
mfem::ElementTransformation *eltrans = pmesh->GetElementTransformation(ele);
  const mfem::FiniteElement *test_fe = vfes.GetFE(ele);
  mfem::DenseMatrix grad(dof, dim);
  test_fe->ProjectGrad(*test_fe, *eltrans, grad);

where ele is the element index. The finite element object vies is defined as:
  mfem::ParFiniteElementSpace vfes(pmesh.get(),
                                   &fec,
                                   num_equation,
                                   mfem::Ordering::byNODES);

and fec is a DG collection:
  mfem::DG_FECollection fec(order,
                            dim,
                            mfem::BasisType::Positive);

Am I correctly using this function? Is it correct to have the same FiniteElement object calling the function and used as an input of the function?
Is this function limited to a set of FiniteElement object?
Thanks,
Marco

It looks like the issue is that you are using the positive basis and ProjectGrad is currently not implemented for that basis. If using the positive basis is not essential for your algorithm, you can try switching to the Gauss-Lobatto basis.
Anothrt option is to switch the basis to Gauss-Lobatto just for the gradient computations using GridFunction::ProjectGridFunction.

@delcmo, were you able to resolve this?

Still working on it. I was on travel last week and this week. Will get back
to it soon.
Marco
On Sun, Nov 3, 2019 at 5:23 PM Tzanio Kolev<EMAIL_ADDRESS>wrote:

@delcmo https://github.com/delcmo, were you able to resolve this?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mfem/mfem/issues/1118?email_source=notifications&email_token=ABD4GCWN6P4QLCDGLUDCERLQR46NNA5CNFSM4JA76NV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC56O4Y#issuecomment-549185395,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABD4GCUDJCAYWLYPL2FOEU3QR46NNANCNFSM4JA76NVQ
.

--
Marc-Olivier Delchini

I believe the question was answered. Since there hasn't been activity on this for a month, I will close this issues. Let us know if you still have questions.

I will give it a try most likely next year and will get back to you if I
have any questions.
Marco
On Tue, Dec 10, 2019 at 8:27 PM Tzanio Kolev<EMAIL_ADDRESS>wrote:

I believe the question was answered. Since there hasn't been activity on
this for a month, I will close this issues. Let us know if you still have
questions.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/mfem/mfem/issues/1118?email_source=notifications&email_token=ABD4GCUPS7MJE3D2G6H4KK3QYA6YZA5CNFSM4JA76NV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGRRIWI#issuecomment-564335705,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABD4GCXDPV7DRGKV22MWSR3QYA6YZANCNFSM4JA76NVQ
.

--
Marc-Olivier Delchini

I gave it a try and below is the logic to compute the first-order, second-order and third-order derivatives in 2D and 3D. I tested it by setting the initial value to a kth-order polynomial and outputting the values of grad_value, grad2_value and grad3_value. The piece of code below is to be included in the loop over the elements and over the quadrature points.
    mfem::ElementTransformation *eltrans =
                                vfes.GetElementTransformation(ele);
    mfem::DenseMatrix dshapedr(dof * dim, dof);
    test_fe->ProjectGrad(*test_fe, *eltrans, dshapedr);

    mfem::DenseMatrix dof_grad(dim, dof);
    dof_grad = 0.0;
    // Compute 'x', 'y' and 'z' derivatives
    for (int d = 0; d < dim; d++)
    {
      for (int r = 0; r < dof; r++)
      {
        mfem::Vector row(dof);
        dshapedr.GetRow(d * dof + r, row);
        dof_grad(d,r) = row * dofVct;
      }
   }

    // Compute second-order derivatives: xx, xy, xz, yy, yz, zz
    mfem::DenseMatrix dof_grad2(dim * dim, dof);
    dof_grad2 = 0.0;
    if (k > 1)
    {
      mfem::Vector row_dof_grad(dof);
      for (int d = 0; d < dim; d++)
      {
        // Get \partial()/\partial(d)
        dof_grad.GetRow(d, row_dof_grad);
        for (int d2 = 0; d2 < dim; d2++)
        {
          for (int r = 0; r < dof; r++)
          {
            mfem::Vector row(dof);
            dshapedr.GetRow(d2 * dof + r, row);
            dof_grad2(d * dim + d2,r) = row * row_dof_grad;
          }
        }
      }
    }

    // Compute third-order derivatives: xxx, xxy, ...
    mfem::DenseMatrix dof_grad3(dim * dim * dim, dof);
    dof_grad3 = 0.0;
    if (k > 2)
    {
      mfem::Vector row_dof_grad2(dof);
      for (int d = 0; d < dim; d++)
      {
        // Get \partial()/\partial(d)
        dof_grad2.GetRow(d, row_dof_grad2);
        for (int d2 = 0; d2 < dim; d2++)
        for (int d3 = 0; d3 < dim; d3++)
        {
          for (int r = 0; r < dof; r++)
          {
            mfem::Vector row(dof);
            dshapedr.GetRow(d3 * dof + r, row);
            dof_grad3(d * dim + d2 * dim + d3,r) = row * row_dof_grad2;
          }
        }
      }
    }

    // Compute first-order derivatives
   mfem::Vector grad_value(dim);
    dof_grad.Mult(test_shape, grad_value);
    // Compute second-order derivatives
    mfem::Vector grad2_value(dim * dim);
    grad2_value = 0.0;
    if (k > 1)
    {
      dof_grad2.Mult(test_shape, grad2_value);
    }

    mfem::Vector grad3_value(dim * dim * dim);
    grad3_value = 0.0;
    if (k > 2)
    {
      dof_grad3.Mult(test_shape, grad3_value);
    }

Note that this code is not optimized yet. It is part of a function to compute a smoothing indicator used in a WENO-type limiter method.

Hello @delcmo
did you calculate the derivatives of NURBs (third order)?
Best!

@delcmo thanks for your reply! does Mfem include third order derivatives of Nurbs (surface/solids)?


resource ownership in Athenz

The following are the use cases that we’re trying to solve with the introduction of resource ownership in Athenz:

The domain administrator is using TF to manage domain data (could be using a personal account or some service identity).

The administrator wants to block access to those resources to be modified using Athenz UI and/or zms-cli since that will create a drift in TF state.

The administrator should have the capability to override the ownership and make changes in case of emergency situations (e.g. TF deleted a service identity from a role but it needs to be re-added asap).

This capability must be available using zms-cli and preferably from Athenz UI as well.

In Athenz UI I should be prompted that the user should not proceed with the move unless explicitly specified.

The roles/policies are created and managed by another service built on-top of Athenz.

The operator wants those resources not to be visible in Athenz UI based on the ownership state and only make them available in their respective solution UIs.

Resource ownership should support partial ownership. For example, with roles and groups, TF can manage either members or meta so the server must support and enforce ownership at that level and not just at the object level.

This indicates that there might be multiple owners of the same resources

https://github.com/AthenZ/athenz/blob/master/docs/design/resource_ownership.md


Tweak ProseMirror menu

Makes ProseMirror's menu hang out at the top of the text box rather than be
inline. This also adds a change to the pdf template (so that it uses the same
class for comment content) and to CSS styles around li/ols (which are
list-style-type: none elsewhere in the app).
Looks like

Resolves eregs/notice-and-comment#200

Linter error: 67:48  error  Inconsistently quoted property 'selectParentNode' found  quote-props


log4j-sniffer: 0.7.0 -> 0.8.0

Motivation for this change
https://github.com/palantir/log4j-sniffer/releases/tag/v0.8.0
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
22.05 Release Notes (or backporting 21.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module
[ ] (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 151395 run on x86_64-linux 1

1 package built:

log4j-sniffer


Promote curation by GigaByte

First curation statement recorded with:
./scripts/record-curation-statement.sh hypothesis:TzbDzhmYEe6XO2Mwn7SCZQ 10.1101/2<IP_ADDRESS>4616 f7a7aec3-8b1c-4b81-b098-f3f2e4eefe58


Table feature doesn't support variable list size

When there is less/more than 30 students on the grading CSV file, the generated PDF will still generate 30 columns.
The problem is that lines and cell dimensions are still hard-coded from the unwraped PDF file that was used to implement this feature.
In order to fix this, the code must:

Take number of students in calculations, when calculating table lines
Take number of students in calculations, when calculating number of cells

Trying to introduce PDF generation library to genpdf-rs, since it has much more high level, user friendly API for generating tables without low level calculations. It should be able to render tables with less code, which will be more maintanable.

Still missing lines when there's 32 students in the group.


Support fit with DDP then test without DDP

🚀 Feature
This issue has been discussed a few times it seems like (#7929, #3325, #3600), but I haven't seen or been able to find a clear response as to whether the PTL team would like to support this workflow.
I would like to train on a dataset with multiple GPUs, then test using a single GPU, without having the call the script multiple times.
Motivation
I do this with fit-test sequence with every run because I want to get the final, accurate validation results using the best checkpoint from training. This means without a "join" context for validation I need to run on a single GPU without a DistributedSampler, as discussed in #7929.
Even with a join context for validation, I usually create my own validation set and use the original validation set as a test set. In this case I want to be able to call trainer.test() on both the validation set and the test set with every run.
I believe this workflow I'm describing is not uncommon and therefore should be considered as something that PTL should be able to handle.
Finally, calling the script multiple times is a significant hassle for me and likely other people as well. For my own workflow I'm launching several grid searches from a single config file using Wandb sweeps behind the scenese. I don't want to wait until all the runs are finished to re-launch everything to generate testing results, since the sweeps can take several days. And adding logic to be able to call the script twice and start testing while some runs are still training adds additional complexities that I really want to avoid so the code is as readable as possible.
Pitch
I have hacked together a solution that re-initializes the trainer for testing on a single GPU without any distributed backend, but it's a bit tricky because DDP is still initialized and things like torchmetrics try to use the default working group to synchronize things in testing still. So in that case I need to pass in my own "dummy gather" function that by-passes the built-in gather function that torchmetrics uses with the default DDP working group. The whole workflow seems fragile at best and if/when it breaks in future updates it will lead to deadlocks and block all of my runs.
Alternatives
One solution is a join context to enable testing with multiple GPUs. If this is robust enough to be used for research papers and production then that would be great. This is likely the best solution long-term but it sounds like this won't be possible in the near future, as discussed in #3325.
A second solution is to somehow "clean up" the acceleartor / DDP initialization so its as if it were never initialized at all. Then the call to trainer.test() would be independent of any DDP environment created by fit(), and I believe everything should just work. My attempts at this failed, but I'm not very experienced with DDP so it could be easy and I'm just missing something. To me this would be just as good as a join context, and perhaps its easy to implement for each accelerator type, in which case this may be the best solution for now.
A third solution would be a more robust version of what I'm currently doing. This might involve wrapping any barrier() calls with a check to see if we're testing on a single GPU, and if so, skip the barrier. Other synchronization methods such as gathers would have to be wrapped as well. This seems like the worst solution of the three since it's the most "hacky" to me.
Thank you!
Side note, I don't think I'm comfortable with a PR for this one, but if I can help in any other way please let me know.
-Collin

Dear @collinmccarthy,
Thanks for sharing your thoughts there. You could use ddp_spawn and re-create a Trainer with different parameters.
DDP created subprocesses and I don t really see how we could make it work without killing them.
Best,
T.C

@tchaton Thank you for the quick response. Do you mean use ddp_spawn for the initial fit(), or just for the test() part? I want to avoid using ddp_spawn for fit because I need the absolute best performance that comes from ddp and ddp2 so I can scale this to many nodes and many GPUs.

And for the subprocesses, I wasn't thinking we would kill them, rather just let them exit the program normally and reset the DDP backend such that torch.distributed.is_initialized() returns False, so future calls to trainer.test() don't try to use an old DDP context.
This does assume that whatever happens after trainer.fit() does not need the DDP context anymore. So I'm sure there are use cases I'm not thinking about here, but this does seem like it should at least work for the case where all you want to do is call fit() once with multiple GPUs and then test() once with one GPU.

Hey @collinmccarthy,
I understand better the use-case now.
Here is a pseudo cod, but with some hacking, I think It could work ! Mind giving it a try ?
trainer = Trainer(accelerator="ddp", gpus=2)
trainer.fit(...)

torch.distributed.destroy_process_group()

if trainer.is_global_zero:
    trainer.training_type_plugin.cluster_environment.set_world_size(1)
else:
    return

trainer.accelerator_connector.replace_sampler_ddp = False
trainer.test(...)

@tchaton Nice! Thank you, this is what I was missing. Working on it now, I'll let you know soon but I'm sure I can get this to work.

@tchaton Okay, I think I've found two pseudo-solutions here.
First, a slight modification of what you have above:
trainer = Trainer(accelerator="ddp", gpus=2)
trainer.fit(...)

torch.distributed.destroy_process_group()

if trainer.is_global_zero:
    trainer.training_type_plugin.num_nodes = 1
    trainer.training_type_plugin.num_processes = 1
else:
    return

# With 1 process, 1 node -> num replicas = 1, so ddp sampler is safe but not necessary
trainer.accelerator_connector.replace_sampler_ddp = False
trainer.test(...)

The big problem is that the state of the DDPPlugin (trainer.training_type_plugin) is mostly "wrong". That includes some environment variables, the interactive ddp processes, parallel devices, etc. It just so happens that this stuff isn't used in test() or didn't cause any obvious issues for me. It feels super fragile at best.
To me the safer solution is to just re-create the trainer with the correct parameters for single GPU testing without any distributed backend. What my earlier attempt at this was missing is the call to torch.distributed.destroy_process_group() which is key to avoiding issues in torchmetrics without resorting to things like a dummy gather function (for dist_sync_fn arguments).
trainer = Trainer(accelerator="ddp", gpus=2)
trainer.fit(...)

torch.distributed.destroy_process_group()
if trainer.is_global_zero:
    trainer = Trainer(gpus=1)
    model = MyModel.load_from_checkpoint(best_ckpt_path)
    trainer.test(model, datamodule=my_datamodule)

If you agree that the second method seems more "robust", one thing PTL could do is abstract away the "clean up" of the training_type_plugin (e.g. the call torch.distributed.destroy_process_group()) and force each plugin type to implement something like it. Then allow the user to specify a flag to trainer.test() to use a single GPU / accelerator type. Then the trainer could call this "clean up" method and initailize a SingleDevicePlugin instead, and then start testing / predicting. Just some thoughts.
I'll circle back to this issue in a couple of weeks and see if there's anything I can do to help out. Maybe at a minimum I could add some documentation and an example with the BoringModel.
Thanks for your help!
-Collin

Hey @ananthsub, any thoughts on this ?

Hey @collinmccarthy,
I am not sure Lightning should provide this out of the box.
But this would definitely be a great addition to the doc.
@awaelchli Any thoughts there ?
Best,
T.C

@awaelchli Any thoughts there ?

I personally would never run the test on multiple GPUs. For correctness, one should always run on a single GPU when evaluating the test set. Uneven inputs are not supported and the distributed sampler will influence the metric computation.
Evaluating the test error in a separate script is the only and best option imo.
If running the test after fit is still desired, one could do
torch.distributed.destroy_process_group()
if trainer.global_rank == 0:
    trainer = Trainer(gpus=1)
    trainer.test(model)

as @collinmccarthy suggested.

https://github.com/PyTorchLightning/pytorch-lightning/pull/8632 addresses one of these concerns, which is the model does not need to be wrapped with DDP for testing.
However, I completely agree with @awaelchli that testing for evaluation purposed ought to be run in a separate script, on a single GPU to avoid any (mis)handling of uneven inputs and the distributed sampler.
the process group destruction also came up in https://github.com/PyTorchLightning/pytorch-lightning/pull/8080 . i think this is potentially something the trainer could do if we are certain that it is the agent which created the initial process group. however, this would still be a breaking change and requires more consideration

Hey @collinmccarthy,
I believe we can close this issue as you have a work around and Lightning won't support this out of the box.
Best,
T.C

Any update on this?


Make a convention for log verbosity levels

Description:
Today we have verbosity levels in Kubevirt. In Kubevirt CR it's possible to set the desired verbosity level for each component (e.g. virt-handler, virt-controller, etc).
It seems that there is no clear convention regarding what each verbosity level represents.
What you expected:
To have a document determining the different verbosity levels and their meaning.
Kubernetes uses something similar to:
0: No logs
1: Warnings
2: Debug
3: Errors

We can standardize that as a start. We should also introduce constants in-code and replace the literal values (log.V(2)) with constants (log.V(DEBUG)).
In the future, we might want to take a non-linear approach and let users specify exactly what they want to see. I'm imagining something like withLibvirtLogs, withLiveMigrationPhases, etc. But as a start we at least need to make a standard of the current position.

/cc @vladikr

/good-first-issue

/good-first-issue

/label good-first-issue

/good first-issue

/remove-lifecycle stale

I want to dive a little into KubeVirt logs  and I think working on this issue would help.  Is it fine for me to take it? @iholder101

I want to dive a little into KubeVirt logs and I think working on this issue would help. Is it fine for me to take it? @iholder101

Yes indeed! Thanks for doing so!
Feel free to ping me about anything :+1:

Hi, finally got time to this!

We can standardize that as a start. We should also introduce constants in-code and replace the literal values (log.V(2)) with constants (log.V(DEBUG)).

What is confusing to me is that we have both Verbosity and Levels (besides being structured!). Based on the log filtering logic we should only use verbosity level for Info Level itself, while we might tweak/enable other log components based on how high is verbosity is. (k8s docs seems to say something similar about -v command line option here plus this quote in JSON log format v - verbosity (only for info and not for error messages, int))
So, I'll start with the following:

V() should only be used for Info messages as we display for >= WARNING levels anyway
Add some verbosity constants and changing the code to follow it
Add some documentation about above items

Hi, nice initiative!
I would recommend to think of how to enforce it as well.
While this is cleaned up and organized, in parallel new such things can get in.


Backport HBASE-26477 Fix MiniMapReduceCluster failure under Java17 to branch-2

https://issues.apache.org/jira/projects/HBASE/issues/HBASE-26477

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 36s
Docker mode activated.

-0 :warning:
yetus
0m  7s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ branch-2 Compile Tests _

+1 :green_heart:
mvninstall
7m 15s
branch-2 passed

+1 :green_heart:
compile
1m  2s
branch-2 passed

+1 :green_heart:
shadedjars
6m 54s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 41s
branch-2 passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
3m 49s
the patch passed

+1 :green_heart:
compile
1m  1s
the patch passed

+1 :green_heart:
javac
1m  1s
the patch passed

+1 :green_heart:
shadedjars
7m  1s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 39s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
160m 31s
hbase-server in the patch passed.

192m 38s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3878/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3878

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux bebe1da88220 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / 52ce87780a

Default Java
AdoptOpenJDK-1.8.0_282-b08

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3878/1/testReport/

Max. process+thread count
3966 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3878/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

+0 :ok:
reexec
1m 13s
Docker mode activated.

-0 :warning:
yetus
0m  7s
Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck

_ Prechecks _

_ branch-2 Compile Tests _

+1 :green_heart:
mvninstall
9m 15s
branch-2 passed

+1 :green_heart:
compile
1m 33s
branch-2 passed

+1 :green_heart:
shadedjars
9m 42s
branch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 56s
branch-2 passed

_ Patch Compile Tests _

+1 :green_heart:
mvninstall
5m 49s
the patch passed

+1 :green_heart:
compile
1m 33s
the patch passed

+1 :green_heart:
javac
1m 33s
the patch passed

+1 :green_heart:
shadedjars
9m 24s
patch has no errors when building our shaded downstream artifacts.

+1 :green_heart:
javadoc
0m 52s
the patch passed

_ Other Tests _

+1 :green_heart:
unit
241m  4s
hbase-server in the patch passed.

283m 31s

Subsystem
Report/Notes

Docker
ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3878/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/3878

Optional Tests
javac javadoc unit shadedjars compile

uname
Linux 5b00b6a8d01c 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Build tool
maven

Personality
dev-support/hbase-personality.sh

git revision
branch-2 / 52ce87780a

Default Java
AdoptOpenJDK-11.0.10+9

Test Results
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3878/1/testReport/

Max. process+thread count
2623 (vs. ulimit of 12500)

modules
C: hbase-server U: hbase-server

Console output
https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3878/1/console

versions
git=2.17.1 maven=3.6.3

Powered by
Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.


X509 verify non-DNS SANs

Description
Fixes #3498 - only the bug part.
The other part of #3498 was a feature request, already addressed in #2906 - which will need to be rebased on this PR, or on development after this PR is merged.
Status
READY
Requires Backporting
Partial: the bug was introduced in 2.18.0, so it's not present in the LTS branches, but let's backport the new test just to be extra sure.

[x] 2.16 #3555
[x] 2.7 #3556

@gilles-peskine-arm @hanno-arm Do you think I should mention in the ChangeLog entry that the bug was introduced in 2.18.0? We don't usually include that kind of information, but in that case it could be useful to explain why there is no corresponding ChangeLog entry in the LTS branches. Wdyt?

Exceptionally merging before the backports are approved, in order to unblock external contribution in #2906.

I don't think it's particularly useful. It wouldn't be the first time we fix a bug that was introduced after the last LTS. There are precedents where we say this, but I'm sure there are precedents where we don't. If someone is concerned about this, they can look at the comments on the pull request. Since this is merged, let's let it be.


Add Windows + Erlang 24 to CI

(moved away from gleam-lang/setup-erlang since it doesn't support Windows with Erlang 24)

Ping.

Thanks!


How do I get TRX balance of an address?

I have some issue by this code.
I am running my website on Windows Server IIS 10 and I am using .NET 5 integrated with Angular.
This code work correctly on localhost but on the server I get below error:
Grpc.Core.RpcException: Status(StatusCode="Cancelled", Detail="Received http2 header with status: 503")
By the way, I'm correctly setup http2 on my IIS


Last updated date for each page on the playbook's mkdocs site

Introduce lasted updated date to each mkdocs page
What are you trying to address

Add last updated date to each page on the playbook's mkdocs

Description of new changes

Added an mkdocs plugin using git revision date (https://github.com/timvink/mkdocs-git-revision-date-localized-plugin)
Removed dead link from one page (failed CI)
Moved one file which was on a nested folder and caused git and mkdocs issues (filename too long)

For all pull requests

[x] Link to the issue you are solving (so it gets closed)
[x] Label the pull request with the appropriate area(s)
[x] Assign potential reviewers (you may also want to contact them on Teams to ensure timely reviews)
[x] Assign the project - Engineering Playbook Backlog
[x] Assign the pull request to yourself

Looking for an additional reviewer @fnocera @TessFerrandez @wbreza. Thanks!

/azp run


[service-bus] Make it possible to abort Sender.createBatch()

Making it possible to abort the createBatch() call (which ultimately is just a fancy init() wrapper).

Instrumenting abortSignal futher in the sender (going through init()).


Update IoT Edge package version to LTS 1.1

The example points users to an old version -
sudo apt-get install iotedge=1.0.10* libiothsm-std=1.0.10*

Please bump the version to 1.1 instead -
sudo apt-get install iotedge=1.1* libiothsm-std=1.1*

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a3b8296d-c923-e990-dcad-35ef2b908a2a
Version Independent ID: 5f85fd7b-f2b6-b3b1-6be2-b6054bb33507
Content: Install Azure IoT Edge
Content Source: articles/iot-edge/how-to-install-iot-edge.md
Service: iot-edge
GitHub Login: @kgremban
Microsoft Alias: kgremban

Thanks for the feedback! We are currently investigating and will update you shortly.

@SatishBoddu-MSFT Hello Satish, could you please check on this bug and update the document? Thanks?

Adding @kgremban for further updates on this issue.
@kgremban Please look into this issue and update the doc with correct version as it needs validation.

@massand I created a pull request to update all examples to use 1.1 versions of IoT Edge. Since this is a straightforward update, I'll have github close this issue automatically when that pull request is merged. If you have any follow-up questions or comments feel free to post here again and we can reopen the issue if necessary.
Thanks

@kgremban It is great to see the versions updated consistently throughout the docs. @veyalla was discussing offline whether we should delete the install instructions for specific package versions from this doc -

Or, if you want to install a specific version of the security daemon, specify the version from the apt list output. Also specify the same version for the libiothsm-std package, which otherwise would install its latest version. For example, the following command installs the most recent version of the 1.0.10 release:
BashCopy
sudo apt-get install iotedge=1.0.10* libiothsm-std=1.0.10*
If the version that you want to install isn't listed, follow the Offline or specific version installation steps later in this article. That section shows you how to target any previous version of the IoT Edge security daemon, or release candidate versions.

If that is already resolved, you can ignore this message.


Choose a topic fixes

MA-1358 - iOS - Discussions - Default text of topic dropdown is incorrect
MA-1361 - iOS - Discussions - Top level topics and their child are appearing in same order
Note:
This is still WIP

Right now, we are displaying the "Choose a topic..." on the options picker when there's no clearly selectable option from the previous screen (When coming from All Posts, Posts I'm following or Top level topics)
So the question here is, do we show "Choose a topic..." as a non-selectable option on the UI?
OR
should it just show up in the label until we select a valid topic?
I've attached screenshots below for clarity.
p.s i know that drop-downs tend to have the placeholder option in the options on the Web (generally)... Just confirming...
CC: @explorerleslie  @andywaldrop
Collapsed

Expanded

@ehmadzubair we should match the defaulting behavior on desktop. Right now, it looks like it defaults to the first coursewide discussion topic, which is ordinarily "General". We do not force students to select a topic, we default one for them. Please check the code for desktop discussions and match the exact logic. Let me know if you have questions.

Okay... it looks like we are selecting the First selectable topic everytime... However, on the app... we were posting a discussion in the most relevant context when possible (e.g if we were in the Chemical Equations topic, the new post already had Chemical Equations selected)


no new refresh token on credentials authentication

Scenario:

Auth with credentials for user 1:
token: abc
refresh_token: xyz

Get new token with refresh_token:
token: def
refresh_token: xyz

token expired, auth with credentials for user 1:
token: ghi
refresh_token: xyz

optionally login from another browser with user 1 credentials:
token: jkl
refresh_token: xyz

I don't think that in step 3) and/or 4) the same token should be returned. Every credentials authentication should generate a new refresh_token.

You're right. I think its better to generate a new refresh token in every scenario.
For example in case 4 if we generate new tokens then you can revoke token only for some clients.
To avoid broke some clients compatibility we need to work on this and tag as a new release.

well, not generate a new refresh_token in every scenario, only when credentials are supplied (step 1, 3 or 4). When refreshing a (access) token using a refresh_token (step 2), the same refresh_token can be used.

Well its a possibility, if you want as a client to revoke the token you can logout.

Please review the pull request I've done concerning this issue. I have one more question. Why did the refresh_tokens extend their validity every time a refresh is done? It seems to me this poses a security risk. Refresh tokens can possibly be valid until the end of time

Hi, please check my comments in the pull request. Thanks