Allow nested metadata

This pr allows for nested metadata (one level deep only). This way someone can pass in metadata for our popover like so
api.upload_run(project_id, run, {
    "chunk_size": "256",
    "popover": {
        "commit_id": "1234"
    }
})

We decided not to do the popover nesting so this PR doesn't need to be reviewed anymore @joeferraratonic


⚠ ClamAV has degraded performance

In af57bd0, ClamAV (https://spamassassin.apache.org/) experienced degraded performance:

HTTP code: 200
Response time: 214 ms

Resolved: ClamAV performance has improved in 046490a.


If I test on YouTube test, what else do I need to change besides --split test?

If I test on YouTube test, what else do I need to change besides --split test? Do I need to change valid to test here?

Just make sure you have the two folders: YouTube/test and YouTube/all_frames/test. You don't have to change that.


Raidboss: anamnesis_anyder

added missing translations fr
corrections translations de and fr on replaceText
remove useless lines on replaceSync
@Akurosia, I have corrected a German translation error, can you confirm that it is correct, please.
@mooondark, 'Hammer/Blade Mark': 'Marque Marteau/Lame',, c'est correct ou je dois mettre Marque de Marteau ?

In the future, removing unused lines causes me a lot of work and is probably the last priority in terms of translations.  This is why it took me a while to get to this patch.  I would recommend against doing that manually in the future if you can avoid it.  Akurosia doing it with a script still is a lot of work, but at least it is done in one big patch.

I have a suggestion here.
Maybe we can make trigger i18n block as (for example a tankbuster on one tank):
return translate('Tank Buster on {0}', data.ShortName(matches.target));

Then we have a separate file named dungeon-name.json, which include:
"Tank Buster on {0}": {
  "de": "Tank buster auf {0}" ,
  "fr": "Tank buster sur {0}",
  "ja": "{0}にタンクバスター",
  "cn": "死刑 -> {0}",
  "ko": "\"{0}\" 탱버",
}

translate function would read out this json file, and find out which localized string to be replaced, based on user's locale settings. And then replace {0} and {1} (and so on) in the localized string to its second and third (and so on) parameters.
( I learned this approach first time in react-i18next, which is an i18n library for react. ( Although I have not seen its source code. ) They also named translate function as t or __, so make it easier to call. )
In this way, we can separate localization and code mechanic into two different files. We can also create a script or GUI tool to convert json file to csv or excel sheet (and vice versa), so even someone who doesn't know any js things can contribute translations.

This sounds like the same approach that was talked about in https://github.com/quisquous/cactbot/issues/1255.  Maybe we can follow up more there?


Implement Basic Project Template

@vilkazz I think this task is already completed. Is it?

I will complete this when i implement delete action (later tonight)!

delete implemented.


[4.0] Collate Codable diagnostics

What's in this pull request?
Cherry-picks #10253 to swift-4.0-branch.
Explanation: Codable failures produced diagnostics and notes explaining what failed during synthesis. These diagnostics were being output as they were generated, which was suboptimal because it meant that NOTEs could be produced before the error they were related to.
This was bad for consumers like Xcode, which associate NOTEs with the last error or warning diagnostic. If the only error in produced was a diagnostic from Codable synthesis, the initial NOTEs would be lost in the Xcode UI.
This collates these notes using a diagnostic transaction and only outputs the errors if something actually goes wrong.
Scope: Improves QoL for everyone adopting Codable and receiving diagnostics.
Radar: rdar://problem/32427064
Risk: Low
Testing: Manual verification of expected ordering of notes.

@swift-ci Please test

This PR should wait for the addition of some unit tests

@swift-ci Please test

Build failed
Jenkins build - Swift Test OS X Platform
Git Commit - 60fa6e5697329c981baa71a517aa96333ec0dffe
Test requested by - @itaiferber

Build failed
Jenkins build - Swift Test Linux Platform
Git Commit - 60fa6e5697329c981baa71a517aa96333ec0dffe
Test requested by - @itaiferber

@tkremenek Is this okay to merge?


How to add personal hosts to /etc/hosts?

Hi there,
I would like to add some personal hosts, but don't want them to be versioned. It makes sense to include this kind of customization? If not, we have always to manage files that should be committed and not.
Idea: add an extra file, something like /configs/hosts.custom that should be appended to /etc/hosts at the install.sh, and add that extra file to .gitignore?

that could work--also I've migrated myself to use https://github.com/StevenBlack/hosts, which might be a better add in replacement for the hosts file in the repo...

yeah, either it could be added as a git submodule like oh-my-zsh, or part of the install script could clone the repo and run the hosts setup. With the submodule approach, the install script should probably do a git submodule update, but with the clone on demand approach it could just clone into /tmp. Either way seems fine.
Feel free to PR it -- you'll likely get to it before I do :)


Update badge

Remove outdated stb_truetype crate
Parse the font once instead of on every request (using once_cell)

Closes https://github.com/rust-lang/docs.rs/issues/869
r? @Kixiron


Follow Azure id format, implement findTreeItem

In this PR:

Made tree item ids mimic Azure ids (just like v1  and pre-unified)
Implemented a find tree item method on the TDP.

I have implemented find item without getting children from the cache. I did this so we could update and use findByIdExperience in utils since it won't have access to the cache. However, without using the cache to get children, find item is extremely slow. For reveal, this hurts the UX quite a bit. I'm open to discussions here, but I think we have to use the cache since reveal (relies on find tree item most times) is just too slow.
PR unblocks #395

Closing as Azure ids have been implemented in #422, and find is implemented in #431


install live server unsuccessfully

Issue Type: Bug
I followed the step to install live server, but I didn't found "Go Live" display on my bar. When I right clicked the html, i didn't see the  "open in live server" sign, either.
VS Code version: Code 1.58.0 (2d23c42a936db1c7b3b06f918cde29561cc47cd6, 2021-07-08T06:54:55.083Z)
OS version: Windows_NT x64 10.0.19041
Restricted Mode: No

@sandy081 any idea what this might be?

oh! I fixed it, thank you so much.


16a OOP: Expressions

We are hitting the limit of what we can do for expressions using only dicts. Up to this point we've been shoehorning all kinds of expressions—call expressions, get expressions, binary expressions, and even unary expressions—into an exprwith a'left', 'oper', and 'right'`.
Let's do it the "proper" way with object-oriented programming (OOP) so that we can easily extend our interpreter to support other kinds of expressions.

Taking a moment to think about all the things we need to do with an Expr class:

resolve() to get a type
evaluate() to get a value

Actually ... that's pretty much it! 'left', 'right', and 'oper' were standardisations we had to impose to make expressions more predictable. Ultimately, though, we should only need to use an object's methods to do/get whatever we need.
So what kinds of expressions are we going to need?

unary expressions (one operator, one operand)
binary expressions (one operator, two operands)
get expressions (a frame, a name)
call expressions(a function/procedure, argument values)

Let's get started.

https://github.com/nyjc-computing/pseudo/blob/0ab95e42597a6ede5e6d88de4ef39be5f8885725/lang.py#L1-L6
To ensure subclasses implement resolve() and evaluate() methods, we raise NotImplementedError from the superclass if they are not implemented by the subclass.
https://github.com/nyjc-computing/pseudo/blob/0ab95e42597a6ede5e6d88de4ef39be5f8885725/lang.py#L10-L35
We initialise the subclasses with attributes.

Adding reprs for debugging: [2837078405fd95b5669d729a1afa2879bf0f389e]
Okay that's just messy. Let's abstract it into the parent class:
Add __slot__s: [5c031794a313a7b85e46928b235b7e122bb8f0e4]
And then use slots to generate a repr from the superclass: [cd908c90e045285eda07263c191f62805c40fa2a]

Literals
https://github.com/nyjc-computing/pseudo/blob/4047fb68ef5cf37ac92f2d63f05085ab4916fb2f/lang.py#L19-L23
Now we can figure out the Literal expr subclass.
https://github.com/nyjc-computing/pseudo/blob/f6548d8a91d2a48a424dd6d3a0e827a497c7e30a/lang.py#L25-L29
The simplest possible resolve() and evaluate(): a Literal is what we use when the user writes a 1 or "Hello" in the pseudocode. It holds an actual, determined value with a known type.

Unarys
https://github.com/nyjc-computing/pseudo/blob/5443c770e4c85c78805745fecf26e21c72847e48/lang.py#L33-L43
Unarys consist of an oper and a right. We are assuming here that the oper does not change right's type. So we delegate resolve() to right by calling its evaluate() method. And we evaluate() a Unary but passing the right value to the oper (which now takes only a single value).

Binarys
https://github.com/nyjc-computing/pseudo/blob/599b5ff77b272438057e72b5dc151a45e9ad615f/lang.py#L47-L61
Taking a page from what we wrote in evaluate(), we check the oper to determine the return type. This will get more complicated when we support REAL types, but that's not happening yet.
Evaluation is much the same as for Unarys. Here I am avoiding complicating the evaluate() method; any required type-checking should be done before evaluate().
We'll handle frame insertion later.

Gets
https://github.com/nyjc-computing/pseudo/blob/d26f41d9a8ac8ff08955680d2c26b01f9cadab82/lang.py#L65-L79
Keeping it simple, resolve() and evaluate() look very similar: both are retrieving a slot from the frame. But we won't be receiving a raw string for name; it is likely to be a name token, which reminds me: We are going to need a Name subclass for Expr as well. So we evaluate() name to get a key for frame.
Name Expr
https://github.com/nyjc-computing/pseudo/blob/98be282229e11c3eb83c0def2cdd19d65aa92e9c/lang.py#L33-L42
Let's make a Name expr too, which resolve()s to a new type, 'NAME'. And it evaluate()s to its name, a raw string.

Calls
https://github.com/nyjc-computing/pseudo/blob/0cbc0094dce4510520fcc44906278d4a18205fd6/lang.py#L96-L109
Lastly, Calls. This takes some explaining.
We are expecting self.callable to be a Get expr, since this is going to be scanned as a name token in the scanner. The actual callable will be stored in a slot in the frame. So we evaluate() this Get to access the slot, and retrieve the type from it (for resolve()).
As for evaluate() ... let's defer that implementation till later :stuck_out_tongue:

Parsing tokens, making Exprs
Now we refactor our parser to make Expr tokens instead of dict tokens.

It's a little daunting just changing all the makeExpr()s into class calls 😰 and what if I change my mind and decide to use some other way to represent expr`s ...?
Let's start with splitting off makeExpr() into six separate functions, each of which return an appropriate Expr object:
https://github.com/nyjc-computing/pseudo/blob/63587e766c13b039b1bfcf136e1fffea8dac3a63/parser.py#L23-L39
Looks excessive, but I plan to merge them back into 1 function later, once I figure out a pattern.

A nice advantage of this is it lets us replace hacky makeToken()s, which have no real business in a parser really:
https://github.com/nyjc-computing/pseudo/blob/707c72220565b595e2ad453c9b6da5924a3bb555/parser.py#L290-L296
With a simpler expression-making function (the design pattern term for this is factory function):
https://github.com/nyjc-computing/pseudo/blob/ac5f17da608a6f890923e76bb1e5f3d7b0f434d9/parser.py#L290-L291

Refactoring
Let's start by replacing the positional arguments in our make*Expr() function calls with keyword arguments: [6f853a8559f8e2a5415e15132a8c90b3c4aebb82]
I am aiming to replace makeLiteralExpr(), makeNameExpr(), etc with a new makeExpr() factory that determines which Expr to return based on the provided keyword arguments:
https://github.com/nyjc-computing/pseudo/blob/e0a595610593c51ae6eb8e5d7e56880dd5322e8c/parser.py#L23-L30
The *, indicates that positional arguments are not accepted; keyword arguments only!
It does so by checking the provided keyword arguments:
https://github.com/nyjc-computing/pseudo/blob/16f8bef1a76a1f859110a6dc2f9db41f25001add/parser.py#L30-L44

Since makeExpr() can now perform the job of any of the make*Expr() functions with the exact same keyword arguments, we replace all those calls with makeExpr() calls: [e98972071abf3df149a6e5fab6dd46ab70ff9e81]

A whole bunch of bugfixes before testing:
b0ccd781f4a21f3d660a0bed49a2daa261bf81f6
6faff09affd2d39bf2f314c1189760242871f9e8
13ea9359428754db49bcf21ddd2f6093882b092b

And more minor bugfixes in our Expr class:
[84efbd334267abe3d1a6f04dc4b151743cbc72d4], [98d7d77bb071d2970008e01debf2fd722e2ea235]
and in the parser:
[d542dba40f917deef4d11c43028319e25e1c5c73], [7e8e13fb3938cedea81056aa4cc5a8792d87f6b2], [f49fd954293e90a8d8659cd0c5068bcad3cb6faa]
And now our parser produces Exprs in place of expr dicts:
{'rule': 'declare', 'name': Name('Number'), 'type': {'line': 1, 'col': 17, 'type': 'name', 'word': 'INTEGER', 'value': None}}
{'rule': 'declare', 'name': Name('Max'), 'type': {'line': 2, 'col': 14, 'type': 'name', 'word': 'INTEGER', 'value': None}}
{'rule': 'assign', 'name': Name('Number'), 'expr': Literal('integer', 0)}
{'rule': 'assign', 'name': Name('Max'), 'expr': Literal('integer', 10)}
{'rule': 'file', 'action': 'open', 'name': Literal('string', 'data.txt'), 'mode': {'line': 5, 'col': 24, 'type': 'keyword', 'word': 'WRITE', 'value': None}, 'data': None}
{'rule': 'while', 'init': None, 'cond': Binary(Name(Name('Number')), <function lte at 0x7f883a501040>, Name(Name('Max'))), 'stmts': [{'rule': 'file', 'action': 'write', 'name': Literal('string', 'data.txt'), 'mode': None, 'data': Name(Name('Number'))}, {'rule': 'assign', 'name': Name('Number'), 'expr': Binary(Name(Name('Number')), <function add at 0x7f883a4f9b80>, Literal('integer', 1))}]}
{'rule': 'file', 'action': 'close', 'name': Literal('string', 'data.txt'), 'mode': None, 'data': None}

Working around frame insertion
This leaves us with a very annoying bug. In makeExpr(), we do an explicit None check to decide what kind of Expr to return:
https://github.com/nyjc-computing/pseudo/blob/98d7d77bb071d2970008e01debf2fd722e2ea235/parser.py#L30-L34
But in value(), we insert a None in place of frame for later insertion by the resolver:
https://github.com/nyjc-computing/pseudo/blob/98d7d77bb071d2970008e01debf2fd722e2ea235/parser.py#L83-L90
This is why we have a Name(Name('Number')) in our while stmt above, where we should be expecting a Get(Name('Number'))!
Resolving this is tricky: replacing frame with anything other than None means we need to use some explicit checking for a string. Instead, let's finally make our own None type. Coincidentally, 9608 pseudocode does define its own None-equivalent, called NULL.

NULL value
We create our own NULL value by calling Python's object(), which gives us a unique object with a unique object id:
https://github.com/nyjc-computing/pseudo/blob/9f709e858f5ab61905851f8f545ad21bdbdb50be/builtin.py#L94
We can use an is NULL check to see if something has a value of NULL.
Import it into our parser:
https://github.com/nyjc-computing/pseudo/blob/97a8f36f5457179386fbaa22b44aeacc23f29091/parser.py#L1
And use it in place of None for our frame:
https://github.com/nyjc-computing/pseudo/blob/97a8f36f5457179386fbaa22b44aeacc23f29091/parser.py#L83-L90
Now we have a Get(<frame>, Name('Number')) expression for our while cond:
{'rule': 'while', 'init': None, 'cond': Binary(Get(<object object at 0x7f3ebcde4e80>, Name('Number')), <function lte at 0x7f3ebcc1e040>, Get(<object object at 0x7f3ebcde4e80>, Name('Max'))), 'stmts': [{'rule': 'file', 'action': 'write', 'name': Literal('string', 'data.txt'), 'mode': None, 'data': Get(<object object at 0x7f3ebcde4e80>, Name('Number'))}, {'rule': 'assign', 'name': Name('Number'), 'expr': Binary(Get(<object object at 0x7f3ebcde4e80>, Name('Number')), <function add at 0x7f3ebcc16b80>, Literal('integer', 1))}]}

Our NULL object is looking pretty ugly there, but fortunately our users (pseudocode programmers) will never get to see it.

Resolving Exprs
Finally we move on to the resolver. Our first hiccup: resolving involves frame insertion. That means the resolve() method of our Exprs must take in the frame as an optional argument: 6d2a54f45b17f3b01025833db36557ec8631d3af]
Literal and Name don't need a frame and therefore no changes there.
Unary must resolve its right, in case it's a GetExpr (e.g.-Var`):
https://github.com/nyjc-computing/pseudo/blob/5008eb80237b454f7f58313263423db7f9babbe8/lang.py#L54-L55
Binary must resolve its left and right:
https://github.com/nyjc-computing/pseudo/blob/4af23ce87b76d3b73d68d207be896702a07b836a/lang.py#L69-L75
Get must replace its frame with the provided one:
https://github.com/nyjc-computing/pseudo/blob/c11ee8f5b304fdd079e6d9a48ba21ec90189b72e/lang.py#L89-L94
Call must resolve its callable, which is not the procedure/function itself but a Get Expr:
https://github.com/nyjc-computing/pseudo/blob/bb74075b1e40908ea4483c8fb57522bd4aaf387e/lang.py#L109-L111
This feels weird: why not evaluate() the Get Expr and replace the callable attribute? Because this lets us use Get.resolve() to retrieve the type, and Get.evaluate() to retrieve the callable itself!

After working out the kinks in lang.py, let's move on to resolver.py.
Having Expr.resolve() methods means we shouldn't need resolve() anymore, right? Let's try to eliminate all resolve() calls and replace them with equivalent Expr.resolve() methods.
INPUT:
https://github.com/nyjc-computing/pseudo/blob/1af3e40e02074d195a1091bc3154e4ed4d76f9e5/resolver.py#L78-L84
We'll also need to fix our Name.resolve() to make it return the word. Yes, that's exactly the same as evaluate(); more things we will have to rethink and refactor later 😩
https://github.com/nyjc-computing/pseudo/blob/b1ab2d9fa121f429dfd73a095ad7a42fb397d37c/lang.py#L41-L42
DECLARE:
https://github.com/nyjc-computing/pseudo/blob/eb366cff74d1961aa8cf413026ac6055ff66a56f/resolver.py#L86-L89
Feels like a step backwards; we are back to grabbing keys from dicts manually. Sometimes we gotta take a step back before we can take a step forwards; we have to do this because our stmts are still dicts and not their own object type yet. That will come later.
Minor fix: forgot to pass frame to the resolve() methods. [fbb89e3185c48423b87b6d0ae5e43439f2c21843]
ASSIGN:
https://github.com/nyjc-computing/pseudo/blob/b8528330f51168c598888d34041169b5013b89e9/resolver.py#L91-L93
CASE/IF/WHILE:
[d56510c65d6b56586b6078d127e7f735dd4f1efe]
These statements rely on a conditional Expr, which we now resolve using the Expr's resolve() method.
PROCEDURE/FUNCTION:
[6e67dcae6dfc31dfa9f252140d04f4eab65efdac]
The changes here primarily concern name-resolving. You can see how it starts to get a bit mindless: having a polymorphic resolve() method makes things much easier!
CALL:
[071446278cb9ccb29b8d1fbfea606982cb80b5c4]
Now our work starts to reveal its value: instead of resolve() having to work differently for different exprs, now we have each Expr decide what it should do to resolve() itself. We've moved the complexity out of resolve(), into the Expr.resolve() methods where it's slightly more manageable.
RETURN:
[fe8ceab03294c47f529ae2100c86557f686bfab3]
*FILE:
[a00fe50f6144ff4a7772a91b8794abcb462fc7ab]
And lastly in our helper functions:
[e92000d0460ad5dab6888b01b0f283f95ebcb4ba]

Testing
The moment of truth: deleting commenting out resolve() and see what happens:
  File "lang.py", line 42, in resolve
    return self.name['word']
TypeError: string indices must be integers

Oops, names don't resolve properly.
> lang.py(42)resolve()
-> return self.name['word']
(Pdb) self.name
'Number'

Turns out instead of passing a token, we passed the word directly to Name. That's a good idea; our resolver should not have to concern itself with tokens already. We'll do another refactoring check for that later.
The fix:
https://github.com/nyjc-computing/pseudo/blob/18533535af4278c97335505cf06ac2541629ce8a/lang.py#L41-L45
expectTypeElseError() is in limbo now; it isn't fully refactored to handle Exprs, so we insert a temporary workaround:
https://github.com/nyjc-computing/pseudo/blob/18533535af4278c97335505cf06ac2541629ce8a/resolver.py#L10-L24
This means we might sometimes raise errors without a token, so we also add another workaround for this: [a81c9b0242334d5115e5be166ced9a979d8facfa], [0cf5095849cfc50f9d6a05d99d43e4fab6372790]
(We'll get round to fixing this, once Exprs work!)

None: Expected 'INTEGER', got 'integer'

Oops, we need to standardise our cases: [80787d159b41a7383c6538751adfe7ded5d2d997], [42ba1f632e89a66b633edefcc47c92092c5ee4f0]
And our resolver works!

Evaluating Exprs
And it's the same story for evaluate()! Let's try to eliminate it, replacing it with Expr.evaluate().
First, we turn our attention to the Expr.evaluate() methods and correct them for recursion: [184eae4c1d1f3208812540c358d68dabede5a35c]
Call presents a thorny edge case: Should we evaluate() the callable here, or elsewhere? Doing it in lang.py may involve pulling in code from the other modules, and we want to keep lang.py as dependency-free as possible. For now, let's just return the callable; it is going to be a Get expr so we evaluate() that to get and return the callable itself.
https://github.com/nyjc-computing/pseudo/blob/eadf6827566620a29631be78c20f49debfb9bc3e/lang.py#L116-L117

Now we do a refactor, converting all evaluate() function calls to Expr.evaluate() method calls. It's quite a straightforward conversion so I won't do a blow-by-blow rundown: [01e97e606595ed80d8c0a79fb7bd87dccd0e7b7d]
And remembering to pass the frames to Expr.evaluate() in lang.py: [855ea90afc67bcab8e06c886c6e645454da94279]
And we're golden: it works!

That tricky Call ...
Now let's turn our attention back to Call Expr:
https://github.com/nyjc-computing/pseudo/blob/855ea90afc67bcab8e06c886c6e645454da94279/interpreter.py#L30-L34
evaluate() calls execCall():
https://github.com/nyjc-computing/pseudo/blob/855ea90afc67bcab8e06c886c6e645454da94279/interpreter.py#L86-L89
and execCall() in turn depends on assignArgsParams and executeStmts():
https://github.com/nyjc-computing/pseudo/blob/855ea90afc67bcab8e06c886c6e645454da94279/interpreter.py#L8-L17
Let's put this code into lang.py and see what we can simplify:
https://github.com/nyjc-computing/pseudo/blob/e096fb32605292b31450d6fb775dc1c528271dfd/lang.py#L106-L134

Unpack those functions and dump them into Call.evaluate():
https://github.com/nyjc-computing/pseudo/blob/741c9cf0c28802e37700a69f8947d9c087897658/lang.py#L116-L125
😣! On line 123 ... we have an execute() dependency! And execute() is a helluva long function:
https://github.com/nyjc-computing/pseudo/blob/741c9cf0c28802e37700a69f8947d9c087897658/interpreter.py#L124-L150
Fine ... we'll temporarily import it from interpreter.py:
https://github.com/nyjc-computing/pseudo/blob/afb66cdfb5de7eb47fa208a81f163e96ebb45b11/lang.py#L4
And continue with encapsulating our statements using OOP as well. Next chapter!


Feature/dataset db model

Added Database model for metadata and tests.

 Create database model for data files links storage


Report files dont build

Version
1
Platform
Linux Ubuntu22.04  x64
Description
Enter your issue details here.
Hi Paradigm team,
The problem is that when I run the command:
flood eth_getBlockByNumber https://metis-rpc.gateway.pokt.network/ --rates 5 10 15 --duration 5 --output /home/test
I only got the JSON file and the figures and other reports are not built.
Then I tried to run the docker version of the Flood:
docker run -v /tmp:/tmp ghcr.io/paradigmxyz/flood eth_getBlockByNumber test1=https://metis-rpc.gateway.pokt.network/ --rates 1 2 1 --duration 5
But I got the same result. Can you please tell me what the solution is?
The screenshots are also attached please take a look.
Thank you for any help you can offer

### Tasks

### Tasks

Hey @HosseinNazem, did you try running:
flood report /home/test
That should generate the html and jupyter notebook report files.


Pre-commit hooks for conventional commits

Depending on what goes into convention commits, tie them into pre-commit hooks.

Too much overhead and difficult to setup.  Unfortunately, not worth the hassle.  :(


Appgwfileshare

Sorry, my fault, i was doing a POC ( no real data on this amount of secrets, etc)

Opened by error


Issue #691: Optimize Top N Implementation

Reduce the per-thread memory to the rows and the ordering values of the top N for that thread.
Change Combine to just do some pointer manipulation.
Recompute the top N for the pipeline in Finalize

Thanks, this looks great!


[EPIC] Comments UI improvements

Megathread for multiple UI/UX improvements we need to make for comments to be truly useful.

[ ] HTML support (#197). Also needs to be supported for descriptions, which can also use HTML albeit rarely.

[ ] Reply button to add a comment to the post from the website

[ ] Expand/collapse button in individual comments

[ ] Possibly a button in each comment to get its link

For 2 & 4, I think we can add an icon on the header to open the comments URL and then maybe a gesture for opening each particular comment.

For 2 & 4, I think we can add an icon on the header to open the comments URL and then maybe a gesture for opening each particular comment.

For 2 I was thinking more along the lines of a FAB, for 4 I agree that a swipeable container is be a good option.


GH-38131: [Swift][CI] Add linting and fix linting errors

Description:

Added swiftlint to swift CI job: swift_test.sh.
Fixed swiftlint errors in swift files.

Closes: #38131

Added swiftlint source and reference artifacts to checkin.  Initially I had attempted to include SwiftLint as a plugin in the Package.swift files but the download took too long during CI build and the plugin didn't seem to fail the arrow swift build using spm.  Also, tried adding git clone of swiftlint in the dockerfile but this took too long during CI build and tried including just the swiftlint source without the artifacts but again this was too slow for CI build (these took over 15 minutes).

@kou Please review again when you get a chance.

@kou I believe I have made all the requested changes.  Please re-review when you get a chance. Thank you!

@kou if everything looks ok, please merge when you get a chance.


Should View Code in API reference link to master?

Currently, the View Code link in API reference links to the master branch, for example the page for StatementCompletedEventArgs links to https://github.com/dotnet/corefx/blob/master/src/System.Data.SqlClient/src/System/Data/RecordsAffectedEventArgs.cs/#L12.
The problem with that is that corefx master changes faster than docs, so when a type is moved (that's what happened for the above type in https://github.com/dotnet/corefx/commit/7888677b3c53596085c2fa53860d9a0519eeecfd), View Code links to a 404 page.
If the View Code link instead pointed to the commit that was used to generate the docs (e.g. https://github.com/dotnet/corefx/blob/f3d05c5/src/System.Data.SqlClient/src/System/Data/RecordsAffectedEventArgs.cs/#L12, assuming the commit was https://github.com/dotnet/corefx/commit/f3d05c5380a6ba430e9151a5a51f71c77bc99392), this would not happen.

Good point.  I think it should link to the current release, not master.  This is going to require proper versioning support in our docs.

When we reach the point where the docs build supports viewing docs for different versions of the product, it should link to the release being viewed (which may or may not be the current release).

They have changed the API build process and it's now linking to release/1.1.0. Which one do we want to show while we don't have the versioning story on docs: latest (release/1.1.0) or stable (release/1.0.0)?

I vote for stable.

I'd agree with stable, until we have versions for the docs site. Then, the show code link should go to the branch (or label) that matches the viewed branch.

I agree. I'll talk to @chenkennt to change this then.

We no longer have View Code for the API reference docs, so closing this one.


Improve Project budgets

Closes #1064

Adds monthly_budget column to the Project table
Adds monthly_budget field to the Project form
Orders can only be successfully placed if it doesn't exceed the monthly budget of the Project

Add pass/fail test for monthly_budget overruns

:+1:


Guard SoilPagedIndexStore>>#pageAt:put:  against overrides

Make sure to never just override an exisiting page in the store.
We change SoilPagedIndexStore>>#pageAt:put: to use #at:ifPresent:ifAbsentPut: to raise an error in this case

(I close the Pr)


[ENH] Fetch posting list operator

Description of changes
Summarize the changes made by this PR.

Improvements & Bug fixes

Operator for fetching the posting list for a given head. Currently clones the posting list. Can revisit in future if it ends up becoming a performance bottleneck.

New functionality

...

Test plan
How are these changes tested?

[x] Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes
None

Merge activity

Dec 5, 6:41 PM EST: A user started a stack merge that includes this pull request via Graphite.


Fix for issue #138 "Can sell used Items: Yes" text outside of button.

Description

Please describe what this PR does and why it does it

Fix for issue #138 "Can sell used Items: Yes" text outside of button.
SolDesktop.java: Refactor member variable name c to applicationConfig.
Changed the permissions on gradlew for *nix systems.

Testing

If the PR is testable, please describe how to test it here

Run the application.
New Game.
New Ship/Prev Ship.
Press escape key.
No menu text should be outside the button.

Notes

Include any extra notes here

Pre Pull Request Checklist:

[x] Code has been scanned with SonarLint
[ ] Methods have been annotated with @ Nullable and @ Nonnull (Read more)
[x] There are no errors present in the project
[x] Code has been formatted and indented
[ ] Methods have appropriate Javadoc (How to write Javadoc)

Can one of the admins please verify this patch?

@GooeyHub: ok to test

Hooray Jenkins reported success with all tests good!

Merged and also added you to the credits! :-)

Awesome. Great game, glad I found it.
Hope you don't mind, I'm going to use it to brush up on my Java, maybe learn some Gradle... etc. etc.

Absolutely! Please learn all the things and help out as you can :D


gitignore config file

add config file to gitignore to keep original repository from being changed... unless you like chaos.

I agree, if this means the git-config.sh file.

I agree, if this means the git-config.sh file.

That's what I mean.

git-config.sh added to .gitignore


Add QueryChannelUpgradeProof or QueryChannelProof/QueryUpgradeProof

Summary
Add a helper function to obtain proofs for testing, either via:
proofChannel, proofUpgrade, proofHeight := endpoint.QueryChannelUpgradeProof()

or
proofChannel, proofHeight := endpoint.QueryChannelProof()
proofUpgrade, _ := endpoint.QueryUpgradeProof

These calls are used by:

endpoint.ChanUpgradeTry
endpoint.ChanUpgradeAck
endpoint.ChanUpgradeOpen
any test providing valid proofs of channels/upgrades (upgrade try, ack, open handlers and sub handlers (startFlushUpgradeHandshake)

A benefit of QueryChannelProof is that it can be reused for proofs used in the opening handshake

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate contributors tagged/assigned

I believe the first approach is the most sensible here. Similar to this, QueryConnectionHandshakeProof also bundles all required QueryProof calls in a single place.

closed by #3731


Can I use Webcam Capture with DCS-2120 IP cam

Issue by @hwathanh:

I used sarxos lib to capture USB webcam successfully but not to my wireless IP cam. I own DCS-2120 IP cam. Could i use sarxos lib to connect to my IP cam via VPN and how to implement. Waiting for your reply. Thanks!

Hi @hwathanh,
You can use Webcam Capture API to capture images from IP camera if and only if your DCS-2120 supports MJPEG streaming. It's enough to get MJPEG image URI and use it in the API. The working (more or less) IP camera examples can be found here:
https://github.com/sarxos/webcam-capture/tree/master/webcam-capture-drivers/driver-ipcam/src/examples/java

On site http://www.ispyconnect.com/man.aspx?n=D-Link, connection type of DCS-2120 is JPEG. It means my IP cam could not be implemented with sarxos API. May be i have to find another API. Thanks for your rapid reply, sarxos!

Hi @hwathanh, please note that it's Webcam Capture API, and yes, you can use it to access your camera (in case of JPEG you need to use PULL mode). Link to examples can be found in my previous post.

Thanks sarxos. I'll try and let you know result later.

I did it. Thank you very much sarxos!


BUG: shap.plots.heatmap() can't handle Explanation without feature_names

Issue Description
In shap.plots.beeswarm(), if the input Explanation does not have specified feature_names, default names are created which I believe it's done here:
https://github.com/shap/shap/blob/ec17a2604127c16b83caaf8e3b4d10eeadaa73ee/shap/plots/_beeswarm.py#L155
But shap.plots.beeswarm() does not. This causes an error at the line below, if the input Explanation doesn't have feature_names specified (it is None):
https://github.com/shap/shap/blob/ec17a2604127c16b83caaf8e3b4d10eeadaa73ee/shap/plots/_heatmap.py#L81
Minimal Reproducible Example
I do not have a minimal reproducible example (also I believe the nature of the problem is clear enough).

Traceback
IndexError                                Traceback (most recent call last)
Cell In[1], line 2
      1 exp = shap.Explanation(shap_values, base_values, data)
----> 2 shap.plots.heatmap(exp)

File ~/.conda/envs/env_name/lib/python3.11/site-packages/shap/plots/_heatmap.py:81, in heatmap(shap_values, instance_order, feature_values, feature_order, max_display, cmap, show, plot_width)
     72 instance_order = convert_ordering(instance_order, shap_values)
     73 # if issubclass(type(instance_order), OpChain):
     74 #     #xlabel += " " + instance_order.summary_string("SHAP values")
     75 #     instance_order = instance_order.apply(Explanation(values))
   (...)
     78 # else:
     79 #     instance_order_ops = None
---> 81 feature_names = np.array(shap_values.feature_names)[feature_order]
     82 values = shap_values.values[instance_order][:,feature_order]
     83 feature_values = feature_values[feature_order]

IndexError: too many indices for array: array is 0-dimensional, but 1 were indexed

Expected Behavior
There should be code in shap.plots.heatmap() similar to that in shap.plots.beeswarm() mentioned above for example which creates default feature names when feature_names is None in the Explanation.
Bug report checklist

[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest release of shap.
[X] I have confirmed this bug exists on the master branch of shap.
[ ] I'd be interested in making a PR to fix this bug

Installed Versions
0.42.1

I do not have a minimal reproducible example (also I believe the nature of the problem is clear enough).

The minimal example is not just for illustrating the issue/bug, it also gives the maintainers a starting point (we'll use the example to make sure that the issue being reported is resolved properly). If you don't provide the example, it means that we need to do so.
Also, it gives us a bit more context, e.g. why is your feature_names None in your Explanation? Are you creating your own, or does it come from one of our Explainers? It could help uncover other underlying problems in the code base.
So please do provide a minimal working example @ivanightingale

The issue is still present in the most current version. I've updated a minimal example. @thatlittleboy


Exception when calling with length = max_message_length+1

the following bombs:
N=1001
p=new_plummer_gas_model(N)
f=Fi(max_message_length=1000)
f.gas_particles.add_particles(p)

while N=1000 or N=1002 doesn't...

the exception is:
Exception when calling function 'new_sph_particle', of code 'FiInterface', exception was ''int' object is not subscriptable'


Support ctest --parallel

Closes #395

I've added 619c5087 to reintroduce the openassetio-python-venv build target, so that the instructions in BUILDING.md are once again valid.  Note that this doc is to be rewritten somewhat in #624 .

Added 509e6534 to upgrade pip. The latest pip versions avoid copying the build directory when creating wheels (i.e. when installing the pure Python component).
This was causing Windows CI breaks due to the now-parallel ctest, whereby the C++ component was still being built when pip wanted to copy the build directory, and Windows has rather strict file permissions preventing copying whist the build was in progress.
I have (so far) re-run CI 4 times with no error.  Previously the failure happened on two consecutive builds (though not the build prior, when the failure could also have occurred).

Added a note about the imminent removal/refactor of the contentious openassetio-python-deploy target in 8410b638


Swap to USB pad trait for thumv7em targets

Supersedes https://github.com/atsamd-rs/atsamd/pull/362, related to https://github.com/atsamd-rs/atsamd/issues/363
An alternative approach that:

Introduces UsbPadD{m,p} marker traits for implementation over USB pads
Implements UsbPadD{m, p} marker trait over GPIO v1 and v2 types
Adds <Dm, Dp> generic arguments to UsbBus and other types that default to existing (v1) pad types

This allows USB pad types to be overridden with something like:
// Override default USB GPIO types to v2
pub type UsbBus = atsamd_hal::usb::UsbBus<Pin<gpio::PA24, Alternate<H>>, Pin<gpio::PA25, Alternate<H>>>;

While (hopefully) maintaining compatibility with previous users of UsbBus and v1 GPIOs.

I thought you looked familiar! I was at Kawaiicon last year + distantly work with Grace's team! :grinning:
Small world ^^

hah, wild how small it is sometimes ^_^

(WRT the PR, LGTM but as mentioned in https://github.com/atsamd-rs/atsamd/issues/363#issuecomment-752292525 I'll wait till @bradleyharden has had a chance to look at it)

@ryankurte,
From #362:

I couldn't find an .into() bound that would work between v1 and v2, i might have just missed it though?

I added an implementation of From in both directions between v1 and v2, but it missed the cut for the v0.11 release. @twitchyliquid64 just released v0.12, which now includes it. Instead of your current approach, you could make the new function generic with bounds like Dm: Into<Pin<PA24, AlternateH>>. That should be backwards compatible and accept both versions.
If you want to stick with a new trait for other reasons, you might want to make crate::typelevel::Sealed a super trait. That way, users won't be able to implement your new trait themselves, even if they try.

@ryankurte,
From #362:

I couldn't find an .into() bound that would work between v1 and v2, i might have just missed it though?

I added an implementation of From in both directions between v1 and v2, but it missed the cut for the v0.11 release. @twitchyliquid64 just released v0.12, which now includes it. Instead of your current approach, you could make the new function generic with bounds like Dm: Into<Pin<PA24, AlternateH>>. That should be backwards compatible and accept both versions.
If you want to stick with a new trait for other reasons, you might want to make crate::typelevel::Sealed a super trait. That way, users won't be able to implement your new trait themselves, even if they try.

The GPIO trait changes are merged in - is this ready to go @ryankurte / @bradleyharden or should we track one of the newer types?

The GPIO trait changes are merged in - is this ready to go @ryankurte / @bradleyharden or should we track one of the newer types?

I'd be happy either way, just haven't yet had a moment to revisit / rebase to use the .into() approach, whatever y'all prefer?

I'd be happy either way, just haven't yet had a moment to revisit / rebase to use the .into() approach, whatever y'all prefer?

Went with the sealed approach which is a bit more complex but... also kindof nice in that you can click the trait and see the types it's implemented over more directly in the docs than looking via .into().

Went with the sealed approach which is a bit more complex but... also kindof nice in that you can click the trait and see the types it's implemented over more directly in the docs than looking via .into().

Thanks! Looks good to merge once we get CI green:
error[E0432]: unresolved import `cortex_m_0_6::peripheral::itm`
  --> /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/cortex-m-0.5.10/src/peripheral/mod.rs:91:46
   |
91 | pub use cortex_m_0_6::peripheral::{cbp, fpb, itm, tpiu};
   |                                              ^^^ no `itm` in `peripheral`

Thanks! Looks good to merge once we get CI green:
error[E0432]: unresolved import `cortex_m_0_6::peripheral::itm`
  --> /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/cortex-m-0.5.10/src/peripheral/mod.rs:91:46
   |
91 | pub use cortex_m_0_6::peripheral::{cbp, fpb, itm, tpiu};
   |                                              ^^^ no `itm` in `peripheral`

Hmmm i may have to fix this on master

Hmmm i may have to fix this on master

According to the embedded rust chat something mad derped in cortex-m, that might have been the issue

According to the embedded rust chat something mad derped in cortex-m, that might have been the issue


needUpdate always true

in https://github.com/ineo6/hosts/blob/master/index.js#L134:
variable prevHost contains real datetime with format "YYYY-MM-DD HH:mm:ss",
but variable content absolutely contains "{update_time}",
which make variable needUpdate always true

Fixed by #4


Make resume logic the same for upload/download sync/async

ResumeDownload Sync and Async look good, but they are missing the try catch that can be seen in ResumeUpload Async,
ResumeUpload Sync is missing, although there is a ResumeUpload Async.

Make a nice try catch aggregate exception (using ResumeDownloadAsync as a model) in the other three cases
Code the missing ResumeUpload for Sync using the others as a model.
Add a log message so that the resume operation is actually noticed by users

Fantastic!

Now we need integration tests to validate our resume logic...


Fill in missing search mode commands in User Guide

Overview
Previously, the search mode section in the User Guide was missing the command add-on in the summary. This has been fixed.
Issues Fixed
Closes #259

LGTM


closeField event doesn't seem to trigger on a field

to duplicate, create a field, set its script to:
on closeField
answer the target
end closeField
Nothing happened to me on a background field
If I have some time this weekend, I'll have a look further and send a PR

@defano I have created a pull request for enabling the CloseField message.


add gdeploy config for fips mode

This configures gluster volumes to use fips-mode-rchecksum.
Validation:
# gluster volume get volume_beta_arbiter_2_plus_1x2 fips-mode-rchecksum
Option                                  Value
------                                  -----
storage.fips-mode-rchecksum             enable

Wouldn't be better to have just one playbook to start fips parametrized with variables? This way we have to maintain multiple files.


ModuleNotFoundError: No module named 'Hinkskalle.util'

In the current containers, downloaded by docker compose, the follwoing error is thrown:
# flask db upgrade
Usage: flask db upgrade [OPTIONS] [REVISION]
Try 'flask db upgrade --help' for help.

Error: While importing 'Hinkskalle', an ImportError was raised:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/flask/cli.py", line 234, in locate_app
    __import__(module_name)
  File "/usr/local/lib/python3.8/dist-packages/Hinkskalle/__init__.py", line 17, in <module>
    from Hinkskalle.util.swagger import register_authenticators
ModuleNotFoundError: No module named 'Hinkskalle.util'

Thanks for reporting this, the working_dir option was not set in the example docker-compose.


feat: allow platform mocking for local development

What is it?

[x] Feature / enhancement
[ ] Bug
[ ] Docs / tests

Description
This PR is to allow platform mocking for local development, i.e. closes #2280. When working with Cloudflare Pages, we'd need to access D1/KV/Durable Objects/etc. As Qwik City is currently using vite dev server which it doesn't know what wrangler/miniflare bindings are available, we'd need to initialise the Cloudflare Pages platform object and pass it to qwikCity() so that we can access it in the routeLoader$().
How To Configure Cloudflare Pages Bindings
vite.config.ts
...
import { Miniflare, Log, LogLevel } from 'miniflare';

async function getMiniflareBindings() {
  const mf = new Miniflare({
    log: new Log(LogLevel.INFO),
    ...
    script: `
      addEventListener("fetch", (event) => {
        event.waitUntil(Promise.resolve(event.request.url));
        event.respondWith(new Response(event.request.headers.get("X-Message")));
      });

      addEventListener("scheduled", (event) => {
        event.waitUntil(Promise.resolve(event.scheduledTime));
      });`
  });

  return mf.getBindings();
}

export default defineConfig(() => ({
  plugins: [
    qwikCity({
      platform: await getMiniflareBindings(),
    }),
    qwikVite(),
    ...
  ]
});

How To Access Cloudflare Pages Bindings
src/routes/index.tsx
export const useLoaderData = routeLoader$(
  async (ev: RequestEventLoader<PlatformCloudflarePages>) => {
    console.log(ev.platform);

    return {};
  },
);

export default component$(() => {
  const signal = useLoaderData();

  return <></>;
});

Checklist:

[x] My code follows the developer guidelines of this project
[x] I have performed a self-review of my own code
[x] I have made corresponding changes to the documentation
[x] Added new tests to cover the fix / functionality

Thank you very much!


How to train with the Bible content?

Hi,
What is the steps to train it with this specific Bible content?
Example:
https://raw.githubusercontent.com/tushortz/variety-bible-text/master/bibles/kjv.txt
Can you show me the steps to train it?
And the other question is: The final file is compatible with LLAMA?
Thanks.

WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
Traceback (most recent call last):
  File "/Users/paulo/Downloads/stanford_alpaca-main/train.py", line 232, in <module>
Traceback (most recent call last):
Traceback (most recent call last):
  File "/Users/paulo/Downloads/stanford_alpaca-main/train.py", line 232, in <module>
  File "/Users/paulo/Downloads/stanford_alpaca-main/train.py", line 232, in <module>
Traceback (most recent call last):
  File "/Users/paulo/Downloads/stanford_alpaca-main/train.py", line 232, in <module>
    train()
  File "/Users/paulo/Downloads/stanford_alpaca-main/train.py", line 194, in train
    train()
train()
  File "/Users/paulo/Downloads/stanford_alpaca-main/train.py", line 194, in train
  File "/Users/paulo/Downloads/stanford_alpaca-main/train.py", line 194, in train
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/hf_argparser.py", line 332, in parse_args_into_dataclasses
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
      File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/hf_argparser.py", line 332, in parse_args_into_dataclasses
model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/hf_argparser.py", line 332, in parse_args_into_dataclasses
    obj = dtype(**inputs)
  File "<string>", line 110, in __init__
    train()
  File "/Users/paulo/Downloads/stanford_alpaca-main/train.py", line 194, in train
  File "/U    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/hf_argparser.py", line 332, in parse_args_into_dataclasses
sers/paulo/Library/Python/3.9/lib/python/site-packages/transformers/training_args.py", line 1172, in __post_init__
        obj = dtype(**inputs)obj = dtype(**inputs)

  File "<string>", line 110, in __init__
  File "<string>", line 110, in __init__
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/training_args.py", line 1172, in __post_init__
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/training_args.py", line 1172, in __post_init__
    obj = dtype(**inputs)
  File "<string>", line 110, in __init__
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/training_args.py", line 1172, in __post_init__
    and (self.device.type != "cuda")
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/training_args.py", line 1556, in device
    and (self.device.type != "cuda")
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/training_args.py", line 1556, in device
    and (self.device.type != "cuda")
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/training_args.py", line 1556, in device
    and (self.device.type != "cuda")
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/training_args.py", line 1556, in device
    return self._setup_devices
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/utils/generic.py", line 57, in __get__
    cached = self.fget(obj)
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/training_args.py", line 1541, in _setup_devices
    return self._setup_devices
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/utils/generic.py", line 57, in __get__
    return self._setup_devices
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/utils/generic.py", line 57, in __get__
    cached = self.fget(obj)
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/training_args.py", line 1541, in _setup_devices
    cached = self.fget(obj)
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/training_args.py", line 1541, in _setup_devices
    return self._setup_devices
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/utils/generic.py", line 57, in __get__
    cached = self.fget(obj)
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/transformers/training_args.py", line 1541, in _setup_devices
    torch.distributed.init_process_group(backend="nccl", timeout=self.ddp_timeout_delta)
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/torch/distributed/distributed_c10d.py", line 761, in init_process_group
    torch.distributed.init_process_group(backend="nccl", timeout=self.ddp_timeout_delta)
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/torch/distributed/distributed_c10d.py", line 761, in init_process_group
    torch.distributed.init_process_group(backend="nccl", timeout=self.ddp_timeout_delta)
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/torch/distributed/distributed_c10d.py", line 761, in init_process_group
    default_pg = _new_process_group_helper(
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/torch/distributed/distributed_c10d.py", line 886, in _new_process_group_helper
    torch.distributed.init_process_group(backend="nccl", timeout=self.ddp_timeout_delta)
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/torch/distributed/distributed_c10d.py", line 761, in init_process_group
    default_pg = _new_process_group_helper(
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/torch/distributed/distributed_c10d.py", line 886, in _new_process_group_helper
    default_pg = _new_process_group_helper(
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/torch/distributed/distributed_c10d.py", line 886, in _new_process_group_helper
    raise RuntimeError("Distributed package doesn't have NCCL " "built in")
RuntimeError: Distributed package doesn't have NCCL built in
    default_pg = _new_process_group_helper(
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/torch/distributed/distributed_c10d.py", line 886, in _new_process_group_helper
    raise RuntimeError("Distributed package doesn't have NCCL " "built in")
RuntimeError: Distributed package doesn't have NCCL built in
    raise RuntimeError("Distributed package doesn't have NCCL " "built in")
RuntimeError: Distributed package doesn't have NCCL built in
    raise RuntimeError("Distributed package doesn't have NCCL " "built in")
RuntimeError: Distributed package doesn't have NCCL built in
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 16486) of binary: /Applications/Xcode.app/Contents/Developer/usr/bin/python3
Traceback (most recent call last):
  File "/Users/paulo/Library/Python/3.9/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/torch/distributed/run.py", line 762, in main
    run(args)
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/torch/distributed/run.py", line 753, in run
    elastic_launch(
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/Users/paulo/Library/Python/3.9/lib/python/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
train.py FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2023-03-15_16:28:33
  host      : <IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.ip6.arpa
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 16487)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2023-03-15_16:28:33
  host      : <IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.ip6.arpa
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 16488)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2023-03-15_16:28:33
  host      : <IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.ip6.arpa
  rank      : 3 (local_rank: 3)
  exitcode  : 1 (pid: 16489)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-03-15_16:28:33
  host      : <IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.<IP_ADDRESS>.ip6.arpa
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 16486)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

It sounds like you don't have a version of pytorch that supports nccl. What hardware are you running on? Here's the relevant documentation with more info: https://pytorch.org/docs/stable/distributed.html. You may have to switch to a different communication backend.

@paulocoutinhox  what is the loss you use here.
secondly, is 40gb single gpu sufficient to train with some >1 bs ?


[Bug]: Run Stopped Uploading, Sync Not Adding Missing Data

Describe the bug
Hi. I was training a run, and half way through I saw the print out about the core service. So I added the line wandb.require("core") and restarted training. It resumed training for a few epochs, it was logging and everything seemed fine.
This morning I came in and saw the run was still going locally, but had stopped syncing to Wandb. Weird. So I stopped the run, updated Wandb from 0.17.X to 0.18, and manually sync'ed all the run folders. The sync said it worked, but the data is not showing up in the UI.
Can you help me figure out if there's a way to sync the missing data, or figure out why it's not working? I've attached the debug logs, although I'm not sure if they append everything everytime. The run is here. Thank you for your help.
debug-cli.cmccarth.log
debug.log
debug-internal.log
$ wandb --version
wandb, version 0.18.0
$ python --version
Python 3.10.14
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.4 LTS
Release:        22.04
Codename:       jammy

Here are the debug logs for the specific run-* subfolder that was in use at the time the uploading stopped.
debug-internal.log
debug.log

Hi Collin, sorry for the trouble and thank you for reporting this! We're actively looking into possible causes.
If the sync completed successfully but some data is missing, it could mean that data never got saved in the transaction log. This would be the "run-0bzd3bun.wandb" file in the run folder and it can be read using https://github.com/wandb/wandb/blob/main/tools/inspect_tool.py. Seeing this file would help a lot, but note that it contains all the data you logged for the run. If the data is not sensitive and you're okay with sharing it, please send it to<EMAIL_ADDRESS>with a link to this comment, and we can debug (and maybe retrieve the data) over email.
In our request logs, it looks like the last nontrivial data upload (for run history / summary, i.e. run.log()) is on 2024-09-13 at 01:39 AM UTC. After that point, there are interactions up until around 4 PM UTC that don't upload data, but that doesn't make a lot of sense based on the timestamps in your logs. Check that there isn't a wandb-core process hanging around---this was a bug that got fixed in 0.17.8.

@timoffex Thank you for the response. Let me dig into this Monday AM with your inspect_tool.py (very happy that exists, I'm sure it will help a lot).
I'm seeing more odd behavior on another run that just crashed with the latest 0.18.0 version. I give each run 30 minutes to upload the checkpoints at the end (can be a ~1GB upload, but that's usually plenty). The run here timed out after 30 minutes, and the last data uploaded shows epoch 122 when the local data shows epoch 130 (end of training). Maybe I should try switching back to the "legacy" service for now?
Here are the debug logs for that run as well.
P.S. I'm using WANDB_CACHE_DIR and WANDB_DATA_DIR and I see warnings in debug.log that they are unknown. Are these no longer valid in version 0.18, or is this a false warning message?
debug-core.log
debug.log
debug-internal.log

Looks like it happened to a third run, here. I switched back to using the legacy service and I haven't had any more issues.

@collinmccarthy Thank you for sending the files!
I'm still working on debugging this, but I noticed something strange that I wanted to ask you about. The transaction log you sent for run hdo222vj is for its first portion, before you resume it. Interestingly, it cuts off at a record corresponding to lines 47875-47887 of the console logs, at 02:16:15, but the uploaded data continues up until 02:16:28. We write the transaction log and upload data concurrently, but it's strange to see file operations lag 13+ seconds behind network operations.

Could there be anything unusual about your file system?
Do you call run.finish() and wait for it to complete?

Note that if you kill the Python process before run.finish() returns, some data may fail to be uploaded or flushed to the transaction log.

It's possible that the legacy service and core handle shutdowns differently resulting in the difference in behavior that you see.

@timoffex I'm not sure about the 13+ second lag. The servers are behind a VPN but I've had an equal number of VPN issues over the last few months when running the legacy service and haven't had any issues. The file system is a standard NFS too, connected via a 40GbE switch.
Yes, I do wait for run.finish() to complete before exiting. GPU 0 calls finish and GPUS 1-N exit, otherwise torchrun will kill the job before GPU 0 finishes uploading the data due to the timeout being reached. If the run gets preempted, I catch SIGCONT/SIGKILL, mark the run as preempting, then call run.finish(exit_code=1). About 30 seconds later the run gets killed, and when it resumes I re-upload all files with policy=now. This workflow has been working through hundreds of runs with and without preemptions over the last few months, without issues.
I'm sure you see this, but here are the gaps in data I see. Of the three runs I posted above, it looks like the core service failed before the run finished in  hdo222vj around epoch 122, so I have missing data from epoch 122-130. In run 0bzd3bun the data was lost between epochs 68-100 ish, and in run mmeqt2q6 the data was lost between epochs 15-25.
I think you may find some clues in run 0bzd3bun because the core service stopped working at epoch 15, and then magically started working again at epoch 25 without the run restarting or getting preempted. The run restarted at epoch 0, and later at epoch 32, but not in between.
I'm going to upload all of the run data I have and email you a link to support (re-using the same email chain we already have). I'll briefly describe the contents of the directories in that email.

I was able to reproduce behavior very similar to this, and in my case it was caused by a very slow artifact upload. https://github.com/wandb/wandb/pull/8429 which will be in the next release (0.18.2) will help diagnose those kinds of issues, if that was the problem in your case. This explanation works for your description of wandb getting stuck at epoch 15 before resuming at epoch 25---it may have been blocked on a slow operation (2 bugs here: it probably shouldn't block, and the operation probably shouldn't be that slow).
The 13s gap was a red herring: the final updates to the transaction log just didn't get flushed. They would have gotten flushed on a successful run.finish(), but of course you can't rely on that if the wandb process is stuck.
For now, feel free to continue to use wandb.require("legacy-service")---it will continue to work until we can get all wandb-core issues resolved.
Thanks a lot for your help debugging!

Got it, thank you for the detailed explanation and for pushing a partial fix! At this point I'll probably freeze my Wandb version to <0.18 until after my paper deadline, but I'll update to 0.18.2 after the deadline and let you know if I run into similar issues.
Thanks again!

Quick update: the fix that prevents artifact uploads from blocking other operations didn't make it to 0.18.2, but will be in 0.18.3 (https://github.com/wandb/wandb/pull/8466). 0.18.2 does have some additional logging that could be helpful.
I've since realized that artifact uploads block even when using legacy-service, so it's unlikely that this explains the problem you were seeing. I'm hoping the additional logging will help us debug further, especially if more people hit this issue.

Got it, thank you for the update! I’ll switch to 0.18.2/3 after my paper deadline and see if I can help narrow down the issue if it’s not entirely fixed by then.

Hey Collin! Have you had a chance to see if the issue still happens? We haven't had reports of similar issues as far as I know, so I'm hoping the change I described before worked. If it's not fixed, we have additional logging in debug-internal.log that may prove useful.

Hey @timoffex, I reverted to 0.17.9 so I haven't gone back to the core service to test it out. My paper deadline is in 4-5 weeks, so I'll probably wait until after that to upgrade to 0.18.3+. Would you like me to close the issue and re-open it if necessary? Or leave it open until I can test it out in December? I'm fine either way.

Let's keep the issue open a little longer, in case anyone else comes across this behavior.


Update coding standard to PSR-12

PSR-12 was recently accepted and replaces PSR-2, which is now deprecated. There aren't many differences between the two, since PSR-12 is an extension of PSR-2. Changes to the src and tests folders were done automatically by phpcbf and I manually changed the references to PSR-2 in CONTRIBUTING.md and Makefile.

Merged. Thanks!


add 1b visit positions

Also improving 'def' consistency?
Edit: was mostly consistent, just missing in the previous set of position analysis games.

also it's odd that fig 32 and fig 33 are separate but on the same page — can we either combine them into one figure or force fig 32 to be on it's own page?

also it's odd that fig 32 and fig 33 are separate but on the same page — can we either combine them into one figure or force fig 32 to be on it's own page?

Good catch. It looks like all the figures in that section can have up to 6 boards. I tried that a while back and it didn't seem to work, not sure what changed, but I've put the new ones in one figure and also (separately) the old ones where possible.


Hours not showing on physical device

Hello, I cannot get the hours to show on the left hours column.
Edit: I tested on emulator and the hours are showing. Default font color on physical android devices is white, which is why the hours do not show because the background is also white.
Solved with: theme={{hourText: {color: 'black'}}}
Thank you

Same issue !

I updated default text color in v1.12.0


Refresh Go Report card - remove step

Why remove it totally? Does it discard?

Why remove it totally? Does it discard?

The Go Report card server had issues that caused reports to show an F score. I'll bring it back tomorrow if it's fixed.

Thanks for input


LaTeX in text blocks

I'm thinking of something like this:

When the user inputs a $ followed by a space, an editable MathLive component will be created as an inline slate element with a dynamic width (long shot but maybe?). Another option is to have a popup for the math input, then pressing Enter will insert an inline slate element (that is not directly editable).
When the user inputs a $$ followed by a space, an editable MathLive component will be created as an block level slate element.

Personal opinion: I like how Notion has a few options about how math is input, both inline and as a block.  One way is: enter a command to open a math popup, (all of its commands start with / and then you type which command you want) and inside of that you can enter LaTeX (or KaTeX or whatever it is, I'm not sure).
Another way is: just type $$ then keep typing the TeX, and when you close it with $$, it grabs everything between the pair of double-dollars and renders it.  I usually use this way because it keeps my fingers on the keyboard, is very fast, etc.  But if you ever need to edit what you already wrote then you click on the rendered TeX and it pops up that editor tool, which is the same thing you get by entering the / command.
Anyway, that seems like best system to me, although I'm sure tastes vary!  Another idea that I kinda like is various non-Tex shortcuts that enter UTF symbols -- this can be nicer because they render one-at-a-time, so you don't have to wait to see what you've written, and can sometimes be easier to edit, copy-paste, etc.  Maybe if these sorts of symbols had shortcuts started by their own command like :, and maybe the same command can enter emojis ... just brainstorming ideas fast than a team can implement them!  :)
But figured you might want to hear some ideas, pick any that sound good to you.  Thank you for building this project!


PR #186 needs updating to use FWLinks in the nuspec

PR #186 needs updating to use FWLinks in the nuspec
https://github.com/xamarin/XamarinComponents/pull/186

Updated to use fwlinks


Progress bar does not stay visible on Windows 10

When running a build with lots of output, the output pushes the progress bar off the visible screen, until the output slows down, and then behaves as expected.
Expected Behavior

Progress bar should always be at the bottom of the terminal screen, as it is on other terminals (or at least Mac)
Current Behavior

Fast output pushes the progress bar down out of sight, and it returns when output slows.
Context

It is more aesthetic than a functional problem.
Steps to Reproduce (for bugs)

Clone gradle/gradle on a Windows 10 machine and run gradlew clean --info --parallel
Your Environment
Windows 10
Command prompt

@w25r Is this only the case when there's lots of output?

I think what happens is that each line of output causes the progress bar to be pushed off the visible screen, and then when the progress bar is updated, it redraws in the right position... but if there is fast enough output that the progress bar isn't updating fast enough, it is pushed out of site altogether.
Is there an equivalent to https://asciinema.org/ for Windows?

It's probably easiest to just record the screen on Windows.

The way it's currently rendering the parallel console is the same on both Windows and *nix system. On Mac, it seems to be fast enough that you don't see any problem. On Windows, I'm not sure if it's due to running in a VM or if it also happens on actual hardware. I remember trying to change the rendering by only updating the output area when we flush the console but didn't see much improvement. We could revisit this solution.

I'm not running in a VM, I'm running on physical hardware.

I tested on an older, slower mac and saw the same problem.  So this may not be an issue at all.  Perhaps the new console just looks better with a super fast processor.

I also cannot see progress bar on 4.02 and 4.1 when running git bash on windows. But when I use embedded terminal in IntelliJ with Git Bash it is visible. Very strange :/

Rich console is still an issue on Windows. I don't think it's wise to disregard these issues.


Django Parler Admin is broken when template cache is enabled

when I try to access a page on Django Admin using Parler, I get a nice 500 error.
http://dpaste.com/1PXT6W0
The problem is here:
    def get_change_form_base_template(self):
        """
        Determine what the actual `change_form_template` should be.
        """
        opts = self.model._meta
        app_label = opts.app_label
        return _lazy_select_template_name((
            "admin/{0}/{1}/change_form.html".format(app_label, opts.object_name.lower()),
            "admin/{0}/change_form.html".format(app_label),
            "admin/change_form.html"
        ))

_lazy_select_template_name = lazy(select_template_name, six.text_type)

Lazy return a proxy instance, and It isn't valid for Django Template Cache. When I replace this return to:
return "admin/change_form.html"

All works nice.

Strange that I haven't seen this error before. This could be a Python 3 issue, as we declare the lazy function nicely with a str return value.
As workaround, you can create your own admin template, and define it as change_form_template. You'll have to include the language tabs manually (copy it from admin/parler/change_form.html)

@Nekmo is this raise the same error with Django 1.8? It can also be a regression in 1.9.

@vstoykov I don't know. I don't use Django 1.8.

@Nekmo Is it possible just to try run in with Django 1.8 to see if it will be broken or you are using 1.9 only features and will be unable to run it?

You can try version 1.6.4 which should solve this issue!


OpenAI LLM call timeout

Not work, and always timeout....
import { ChatOpenAI } from "langchain/chat_models/openai";
import { HumanChatMessage, SystemChatMessage } from "langchain/schema";

//
const key = "sk-xxx";
const chat = new ChatOpenAI({ openAIApiKey: key, timeout:2000 });
const response = await chat.call([
  new SystemChatMessage(
    "You are a helpful assistant that translates English to French."
  ),
  new HumanChatMessage("Translate: I love programming."),
]);

console.log(response);

But, with openai official API, it works fine:
import { Configuration, OpenAIApi } from "openai";
const key = "sk-xxx";
const configuration = new Configuration({
  apiKey: key,
});

const messages = [{ "role": "user", "content": "Hello!" }];

const openai = new OpenAIApi(configuration);
const chatRequestOpts = {
  model: "gpt-3.5-turbo",
  messages: messages,
  temperature: 0.7,
};
const response = await openai.createChatCompletion(chatRequestOpts);

console.log(response.data);

Any ideas? Thanks!

Also experiencing this issue.

Solved by this this comment.
Turn your clash client mode to enhanced mode


Feature request: Profiling with multiple workers in distributed settings and visualizing them individually

Hi,
I am running Tensorflow in distributed settings only using CPUs with multiple workers and parameter servers. I would like to find a way to generate timeline information of individual workers/parameter servers and visualize them together to check on potential scaling issues.
Currently, I can generate timeline traces containing the profiles of all the parameter servers and a single worker.  But, visualizing all the workers in the same timeline trace would be beneficial.
Thank you.

This question is better asked on  StackOverflow since it is not a  bug or feature request. There is also a larger community that reads questions there. Thanks!


Data scientist/Analyst: Determine how to customize alerts for each individual firefighter profile

Note: in order to work out the tech (algos / ML) behind the alerts, we're going to need to do some of the storyboarding described in Issue 89
e.g. (illustrative example)
In the command center, an alert goes red for Alfonso. What happens next?

Does Alfonso's response depend on knowing what the underlying condition, or conditions were? e.g. will he react differently if
** a sensor just saw a toxic level of NO2 for a few seconds and then stopped
**  a firefighter is experiencing a combination of {high temp + CO}?
**  a firefighter has been exposed to too much NO2 cumulatively over the last 15 minutes?

Labelling data is one of the big costs in machine learning and the 'type' of explainability required determines how we do the labels (as well as how feedback is gathered at runtime.)  e.g. red / yellow /green is simple, but is it sufficiently-explained to enable correct follow-up actions? When context/explanation is essential, we often choose to use machine learning to learn the explanations (like 'critical exposure to NO2 over 15 mins') rather than the decisions ('red: get the firefighter out').  Need to know this up-front, as labelling 100s/1000s of examples is expensive is usually not cost-effective to repeat / fix.

I think we'll need to include this in a set of assumptions and future recommendations based on the research we've done.


Display data tags available at grid node

Is your feature request related to a problem? Please describe.
We want  to show data tags Infos at the grid node.
Describe the solution you'd like
Use grid node UI to display data tags available showing a brief description of them.

I'd like to work on it

Ok!


options menu icon area is weired

@dragove it is regression and should be fixed in the next release of IDEA. Details here https://youtrack.jetbrains.com/issue/IDEA-316472/Welcome-Screen-settings-icon-background-regression-with-custom-theme

Thanks for telling me that, I'll close this issue as it is not caused by the theme.


Rename invalid Latin-1 filenames

As mentioned in #184 you have some files in your repository whose names are not UTF-8.  This is not valid.  These files appear to be Latin-1 (\344 is ä in Latin-1.)
I have renamed them to be UTF-8 strings, using ä (U+00e4).
Fixes #184

Hi @ethomson, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!

It looks like you're a Microsoft contributor (Edward Thomson). If you're full-time, we DON'T require a Contribution License Agreement. If you are a vendor, please DO sign the electronic Contribution License Agreement. It will take 2 minutes and there's no faxing! https://cla.microsoft.com.

TTYL, MSBOT;

I'm getting Connect bugs assigned to me because of this issue, GitHub is getting support requests for this issue, and so is the LibGit2Sharp project.
I'm a Microsoft FTE and do not believe that I require a CLA.
Thanks for your attention to this matter!

It seems that it doesn't fix the problems entirely, at least on VS (libgit2sharp?) side of things. I tried pulling your branch and then opening it in VS. Here's what git says about it:

And here's what VS says about it:

I think that, to avoid all these headaches, we'd rather just switch to not having files with Unicode names, and have the test generate them as needed at runtime. I'll take care of it.
But it is still worth figuring out what's going on with the above two screenshots on VS side of things...

I also had trouble finding the renamed files from our tests. Changing the test to generate the file as necessary seems a better approach.

Oops, that was my bad, I missed renaming the Program.py and left its folder name as Latin-1.  Fixed that in the PR, thanks!

Thanks, this works great!

Cool, thanks!  :dancers:


[MM-36431] Logic to support multiple configurable tabs per server

Summary
This PR contains just the logic to control multiple tabs based on a single server. The relationship is now setup such that we have any number of servers with any number of tabs, and each of those tabs maps to a view. Currently we have 3 different tab types, for each of the new products, but they all just map back to the root server type. Follow-up PRs will expand upon this logic and add behaviours that the app will follow with respect to the different tabs.
This should just be a logic PR so only Dev Review is needed at this time.
Ticket Link
https://mattermost.atlassian.net/browse/MM-36431
Release Note
Converted the tabs interface to support multiple configurable tabs based on the server added.

The file .circleci/config.yml is in the blocklist for external contributors. Hence, these changes are not tested by the CI pipeline active until the build is re-triggered by a core committer or the PR is merged. Please be careful when reviewing it.
/cc @mattermost/core-security @mattermost/core-build-engineers


Training cascade_rcnn failed because of get_loss

Traceback (most recent call last):
File "detection_train.py", line 277, in                                                                                train_net(parse_args())
File "detection_train.py", line 21, in train_net                                                                                transform, data_name, label_name, metric_list = config.get_config(is_train=True)
File "/simpledet/simpledet/simpledet/config/cascade_r101v1_fpn_1x_mine.py", line 198, in get_config                                                                     bbox_head_3rd
File "/simpledet/simpledet/simpledet/models/cascade_rcnn/builder.py", line 28, in get_train_symbol                                                                      rpn_loss = rpn_head.get_loss(rpn_feat, rpn_cls_label, rpn_reg_target, rpn_reg_weight)
TypeError: get_loss() takes 4 positional arguments but 5 were given
Hi, I used cascade_r101v1_fpn_1x as my model to train custom data, but got error like this.

Bug confirmed and I am working on it.

Close by #162


Added missing contains module requirement

'mout/array/contains' is used when hints with possible_format is specified, but was missing in required modules.

Thanks a lot!


Fix missing favicon for component examples

Fixes https://github.com/alphagov/govuk-design-system/issues/2008
What

Update the layout-example.njk template to use the GOV.UK Frontend page template instead of hard coding it. This means we don't have to hardcode a link to the favicon (which is missing at the moment) and we also get other <head> tags automatically, e.g: social sharing image.
Update other layouts to make sure they have the correct assetUrl set.

Why
Anything using layout-example (for example, tabs) had a missing favicon and no social share image.
Anything using a layout with a missing assetUrl had a missing social share image as the asset could not be found (for example, question pages)
Favicon / layout change
There should be no visual change for examples, other than that a favicon should now appear.
Social share images
Social share previews tested using https://socialsharepreview.com
Social share preview, before (component example)

Social share preview, after (component example)

Social share preview, before (full page example)

Social share preview, after (full page example)

Using the govuk-frontend template for the example pages seems like a really good idea. I'm just wondering if there's a way to be able to use the _generic.njk template in this repo, to avoid code duplication 🤔 Do you think that would work? We'd have to have some way to override the robots line, of course.

Using the govuk-frontend template for the example pages seems like a really good idea. I'm just wondering if there's a way to be able to use the _generic.njk template in this repo, to avoid code duplication 🤔 Do you think that would work? We'd have to have some way to override the robots line, of course. We could then perhaps even base the other example layouts off this one.

@lfdebrux That's a good point. Unfortunately I'm not sure it'll help with code duplication as (from a quick look) I think we'd end up having to override almost every code block as there are subtle differences in things like the head, htmlClasses, main etc

Using the govuk-frontend template for the example pages seems like a really good idea. I'm just wondering if there's a way to be able to use the _generic.njk template in this repo, to avoid code duplication 🤔 Do you think that would work? We'd have to have some way to override the robots line, of course. We could then perhaps even base the other example layouts off this one.

@lfdebrux That's a good point. Unfortunately I'm not sure it'll help with code duplication as (from a quick look) I think we'd end up having to override almost every code block as there are subtle differences in things like the head, htmlClasses, main etc

Ah, fair enough! Sorry to send you barking up the wrong tree there.

@lfdebrux Not at all, it was a good thought! I think a tidy-up in the future might be a good idea, but might require a bit of an overhaul - something to keep in mind perhaps.


cvxpy 1.0.3 traceback using CVXOPT

When solving a 3 variable l1-regularized least squares problem, I see python traceback.
/usr/local/lib/python3.6/site-packages/cvxpy/problems/solvers/cvxopt_intf.py in format_results(self, results_dict, data, cached_data)
415                     new_dual[:P_rows] = 0
416                 else:
--> 417                     new_dual[:P_rows] = data["P_leq"].T*z
418                 new_dual[P_rows:] = new_results[s.INEQ_DUAL][leq_len:]
419                 new_results[s.INEQ_DUAL] = new_dual
testcvxopt.html.zip

Please don't post zip files. I don't like downloading random zip files.

I had to zip it because github did not allow me to include .html output from Jupyter notebook.
How would you like the bug report?   I am trying to be helpful.

Here it is:

import numpy as np
import cvxpy as cvx
n=3
m=2
k=1
np.random.seed(1)
xTrue=np.random.rand(m,k)
A=np.random.rand(n,m)
b0=A@xTrue
noise=np.random.randn(n,k)
b=b0+noise
xEst=cvx.Variable((m,k))
gamma=cvx.Parameter(nonneg=True)
err=cvx.sum_squares(A*xEst-b)
penalty1=cvx.norm(xEst,1)
obj=cvx.Minimize(err+penalty1)
prob=cvx.Problem(obj)
TRACEBACK for l1-regularized penalty
resCVX2=prob.solve(solver='CVXOPT',verbose=True)
--------------------------------  Output  --------
pcost       dcost       gap    pres   dres   k/t
0:  0.0000e+00 -1.0000e+00  6e+01  2e+00  1e+00  1e+00
1:  1.4024e+01  1.6917e+01  1e+02  2e+00  2e+00  5e+00
2:  6.3266e+00  9.0873e+00  3e+00  3e-01  2e-01  3e+00
3:  8.6975e+00  9.9456e+00  2e+00  1e-01  9e-02  1e+00
4:  9.6416e+00  9.6626e+00  6e-02  3e-03  2e-03  2e-02
5:  9.6542e+00  9.6546e+00  1e-03  7e-05  4e-05  5e-04
6:  9.6545e+00  9.6546e+00  2e-04  9e-06  5e-06  7e-05
7:  9.6546e+00  9.6546e+00  8e-06  4e-07  3e-07  2e-06
8:  9.6546e+00  9.6546e+00  3e-07  1e-08  8e-09  7e-08
Optimal solution found.
TypeError                                 Traceback (most recent call last)
 in ()
6 print(obj.value)
7 #  TRACEBACK for l1-regularized penalty
----> 8 resCVX2=prob.solve(solver='CVXOPT',verbose=True)
/usr/local/lib/python3.6/site-packages/cvxpy/problems/problem.py in solve(self, *args, **kwargs)
244         else:
245             solve_func = Problem._solve
--> 246         return solve_func(self, *args, **kwargs)
247
248     @classmethod
/usr/local/lib/python3.6/site-packages/cvxpy/problems/problem.py in _solve(self, solver, ignore_dcp, warm_start, verbose, parallel, **kwargs)
358         data, inverse_data = self._solving_chain.apply(self)
359         solution = self._solving_chain.solve_via_data(self, data, warm_start, verbose,
--> 360                                                       kwargs)
361         self.unpack_results(solution, self._solving_chain, inverse_data)
362         return self.value
/usr/local/lib/python3.6/site-packages/cvxpy/reductions/solvers/solving_chain.py in solve_via_data(self, problem, data, warm_start, verbose, solver_opts)
231         """
232         return self.solver.solve_via_data(data, warm_start, verbose,
--> 233                                           solver_opts, problem._solver_cache)
/usr/local/lib/python3.6/site-packages/cvxpy/reductions/solvers/conic_solvers/cvxopt_conif.py in solve_via_data(self, data, warm_start, verbose, solver_opts, solver_cache)
106             warm_start,
107             verbose,
--> 108             solver_opts)
/usr/local/lib/python3.6/site-packages/cvxpy/problems/solvers/cvxopt_intf.py in solve(self, objective, constraints, cached_data, warm_start, verbose, solver_opts)
192         # Restore original cvxopt solver options.
193         self._restore_solver_options(old_options)
--> 194         return self.format_results(results_dict, data, cached_data)
195
196     def cpl_solve(self, data, kktsolver):
/usr/local/lib/python3.6/site-packages/cvxpy/problems/solvers/cvxopt_intf.py in format_results(self, results_dict, data, cached_data)
415                     new_dual[:P_rows] = 0
416                 else:
--> 417                     new_dual[:P_rows] = data["P_leq"].T*z
418                 new_dual[P_rows:] = new_results[s.INEQ_DUAL][leq_len:]
419                 new_results[s.INEQ_DUAL] = new_dual
TypeError: incompatible dimensions

By the way, the same problem works with other solvers (ECOS, OSQP).
On dropping the l-1 penalty term in the objective, CVXOPT succeeds.
Hope this helps.

I think there are two bugs in cvxopt_intf.py.

On line 409, P_rows is set to the number of columns of a matrix data["P_leq"].
On line 417, there is a multiplication data["P_leq"].T * z. This disagrees with a comment on line 398, that says we need to multiply duals by P_leq (no transpose).

I have changed line 409 to correctly get the number of rows, and changing line 417 to remove the transpose. All tests pass (including an additional test I wrote to check dual variables for SOCPs and LPs), and @sunilshah 's example now runs without generating an error.
I will be including these changes in my next pull request.


Fix missing f-strings

Self explanatory. Search for the strings in element.py and check them.

Closed by #48


Add garbage collector aware recycler

Testing against this patch indicates that the pooling of byte arrays when using the modern, low-pause collectors is unnecessary, because there are no pauses for evacuation failures to cause an impact to application latency.
For a 2,500 second runtime, the baseline generational ZGC cluster:

Versus generational ZGC w/ patch:

Example of overhead:

Thanks for the PR.
One callout, this may result in a small bloat in how much extra heap a delta transition allocates- currently a delta transition can reallocate a long[] from the memory pool that is already being referenced by reader threads and thats ok because those reads get invalidated later https://github.com/Netflix/hollow/blob/1cdd463c93c085b03034f7f7c654b52ad6eb2c6e/hollow/src/main/java/com/netflix/hollow/core/read/engine/object/HollowObjectTypeReadStateShard.java#L361
Also thinking from a rollout perspective maybe good to gate this behind a feature flag.
Other than that, just noting that the status quo seems appealing from the standpoint that java GC has a smaller region to manage so maybe able to work more efficiently, making impact to read performance a little less likely. But I do see the efficiency argument with the patch that when data size shrinks hollow wont hold on to more heap than necessary. Would be good to get @dkoszewnik 's buy in.

result in a small bloat in how much extra heap a delta transition allocates

maybe able to work more efficiently, making impact to read performance a little less likely.

Neither will be a concern with the low pause collectors. I bet G1 was the reason this optimisation was required, because evacuation failures are so expensive. The heuristics that determine the young/old balance would have had a hard time accomodating the infrequent allocation spikes caused by delta updates. It no longer matters, and in fact, promotion failures are something than Generational Shenandoah expects, IIRC it can fall back to promoting a region in place, instead of copying.
These collectors have essentially constant pause times, in the case of ZGC well below 1ms, that don't scale with heap size or allocation rates. Any impact to application latency is in things like store/load barriers, and the impact of the collector running concurrently with the application, but you've already made that trade off by your selection of GC.

The use of the recycler predates G1, the rationale dates back to the Parallel collector.  If this complexity is no longer necessary, I'm all for simplifying.
A side note -- before G1 was GA (and slightly after, IIRC), we tested with candidate versions and occasionally ran into ArrayIndexOutOfBounds exceptions where they should not have been possible -- I am certain we were running into bugs in early versions of G1.  I hope these newer collectors don't have similar issues.  But I don't think the use of an array segment recycler would make us any more or less likely to encounter them.


DMSDK Batcher to take action when a job starts and when a job ends

As a developer
I want a TITLE
So that I can log data onto the filesystem OR back into MarkLogic upon these events occurring
Experimenting with creating custom listeners.  I created one listener that would write back documents to MarkLogic upon a batch success.  Developers would want notification when a job starts and ends in addition to the batch complete/failure events.

Related to #612

Looks like this is a dup of #614


Tablet recognized as Mobile?

Hello.
When I call isMobile() on an iPad it returns true. Should this be the case?
I am using isMobile() and isTablet() to find out if the user is visiting my site with a mobile (eg: iPhone), tablet (eg: iPad) or PC, so I can serve different content, based on each device.
Why isn't iPad recognized as Tablet?
Thank you,
Ilias

The Mobile Detect Library returns true for isMobile for both Mobile and Tablets. See first example example on the mobile detect page.


[1354] Download eligibility checks CSV

Context
We need to monitor the effect of some changes to the eligibility checker but don't have the data in an easily accessible place currently.
Changes proposed in this pull request

Add 'Download CSV' link to the eligibility checks index page

Stream the CSV as this is being generated from the transactional DB. As this data grows this page will start to timeout if we generate the whole CSV before responding.
Download link:

Example CSV format:

 Add CSV download link for eligibility checks.


⚠ Spryr has degraded performance

In a225619, Spryr (https://spryr.com) experienced degraded performance:

HTTP code: 200
Response time: 3350 ms

Resolved: Spryr performance has improved in 413b4ff after 10 minutes.


Fix formatting of Quaternion comment

;-;
really

Maybe consider adding formatting check in this repo and not making the Server repo check for formatting of its submodules

Maybe consider adding formatting check in this repo and not making the Server repo check for formatting of its submodules

poke butlah?


Contacting Nextdoor

@schowdhury8
(submitted a contact form and contacted the CEO thru LinkedIn)

ZenHub Information
This information is updated automatically. To modify it, please use ZenHub.

Never heard back... Ah well.


Compilation error with 4.05.0

I am trying to compile v1.2.0 or develop I get the following error when executing the ./build step
make[1]: Leaving directory '/home/guedemann/source/git/HLL/kind2/ocamlczmq/src'
~/source/git/HLL/kind2
make -C src
make[1]: Entering directory '/home/guedemann/source/git/HLL/kind2/src'
sed s/%%%GIT_DESCRIBE%%%/v1.2.0/ < version.ml.in > version.ml
ocamlbuild -j 0 -pkg yojson -cflags -w,@P@U@F kind2.native
Finished, 1 target (0 cached) in 00:00:00.
+ ocamlopt.opt -c -w @P@U@F -I /home/guedemann/.opam/default/lib/biniou -I /home/guedemann/.opam/default/lib/easy-format -I /home/guedemann/.opam/default/lib/yojson -annot -bin-annot -color always -I induction -I utils -I smt -I testgen -I terms -I certif -I lustre -I qe -I invgen -I c2i -I ic3 -I doc -o transSys.cmx transSys.ml
File "transSys.ml", line 1594, characters 44-46:
Warning 40: BV was selected from type TermLib.feature.
It is not visible in the current scope, and will not
be selected if the type becomes unknown.
+ ocamlc.opt -c -w @P@U@F -I /home/guedemann/.opam/default/lib/biniou -I /home/guedemann/.opam/default/lib/easy-format -I /home/guedemann/.opam/default/lib/yojson -annot -bin-annot -color always -I lustre -I induction -I utils -I smt -I testgen -I terms -I certif -I qe -I invgen -I c2i -I ic3 -I doc -o lustre/lustreParser.cmi lustre/lustreParser.mli
File "lustre/lustreParser.mli", line 129, characters 60-66:
Error: Unbound module A
Command exited with code 2.
Compilation unsuccessful after building 207 targets (0 cached) in 00:00:05.
Makefile:8: recipe for target 'kind2' failed
make[1]: *** [kind2] Error 10
make[1]: Leaving directory '/home/guedemann/source/git/HLL/kind2/src'
Makefile:39: recipe for target 'kind2' failed
make: *** [kind2] Error 2

What I am doing wrong here?
I am using ocaml  4.05.0 and menhir menhir, version 20200612

Thank you for reporting this issue. It seems that menhir 20200525 introduced some change that broke our build process. We will investigate how to fix it. In the meantime, a temporary workaround is to downgrade menhir to 20200211 by running opam install "menhir<=20200211".

Thanks for your quick answer, that helped and I managed to compile kind2


pandas 1.x?

Is support for pandas 1.x in the scope of zipline-trader? Is this up to  quantopian/zipline? (https://github.com/quantopian/zipline/issues/2243 https://github.com/quantopian/zipline/issues/2616)
I started to see what the upgrading looks like
https://github.com/WillForan/zipline-trader/commits/pandas_update
nosetests -e zipline.utils.numpy_utils
...
FAILED (SKIP=6, errors=333, failures=66)

eek

Hi, not currently no. once everything is stable we could start looking at upgrades.

For what it's worth, shortly before Quantopian announced their shutdown, one of zipline's devs had started a branch  with the obvious (well, at least to me, an aspiring Python noob...) intention of making zipline compatible with the latest pandas versions. I guess this will never get completed now, but hopefully it's a decent starting point.

Hi, @WillForan did you happen to make this transition?
I would like to understand if this is possible and what is the effort required to move to pandas 1.X

@WillForan i started working on this, and except for quarter-estimates all tests are passing. https://github.com/shlomikushchi/zipline-trader/pull/165

merged in 1.6.0


README: fix typos, improve wording

I also removed the chevrons in front of shell commands to allow for easier copy/pasting.

Thanks!