Clicking 'add metadata' when files are uploading breaks the metadata

Background
Link to issue
https://biomage.atlassian.net/secure/RapidBoard.jspa?rapidView=1&projectKey=BIOMAGE&modal=detail&selectedIssue=BIOMAGE-1048
Link to staging deployment URL
https://ui-stefan-ui341.scp-staging.biomage.net/
Links to any Pull Requests related to this
Anything else the reviewers should know about the changes here
Changes
Code changes
Definition of DONE
Your changes will be ready for merging after each of the steps below have been completed:
Testing

[ ] Unit tests written
[ ] Tested locally with Inframock (with latest production data downloaded with biomage experiment pull)
[ ] Deployed to staging

To set up easy local testing with inframock, follow the instructions here: https://github.com/biomage-ltd/inframock
To deploy to the staging environment, follow the instructions here: https://github.com/biomage-ltd/biomage-utils
Documentation updates
Is all relevant documentation updated to reflect the proposed changes in this PR?

[ ] Relevant Github READMEs updated
[ ] Relevant wiki pages created/updated

Approvers

[ ] Approved by a member of the core engineering team
[ ] (UX changes) Approved by vickymorrison (this is her username, tag her if you need approval)

Just before merging:

[ ] After the PR is approved, the unstage script in here: https://github.com/biomage-ltd/biomage-utils is executed. This script cleans up your deployment to staging

Optional

[ ] Photo of a cute animal attached to this PR

I can't test if it's fixed in the staging link because this error happens when trying to upload files to a new project. Is this expected?

You can ignore my comment above ^^ - it's an issue with using Firefox (I'll make a ticket for it). Your fix works in Chrome.


UnicodeDecodeError for LotV beta replay

From this replay of Puck in a PvZ for the new LotV beta
http://lotv.spawningtool.com/4/download/
Python 3.4.0 (default, Apr 11 2014, 13:05:18)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sc2reader
>>> replay = sc2reader.load_replay('replays/lotv.SC2Replay')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kevin/sc2reader/sc2reader/factories/sc2factory.py", line 85, in load_replay
    return self.load(Replay, source, options, **new_options)
  File "/home/kevin/sc2reader/sc2reader/factories/sc2factory.py", line 137, in load
    return self._load(cls, resource, filename=filename, options=options)
  File "/home/kevin/sc2reader/sc2reader/factories/sc2factory.py", line 146, in _load
    obj = cls(resource, filename=filename, factory=self, **options)
  File "/home/kevin/sc2reader/sc2reader/resources.py", line 271, in __init__
    self._read_data(data_file, self._get_reader(data_file))
  File "/home/kevin/sc2reader/sc2reader/resources.py", line 601, in _read_data
    self.raw_data[data_file] = reader(data, self)
  File "/home/kevin/sc2reader/sc2reader/readers.py", line 33, in __call__
    ) for i in range(data.read_bits(5))],
  File "/home/kevin/sc2reader/sc2reader/readers.py", line 33, in <listcomp>
    ) for i in range(data.read_bits(5))],
  File "/home/kevin/sc2reader/sc2reader/decoders.py", line 252, in read_aligned_string
    return self._buffer.read_string(count, encoding)
  File "/home/kevin/sc2reader/sc2reader/decoders.py", line 108, in read_string
    return self.read_bytes(count).decode(encoding)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 33: invalid start byte

Currently working on fixing it, and I'll try to post a fix if I can figure things out (probably not). Just thought I would share so we could work on these things together

Alright, I am of the impression that we just need blizzard to tell us what has changed in the format to be able to fix this. Or maybe we just need someone smarter than me looking at this.

I mean, it would certainly be helpful if they just told us what changed. They have no record of providing information on non-official builds though so I wouldn't hold your breath. I'll try to take a look at it later this week. If things haven't changed too dramatically I can probably spot the difference.

@koalaling just posted a new s2protocol for LotV: https://github.com/Blizzard/s2protocol/commit/2c1d7fff4fadee17314a8960f21321ff6e654267

What a champ! I'll take a stab at it, though I'm not actually good at this stuff, so if you don't hear more from me, it is because I failed

I've started doing work for this, you can follow my progress in the LotV branch. It isn't quite able to parse the replays yet and its not clear how many non-parsing related changes will be required to make for a good read.
You should checkout the changes in the readers file. Pretty interesting I'd say. Maybe this will also parse Heroes of the Storm replays?

I guess hero_talent_tree_selection_panel_toggled_event is pretty clear to me!

Wow, quick work! I'm playing around with it a bit to see what i can find.
One more thing I'll note for your TODO list is to set the expansion https://github.com/GraylinKim/sc2reader/blob/722ae4f16430c023bafa32fa69a1773909b1f867/sc2reader/resources.py#L344
Thanks for your continued effort to maintain this library!

So this probably won't make you feel really comfortable, but I'm using this branch in production right now since I want to run more replays through it. So far, so good! I'll report any issues I come up against as they pop up.
For others who are building on top of sc2reader, you can check out my branch of spawningtool where I'm going through the types that blizzard is using
https://github.com/StoicLoofah/spawningtool/tree/32-lotv

Obviously a lower priority, but I got an archon mode replay that doesn't appear to work so well

I haven't dug much into it, but I got an archon mode replay that isn't quite working correctly
http://lotv.spawningtool.com/81/download/
As a sample of what this looks like:
>>> replay.players
[Player 1 - Amshel (Zerg), Player 2 - DrennoC (Zerg), Player 3 - Filleriste (Terran), Player 4 - Sudhish (Terran)]
>>> replay.players[0].units[-10:]
[Zergling [A240005], Zergling [9540004], Zergling [C840004], Zergling [1BC0007], Zergling [A440004], Zergling [C380004], Zergling [C100007], Zergling [C040002], Zergling [BA00005], Larva [4840002]]
>>> replay.players[1].units[-10:]
[Marine [3980003], Marine [6040003], Marauder [A4C0004], Marine [D00003], Marine [AE00004], Marine [A240004], Marauder [A540009], Marine [AA40003], Marine [AAC0002], Marine [AE80006]]

So the 2 zerg players are on the first team, but the units are associated with those 2 players rather than with the players on the 2 teams respectively.
This possibly should be addressed separately, and I haven't looked deeply into why this is happening, though my gut says that blizzard got this working through some creative indexing within the lobby that is throwing us off. I am as useless as ever in actually understanding how to fix these things but am happy to dig more or provide better reporting as desired

It may seems obvious but what is the size of the players list? If it is 4 it's awkward but if it's 2, the two players of each team may have been merged. In the former case, is players[0].units equals to players[2].units? I cannot test it myself ATM

Another fun fact that isn't too systemic in sc2reader but is probably a big assumption for others is that there are 16 frames per second. One change was that Blizzard slowed down the game clock so that it matches real-time. I verified the differences by comparing unit build times between
https://docs.google.com/spreadsheets/d/1JtL6Wd9q5Qxm3KEewnOXSz1KxXp6d6OF6KmrgO183Lw/edit#gid=410665689
https://github.com/StoicLoofah/spawningtool/blob/43c2ca0574c9dac3ce365853419a634962afa2ff/spawningtool/constants.py
Doing the math, it seems like they sped it up by roughly 1.38, which naively is around 22 frames per second. Of course, I have no idea whether they actually implemented it that way since I haven't easily been able to find a replay and matching VOD (and don't have beta access to test it), but that may be something we have to integrate into our thinking as well.

@EHadoux currently the size of players is 4

I'm pretty sure your maths are right as 1.38 was the speed factor between real time and fastest speed (is that the right name? I have the game in French) AFAIK. I watched some streams and it seems like the clock went slower.

Okay @StoicLoofah I pushed a couple small changes. A fixed an attribute mapping that was causing parse failure and fixed the replay.expansion attribute. Please forward all errors that pop into your logs to my email so I can diagnose.
If you guys can fill-in the correct values for this table we can fix the game timings for LotV as well:
GAME_SPEED_FACTOR = {
    'Slower':   0.6,
    'Slow':     0.8,
    'Normal':   1.0,
    'Fast':     1.2,
    'Faster':   1.4
}

It seems that I have a choice between looking into archon mode, adding data for the new units/abilities, and wrapping data for the new events. Anyone have a preference?

If anyone wants to help out right now they can:

Send me replays
Do some research on the new initData flags (fb285c6). Which game settings on bnet flip which flags, if any?

I am guessing that the tandem_leader_user_id is related to archon mode. I am hoping the other flags are used so that we can better differentiate game modes in replays now.

Here is the table for game speed: http://wiki.teamliquid.net/starcraft2/Game_Speed
However, it reads it can sometimes be a tiny bit different.

Speed
Time Factor

Slower
0.599

Slow
0.83

Normal
1

Fast
1.21

Faster
1.38

@EHadoux 's numbers for game speed look right. I don't really have a good sense for what Blizzard did to get the time adjusted. It would seems strange to me that there should be a non-integer number of frames per second, and it could be a pain to deal with, but I don't have a good way to verify that externally.
@GraylinKim I have not yet fully tested out this branch, but at least the last batch of commits appeared to have been stable in production, so no errors there, yet!
For how you spend your time, archon mode is probably the most applicable for me, but I can see the value of looking into the new events as well. For the data for new units/abilities, that might be too much in flux to spend too much time there. But of course, do what works best for you.
Looking at the new initData flags, I actually think those might be for Heroes of the Storm. "practice" and "cooperative" are definitely in Heroes, and the other 3 could map to other modes (amm = quick match, ranked = hero league, competitive = team league?). I couldn't say, but since all LotV games are unranked or custom, I doubt we will see variance there.
Most of the replays I have are probably uninteresting (since they all work), but if you like, I can send you a zip or list of URLs or whatever else works for you. Let me know.
Again, I really appreciate that you continue to maintain this package!

Those ratios seem to be empirical from the MULE. After reading the liquipedia page and this (http://us.battle.net/sc2/en/forum/topic/628076627) one, the theoretical ratios are more likely to be 0.6x, 0.8x, 1.0x, 1.2x and 1.4x.
With those figures, seconds are round for every speed.

Just ran through the latest version of the code, and I ran into some issues with TrackerEvents not having units associated with them. For example, there were dicts that looked like this
{'upkeep_pid': 1, 'unit_id_recycle': 1, 'unit_upkeeper': Player 1 - Pseudorandom (Zerg), 'unit_type_
name': 'Drone', 'location': (158, 30), 'control_pid': 1, 'second': 19, 'unit_id_index': 241, 'unit_controller': Player 1 - Pseudorandom (Zerg), 'x': 158, 'name': 'UnitBornEvent', 'unit_id': 63176705, 'unit': None, 'frame': 305, 'y': 30}

that presumably should have had a unit, but did not. This particular one manifested in http://lotv.spawningtool.com/5/download/ and I saw it on UnitBornEvent, UnitInitEvent, and UnitTypeChangeEvent.
Also relevant is that this was working as of 5da22b473b8a77390d825a75e9963cd2342cce26 . I didn't see anything in the changes since then that would indicate what the problem was. My only guess is that if you tie the types to the expansions, then maybe the fix is to generate that data?
Now that I read my comment, I'm getting the suspicion that this is exactly what was on your TODO list and that you know all of this. I completely accept my ignorance should this be the case

@GraylinKim : I'm going to continue to be "that guy". I am running into a regression on this branch compared to master: as of a few days ago, it looks like HotS replays aren't parsing correctly on the lotv branch. For example,
http://spawningtool.com/22312/download/
>>> import sc2reader
>>> sc2reader.load_replay('replays/fightnight.SC2Replay')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kevin/sc2reader/sc2reader/factories/sc2factory.py", line 85, in load_replay
    return self.load(Replay, source, options, **new_options)
  File "/home/kevin/sc2reader/sc2reader/factories/sc2factory.py", line 137, in load
    return self._load(cls, resource, filename=filename, options=options)
  File "/home/kevin/sc2reader/sc2reader/factories/sc2factory.py", line 146, in _load
    obj = cls(resource, filename=filename, factory=self, **options)
  File "/home/kevin/sc2reader/sc2reader/resources.py", line 271, in __init__
    self._read_data(data_file, self._get_reader(data_file))
  File "/home/kevin/sc2reader/sc2reader/resources.py", line 602, in _read_data
    self.raw_data[data_file] = reader(data, self)
  File "/home/kevin/sc2reader/sc2reader/readers.py", line 174, in __call__
    ) for p in details[0]],
KeyError: 10

I tried a few replays from the past few days, nad they're exhibiting the same behavior but are otherwise working fine on master.
This is present on the HEAD of lotv, but as a heads up, I have been using 5da22b4 in production because of the issues with unit types as noted above.
Also let me know if you want to talk through this offline. I wish I could be more helpful and independent here in addressing these, but the fact is that I'm just not knowledgeable enough to do that.

Here's was the patch I have for it locally. It seems like it works, but I question everything
diff --git a/sc2reader/readers.py b/sc2reader/readers.py
index b922dd6..ac99933 100644
--- a/sc2reader/readers.py
+++ b/sc2reader/readers.py
@@ -170,7 +170,7 @@ class DetailsReader(object):
                 observe=p[7],
                 result=p[8],
                 working_set_slot=p[9] if replay.build >= 24764 else None,
-                hero=p[10] if replay.build >= 34784 else None,
+                hero=p[10] if replay.build >= 34784 and 10 in p else None,
             ) for p in details[0]],
             map_name=details[1].decode('utf8'),
             difficulty=details[2],

That patch looks fine to me, I'll apply it when I get home tonight. I'm sure the issue is that "hero" isn't written out unless you are playing Heroes right now. Interesting to see a divergence in what is written based on the game though.
re: The unit being missing you are exactly right. All unit types and therefore units are tied to a datapack. When I fixed the expansion to read LotV it could no longer find a datapack and no longer resolve units. In the past I generated the datapack automatically by scrapping the RAM of the running game. Since I don't have beta access I can't do that now. It would be possible to hand-build that datapack if you wanted to take on that task, it is just a simple CSV format mapping in-game codes to names.

To restore the previous behavior I could just copy the HotS datapacks into LotV and write the glue code. There are certainly problems with that approach but maybe it is better than nothing for now.

@GraylinKim That would be a good solution for now. If you're busy, I'm happy to take a shot at it myself, though I may not get around to it this weekend. I at least have taken a look at the types in the process of putting together
https://github.com/StoicLoofah/spawningtool/blob/32-lotv/spawningtool/lotv_constants.py
You will see a PR from me if I manage to get something working

Anyone have any thoughts or any success in handling archon mode? I see they added that new archon_leader_id, but I'm not sure how that would get integrated into the system. Again, I would tentatively be interested in helping out, but I have no idea where to start

I'd have to look into things to see how it was laid out, but my best guess is as follows:

Teams are normal, only 1 person is flagged as the leader indicating archon mode active
The leaders own all the units, their team mates own no units.
Players can now bulk select, hotkey, and command their leader's units.
Non-leader players get no credit for kills, resources, etc

If I am right, it is just a matter of pulling all the information for the teams from the archon_leader_id players. Not much else should need to change for most use cases.
I haven't looked into it myself yet though so this is all guess work. If you could poke around the events for an archon game a little bit and confirm or refute that would be a good start.

Sorry for being slow to respond here. I took a look at another archon mode replay, and the situation is roughly the same as discussed in https://github.com/GraylinKim/sc2reader/issues/185#issuecomment-89498669
The biggest issue is that the units are associated with the wrong users: they're associated with sid 0 and 1, but they should be associated with sid 0 and 2. It looks like the teams otherwise are properly configured.
I wonder if it might be related to https://github.com/GraylinKim/sc2reader/issues/176 where we're not mapping lobby slots into in-game ids correctly.
Do you have any suggestions for where I might start digging into this?

There's a new protocol out https://github.com/Blizzard/s2protocol/commit/2827bad0986c791201423603dac61ea29a3c3886
And here's an example of a replay http://lotv.spawningtool.com/903/download/
@GraylinKim I would be willing to help out for these things when they come up, but i literally have no idea what I'm looking at. If it's easy for you to do and you don't mind, that's cool. If you want to offload it, let me know

Anyone have any  other thoughts on how we can start tackling archon mode? I have been getting more requests to support it, and I'm still a little lost on where to start with it

This is out of my knowledge. However, I can tell that, even on Blizzard's side, it's not really clear.
I mean, in game I always have toggled the button to be green and all the enemies in red (the second or third one on the right of the minimap). However, when doing that, my enemies are red but so do I. Only my friend is in the right color. It is almost (?) always the same problem for the colors in the graph screen after each game.
That is to say that there are still some bugs so I don't know if it really the worth the time as everything may change anytime.


xcodegen command gives spec validations error

im getting this error when i run xcodegen. Can someone please help?

Can you post the list of your configs in the project.yml?

And what version of XcodeGen is this on?
xcodegen version

configs:
Debug (Development): debug
Debug (Staging): debug
Debug (Production): debug
Release (Development): release
Release (Staging): release
Release (Production): release
configFiles:
Debug (Development): Config/Development.xcconfig
Debug (Staging): Config/Staging.xcconfig
Debug (Production): Config/Production.xcconfig
Release (Development): Config/Development.xcconfig
Release (Staging): Config/Staging.xcconfig
Release (Production): Config/Production.xcconfig
$ xcodegen version
Version: 2.22.0
Thank you yonas

I am guessing this is causing the problem?

@DavidWoohyunLee I had the same problem when using "Devel Archive" and "Devel Run" in my configuration names.
It appears that the definition of "Config Variant" assumes that you always include the words "Debug" and "Release" in your config name, right after the "variant" name and not before.
In my case I had to switch "Devel Run" to "Devel Debug" and "Devel Archive" to "Devel Release" and everything started to work again.
From documentation:

This generates a scheme for each entry, using configs that contain the name with debug and release variants. This is useful for having different environment schemes.

I guess it's assumed that "debug" and "release" should be placed at the end of config name, not at start.

@yonaskolb Could you tell me when you will be releasing a new version?

@DavidWoohyunLee this is now released

In my case I was using config variants like following and it was working; now I removed parenthesis to fix the issue.
configVariants:
- (Development)
- (Staging)
- (Production)


Update README.md from branch example

updates
I've updated README file. please review and accept

Thoughts
I probably shouldn't. But okay :+1:


"Hacktoberfest-accepted" Handwritten font

A regular handwritten  font.This will ease in copying assignments from the online and to print that as if that is handwritten.

@vicky-co This doesn't fall under any task category. Closing this PR


Version 2.5.5

Please, merge this first.

This review/aproval system with 2 developers makes no sense.

:+1:

I guess it's so that at least someone reviewed it, but for a project such as this that is in maintenance mode, I would go without required approvals but I don't have permissions for it.


Error: undefined local variable or method `formulae_to_install' for Homebrew:Module

[x] ran brew update and can still reproduce the problem?
[x] ran brew doctor, fixed all issues and can still reproduce the problem?
[x] ran brew config and brew doctor and included their output with your issue?

What you were trying to do (and why)
Trying to update some outdated formulas. Just business as usual system maintenance.
What happened (include command output)
As I normally do, I ran brew update followed by brew outdated to see what was available for an update. Then I ran brew upgrade elixir go imagemagick node. The upgrade process started as normal and then terminated with an error (exit code != 0). It seemed a bug in HomeBrew's code.

Command output

$ brew update
Updated Homebrew from 505c1e724 to ed8c2616c.
Updated 2 taps (homebrew/core and homebrew/cask).
==> Updated Formulae
elixir ✔                    eslint                      jhipster                    n                           scons
imagemagick ✔               evince                      juju                        neatvi                      simple-scan
python ✔                    falcon                      kitchen-sync                node-build                  suite-sparse
aliyun-cli                  fastlane                    lc0                         odpi                        tarantool
arduino-cli                 fdupes                      libgusb                     opencoarrays                tnftpd
atomicparsley               flowgrind                   libmpdclient                osm2pgsql                   ungit
balena-cli                  glooctl                     librsvg                     ott                         virustotal-cli
bettercap                   gor                         libuv                       pc6001vx                    vulkan-headers
cdargs                      gowsdl                      libvorbis                   pnpm                        xclogparser
circleci                    html2text                   liquibase                   publish                     xmlrpc-c
deno                        imagemagick@6               mafft                       pwntools                    zoxide
diffoscope                  include-what-you-use        mikutter                    remake
dvc                         ios-webkit-debug-proxy      mill                        rtorrent
dxpy                        ipython                     mksh                        scipy
==> Updated Casks
baidunetdisk           eddie                  keepingyouawake        netron                 resolume-arena         superproductivity
bitwig-studio          electerm               kui                    onedrive               riot                   thedesk
chromium               electrum               lifesize               osxfuse                screen                 tiger-trade
chrysalis              endurance              lightproxy             qownnotes              sejda-pdf              timing
codeexpander           exodus                 lmms                   quicken                sensei                 wickrme
dashlane               fedora-media-writer    loaf                   raven-reader           sqlpro-for-mssql
dbschema               gloomhaven-helper      loom                   refined-github-safari  sqlpro-studio
detectx-swift          hex-fiend              meteorologist          rekordbox              stats
$ brew outdated
.......
$ brew upgrade elixir go imagemagick node
==> Upgrading 4 outdated packages:
elixir 1.10.3 -> 1.10.4
go 1.14.2_1 -> 1.14.4
imagemagick 7.0.10-22 -> 7.0.10-23
node 14.2.0 -> 14.5.0
==> Upgrading elixir 1.10.3 -> 1.10.4
==> Downloading https://homebrew.bintray.com/bottles/elixir-1.10.4.catalina.bottle.tar.gz
==> Downloading from https://akamai.bintray.com/ed/ed5f1be4059b41113c35f0a3ae1f01ae5042b143869d22457428507c4a976812?gda=exp=1593960269~hm
######################################################################## 100.0%
==> Pouring elixir-1.10.4.catalina.bottle.tar.gz
🍺  /usr/local/Cellar/elixir/1.10.4: 430 files, 5.9MB
Removing: /usr/local/Cellar/elixir/1.10.3... (430 files, 5.9MB)
Removing: /Users/Tom/Library/Caches/Homebrew/elixir--1.10.3.catalina.bottle.tar.gz... (4.2MB)
==> Upgrading go 1.14.2_1 -> 1.14.4
==> Downloading https://homebrew.bintray.com/bottles/go-1.14.4.catalina.bottle.tar.gz
==> Downloading from https://akamai.bintray.com/93/935214ab166d8da148af812614545a25778e4ee11c2ea5cfd332bff72ec4046e?gda=exp=1593960272~hm
######################################################################## 100.0%
==> Pouring go-1.14.4.catalina.bottle.tar.gz
🍺  /usr/local/Cellar/go/1.14.4: 9,443 files, 424.7MB
Removing: /usr/local/Cellar/go/1.14.2_1... (9,440 files, 424.4MB)
==> Upgrading imagemagick 7.0.10-22 -> 7.0.10-23
==> Downloading https://homebrew.bintray.com/bottles/sqlite-3.32.3.catalina.bottle.tar.gz
==> Downloading from https://akamai.bintray.com/98/98f798c4a62c9db46cc6ac82c499a0a1b8016dfebdfaa54eb4bf5c95a90117b0?gda=exp=1593960307~hm
######################################################################## 100.0%
==> Downloading https://homebrew.bintray.com/bottles/imagemagick-7.0.10-23.catalina.bottle.tar.gz
==> Downloading from https://akamai.bintray.com/bf/bf19ce6c826be9ab5a42d4e02b74d2f6cd8f5d9402276dae5428ac3c1dc08c6d?gda=exp=1593960308~hm
######################################################################## 100.0%
==> Installing dependencies for imagemagick: sqlite
==> Installing imagemagick dependency: sqlite
==> Pouring sqlite-3.32.3.catalina.bottle.tar.gz
==> Caveats
sqlite is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.
If you need to have sqlite first in your PATH run:
echo 'export PATH="/usr/local/opt/sqlite/bin:$PATH"' >> ~/.profile
For compilers to find sqlite you may need to set:
export LDFLAGS="-L/usr/local/opt/sqlite/lib"
export CPPFLAGS="-I/usr/local/opt/sqlite/include"
For pkg-config to find sqlite you may need to set:
export PKG_CONFIG_PATH="/usr/local/opt/sqlite/lib/pkgconfig"
==> Summary
🍺  /usr/local/Cellar/sqlite/3.32.3: 11 files, 4MB
==> Installing imagemagick
==> Pouring imagemagick-7.0.10-23.catalina.bottle.tar.gz
🍺  /usr/local/Cellar/imagemagick/7.0.10-23: 1,497 files, 25.6MB
Removing: /usr/local/Cellar/imagemagick/7.0.10-22... (1,497 files, 25.6MB)
Removing: /Users/Tom/Library/Caches/Homebrew/imagemagick--7.0.10-22.catalina.bottle.tar.gz... (9.9MB)
==> Upgrading node 14.2.0 -> 14.5.0
==> Downloading https://homebrew.bintray.com/bottles/icu4c-67.1.catalina.bottle.tar.gz
==> Downloading from https://akamai.bintray.com/2d/2d1e91b5127f66e7941790c004817c94c892725c88f84f1e4c37297fcbc0c72f?gda=exp=1593960317~hm
######################################################################## 100.0%
==> Downloading https://homebrew.bintray.com/bottles/node-14.5.0.catalina.bottle.tar.gz
==> Downloading from https://akamai.bintray.com/91/91096144949902e76d46a3c0cfa26f5f55665da838f77ac96c58c416940d28d0?gda=exp=1593960319~hm
######################################################################## 100.0%
==> Installing dependencies for node: icu4c
==> Installing node dependency: icu4c
==> Pouring icu4c-67.1.catalina.bottle.tar.gz
==> Caveats
icu4c is keg-only, which means it was not symlinked into /usr/local,
because macOS provides libicucore.dylib (but nothing else).
If you need to have icu4c first in your PATH run:
echo 'export PATH="/usr/local/opt/icu4c/bin:$PATH"' >> ~/.profile
echo 'export PATH="/usr/local/opt/icu4c/sbin:$PATH"' >> ~/.profile
For compilers to find icu4c you may need to set:
export LDFLAGS="-L/usr/local/opt/icu4c/lib"
export CPPFLAGS="-I/usr/local/opt/icu4c/include"
For pkg-config to find icu4c you may need to set:
export PKG_CONFIG_PATH="/usr/local/opt/icu4c/lib/pkgconfig"
==> Summary
🍺  /usr/local/Cellar/icu4c/67.1: 258 files, 71.2MB
==> Installing node
==> Pouring node-14.5.0.catalina.bottle.tar.gz
==> Caveats
Bash completion has been installed to:
/usr/local/etc/bash_completion.d
==> Summary
🍺  /usr/local/Cellar/node/14.5.0: 4,659 files, 61.0MB
Removing: /usr/local/Cellar/node/14.2.0... (4,659 files, 60.8MB)
==> Upgrading 1 dependent:
python 3.7.7 -> 3.7.8
==> Upgrading python 3.7.7 -> 3.7.8
==> Downloading https://homebrew.bintray.com/bottles/python-3.7.8.catalina.bottle.tar.gz
==> Downloading from https://d29vzk4ow07wi7.cloudfront.net/f7fb660ba03f202d739f38ded3ac5952569103855d8a8e9aa1a7bfff764bdfc2?response-content-
######################################################################## 100.0%
==> Pouring python-3.7.8.catalina.bottle.tar.gz
==> /usr/local/Cellar/python/3.7.8/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python
==> /usr/local/Cellar/python/3.7.8/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python
==> /usr/local/Cellar/python/3.7.8/bin/python3 -s setup.py --no-user-cfg install --force --verbose --install-scripts=/usr/local/Cellar/python
==> Caveats
Python has been installed as
/usr/local/bin/python3
Unversioned symlinks python, python-config, pip etc. pointing to
python3, python3-config, pip3 etc., respectively, have been installed into
/usr/local/opt/python/libexec/bin
You can install Python packages with
pip3 install
They will install into the site-package directory
/usr/local/lib/python3.7/site-packages
See: https://docs.brew.sh/Homebrew-and-Python
==> Summary
🍺  /usr/local/Cellar/python/3.7.8: 4,007 files, 61.4MB
Removing: /usr/local/Cellar/python/3.7.7... (4,006 files, 61.0MB)
Removing: /Users/Tom/Library/Caches/Homebrew/python--3.7.7.mojave.bottle.tar.gz... (14.9MB)
==> Checking for dependents of upgraded formulae...
Error: undefined local variable or method formulae_to_install' for Homebrew:Module Please report this issue:   https://docs.brew.sh/Troubleshooting /usr/local/Homebrew/Library/Homebrew/upgrade.rb:164:in block in check_installed_dependents'
/usr/local/Homebrew/Library/Homebrew/cache_store.rb:17:in use' /usr/local/Homebrew/Library/Homebrew/upgrade.rb:163:in check_installed_dependents'
/usr/local/Homebrew/Library/Homebrew/cmd/upgrade.rb:114:in upgrade' /usr/local/Homebrew/Library/Homebrew/brew.rb:111:in '

What you expected to happen
I expected the brew upgrade command to terminate successfully, as it usually does.
Step-by-step reproduction instructions (by running brew commands)
I've tried to reproduce the issue. I haven't been able to, but I think it's because it only happens when upgrading some specific formulas. Looking at the error message, it seems to originate from a bad reference introduced in this commit: https://github.com/Homebrew/brew/commit/2c133a3d45366a503c99a02e7e255e4387e8bff7#diff-2cfb89cd16b5d61e1c8b256f9b28c713R164
Output of brew config and brew doctor commands

$ brew config

HOMEBREW_VERSION: 2.4.3-53-ged8c261
ORIGIN: https://github.com/Homebrew/brew.git
HEAD: ed8c2616ca9305a8fc120103a129afdd5e659760
Last commit: 8 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: bc53daee964234f1d06e147668825c0debd68c77
Core tap last commit: 45 minutes ago
HOMEBREW_PREFIX: /usr/local
HOMEBREW_DISPLAY: /private/tmp/com.apple.launchd.sPUGWzUy7Y/org.macosforge.xquartz:0
HOMEBREW_MAKE_JOBS: 8
CPU: octa-core 64-bit ivybridge
Homebrew Ruby: 2.6.3 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
Clang: 11.0 build 1103
Git: 2.27.0 => /usr/local/bin/git
Curl: 7.64.1 => /usr/bin/curl
macOS: 10.15.5-x86_64
CLT: 1<IP_ADDRESS>
Xcode: N/A
XQuartz: 2.7.11 => /opt/X11

brew doctor is fine.

Fixed in #7903.


Missing facebookcatalog tag

According to this page: https://developers.facebook.com/docs/sharing/webmasters/crawler/
facebook also has the facebookcatalog tag.

What exactly is the issue here, this user agent is identified as facebookcatalog and bot which is correct?
It doesn't have its own constant in package, but only most common user agents does. I haven't seen facebookcatalog in the wild, I don't know what it is used for. They use facebookexternalhit for fetching pages.

I don't actually know if facebookcatalog is used, but I noticed it in their doc and saw that you had no constant for it. Feel free to close the issue.

It is identified correctly, and constants are assigned only to the most common cases for easier use.
I will close the issue.


Add documentation and example app

Since colleagues around government have expressed an interest in using this, I thought I should add some docs.

Looks good!


Arcade stick unresponsive on PC and Switch with GP2040 - any advice?

Hi there,
I bought a HRAP3 on eBay with the Pi Pico board and GP2040 pre-installed, and it is currently unresponsive on PC, MiSTer and Switch, save for the L3 and R3 buttons, which light up when I test the stick under 'game controllers' on PC. The HRAP shows up as an Xbox 360 controller on PC, a console the firmware doesn't support as far as I'm aware, so I'm wondering if this is an issue with the xinput/dinput/console settings - could you please let me know how to switch between these without opening up the stick, if possible? I've tried uninstalling the device on my PC and seeing if I can get it to register as a generic controller instead, but this didn't help. Is there anything else I can try? (I'm completely unfamiliar with GP2040 itself, but would be up for reinstalling it and doing some tinkering if necessary - please point me in the right direction!)
Thank you very much for your help!
Mike

Try this:
https://gp2040.info/#/usage?id=input-modes

Thanks for your reply! Sadly, I tried this with every button on the stick and on every compatible machine I have (since the Brook UFB changes modes the same way), but I was still seeing no response from anything but the L3 and R3 buttons.
I didn't have long to test it, but the stick seemed to be working fine when I first opened it, so I'm a bit stumped...

So if it shows as a 360 controller, that’s XInput mode. It’s generally the best for use on PC (in my opinion). If the other buttons aren’t working, my first guess would be that they’re disconnected. It’s possible they were loose and something popped off. If you lost a ground connector, that could cause multiple buttons to stop responding at once.

Thank you! Yeah, I just need to get in there and have a look around. I was hoping there might be other avenues to explore, as the old HRAP bodies are a nightmare to open and put back together, but this one could definitely have been thrown around a lot in the post!

No worries! Glad you figured it out.


feat: Provision examples with .env file for OPENAIKEY

Provision examples from npx create-fabrice-ai with an .env file to hint users where they should add their OPENAI_API_KEY. Ideally this env file could come with entries for different providers as well (assuming they also work out of the box).

Fixed with: https://github.com/callstackincubator/fabrice-ai/pull/115/files


Fixed switch custom color with disabled.

Please makes sure these boxes are checked before submitting your PR, thank you!

[x] Make sure you follow Element's Contributing Guide.
[x] Make sure you are merging your commits to dev branch.
[x] Rebase before creating a PR to keep commit history clear.
[x] Add some descriptions and refer relative issues for you PR.

switch 组件 disabled 状况下自定义颜色不正常，如此例中：http://jsfiddle.net/j95axgwh/

mounted 时 disabled 的组件取消禁用时自定义颜色不生效
一旦触发自定义颜色生效则 disabled 样式的颜色不生效

gogu 又开始进这个坑了 😉


feat: add proportion exceeding wrapper for flip flop index

Adding proportion exceeding for future papers and notebook demos.

@tennlee  and @nicholasloveday this PR is ready for review. On to #106...

@nicholasloveday take another look, I've addressed your comments, mainly around the preserve and reduce dim parameters you requested.


feat: DEV-2885: Add new dm columns

Hi @Travis1282!
This PR was created in response to a PRs in upstream repo:

https://github.com/heartexlabs/dm2/pull/83

already merged


missing sqlite schema function

github.com/facebookincubator/ent@v0.1.1/dialect/sql/schema/sqlite.go:152
func (d *SQLite) table(context.Context, dialect.Tx, string) (*Table, error) { return nil, nil }

If the table already exists the situation will cause the program to panic.

Thanks for reporting this issue.
Closing since it’s a duplicate of #200, and there’s an ongoing PR for adding this support.


Remove org.http4s.batteries

Fixes #1147

That is fine by me. I didn't know it was a recommendation, so I just went with the minimal imports.


[Device Support Request] Avatto TRV16 Smart Radiator Thermostat

Problem description
I just bought an Avatto TRV16 thermostat. I was able to connect it to ZHA, but it does not show any controls.
Solution description
I would like to see this device being supported by zigpy/ZHA with reading and setting heating parameters.
Screenshots/Video
Screenshots/Video
[Paste/upload your media here]

Device signature
Device signature
{
  "node_descriptor": {
    "logical_type": 2,
    "complex_descriptor_available": 0,
    "user_descriptor_available": 0,
    "reserved": 0,
    "aps_flags": 0,
    "frequency_band": 8,
    "mac_capability_flags": 128,
    "manufacturer_code": 4417,
    "maximum_buffer_size": 66,
    "maximum_incoming_transfer_size": 66,
    "server_mask": 10752,
    "maximum_outgoing_transfer_size": 66,
    "descriptor_capability_field": 0
  },
  "endpoints": {
    "1": {
      "profile_id": "0x0104",
      "device_type": "0x0051",
      "input_clusters": [
        "0x0000",
        "0x0004",
        "0x0005",
        "0xef00"
      ],
      "output_clusters": [
        "0x000a",
        "0x0019"
      ]
    }
  },
  "manufacturer": "_TZE200_rxntag7i",
  "model": "TS0601",
  "class": "zigpy.device.Device"
}

Diagnostic information
Diagnostic information
 "data": {
    "ieee": "**REDACTED**",
    "nwk": 15646,
    "manufacturer": "_TZE200_rxntag7i",
    "model": "TS0601",
    "name": "_TZE200_rxntag7i TS0601",
    "quirk_applied": false,
    "quirk_class": "zigpy.device.Device",
    "quirk_id": null,
    "manufacturer_code": 4417,
    "power_source": "Battery or Unknown",
    "lqi": null,
    "rssi": null,
    "last_seen": "2024-09-25T07:37:07",
    "available": true,
    "device_type": "EndDevice",
    "signature": {
      "node_descriptor": {
        "logical_type": 2,
        "complex_descriptor_available": 0,
        "user_descriptor_available": 0,
        "reserved": 0,
        "aps_flags": 0,
        "frequency_band": 8,
        "mac_capability_flags": 128,
        "manufacturer_code": 4417,
        "maximum_buffer_size": 66,
        "maximum_incoming_transfer_size": 66,
        "server_mask": 10752,
        "maximum_outgoing_transfer_size": 66,
        "descriptor_capability_field": 0
      },
      "endpoints": {
        "1": {
          "profile_id": "0x0104",
          "device_type": "0x0051",
          "input_clusters": [
            "0x0000",
            "0x0004",
            "0x0005",
            "0xef00"
          ],
          "output_clusters": [
            "0x000a",
            "0x0019"
          ]
        }
      },
      "manufacturer": "_TZE200_rxntag7i",
      "model": "TS0601"
    },
    "active_coordinator": false,
    "entities": [
      {
        "entity_id": "update.tze200_rxntag7i_ts0601_firmware",
        "name": "_TZE200_rxntag7i TS0601"
      }

Logs
Logs
[Paste the logs here]

Custom quirk
Custom quirk
[Paste your custom quirk here]

Additional information
No response

Please, I also need this device to be supported by HA, would it be possible to add it?
Thank you very much in advance.

I need this too and am new to home assistant. It's taken me a year to get round to setting this up I also have the TRV16 but as it's also listed as TS0601 albeit with _TZE200_bvu2wnxz instead of _TZE200_rxntag7i but as they are both Avatto I'm hoping it's the same.. enough.
Did you see that Zigbee Device Compatibility Repository - Avatto_TRV06 points to Custom Quirk? It seems to have a lot of information about creating and editing quirks too. I have no experience in doing this yet or even installing a quirk but it appears to be a good place to start.


OSSL_PARAM_construct_utf8_string computes the string length.

If the passed string length is zero, the function computes the string length
from the passed string.

[x] documentation is added or updated
[x] tests are added or updated

I think I was overly aggressive about the changes, I've backed the test/ ones out which ought to appease the CIs.  These didn't use strlen(2) to compute the length of the string, so the output has changed.
A re-approval is required.

Merged, thanks.  This irksome little thing is gone now.


usb_serial_jtag read/write issues  (IDFGH-11373)

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.
v5.1.1 and master b4268c874a4cf8fcf7c0c4153cffb76ad2ddda4e
Espressif SoC revision.
ESP32-S3 (QFN56) (revision v0.1)
Operating System used.
Linux
How did you build your project?
Command line with idf.py
If you are using Windows, please specify command line type.
None
Development Kit.
esp32s3 tdisplay s3
Power Supply used.
USB
What is the expected behavior?
read/write work more or less as uart driver functions
What is the actual behavior?
uart via serial_jtag doesn't seem to work in v5.1.1 without first a new line output on stdout.
On master b4268c874a4cf8fcf7c0c4153cffb76ad2ddda4e even with that the code didn't work.
Steps to reproduce.
With menuconfig setup with console output primary set to uart0 and secondary set to serial_jtag
#include <stdio.h>
#include "sdkconfig.h"
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "driver/usb_serial_jtag.h"

#include "esp_vfs_usb_serial_jtag.h"
#include "esp_vfs_dev.h"
#include <fcntl.h>

/* master is b4268c874a4cf8fcf7c0c4153cffb76ad2ddda4e */
void app_main(void)
{
    /*
     * this doesn't help in master not in v5.1.1 for me
     *

    setvbuf(stdin, NULL, _IONBF, 0);
    esp_vfs_dev_usb_serial_jtag_set_rx_line_endings(ESP_LINE_ENDINGS_CR);
    esp_vfs_dev_usb_serial_jtag_set_tx_line_endings(ESP_LINE_ENDINGS_CRLF);
    fcntl(fileno(stdout), F_SETFL, O_NONBLOCK);
    fcntl(fileno(stdin), F_SETFL, O_NONBLOCK);

    */

    usb_serial_jtag_driver_config_t config = USB_SERIAL_JTAG_DRIVER_CONFIG_DEFAULT();
    esp_err_t err = usb_serial_jtag_driver_install(&config);
    ESP_ERROR_CHECK(err);

    /*
     * this doesn't help in master not in v5.1.1 for me
     *
    esp_vfs_usb_serial_jtag_use_driver();
    // this instead of esp_vfs_usb_serial_jtag_use_driver makes no difference in v5.1,1 nor master
    // esp_vfs_usb_serial_jtag_use_nonblocking();
    esp_vfs_dev_uart_register();

    */
    /* without this line the following loop doesn't do anything on v5.1.1.
     * on master nothing happens regardless ? */
    putc('\n', stdout);
    /* with the line above on v5.1.1 i get echo back */

    uint8_t buf[64];
    while(1) {
        int read = usb_serial_jtag_read_bytes(buf , sizeof(buf), 10);
        assert(read < sizeof(buf));
        if (read) {
            usb_serial_jtag_write_bytes(buf , read, 10);
        }
    }
}

Debug Logs.
No response
More Information.
It seems a printf("something\n") with a new line is necessary for things to work on v5.1.1
A putc('\n', stdout); also works.
But writing a \n via jtag write_bytes didn't help
In master at b4268c874a4cf8fcf7c0c4153cffb76ad2ddda4e not even the new line or a printf helped.
Am I using the library wrong?

What is the setting for primary and secondary console output you have in menuconfig?
Also, can you describe what you want to achieve? The code looks a bit odd, since first you are configuring a high-level method for accessing the console (via the VFS driver stdio) but then you are using low-level driver functions to read and write bytes. Understanding the goal behind this code might help with troubleshooting this.

@igrr

What is the setting for primary and secondary console output you have in menuconfig?

I have the primary set to uart0 and secondary set to usb serial jtag port.
I also have tried with usb serial jtag port as primary.

Also, can you describe what you want to achieve?

I want to use the esp32s3 as I am using an esp32 via the usb-serial converter chips, that includes using the "serial" for firmware flashing but also for application needs with binary data (i.e. not for logging).
I tried the software cdc which works great but when I use that I have to use the reset/boot button to flash which breaks in CI contexts without some additional hardware to emulate button pressing.
In the code I provided most of it is commented out - that's just to show things I tried that didn't work for me.
If I understand correctly the vfs abstraction allows to use of the usb port as serial by reading/writing to a file unix style but I would rather use the functions available in usb_serial_jtag - but they don't work for me. It seems that in 5.1.1 without outputing first a new line, nothing work.
putc('\n', stdout); or printf("\n");
In master not even that works.
To show better what I mean let me remove the commented out code -I'd expect this code to work on v5.1.1 and it does not
#include <stdio.h>
#include "sdkconfig.h"
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "driver/usb_serial_jtag.h"

/* master is b4268c874a4cf8fcf7c0c4153cffb76ad2ddda4e */
void app_main(void)
{
    usb_serial_jtag_driver_config_t config = USB_SERIAL_JTAG_DRIVER_CONFIG_DEFAULT();
    esp_err_t err = usb_serial_jtag_driver_install(&config);
    ESP_ERROR_CHECK(err);

    // output here a new line with putc or printf to make things "work"

    uint8_t buf[64];
    while(1) {
        int read = usb_serial_jtag_read_bytes(buf , sizeof(buf), 10);
        assert(read < sizeof(buf));
        if (read) {
            usb_serial_jtag_write_bytes(buf , read, 10);
        }
    }
}

it starts working if after the driver_install you print a new line.
Thank you!

I tested this with idf.py monitor as well as with screen

I found a 'fix' for my last "minimal" example but doesn't look very pretty IMHO
adding fflush and fsync after usb_serial_jtag_write_bytes
            usb_serial_jtag_write_bytes(buf , read, 10);
            fflush(stdout);
            fsync(fileno(stdout));

@igrr shouldn't the driver take care of this? v5.1.1

You can get rid of fflush if you switch the stream to a non-buffered mode using setvbuf function.
I don't think we can get rid of fsync. The driver will flush the USB endpoint buffer every time it fills (64 bytes) or when it encounters a newline character. So for binary data, it's up to you to inform the driver that you'd like the data to be sent to the host immediately.

@igrr I tried various things but it seems that calling fsync just after the install is enough and is not required to call it every time a write is done.
Do you agree? or do i need to do it after every write?


Question: Do I have to pay for embeds?

In my blog post on Medium I speak favorably of Carbon.
However, one of my readers pointed out to me that embedding Carbon snippets does not seem to work unless one has a paid account.
This seems to be reflected on the Account page:

However, I've never had any issues with embedding snippets from my free account. I also noticed that the git commit for the "Carbon Diamond is not quite ready yet." message is 3 years old.
So now I am confused... Is a paid account now required for embedding snippets?

Thanks for the question. That feature is now included in free tiers. Feel free to submit a PR to fix this 🙂


Update Update-PnPSiteDesignFromWeb.md

Before creating a pull request, make sure that you have read the contribution file located at
https://github.com/pnp/powerShell/blob/dev/CONTRIBUTING.md
Type

[ ] Bug Fix
[ ] New Feature
[ ] Sample
[x] Documentation

Related Issues?
N/A
What is in this Pull Request ?
Added description for -Url parameter, removed description of the non-existent -WhatIf parameter

Thanks @PowershellScripts , merged it !


SecRequestBodyAccess Off doesn't fully disable request body processing

Hi!
Why setting SecRequestBodyAccess Off  doesn't fully disable parsing request body and filling ModSecurity variables?
My case:
I'm using ModSecurity with Nginx for proxying files as multipart/form-data. I don't want to process body content and save memory,
so I set SecRequestBodyAccess to  Off  and when I send a big file I get bad_alloc error and core dump:
(gdb) bt full
#0 0x00007f939d2b9377 in raise () from /usr/lib64/libc.so.6
No symbol table info available.
#1 0x00007f939d2baa68 in abort () from /usr/lib64/libc.so.6
No symbol table info available.
#2 0x00007f939b74d7d5 in __gnu_cxx::__verbose_terminate_handler() ()
from /usr/lib64/libstdc++.so.6
No symbol table info available.
#3 0x00007f939b74b746 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#4 0x00007f939b74b773 in std::terminate() () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#5 0x00007f939b74b993 in __cxa_throw () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#6 0x00007f939b74bf2d in operator new(unsigned long) () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#7 0x00007f939b7aaa19 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#8 0x00007f939b7aabd6 in std::string::_M_mutate(unsigned long, unsigned long, unsigned long) () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#9 0x00007f939b7ab19e in std::string::_M_replace_safe(unsigned long, unsigned long, char const*, unsigned long) () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#10 0x00007f939f364683 in modsecurity::AnchoredVariable::set(std::string const&, unsigned long) () at anchored_variable.cc:71
No locals.
#11 0x00007f939f355155 in modsecurity::Transaction::processRequestBody() ()
at transaction.cc:873
Python Exception <class 'gdb.error'> There is no member or method named _M_head_impl.:
a =
^[[ fullRequest = "Accept: */*\nContent-Length:<PHONE_NUMBER>\nContent-Type: multipart/form-data; boundary=yBnB6soCHCHTWWhy6kHZ-T6lUqGnDkGXJdrd\nConnection: Keep-Alive\nUser-Agent: Apache-HttpClient/4.5"..
l = std::vector of length 7, capacity 8 = {0x9e6df0, 0x9e6d70, 0x9e6cf0,
0x9e6c90, 0x9e6c30, 0x9e5a30, 0x9e5b40}
#12 0x000000000055111f in ngx_http_modsecurity_pre_access_handler ()
No symbol table info available.

For me it's seems like I could have to have setting to fully disable body processing.
I've analyzed code a bit and found available memory checking condition:
https://github.com/SpiderLabs/ModSecurity/blob/v3/master/src/transaction.cc#L977
But following REQUEST_BODY variable setting doubles occupied memory size what leads bad_alloc error:
https://github.com/SpiderLabs/ModSecurity/blob/v3/master/src/anchored_variable.cc#L71
If it's a bug I could contribute and make pull request.
But now for me it's not really obvious what approach is better to fix it.

Hi @slinikma,
That is something that only happens within the intermediate file? or something that happens via memory exchange?

@zimmerle, sorry I mixed up SecRequestBodyAccess Off and ctl:requestBodyAccess=Off. So, core dump I described above was with SecRequestBodyAccess set On and rules with ctl:requestBodyAccess set Off.
In such case it's not really obvious that ModSecurity set request body to REQUEST_BODY and FULL_REQUEST variables and occupies memory size  three times (?) bigger than file size. Setting variables happens without available memory checking (std::string::reserve for example) and it could lead to core dump.
As for SecRequestBodyAccess On it prevents setting REQUEST_BODY and FULL_REQUEST, so if available server memory is going to end while file downloading, nginx worker process could be killed by OOM killer. But it's a problem with server resource sizing, not with ModSecurity. I just met this problem on dev server.
As for your question - nginx dumps file on disk and nginx-modsecurity connector calls msc_request_body_from_file function.
Now I don't think that my problem is a bug, but I didn't expect core dump in such case.


Trailing Comma "all" setting breaking JSON

Having the trailing comma option set to all sets it on JSON entries as well. This is causing syntax errors and I have to disable prettier to edit JSON files unless I turn of trailing commas.

I have a very similar issue... but it is doing it if I have any trailing comma setting.  It also seems to be saving my settings file even if I don't have it open and am doing something completely different.  It constantly resaves and reformats my settings.json file (with trailing commas).

@kkemple @Goblinlordx I have the same issue here.
Maybe this is a VSCode issue? https://github.com/Microsoft/vscode/issues/29701

I m pretty sure it started occurring when I updated prettier though and not a VScode update.  I really wish there was an easy way to revert to an older plugin version to test =/.

For trailing-comma all, could you reopen prettier/prettier#2308 @kkemple
For the other part of the issue, I don't get it for now.

I see they pushed patch for this 22 hours ago. Is it possible to update plugin, please? Thanks a lot.

quick fix: add "prettier.jsonEnable": [] to your vscode settings

@infernojs thanks

@dacz The patch is on master but not yet released. Diff between master and 1.5.2 (we bundle 1.5.1 for now) https://github.com/prettier/prettier/compare/1.5.2...master
@vjeux Do you plan on a patch release soon?

@rafaelkallis, do you get this error even after turning JSON off?
Unexpected token, expected ; (2:9)

@cuth having it only when editing json files. json still get saved though.

@cuth your error seems weird to me. This doesn't seem to be a JSON error.

@CiGit, looks like it is just related to updating settings and upgrading prettier in node_modules but not restarting vscode. Thanks!!


change  private String mFormat;  to be  private final String mFormat;

https://alluxio.atlassian.net/browse/ALLUXIO-2260
Mark member variable mFormat as final in PropertyKeyFormat.java
ALLUXIO-2260

Can one of the admins verify this patch?

ok to test

Jenkins, add to whitelist

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/11781/
Test PASSed.

@zcw01 Thanks! Could you sign the CLA agreement?

LGTM


Database

How can I have a database??
Is realm supported??
Is sqlite suppoted??

Only if you're able to write some native code, unfortunately.
Not directly, this https://github.com/realm/realm-js/tree/master/src/RealmJS.xcodeproj should be updated before using, but I' never have a chance to work with Realm.
Not directly, it also requires writing wrappers in native code (Objective-C or Swift).


feat(query): annotate logger with trace id and record query errors in trace log

This PR connects logged errors to their corresponding trace by annotating the logger with the request's trace ID. Additionally, errors bubbling up to the query handler are captured in the trace logs.
A small, unrelated README update is also included.

[ ] CHANGELOG.md updated with a link to the PR (not the Issue)
[x] Well-formatted commit messages
[x] Rebased/mergeable
[ ] Tests pass
[ ] Use @affo's TraceInfo method once its merged

looks good once you fix what is causing tests to fail, imo.


Frame size is wrong

Frame(0, 100).size = Point(100, 100) where its actual size is 101x101
This happens due to the classic errors that arise when thinking about 0 indexed  vs 1 indexed arrays
I think this should be handled together with #22

There are other lurking issues and this is going to screw many things up in the future:
for example:
const array1 = [
    [1, 2, 3],
    [4, 5, 6]
]
const s = new Sheet([0, 0], [0, 0)])
s.dataFrame.loadFromArray(array1);
s.size // (2, 1) as expected
s.corner.x // 2 as expected
s.corner.y  // 1 as expected

const array2 = [
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]
]
s.dataFrame.loadFromArray(array1);

s.size // (2, 3) wtf!
s.corner.x // 2 as expected
s.corner.y  // 3 wtf!

the point (2,3) is not even in the sheet, and it's corner should be (2x2) - this. is pretty messed up

ap-sheet

ap-sheets -- meaning the actual element -- do not have a size method, so I'm not sure what the example code is trying to do. Do you mean to get the size of the DataFrame?

s = ap-sheet
everything above refers to the data frame ie s.dataFrame (there is a typo there, should be s.dataFrame for everything)


Update kafka.go to add transport.kafka.compression.type as an option

Addresses Issue: https://github.com/netsampler/goflow2/issues/92

Fixed build issue @lspgn can you re-run workflow again?

Implemented the changes as part of #128 (not sure if there's a better way to do with GitHub). Closing this PR.
Thank you!


calculating value of an upvote

Requests for a such a method come in often, with solutions seemingly scattered and hard to find.

Closing in favor of the above.


Decode query values of URI

Solves https://github.com/bcit-ci/CodeIgniter4/pull/953#pullrequestreview-100302997 and by the default the query values will be decode.

@natanfelles I don't believe this is valid anymore since I removed the urlencoding in 2604ea75b7938e0cfa5efdfb022c8ab14608965a. Can you confirm?

Yes, @lonnieezell . I can confirm this issue in my environment.
The URL Queries are re-encoded every new request.
Please, test this:
$pager = \Config\Services::pager();
echo $pager->makeLinks($_GET['page'] ?? 0, 10, 200);

In the browser try something like: http://localhost:8080/?q=you+alice+to+the+little
Navigate by the links. Look that the URI continues changing:
Page 2: http://localhost:8080/?q=you%2Balice%2Bto%2Bthe%2Blittle&page=2
Page 3: http://localhost:8080/?q=you%252Balice%252Bto%252Bthe%252Blittle&page=3
Page 4: http://localhost:8080/?q=you%25252Balice%25252Bto%25252Bthe%25252Blittle&page=4
And continues re-encoding... I'm testing with PHP and nginx servers, and same results.

Looks like this happens when the getQuery() method is called, within the http_build_query() call. I think what you've done is likely fine, but need to think through the ramifications a bit. There's something tickling the back of my mind saying that we might be overwriting someone's values if it happens to include strings that it shouldn't.
After looking through everything again, I'm a little concerned about how easy it would be to double-encode something. We don't currently use the parser for pagination output, though we likely should. In that case, though, it should be escaped as a URL, which I believe encodes it again. That could definitely cause issues, too. Hmm....


Update HttpBinApi.credentials.ts

Replace the docs URL. We need to be clear that the URL must point to their site. The automatic linking won't work as obviously we won't have official docs for community nodes.

Hey @StarfallProjects I've already changed the documentation URL to point to an external website.
Do you think we should use a placeholder instead? This will cause the linter to fail right out of the box, pointing the user to the correct place for fixing.
Let me know what you think is best.

Agreed, let's use the placeholder.


Add --memory-leak-check to deallocator proofs

We should review all of the deallocator harnesses, e.g. aws_hash_table_clean_up_harness, and verify that all of the memory is correctly freed.
Also see: https://github.com/awslabs/aws-c-common/pull/802#discussion_r635648340 and aws/s2n-tls#2774.

Has this been added with the merged PR?


[Feature request]RadosGW multi site replication + HAPROXY with Ansible deployment.

Hey,
Recently i've encoutered few companies with the same interest about adding a quiet valueable and a reasonable feature which me and the community aswell would be gladly to have in our Ceph inventory.
The feature is RadosGW multisite along with HAPROXY support towards the Playbooks deployment.
Thanks.

The only problem is that we will have a dependency with the HAProxy module. It is always really annoying to work with external playbooks. Perhaps I can add an example directory for the HAProxy config template.
I don't really have time for this at the moment, I'll pick up that task later.
Thanks!

Is it still on the table? I would like to have that feature as well.

Hum yes and no, at some point it might be but I'll not integrate the HAProxy role in this repo.

I'm closing this since this should not be part of this repo.


Writing seperate loader for *.cucumber files results in esbuild-preprocessor error (headless mode only)

Hi,
Not too sure that this thread should belong in this forum or the one for the cypress-cucumber-preprocessor plugin, but here's the problem.
So basically, I'm trying to modify the es-build-preprocessor to convert *.cucumber files instead of *.feature to the needed .spec.ts files and I hope I'm doing it completely wrong and there's a better solution/approach than the following:
According to the readme's, I should write my own loader to handle *.cucumber files. This looks as follows (based on the existing cucumber-processor code):
import * as fs from 'fs';
// @ts-expect-error will error
import { compile } from '@badeball/cypress-cucumber-preprocessor/browserify';

export function createCucumberESBuildPlugin(config) {
  return {
    name: 'cucumber',
    setup(build) {
      // @ts-expect-error will error
      build.onLoad({ filter: /\.cucumber$/ }, async (args) => {
        const content = fs.readFileSync(args.path, 'utf8');
        return {
          contents: await compile(config, content, args.path),
          loader: 'js',
        };
      });
    },
  };
}

I'm using this plugin as follows:
import { addCucumberPreprocessorPlugin, } from "@badeball/cypress-cucumber-preprocessor";
import { createCucumberESBuildPlugin } from "./cucumber-plugin";
import createBundler = require("@bahmutov/cypress-esbuild-preprocessor");

export async function setupNodeEvents(
    cypressOn: Cypress.PluginEvents,
    config: Cypress.PluginConfigOptions,
): Promise<Cypress.PluginConfigOptions> {

    // https://github.com/bahmutov/cypress-on-fix
    // eslint-disable-next-line @typescript-eslint/no-var-requires
    const on = require('cypress-on-fix')(cypressOn)

    await addCucumberPreprocessorPlugin(on, config);

    on(
        'file:preprocessor',
        createBundler({
            plugins: [createCucumberESBuildPlugin(config)],
        },),
    );

As you can see, I'm using the cypress-on-fix package to fix possible errors for the 'on' listeners.
When executing the Cypress tests in headless mode, I can see that the *.cucumber files are being converted to the correct format in the generated Cypress bundles, but I get the following error:
CypressError: `cy.task('cypress-cucumber-preprocessor:spec-envelopes')` failed with the following error:
> Unexpected state in specEnvelopesHandler: before-run. This almost always means that you or some other plugin, are overwriting this plugin's event handlers. For more information & workarounds, see https://github.com/badeball/cypress-cucumber-preprocessor/blob/master/docs/event-handlers.md (if neither workaround work, please report at https://github.com/badeball/cypress-cucumber-preprocessor)
https://on.cypress.io/api/task

I have tried both workaround, but without succes. The same error keeps popping up, but only in headless mode, perhaps because that timings in headless mode are conflicting with eachother but not in headed mode.
I'm very curious what is going wrong so some help in this would be very appreciated!
Cheers,
Thomas

You need to provide a minimal, reproducible example that illustrates the issue.

Thanks for your response. I will work on an example ASAP.

Thanks for your response. I will work on an example ASAP.

Hello, Have you been able to find a workaround for this please ?
I'm also using cypress-on-fix to be able to register cucumber and allure as plugin, but i'm stuck with this error

At our project, we have chosen to rename the .feature files not to .cucumber files, but to .gherkin.feature files so we can configure IntelliJ to use .gherkin.feature files so the preprocessor plugin does not need to be altered. Hereby we solved our challenge in to use Gherkin feature files together with Karate feature files.
No really a solution for our problem, but the workaround is OK by now.


the actual version is V2 as of 09/11/2018

your article is a bit outdated

Document details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 36d01d11-eea9-7f49-693d-25c3fecc6253
Version Independent ID: 6749a619-2b75-1561-81e9-b45888c58c4b
Content: How to use the SharePoint Migration Tool
Content Source: migration/how-to-use-the-sharepoint-migration-tool.md
Product: sharepoint-server-itpro
GitHub Login: @JoanneHendrickson
Microsoft Alias: jhendr

@dplotnikov  - Thank you for submitting this feedback.
We greatly appreciate your due diligence on letting us know this article is out of date.
I was able to get this into the backlog for the SharePoint team. The team will get this prioritized into a sprint plan. Thank you for reporting and making the docs better. Much appreciated.
I made a note to request the team to update this when the work is complete.


Feature/license

@silas it appears as if your CI process is broken, but this should be good to go

Thanks! Published to npm<EMAIL_ADDRESS>


Fix issue with unprotected concurrent access to gitRepository state

This is a replacement for https://github.com/GoogleContainerTools/kpt/pull/3983. It moves the logic for fetching and updating the lifecycle of a PackageRevision from the gitPackageRevision object to the gitRepository object, as it involves manipulating the state of the gitRepository.
To make sure all access to the shared state is protected by the gitRepository mutex and no calls tries to take the lock twice, all calls from gitRepository will call versions of the functions that expects that the caller already has the lock, while calls from elsewhere call the versions of the functions that takes the lock.

There are still some concurrency issues here, which is the cause of the failing e2e tests. I'm looking into it.


export key log files from client and server in the interop runner

Codecov Report

Merging #2285 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #2285   +/-   ##
======================================
  Coverage      86%     86%
======================================
  Files         108     108
  Lines        8074    8074
======================================
  Hits         6944    6944
  Misses        852     852
  Partials      278     278

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 07d76c5...2c49904. Read the comment docs.


Pass migrated annotation for apps

Description:
Helmv3 apps that were originally made through norman and are migrated to dashboard will have an annotation added to their  spec.chart.metadata.annotations field. This annotation needs to be passed on upgrade if it can be found on the app similar to ui-repo and repo-type annotations. Annotation is "apps.cattle.io/migrated".
Backend Issue:
note: backend does not need to be merged first.
https://github.com/rancher/rancher/issues/34700

https://github.com/rancher/rancher/pull/35202

@rmweir I'm not seeing the annotation - I create a helm 3 app from the ember UI (e.g. openebs). I then look at the app in the Dashboard UI (via Steve) and I don't see any annotations as described above.
I was expecting to see a apps.cattle.io/migrated annotation.
I thought from the description, if I retrieve an app created with Norman via the Steve API, I should see an annotation indicating that it has been migrated - which should then be retained when upgrading.

@nwmac the annotation is one that would be added by the user as part of a manual process we are using for now to migrate apps. Can test by using kubectl to add "apps.cattle.io/migrated": "true" to an apps spec.metadata.annotations field. Then when you go to edit that apps, you'll notice the request omits the annotation from the chart.

@rmweir So its a manual migration process - is this documented? So, if we see this annotation on an app, we should carry it forward when upgrading... should it also be passed through to any dependencies?

@rmweir Can you clarify where the annotation should be placed? The description says spec.chart.metadata.annotations where as you latest comment says spec.metadata.annotations.
I see 2 places the annotation could go on the app resource:
spec.chart.metadata.annotations
metadata.annotations
can you let me know which one it is, please.

@nwmac you will find the annotation on spec.chart.metadata.annotations, when you pass it to the upgrade action it will be be charts.annotations, similar to how the other two annotations are passed.

Re-opening the issue as the following issue is seen while upgrade:
Rancher version: v2.6-head 44b8030

Create a cluster with 3 nodes 1 role each
Install the app aws-event-sources from the ember UI in the default project
Add the annotation via kubectl on the downstream cluster in spec.chart.metadata.annotations

catalog.cattle.io/ui-source-repo: helm3-library
catalog.cattle.io/ui-source-repo-type: cluster
apps.cattle.io/migrated: "true"

From the local cluster kubectl file or execute into the local cluster --> kubectl shell --> kubectl edit apps.project.cattle.io aws-event-sources -n <ProjectIDAppInstalled>
Add the annotation cattle.io/skipUninstall: "true" under metadata.annotations
Delete the aws-event-sources from the downstream cluster --> default project
Navigate to the explore --> Apps & market --> Installed apps --> aws-event-sources --> Edit/upgrade
The page won't load and a 404 response is returned:

baseType: "error"
code: "NotFound"
message: "apps.project.cattle.io \"aws-event-sources\" not found"
status: 404
type: "error"

Re-opening as I expected the upgrade to go through without any errors.

@anupama2501 @rmweir I don't understand the use case that the set of steps represents. Can you please explain. Although related, I think this is a different issue to not passing down the annotation.

@nwmac the use case is migrating from using a norman app to the dashboard app. Adding the annotation is part of this. The annotation tells rancher that certain steps need to be taken because the helm chart was originally installed with a norman app. Maybe the issue is not related to adding the annotation but the functionality that the annotation is trying to enable is unusable as long as this issue is here.

Verified on ac67f83 v2.6-head

Created an app aws-event on rancher serve v2.5.11 in the downstream cluster
Upgraded the rancher server to v2.6-head ac67f83
Added the following annotation to the app
kubectl edit apps aws-event-sources -n aws-event-sources
Add the chart repo name: helm3-libarary , path: https://git.rancher.io/helm3-charts and branch: master.
Add the annotation "cattle.io/skipUninstall": "true" via `kubectl edit apps.project.cattle.io aws-event-sources -n  from the local cluster
Delete the app from the downstream cluster default project
Verify the app is not removed from the  apps & market --> installed apps
Upgraded the app and the upgrade goes through without an error.

Verified on v2.6-head e62e1ae upgraded from 2.5.11
Validated few more scenarios with respect to the above:
To migrate:
kubectl get apps -A | grep aws
kubectl edit apps aws-event-sources -n aws-event-sources

Add the annotation via kubectl on the downstream cluster in spec.chart.metadata.annotations
catalog.cattle.io/ui-source-repo: helm3-library
catalog.cattle.io/ui-source-repo-type: cluster
apps.cattle.io/migrated: "true"

From local cluster:
kubectl get apps.project.cattle.io -A | grep aws
kubectl edit  apps.project.cattle.io aws-event-sources -n p-mr57l

Delete the app from the ember UI:
kubectl delete apps.project.cattle.io aws-event-sources -n p-mr57l

From apps  & market place --> Repositories --> name: `helm3-library`, URL: `https://git.rancher.io/helm3-charts`, branch: `master`

Uninstall post migration:

Migrated the app hpe-csi-driver from ember to dashboard by following the above mentioned steps
Uninstalled the app and verified the uninstallation went through without errors

Update the app post migration:

edited the app aws-event-sources by changing the default images to false
Verified the update went through as expected

Upgraded the app post migration

Upgraded the app externalip-webhook from version 0.1.4 to 0.1.5
Upgrade went through without any errors

Upgraded the app and uninstalled

From the above test case post upgrade, uninstalled the app
App was deleted and all the workloads were removed from the project


There are no integration tests for the Sample Project registration code

Done - see the Sample.Tests project


Path separators are mixed for windows vs *nix

If I use mixed paths in User input path variables, the tasks are failing to resolve the path correctly and thus failing.
This is scenario for most of the tasks where they use "path input" and "find files".

@ericsciple is looking into this issue as per https://github.com/Microsoft/vsts-task-lib/pull/83
I am closing issue from our side as we have bug against our tasks [Ant/Maven/Gradle/VsTest]

Can you send me the bug? Does it have repro steps?

internal tracking in mseng: 592128

Use forward slash in cross platform tasks.  Node will handle correctly in all platforms


libtool, autoconf, automake needed

Expected behavior
Documentation should mention that libtool, automake, autoconf are needed to npm the lisk-sdk. sudo apt install libtool automake autoconf fixes it.
Actual behavior
Documentation doesn't mention it. The installation of the lisk-sdk fails because those tools are missing.
Steps to reproduce
Try to npm install --save lisk-sdk on a brand new ubuntu (based) OS.
Which version(s) does this affect?
Newer ubuntu (based) linuxs, maybe others too

Same issue on OSX Catalina Beta 6, needed packages can be installed with homebrew

This is a duplicate of #336


Add documentation for external storage

We recently introduced the use of temporary storage to store intermediate query results. This is useful when you are executing a memory-intensive task and there is not enough memory to complete this task (error returned was pq: memory budget exceeded...). The use of temporary storage has been implemented for ORDER BYs without LIMITs (see https://github.com/cockroachdb/cockroach/pull/16717) and joins (see https://github.com/cockroachdb/cockroach/pull/17192).
This enhancement is currently turned off by default although I think the plan is to turn it on by default for 1.1. In any case, sql.defaults.distsql.tempstorage is a boolean setting that turns this enhancement on for the whole cluster. There are also fine-grained settings for sorts and joins, in the case that some bug crops up in one but the operator would still like to keep the other one on. These are sql.defaults.distsql.tempstorage.sorts and sql.defaults.distsql.tempstorage.joins.
There is also a per-node environment variable COCKROACH_WORK_MEM that defaults to 64MB. It limits the "working memory" of a processor. For example, if a sorter processor is sorting a bunch of rows, the sorter processor can only consume up to 64MB of rows before hitting this limit and falling back to disk. It is important to note that this limit only applies when sql.defaults.distsql.tempstorage = true.

Temp storage on by default: https://github.com/cockroachdb/cockroach/pull/17665

@asubiotto Can you give me an overview of how this interacts with the IMPORT statement's use of "temp" storage? I have a PR open to document that feature #1914

@sploiselle I believe that the IMPORT statement would use the same directory as this, @mjibson can confirm. Both distsql and the IMPORT statement use the same RocksDB instance.

There's also a cli flag for overriding the temp store directory: https://github.com/cockroachdb/cockroach/issues/17859
@sploiselle, are you able to cover this feature as a whole, or just in relation to import?

Here's a rough diagram from a conversation with @asubiotto

We already have docs for the --max-disk-temp-storage flag on cockroach start.
We need to add details about the cluster settings for turning off temp on-disk storage for joins and sorts, and for changing the max memory size before spilling to disk.
For 2.0, we need to document the --temp-dir flag on cockroach start, which users can use to specify the location for temp storage (defaults to the first store's data directory).

@knz, I'm trying to understand the relationship between --max-disk-temp-storage, --max-sql-memory, and the sql.distsql.temp_storage.workmem cluster setting.
My understanding is that --max-disk-temp-storage defines "the maximum on-disk storage capacity available to store temporary data for SQL queries that exceed the memory budget." But is that "memory budget" referring to --max-sql-memory, sql.distsql.temp_storage.workmem, some combination, or something else entirely?

If I understand well the --max-sql-memory is a constraint on RAM usage, whereas --max-disk-temp-storage and sql.distsql.temp_storage.workmem are a constraint on disk usage. @asubiotto is this understanding correct?

@knz, that's my understanding of --max-disk-temp-storage, but the description for sql.distsql.temp_storage.workmem seems to refer to memory and not disk usage...
| sql.distsql.temp_storage.workmem                  | 64 MiB        | z    | maximum amount of memory in bytes a processor can use before falling back to temp storage                                                       |

Oh I see. Yes I vaguely understand what's going on here, but I wouldn't be able to explain.

--max-sql-memory is a constraint on RAM usage for the node as a whole (i.e. all processors), sql.distsql.temp_storage.workmem is a limit per processor that can use external storage (i.e. not applicable to all processors). This means that a processor that can fall back to disk will use at most sql.distsql.temp_storage.workmem. However, processors may hit --max-sql-memory before this if there are other memory-intensive processors running and will fall back to disk.
--max-disk-temp-storage is the only flag that relates to temp storage disk budget. So once a processor falls back to disk, they will use at most this budget before failing a query (to avoid disk blow-ups).

Excellent. Thank you, @asubiotto. Can you give me a few examples of processors limited by sql.distsql.temp_storage.workmem?

The HashJoiner and the Sorter (however, only for sorting without a limit for now)

This goes back to 1.1, but moving to the 2.1 milestone since we'll work on it in that timeframe.

Note that external storage has now been added to all sort strategies: https://github.com/cockroachdb/cockroach/pull/27083

@rmloveland, @awoods187, when you look at the full docs backlog for sql, this is another one that probably warrants prioritization.

I'm wondering what the status of this issue is. Can it be closed?

I'm wondering what the status of this issue is.

@asubiotto I haven't looked at this in a very long time.  Based on a quick look at the docs, we have:

Docs for the --max-disk-temp-storage flag on the "Start a node" page

Rudimentary cluster settings docs for sql.distsql.temp_storage.{joins, sorts,workmem}

Can it be closed?

I don't claim to understand all of the discussion on this issue, but It looks like yes?  Do you think the docs we already have (linked above) cover this issue?

Thanks for checking! It seems like it can.


How to pass external textures instead of using WebCamTexture

Hi
An ARSession is already running on phone. I get camera texture from AR camera preview. I am trying to pass it to MediaPipe graph. Is it possible? If possible can you please suggest the process. Thank you.

If the graphics API is OpenGL ES, yes.
But note that it's only available on Android currently (see #126 for iOS support).
To pass an external texture, you need to get its texture name (cf. https://docs.unity3d.com/ScriptReference/Texture.GetNativeTexturePtr.html) first.
https://github.com/homuler/MediaPipeUnityPlugin/blob/92cad4bbe9ba52514034c52ac5b5f0a99accab06/Assets/Mediapipe/Samples/Scripts/DemoGraph.cs#L79-L82
If you have it, you can build GpuBuffer from it, and pass that to MediaPipe through CalculatorGraph#AddPacketToInputStream API.
https://github.com/homuler/MediaPipeUnityPlugin/blob/92cad4bbe9ba52514034c52ac5b5f0a99accab06/Assets/Mediapipe/Samples/Scripts/DemoGraph.cs#L99-L112

Ok. Let me tell you my requirement here.
Because ARKit already uses device camera already to show camera preview, I am not able to use WebCamTexture.
ARKit provides me a texture2D.
I need to pass this texture to MediaPipe Graph and render joints/lines on top of it.
And doing it on iPhone is my first priority. Android is later priority.
FYI, Unity iOS OpenGLES is deprecated. So I am using Metal Graphics API and samples are working fine with Metal graphics.
Thank you.

FYI, Unity iOS OpenGLES is deprecated.

Thank you. I know that but to pass texture's native pointer to MediaPipe is not supported yet on iOS.

Okay. Thanks

Actually I am trying to integrate mediapipe with ARKit to use iPhone depth and get 3D world coordinates of the human body tracking.
If mediapipe already provides 3D world coordinates or depth (z), I don't even need to do this.
I see all the annotations rendered in 2D on image. Does Mediapipe also provide Z/Depth value for pose tracking. Please clarify me on this.
Thanks.

Hi. did you get a chance to read my above message. I appreciate you suggestions. thank you.

Does Mediapipe also provide Z/Depth value for pose tracking.

Please see the official information.
https://google.github.io/mediapipe/solutions/pose.html#output

Able to pass external textures.

Disable webcamtexture.play
Assign external texture to output texture

@manisharan-ys Hi, Have you successfully integrate mediapipe with ARKit to use iPhone depth? And may i ask how  you made it


Directive doesn't but should update when timestamp changes

The directive should update its text when the timestamp (passed in via am-time-ago) changes, but it doesn't.
I think the problem is here:
https://github.com/urish/angular-moment/blob/master/angular-moment.js#L257
This is scope.$watch() but should be attr.$observe() like the others?  Making that small change fixes the problem for me.
I'm writing this issue rather than submitting a patch because it seems unlikely to me that I'd be the only one having this problem, so I'm wondering if I'm doing something wrong or non-standard.  (FTR: I'm using Angular 1.4.)

Hi, thanks for reporting!
We actually have a test case for this:
https://github.com/urish/angular-moment/blob/master/tests.js#L86
Can you please try to create a test case that reproduces the issue?
Thanks!

I think the problem is that I'm passing into the am-time-ago attribute not a scope variable name but rather the value itself.  It's surprising that angular-moment (mostly) works when passing in a value, rather than a scope variable name, given that it's designed only for the latter!
My particular use case unfortunately doesn't lend itself to passing in the name of a scope variable.  I'm passing in the value of calling a method on an object.  That object gets updated by various other parts of my program.
Is there any way to pass in a value (i.e., to specify an expression as the directive attribute value) and have the directive auto-update when the expression's value changes?

I see no reason why it wouldn't work. Can you try to create a test case of a plunker reproducing the issue?
Thanks!

Sure: http://plnkr.co/edit/KN5Lbo1ZSOw2y05sz7gj?p=preview

Thanks, now I understand what you were referring to :)
am-time-ago is not supposed to be used with the curly braces, you can write any expression that you like without curly braces, e.g. <time am-time-ago="myCtrl.someFunction()"> should do the trick.
Does this help?

Hi @KurtSteinkraus , I see that you have updated the plunker and it seems to be working there. Do you need any further discussion or can we close the issue?
Thanks!

It helps and fixes the problem I was seeing.  I'm not sure why I put curly braces in the first place.  I don't understand Angular very well... :-/  Thanks!


Convert all typedefs to CamelCase

as requested in https://github.com/openlayers/ol3/pull/5343#issuecomment-220066248

Thanks a lot for this effort @probins.


[SPARK-31511][SQL] Make BytesToBytesMap iterators thread-safe

What changes were proposed in this pull request?
This PR increases the thread safety of the BytesToBytesMap:

It makes the iterator() and destructiveIterator() methods used their own Location object. This used to be shared, and this was causing issues when the map was being iterated over in two threads by two different iterators.
Removes the safeIterator() function. This is not needed anymore.
Improves the documentation of a couple of methods w.r.t. thread-safety.

Why are the changes needed?
It is unexpected an iterator shares the object it is returning with all other iterators. This is a violation of the iterator contract, and it causes issues with iterators over a map that are consumed in different threads.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Existing tests.

Test build #121595 has started for PR 28286 at commit c4d6c7b.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/26275/
Test PASSed.

Thanks, @hvanhovell . The proposed change looks reasonable to me. Is this PR for Apache Spark 3.1.0 as an improvement?

Test build #121595 has finished for PR 28286 at commit c4d6c7b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121595/
Test PASSed.

LGTM too

I encountered a problem with incorrect results in the production environment. After several days of research, I found that the patch can solve the problem.
It is recommended to fix it in the previous version, because before this patch, the result may be incorrect.
@HyukjinKwon @dongjoon-hyun
Before this patch, BytesToBytesMap.iterator was not thread-safe.
Because BroadcastExchangeExec is multi-threaded,
Thread 1 calls BytesToBytesMap.iterator in broadcast UnsafeHashedRelation.
Thread 2 needs to write the UnsafeHashedRelation of thread 1 to disk because of insufficient driver memory, and calls BytesToBytesMap.iterator.
https://github.com/apache/spark/blob/8749b2b6fae5ee0ce7b48aae6d859ed71e98491d/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashedRelation.scala#L368
broadcast-exchange-0
2020-08-27 23:06:00.270000
BytesToBytesMap Addr:1328576284
java.lang.Thread.getStackTrace(Thread.java:1556)
org.apache.spark.unsafe.map.BytesToBytesMap.iterator(BytesToBytesMap.java:418)
org.apache.spark.sql.execution.joins.UnsafeHashedRelation.org$apache$spark$sql$execution$joins$UnsafeHashedRelation$$write(HashedRelation.scala:225)
...
org.apache.spark.broadcast.TorrentBroadcast$.blockifyObject(TorrentBroadcast.scala:277)
org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:126)
org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:88)
org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:56)
org.apache.spark.SparkContext.broadcast(SparkContext.scala:1500)

broadcast-exchange-1
2020-08-27 23:06:00.429000
BytesToBytesMap Addr:1328576284
java.lang.Thread.getStackTrace(Thread.java:1556)org.apache.spark.unsafe.map.BytesToBytesMap.iterator(BytesToBytesMap.java:418) org.apache.spark.sql.execution.joins.UnsafeHashedRelation.org$apache$spark$sql$execution$joins$UnsafeHashedRelation$$write(HashedRelation.scala:225)
org.apache.spark.sql.execution.joins.UnsafeHashedRelation$$anonfun$write$1.apply$mcV$sp(HashedRelation.scala:198)
...

org.apache.spark.storage.DiskStore.put(DiskStore.scala:69)
org.apache.spark.storage.BlockManager.dropFromMemory(BlockManager.scala:1378)
org.apache.spark.storage.memory.MemoryStore.org$apache$spark$storage$memory$MemoryStore$$dropBlock$1(MemoryStore.scala:524)
org.apache.spark.storage.memory.MemoryStore$$anonfun$evictBlocksToFreeSpace$2.apply(MemoryStore.scala:545)
org.apache.spark.storage.memory.MemoryStore$$anonfun$evictBlocksToFreeSpace$2.apply(MemoryStore.scala:539)
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
org.apache.spark.storage.memory.MemoryStore.evictBlocksToFreeSpace(MemoryStore.scala:539)
org.apache.spark.memory.StorageMemoryPool.acquireMemory(StorageMemoryPool.scala:92)
org.apache.spark.memory.StorageMemoryPool.acquireMemory(StorageMemoryPool.scala:73)
org.apache.spark.memory.UnifiedMemoryManager.acquireStorageMemory(UnifiedMemoryManager.scala:179)
org.apache.spark.memory.UnifiedMemoryManager.acquireUnrollMemory(UnifiedMemoryManager.scala:186)
org.apache.spark.storage.memory.MemoryStore.reserveUnrollMemoryForThisTask(MemoryStore.scala:582)
org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:223)
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1039)
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1030)
org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:970)
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1030)
org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:793)
org.apache.spark.storage.BlockManager.putSingle(BlockManager.scala:1351)
org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:122)
org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:88)
org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:56)
org.apache.spark.SparkContext.broadcast(SparkContext.scala:1500)

Thanks for reporting it! I think we should backport this patch according to the bug report from the real world.

Thank you for sharing, @cxzl25

BTW, @cxzl25 . Since it's a correctness bug report, +1 for backporting.
However, in order to backport branch-3.0 and branch-2.4, we need a valid test case for your reporting.
SPARK-31511 was an improvement PR and only was merged for 3.1.0.
cc @gatorsmile

@hvanhovell . Could you switch SPARK-31511 from Improvement to Bug and label correctness, please?

Also, cc @zhengruifeng for 3.0.1 and @ScrapCodes for 2.4.7
Since there is no Jira issue which is marked as Blocker, I guess this is up to the release manager's decision.

@dongjoon-hyun let's just backport it. This is tricky to test, you need multiple consumers of a hash map in different threads that have to go through a fairly large amount of data, and then you might catch it. A test for this is likely to test nothing, and be fairly expensive.

Got it, @hvanhovell . Okay, if you think so.

This is backported to branch-3.0. Could you make a backporting PR for branch-2.4 because there is a conflict?

I added a new fixed version 3.0.2 to SPARK-31511 because 3.0.1 RC3 vote is still alive.

If you want to use sql to reproduce this bug, it is really difficult.
I write a ut to test UnsafeHashedRelation.
I'm not sure if this can help.
org.apache.spark.sql.execution.joins.HashedRelationSuite
  test("SPARK-31511: Make BytesToBytesMap iterators thread-safe") {
    val ser = sparkContext.env.serializer.newInstance()
    val key = Seq(BoundReference(0, LongType, false))

    val unsafeProj = UnsafeProjection.create(
      Seq(BoundReference(0, LongType, false), BoundReference(1, IntegerType, true)))
    val rows = (0 until 10000).map(i => unsafeProj(InternalRow(Int.int2long(i), i + 1)).copy())
    val unsafeHashed = UnsafeHashedRelation(rows.iterator, key, 1, mm)

    val os = new ByteArrayOutputStream()
    val thread1 = new Thread {
      override def run(): Unit = {
        val out = new ObjectOutputStream(os)
        unsafeHashed.asInstanceOf[UnsafeHashedRelation].writeExternal(out)
        out.flush()
      }
    }

    val thread2 = new Thread {
      override def run(): Unit = {
        val threadOut = new ObjectOutputStream(new ByteArrayOutputStream())
        unsafeHashed.asInstanceOf[UnsafeHashedRelation].writeExternal(threadOut)
        threadOut.flush()
      }
    }

    thread1.start()
    thread2.start()
    thread1.join()
    thread2.join()

    val unsafeHashed2 = ser.deserialize[UnsafeHashedRelation](ser.serialize(unsafeHashed))
    val os2 = new ByteArrayOutputStream()
    val out2 = new ObjectOutputStream(os2)
    unsafeHashed2.writeExternal(out2)
    out2.flush()
    assert(java.util.Arrays.equals(os.toByteArray, os2.toByteArray))
  }

before:

after:

Thank you so much, @cxzl25

@cxzl25 can you open a PR to add this test? You can also help with the branch 2.4 backport if you have time. Thanks!

@cxzl25 Can you show what kind of production environment  and use case did you have this problem


Should be able to resize items by resizing current area selection

Version: 0.0.14
Resizing the current area selection should resize the top item it encloses

mostly implemented, but disabled for now since it behaves in unstable way


Improvements in the AWS ECS documentation

Summary
We have identified a few improvements in AWS ECS documentation.

Update the order of the Troubleshooting AWS ECS Deployment section.
Add reference for the Opsgenie service integration with the AWS CloudWatch in Opsgenie Alarms section.
Add rate-limiting cloud formation template in the Deployment section.

Why do we need this ?
...
What is already there? What do you see now?
AWS ECS documentation.
What is missing? What do you want to see?

Update the order of the Troubleshooting AWS ECS Deployment section.

Currently, the Troubleshoot section is in the middle of the other sections. Please move it to the end, i.e., after the Monitoring section.

Add reference for the Opsgenie service integration with the AWS CloudWatch in Opsgenie Alarms section.

Add the following reference to show users how to integrate the Opsgenie service with AWS CloudWatch and get Opsgenie CloudWatch API Key and Opsgenie Alerts Endpoint.
Ref: https://support.atlassian.com/opsgenie/docs/integrate-opsgenie-with-amazon-cloudwatch/

Add rate-limiting cloud formation template in the Deployment section.

Currently, there is no rate-limiting cloud formation template in the deployment section. Please add the rate-limiting template section
How do you propose to document this?
...
Can you do this yourself and submit a Pull Request?
No

Docs should update within an hour.

Additional Scenario:

Database migrations when updating a deployment.
Scenario: A user is updating a deployment from an older version that is 2 patches & 1 minor version behind the latest version. As per release notes, it is mentioned to do database migration of IS in 2 versions. In this case, the user is confused if they need to perform database migrations twice (on each version) or only once (after upgrading to the target version).
Resolution: One-time execution (after upgrading to the target version) is enough for all the schema migrations in between the upgraded versions (v3.a.b to v3.x.y). I.e., the user can upgrade a CloudFormation template to v3.x.y directly and run the database migration once. It will do the necessary database migrations required from v3.a.b to v3.x.y.

@michalborkowski96, please add this as a Note point in the Updating documentation.

A new note section should appear at https://www.thethingsindustries.com/docs/getting-started/aws/ecs/updating/ within an hour


Rename enabled flag to use_cdms_auth

Codecov Report

Merging #258 into develop will not change coverage.
The diff coverage is 100%.

@@           Coverage Diff            @@
##           develop     #258   +/-   ##
========================================
  Coverage    93.35%   93.35%
========================================
  Files           56       56
  Lines         2032     2032
  Branches       189      189
========================================
  Hits          1897     1897
  Misses         114      114
  Partials        21       21

Impacted Files
Coverage Δ

datahub/company/management/commands/manageusers.py
100% <100%> (ø)
:arrow_up:

datahub/company/admin.py
88.88% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 77ba360...5511439. Read the comment docs.