[jvm-packages] Hanging up during "mvn package" test,  And an Error ClassNotFound in spark-cluster

I have deal with xgboost-spark for two weeks. but still can't running it in spark. There's two problem.  First problem is "mvn package" ,  I run it in a centos6.5 machine. It always hang up at spark-tests step.  after two hours,  I have to press ctr-c.
16/07/20 13:30:19 INFO CacheManager: Partition rdd_10_1 not found, computing it
16/07/20 13:30:19 INFO CacheManager: Partition rdd_10_0 not found, computing it
16/07/20 14:00:19 INFO BlockManagerInfo: Removed broadcast_0_piece0 on localhost:35587 in memory (size: 1667.0 B, free: 1087.1 MB)
^C16/07/20 16:59:09 INFO RabitTracker$TrackerProcessLogger: Traceback (most recent call last):
16/07/20 16:59:09 INFO RabitTracker$TrackerProcessLogger:   File "/tmp/tracker5428704971120844141.py", line 480, in
16/07/20 16:59:09 INFO RabitTracker$TrackerProcessLogger:     main()
16/07/20 16:59:09 INFO RabitTracker$TrackerProcessLogger:   File "/tmp/tracker5428704971120844141.py", line 475, in main
16/07/20 16:59:09 INFO RabitTracker$TrackerProcessLogger:     start_rabit_tracker(args)
16/07/20 16:59:09 INFO RabitTracker$TrackerProcessLogger:   File "/tmp/tracker5428704971120844141.py", line 442, in start_rabit_tracker
16/07/20 16:59:09 INFO RabitTracker$TrackerProcessLogger:     rabit.join()
16/07/20 16:59:09 INFO RabitTracker$TrackerProcessLogger:   File "/tmp/tracker5428704971120844141.py", line 332, in join
Another problem occured when I skip tests and run a test program in spark1.5 cluster.  An error called "ClassNotFound:XGBoostError.class"  always arise.  and program hang up too. I think there may be some relation between these two program. So I put them together. I will appreciate If anyone can help me.
16/07/20 17:07:25 WARN ThrowableSerializationWrapper: Task exception could not be deserialized
java.lang.ClassNotFoundException: ml.dmlc.xgboost4j.java.XGBoostError
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:274)
at org.apache.spark.serializer.JavaDeserializationStream$$anon

Uploading ClassNotFound.txt…

Uploading MvnPackagError.txt…
Uploading MvnPackagError.pdf…

I cannot see what happened in mvn package case, it seems that your tracker script cannot be run

@CodingCat  thinks for your reply. There's one things I have forgot to say. The spark-cluster use a python2.6, and xgboost has some code in python2.7. I don't have permission to change the cluster. So I have changed the tracker.py to run in cluster. The Only thing I have changed is "import argparse" --> "import optparse"  and change some code about argparse, I dont think it's matters. Is this problem comes with python2.6 and I have to use python2.7?  thinks a lot.

Logs before task stop.
16/07/21 11:50:38 INFO Client: Application report for application_1468926081807_414233 (state: RUNNING)
16/07/21 11:50:39 INFO Client: Application report for application_1468926081807_414233 (state: FINISHED)
16/07/21 11:50:39 INFO Client:
client token: Token { kind: YARN_CLIENT_TOKEN, service:  }
diagnostics: N/A
ApplicationMaster host: <IP_ADDRESS>
ApplicationMaster RPC port: 0
queue: root.hadoop-waimai.etltest
start time:<PHONE_NUMBER>519
final status: FAILED
tracking URL: http://rz-data-hdp-rm01.rz.sankuai.com:8088/proxy/application_1468926081807_414233/
user: hadoop-waimai
Exception in thread "main" org.apache.spark.SparkException: Application application_1468926081807_414233 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:920)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:966)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:672)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/07/21 11:50:39 INFO ShutdownHookManager: Shutdown hook called
16/07/21 11:50:39 INFO ShutdownHookManager: Deleting directory /tmp/spark-656cdaf9-7adb-477d-9e00-8a9e0ff85366
KILL SPARK APPLICATION:
export HADOOP_JAR_AUTHENTICATION=KERBEROS
export HADOOP_JAR_KERBEROS_KEYTAB_FILE=/etc/hadoop/keytabs/hadoop-launcher.keytab
export<EMAIL_ADDRESS>export HADOOP_PROXY_USER=hadoop-waimai
/opt/meituan/hadoop/bin/yarn application -kill application_1468926081807_414233
16/07/21 11:50:40 INFO client.RMProxy: Connecting to ResourceManager at rz-data-hdp-rm01.rz.sankuai.com/<IP_ADDRESS>:8032
Application application_1468926081807_414233 has already finished
Traceback (most recent call last):
File "/opt/meituan/warehouse/ETL/sqlweaver/../sqlweaver/step.py", line 119, in act
self.actionclass.run(step = self)
File "/opt/meituan/warehouse/ETL/sqlweaver/../sqlweaver/action/base.py", line 60, in run
cls.run(step)
File "/opt/meituan/warehouse/ETL/sqlweaver/../sqlweaver/action/sparkjobsubmit.py", line 43, in run
handler.submit_job(step.template, step.args['submit_user'])
File "/opt/meituan/warehouse/ETL/sqlweaver/../sqlweaver/handler/SparkHandler.py", line 128, in submit_job
raise RuntimeError(msg_f)
RuntimeError: <StringIO.StringIO instance at 0x2dfa9e0>
Traceback (most recent call last):
File "/opt/meituan/warehouse/ETL/sqlweaver/../sqlweaver/engine/koala.py", line 34, in process
step.act()
File "/opt/meituan/warehouse/ETL/sqlweaver/../sqlweaver/step.py", line 128, in act
raise e
RuntimeError: <StringIO.StringIO instance at 0x2dfa9e0>
Fail in Action Step(SparkJobSubmit)

I think it is likely the modification of the tracker introduces some bugs
@tqchen , the second case where python 2.6 brings the issue

I find the reason: one test cant pass. The test is in xgbosot-spark test, called "test with dense vectors containing missing value". I will continue debug this.

it is a problem about python2.6

I actually have this issue with python 2.7.9 on google cloud dataproc cluster, running Debian 8 jessie.
Stops at:
16/08/24 21:23:26 INFO CacheManager: Partition rdd_10_0 not found, computing it
16/08/24 21:23:26 INFO CacheManager: Partition rdd_10_1 not found, computing it

@omdv Can you share how to run xgboost on google cloud dataproc, I got stuck.. Thanks!


Not all displays listed

Hello,
I'm using srandrd-0.6.1 on an xorg-server-<IP_ADDRESS> on a laptop with one internal and one external display. When I call xrandr, I get this:

bash-5.0# xrandr
Screen 0: minimum 8 x 8, current 1920 x 1080, maximum 32767 x 32767
eDP1 connected primary 1920x1080+0+0 (normal left inverted right x axis y axis) 310mm x 170mm
   1920x1080     60.01*+  59.93
   1680x1050     59.88
   1400x1050     59.98
   1600x900      60.00    59.95    59.82
   1280x1024     60.02
   1400x900      59.96    59.88
   1280x960      60.00
   1368x768      60.00    59.88    59.85
   1280x800      59.81    59.91
   1280x720      59.86    60.00    59.74
   1024x768      60.00
   1024x576      60.00    59.90    59.82
   960x540       60.00    59.63    59.82
   800x600       60.32    56.25
   864x486       60.00    59.92    59.57
   640x480       59.94
   720x405       59.51    60.00    58.99
   640x360       59.84    59.32    60.00
DP1 disconnected (normal left inverted right x axis y axis)
DP2 disconnected (normal left inverted right x axis y axis)
HDMI1 connected 1920x1080+0+0 (normal left inverted right x axis y axis) 470mm x 260mm
   1920x1080     60.00*+  60.00    50.00    59.94
   1920x1080i    60.00    50.00    59.94
   1680x1050     59.88
   1280x1024     75.02    60.02
   1440x900      74.98    59.90
   1280x960      60.00
   1280x800      59.91
   1152x864      75.00
   1280x720      60.00    50.00    59.94
   1024x768      75.03    60.00
   832x624       74.55
   800x600       75.00    60.32
   720x576       50.00
   720x576i      50.00
   720x480       60.00    59.94
   720x480i      60.00    59.94
   640x480       75.00    60.00    59.94
   720x400       70.08
HDMI2 disconnected (normal left inverted right x axis y axis)
VIRTUAL1 disconnected (normal left inverted right x axis y axis)
DVI-I-1-1 disconnected (normal left inverted right x axis y axis)

When I call "srandrd list", I only get the internal display:

bash-5.0# srandrd list
eDP1 CF268C7800000004

When I disconnect the external Display, then start srandrd as daemon, and then connect the external display again, I get the correct event and my display handling script is called:

bash-5.0# srandrd -v -n /usr/SUPERSHEEP/bin/dsp_hndl.sh
Event: HDMI1 connected
Time: 3633511
CRTC: 63
Size: 1920x1080
EDID (vendor, product, serial): D109790C00005445

When I now I call "srandrd list", I still get only the internal display listed.
Is there anything obvious I'm doing wrong here?
Thanks and best regards,
Christian

Thanks for the reporting the issue. I don't have any issues on my computer. Would you be willing to dig a little deeper on yours to identify the issue?

Sure, if you tell me where to dig ;)
I already did a little debugging, and I found out that in iter_crts after these calls:

	si = XineramaQueryScreens(dpy, &nscreens);
	mi = XRRGetMonitors(dpy, DefaultRootWindow(dpy), True, &nmonitors);

both nscreens and nmonitors are 1. But I'm not much of an X11 expert, so maybe this is ok?
And I have to add that this happens on a self-tailored linux system, where everything is compiled and configured by myself; so maybe it's a just configuration issue from my side...
Thanks and regards,
Christian

Hi,
any more test I could do?
Thanks,
Christian

Just some comment on how to reproduce this on my end:
$ xrandr
Screen 0: minimum 320 x 200, current 3840 x 1080, maximum 16384 x 16384
eDP connected 1920x1080+1920+0 (normal left inverted right x axis y axis) 309mm x 174mm
   1920x1080     60.05*+
   1680x1050     60.05
   1280x1024     60.05
   1440x900      60.05
   1280x800      60.05
   1280x720      60.05
   1024x768      60.05
   800x600       60.05
   640x480       60.05
HDMI-A-0 disconnected (normal left inverted right x axis y axis)
DisplayPort-0 disconnected (normal left inverted right x axis y axis)
DisplayPort-1 connected primary 1920x1080+0+0 (normal left inverted right x axis y axis) 531mm x 299mm
   1920x1080     59.93*+  60.00    50.00    59.94
   1680x1050     59.88
   1280x1024     60.02
   1280x960      60.00
   1280x720      60.00    50.00    59.94
   1024x768      60.00
   800x600       60.32
   720x576       50.00
   720x480       60.00    59.94
   640x480       60.00    59.94
DisplayPort-2 disconnected (normal left inverted right x axis y axis)
DisplayPort-3 disconnected (normal left inverted right x axis y axis)

Above is two monitors arranged side by side and it works as expected:
$ srandrd list
DisplayPort-1 6D1E56CC0000DA41
eDP AF06403D00000000

If I change them to be on top of each with the following xrandr output
$ xrandr
Screen 0: minimum 320 x 200, current 1920 x 1080, maximum 16384 x 16384
eDP connected 1920x1080+0+0 (normal left inverted right x axis y axis) 309mm x 174mm
   1920x1080     60.05*+
   1680x1050     60.05
   1280x1024     60.05
   1440x900      60.05
   1280x800      60.05
   1280x720      60.05
   1024x768      60.05
   800x600       60.05
   640x480       60.05
HDMI-A-0 disconnected (normal left inverted right x axis y axis)
DisplayPort-0 disconnected (normal left inverted right x axis y axis)
DisplayPort-1 connected primary 1920x1080+0+0 (normal left inverted right x axis y axis) 531mm x 299mm
   1920x1080     59.93*+  60.00    50.00    59.94
   1680x1050     59.88
   1280x1024     60.02
   1280x960      60.00
   1280x720      60.00    50.00    59.94
   1024x768      60.00
   800x600       60.32
   720x576       50.00
   720x480       60.00    59.94
   640x480       60.00    59.94
DisplayPort-2 disconnected (normal left inverted right x axis y axis)
DisplayPort-3 disconnected (normal left inverted right x axis y axis)

srandrd list only outputs the primary monitor.
$ srandrd list
DisplayPort-1 6D1E56CC0000DA41

For me the above case works as expected when dropping breaking the loop - but far from being an expert in X.
$ git diff
diff --git a/srandrd.c b/srandrd.c
index d263573..b0631aa 100644
--- a/srandrd.c
+++ b/srandrd.c
@@ -226,9 +226,6 @@ iter_crtcs(Display * dpy, void (*f) (Display *, char *, char *, int))
                                        f(dpy, info->name, edid, sid);
                                        XRRFreeOutputInfo(info);
                                }
-                               if (i == sid) {
-                                       break;
-                               }
                        }
                }
                XFree(si);

@jceb Does this sound reasonable?

Ah, interesting. Thank you for digging into it! I can see that this makes sense. I'll take a look at it.
If you could continue to test the code with the patch, I'd be very happy. Hope there will be no side effects.

Hi, I had the same issues but much simpler. (This is on a HP all-in-one device, but 'normal' pc and monitor is the same)
Disconnected HDMI or connected HDMI the output shows always the build-in screen only:
$ ./srandrd list
eDP-1 0E222D4000000000
When applying the patch from  RainerKuemmerle it works as expected:
(Connected HDMI)
$ .srandrd list
eDP-1 0E222D4000000000
HDMI-1 2D4C0C3900000000
(Disconnect HDMI)
$ ./srandrd list
eDP-1 0E222D4000000000
PS, this is just trial and error, I have no clue what the original piece code should do ;-)

Thanks for verifying. I applied the patch.

Just FYI: without the patch 'srandrd list' still has issues but with or without the patch 'srandrd display_handler' does receive the correct connected/disconected device.
And with or without the patch the display_handler receives the event twice. Not sure why that is.
So the patch seems to only affect the 'list' command.


blog should be openaccessbutton.org/blog to fit hyperlinks in app

the blog does appear at /blog, so assuming this can be closed.


update mud faucet CLI to use the new faucet

I think our old testnet infra still uses the old gRPC faucet, which is why I've avoided updating this. We haven't opened up the newer testnets/infra to folks yet.

since we're not intending to run a public faucet for Redstone Holesky, I wonder if we should just remove the faucet command?
or are we going to have a Redstone Devnet?

I removed the faucet command/usage for now: https://github.com/latticexyz/mud/pull/2811
I wanna rethink our faucet API (i.e. not trpc) before we expose it more broadly.


Illegal reflective access operation with Java 11

Yes, still making use of this excellent tool, but have now moved to Java 11. Working fine, but may not on newer than 11.
PackUpdateUpdater Version: 3.0
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by at.chaosfield.updaterupdater.UpdaterClassLoader (file:/PackUpdateUpdater.jar) to method java.net.URLClassLoader.addURL(java.net.URL)
WARNING: Please consider reporting this to the maintainers of at.chaosfield.updaterupdater.UpdaterClassLoader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

This is a bug affecting the self-updater. I have pushed an update, can you please check with the latest release from https://github.com/XDjackieXD/PackUpdateUpdater/releases? You can do this by either manually swapping out the self-updater or by generatring a new instance zip file.

That warning appears to have been fixed, and things are working fine, I'm seeing this now though.

Client, Server or Generate?

This is on Windows 10, on a test server I have set up.
I put the new version on my production server that runs Debian Linux headless and I don't see it there.

Ah, this is a windows specific thing.
Windows does not support unix style ttys, and appachently jline does not work well with windows. There is no real issue, except that progress bars may look ugly. I may look into this in the future, but its probably not worth it.
I'll keep the issue open for now, but i doubt many servers run on windows. Also you can tell packupdate to use the gui used on the client for the server side as well, you might want to do that on windows.

Doesn't bother me any, just something I noticed on my test server, feel free to close.


create-react-app  failed.

Is this a bug report?
(write your answer here) Yes.

Did you try recovering your dependencies?
Yes.

(Write your answer here.)
node v10.14.1
npm  v6.5.0
Which terms did you search for in User Guide?

(Write your answer here if relevant.)
Environment

(paste the output of the command here)
Environment Info:
(node:8032) UnhandledPromiseRejectionWarning: Error: The system cannot find the path specified.
at Function.e.exports.sync (C:\Users\Xiaozhou Huangfu\AppData\Roaming\npm\node_modules\create-react-app\node_modules\envinfo\dist\envinfo.js:1:7778)
at Object.copySync (C:\Users\Xiaozhou Huangfu\AppData\Roaming\npm\node_modules\create-react-app\node_modules\envinfo\dist\envinfo.js:1:104976)
at Object.t.writeSync.e [as writeSync] (C:\Users\Xiaozhou Huangfu\AppData\Roaming\npm\node_modules\create-react-app\node_modules\envinfo\dist\envinfo.js:1:123499)
at C:\Users\Xiaozhou Huangfu\AppData\Roaming\npm\node_modules\create-react-app\node_modules\envinfo\dist\envinfo.js:1:124274
at Promise.all.then.e (C:\Users\Xiaozhou Huangfu\AppData\Roaming\npm\node_modules\create-react-app\node_modules\envinfo\dist\envinfo.js:1:124289)
at process._tickCallback (internal/process/next_tick.js:68:7)

(node:8032) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:8032) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Steps to Reproduce

(Write your steps here:)

create app through create-react-app and npm install
npm run build

Expected Behavior

(Write what you thought would happen.)
Successful Build
Actual Behavior

(Write what happened. Please add screenshots!)
failed to compile.

log:
0 info it worked if it ends with ok
1 verbose cli [ 'C:\Program Files\nodejs\node.exe',
1 verbose cli   'C:\Users\Xiaozhou Huangfu\AppData\Roaming\npm\node_modules\npm\bin\npm-cli.js',
1 verbose cli   'run',
1 verbose cli   'build' ]
2 info using<EMAIL_ADDRESS>3 info using<EMAIL_ADDRESS>4 verbose run-script [ 'prebuild', 'build', 'postbuild' ]
5 info lifecycle<EMAIL_ADDRESS><EMAIL_ADDRESS>6 info lifecycle<EMAIL_ADDRESS><EMAIL_ADDRESS>7 verbose lifecycle<EMAIL_ADDRESS>unsafe-perm in lifecycle true
8 verbose lifecycle<EMAIL_ADDRESS>PATH: C:\Users\Xiaozhou Huangfu\AppData\Roaming\npm\node_modules\npm\node_modules\npm-lifecycle\node-gyp-bin;C:\Users\Xiaozhou Huangfu\Documents\GitHub\scdy\node_modules.bin;C:\Users\Xiaozhou Huangfu\AppData\Roaming\npm;C:\Program Files\nodejs;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Windows\System32\OpenSSH;C:\Program Files (x86)\bin;C:\Program Files\nodejs;C:\Users\Xiaozhou Huangfu\AppData\Local\Microsoft\WindowsApps;C:\Users\Xiaozhou Huangfu\AppData\Local\GitHubDesktop\bin;C:\Users\Xiaozhou Huangfu\AppData\Local\Yarn\bin;C:\Users\Xiaozhou Huangfu\AppData\Roaming\npm
9 verbose lifecycle<EMAIL_ADDRESS>CWD: C:\Users\Xiaozhou Huangfu\Documents\GitHub\scdy
10 silly lifecycle<EMAIL_ADDRESS>Args: [ '/d /s /c', 'react-scripts build' ]
11 silly lifecycle<EMAIL_ADDRESS>Returned: code: 1  signal: null
12 info lifecycle<EMAIL_ADDRESS>Failed to exec build script
13 verbose stack Error<EMAIL_ADDRESS>build: react-scripts build
13 verbose stack Exit status 1
13 verbose stack     at EventEmitter. (C:\Users\Xiaozhou Huangfu\AppData\Roaming\npm\node_modules\npm\node_modules\npm-lifecycle\index.js:301:16)
13 verbose stack     at EventEmitter.emit (events.js:182:13)
13 verbose stack     at ChildProcess. (C:\Users\Xiaozhou Huangfu\AppData\Roaming\npm\node_modules\npm\node_modules\npm-lifecycle\lib\spawn.js:55:14)
13 verbose stack     at ChildProcess.emit (events.js:182:13)
13 verbose stack     at maybeClose (internal/child_process.js:962:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:251:5)
14 verbose pkgid<EMAIL_ADDRESS>15 verbose cwd C:\Users\Xiaozhou Huangfu\Documents\GitHub\scdy
16 verbose Windows_NT 10.0.17134
17 verbose argv "C:\Program Files\nodejs\node.exe" "C:\Users\Xiaozhou Huangfu\AppData\Roaming\npm\node_modules\npm\bin\npm-cli.js" "run" "build"
18 verbose node v10.14.1
19 verbose npm  v6.5.0
20 error code ELIFECYCLE
21 error errno 1
22 error<EMAIL_ADDRESS>build: react-scripts build
22 error Exit status 1
23 error Failed at the<EMAIL_ADDRESS>build script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]
package.json:
{
"name": "scdy",
"version": "0.1.0",
"private": true,
"dependencies": {
"axios": "^0.18.0",
"html-entities": "^1.2.1",
"react": "^16.6.0",
"react-dom": "^16.6.0",
"react-icons": "^3.2.2",
"react-router-dom": "^4.3.1",
"react-scripts": "2.1.1"
},
"scripts": {
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test",
"eject": "react-scripts eject"
},
"eslintConfig": {
"extends": "react-app"
},
"browserslist": [
">0.2%",
"not dead",
"not ie <= 11",
"not op_mini all"
]
}
Reproducible Demo

(Paste the link to an example project and exact instructions to reproduce the issue.)
https://github.com/xhf394/scdy

I've cloned your project, installed dependencies using npm install and was able to get your project up and running. Therefore, your node/npm installation might wonky. Have you tried reinstalling it, maybe try nvm-windows.
Would love to help, but I'm not an expert in developing on windows machines. 😅

Thanks for helping. Find a pseudo-class issue in css file. add one more space caused error.

Thanks a lot for helping. Yes. I tried reinstall and ran it under yarn and finally found out an error in css file. compiled successfully now. Thanks!!
Xiaozhou
| |
Xiaozhou Huangfu
|
|
<EMAIL_ADDRESS>|
签名由网易邮箱大师定制
On 12/11/2018 17:00，Julian<EMAIL_ADDRESS>wrote：
I've cloned your project, installed dependencies using npm install and was able to get your project up and running. Therefore, your node/npm installation might wonky. Have you tried reinstalling it, maybe try nvm-windows.
Would love to help, but I'm not an expert in developing on windows machines. 😅
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Glad you figured it out, I've misread your post first and thought you were having trouble when running npm run start. Cheers! 🍻


chrome.runtime is not a function.....

screen-sharing apps need chrome.runtime function.
it seems not exist.
anyone knows how to add this feature?

As per https://www.magpcss.org/ceforum/viewtopic.php?f=6&t=17042 this function is currently not implemented.
CEF has limited support for screen sharing to enable see https://www.magpcss.org/ceforum/viewtopic.php?f=6&t=16952#p42886
You won't be able to use any extensions that reply on chrome.runtime until it's implemented, you can subscribe to https://bitbucket.org/chromiumembedded/cef/issues/1947/ for updates.


Tutorials page calendar slots do not point to individual tutorial pages

@jcklie  I was trying to find what change in this commit  caused this. But I am not able to find it. Can you look into this?

I think I found the solution for this. I am submitting a PR.

Submitted a PR for this https://github.com/acl-org/acl-2020-virtual-conference/pull/241


Bring back new monthly quota fixtures

What is the purpose of this Pull Request?
As I said in #107, I merged @rennerocha PR without the lasts fixtures in order to add its structure so it would be better for me to add again.
What was done to achieve this purpose?
Bring back the fixtures that I had remove, with its corresponding tests.
How to test if it really works?

Install Rosie
Run its tests:

$ python rosie.py test

Who can help reviewing it?
@cuducos
Related to #107

And bring coveralls back to green ;)


Test that char sequences can be read in multiple encodings

The bug occurs when the default character set is different than the specified character set. Since we don't know the default character set in which the test runs a priori, test two incompatible character sets against each other. They can't both be the default character set.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (64f321b) 85.95% compared to head (06cbe86) 85.99%.
Report is 1 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master     #543      +/-   ##
============================================
+ Coverage     85.95%   85.99%   +0.03%
- Complexity     3478     3479       +1
============================================
  Files           230      230
  Lines          8223     8223
  Branches        960      960
============================================
+ Hits           7068     7071       +3
+ Misses          865      863       -2
+ Partials        290      289       -1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Hi @elharo
OK to close then?


Add integration test to verify workflow of redis with OpenEBS

Is this a BUG REPORT or FEATURE REQUEST?
FEATURE REQUEST

What happened:
OpenEBS can be used as a Persistent Volume for Redis. The same can be setup using the YAML file provided at https://github.com/openebs/openebs/tree/master/k8s/demo/redis
What you expected to happen:
Implement a test suite in ansible to deploy redis standalone & redis cluster on Kubernetes using OpenEBS as volume.

Explore using the redis operator.

The OpenEBS-specific app deployment litmusbooks will be developed & maintained at https://github.com/mayadata-io/litmus


[Spark] Adding support for partition pruning in vacuum command

Which Delta project/connector is this regarding?

[x] Spark
[ ] Standalone
[ ] Flink
[ ] Kernel
[ ] Other (fill in here)

Description

Currently, users have large tables with daily/hourly partitions for many years, among all these partitions only recent ones are subjected to change due to job reruns, corrections, and late arriving events.

When Vacuum is run on these tables, the listing of files is performed on all the partitions and it runs for several hours/days. This duration grows as tables grow and vacuum becomes a major overhead for customers especially when they have hundreds or thousands of such delta tables. File system scan takes the most amount of time in Vacuum operation for large tables, mostly due to the parallelism achievable and API throttling on the object stores.

This change does a parallel listing of partitions directories (not files) using partition column info from the delta log manifest. Then constructs a data frame to apply the filtering clause. The partition predicate filtering is only supported for the latest partition structure and when there is a change in partitioning user should run a normal vacuum first.

The parallel listing of partitions also helps default vacuum as it distributes file listing when there are too many partitioning columns.

Design Doc : https://docs.google.com/document/d/1vRTfMk3bRmCWLa-E4W-UaNOgFo_DyFCcCMVjB1GzrcU/edit?usp=sharing

If this PR resolves an issue be sure to include "Resolves #1691" to correctly link and close the issue upon merge.
How was this patch tested?

Unit Testing  ( build/sbt 'testOnly org.apache.spark.sql.delta.DeltaVacuumSuite')

SQL-based Vacuum on a table using complex predicates. eg: vacuum db.table where v1=20 and v2>=3
Vacuum (with filter) on a non-partitioned table should return Delta Vacuum filtering clause expects partition predicates and only supports partitioned tables
Vacuum on the partitioned table with a non partition predicate should fail with the Predicate references non-partition column Error

End  to End Test using Spark 3.4.1/3.3.1 on S3A FS (30 Executors with 25GB memory and 4 Cores)

Created a test delta table partitioned by 4 columns (year, month, day, hour). Added 10K mock files to each of the 48 partitions (2 days of data). Then executed vacuum dry run using
Delta 2.1 and Spark 3.3.1

Scanning ~480K files
time taken:<PHONE_NUMBER>50 nanoseconds (~ 2.37 Minutes)

Delta 3 (this patch) and Spark 3.4.1 without partition filer predicate

Scanning ~480K files
time taken:<PHONE_NUMBER>7 nanoseconds (~ 0.69 Minutes)

Delta 3 (this patch) and Spark 3.4.1 with partition filtering of 1 partition

Scanning ~10K Files
time taken:<PHONE_NUMBER> nanoseconds (~ 0.139 Minutes)

Does this PR introduce any user-facing changes?
yes, the MR accepts an optional partition predicate filter from the user (similar to Optimize command).
VACUUM table_name [WHERE partition_predicate] [RETAIN num HOURS] [DRY RUN]
eg:  VACUUM test_db.table where year||month >= '202307' RETAIN 168 HOURS dry run

@rajeshparangi can you take a look?

I will be out of office for two weeks. I will be able to take a look once I come back.

I worry that the design for VACUUM with partition filters is incompatible with the Delta Lake protocol spec. The problem is that Delta Lake does not require the files to be distributed according to the traditional Hive style partitioning. So in theory you might have a file in directory date=2023-09-26 that is actually for partition date=2023-09-25 according to the Delta log (which is the source of truth). This design seems to require that partitions actually follow that naming scheme. Because otherwise you might delete files from a directory date=X that are actually referenced by the table but in another partition with date Y. There are also implementations that don't follow the hive naming scheme at all.
That said, the parallelization of file listing is a good idea! If we can separate that out, then it can be considered separately. Does that make sense?

@bart-samwel  thank you for your comments. I understand that the delta tables need not always follow hive-style partitioning. So far my understanding was this would happen in the following cases and I believe that it is handled in the code.

Random Prefix Tables: This was not available in OSS delta and was ignored
When Partition columns are renamed: The above logic gets the logical to physical folder name mapping from delta log and I believe it is handled.
When Tabled are liquid partitioned: The liquid partitioned tabled don't have partition column spec in delta log and hence would not be able to use this feature and has graceful exit handled in the code.

It would be useful if you could point me to other cases where

physical partitions like hive style will be missing

in theory you might have a file in directory date=2023-09-26 that is actually for partition date=2023-09-25 according to the Delta log (which is the source of truth)

Finally to answer you about the deletion of files from another directory (>you might have a file in directory date=2023-09-26 that is actually for partition date=2023-09-25 according to the Delta log ). The source of truth for all valid files are delta log and there are no filters applied while getting this dataframe. The filter is only applied to scan for all files in the table. So the subsequent left-anti join all files data frame with valid files data frame would protect from deletion of any file referenced in delta log (the source of truth)

That said, the parallelization of file listing is a good idea! If we can separate that out, then it can be considered separately. Does that make sense?

if there are still concerns about the filtering and improving the parallel listing could address or improve the performance however it won't address the long-term API cost for maintaining the delta lake.
Do you think my alternate proposal (in the design doc attached) for using cloud storage inventory would be acceptable to get around the API cost if we always have to do full listing?

@arunravimv

@bart-samwel thank you for your comments. I understand that the delta tables need not always follow hive-style partitioning. So far my understanding was this would happen in the following cases and I believe that it is handled in the code.

Random Prefix Tables: This was not available in OSS delta and was ignored
When Partition columns are renamed: The above logic gets the logical to physical folder name mapping from delta log and I believe it is handled.
When Tables are liquid partitioned: The liquid partitioned tables don't have partition column spec in delta log and hence would not be able to use this feature and has graceful exit handled in the code.

It would be useful if you could point me to other cases where
physical partitions like hive style will be missing

I think you've probably covered the current cases pretty well. The problem is that these are current cases, but we need to consider all cases, including cases from the future that we don't know about yet but that do produce valid Delta tables.

in theory you might have a file in directory date=2023-09-26 that is actually for partition date=2023-09-25 according to the Delta log (which is the source of truth)

Finally to answer you about the deletion of files from another directory (>you might have a file in directory date=2023-09-26 that is actually for partition date=2023-09-25 according to the Delta log ). The source of truth for all valid files are delta log and there are no filters applied while getting this dataframe. The filter is only applied to scan for all files in the table. So the subsequent left-anti join all files data frame with valid files data frame would protect from deletion of any file referenced in delta log (the source of truth)

So it would simply limit the deletions to the set of files in this folder, but it would consider the entire log, including other partitions? That sounds correct!
I think the trouble is that if we add this new VACUUM syntax, it looks like logical syntax that operates on the logical partitions of the table. As a user, I would expect this to work on the logical partitions, and I'd expect it to always work, and not only if the table happens to be written by particular clients with particular settings.

That said, the parallelization of file listing is a good idea! If we can separate that out, then it can be considered separately. Does that make sense?

If there are still concerns about the filtering and improving the parallel listing could address or improve the performance however it won't address the long-term API cost for maintaining the delta lake.
Do you think my alternate proposal (in the design doc attached) for using cloud storage inventory to avoid listing of files would be acceptable to get around the API cost if we always have to do full listing?

FWIW I did the math on this. LIST costs $0.005 per 1000 requests. Each request returns 1000 objects. For a bucket with 100M objects, this costs $0.005 / 1000 * (100M / 1000) = $0.50.  The equivalent Inventory costs $0.0025 per 1M objects listed, so it costs $0.25. So the difference is $0.25 per day. For a daily vacuum, I don't think that that price difference is worth this type of investment?
I think it's not about API cost. If you're running this from within the cloud, then the VM cost of doing the LISTing is probably the more expensive part. If you use Inventory, then you can start VACUUMing much sooner because you don't have to wait for the LISTing to return. Could we get LIST to be fast enough so that we don't need this extra complexity? Because inventory is hard to use. At standard S3 throttling rates (3500 requests per second), LIST for a 100M objects bucket will take at least 100M / 1000 / 3500 = 28 seconds and $0.25.  But if you do this serially, then it's going to take a lot of time indeed. :)
I think we can come up with an algorithm that lists the entire key space in parallel without knowing anything about partitions or how the files are distributed. How about something that divides and conquers the LISTing space?
E.g.:
Keep a data structure that lists all the ranges that we divided the file name space into, and how far along we are with listing them. I.e., fields for startKey, endKey, lastSeenKey.

Start with [(startKey = "", endKey = None, lastListedKey = None)] where "" sorts before everything and None means the end key that sorts after everything.
Repeatedly pick a range from the list. List the files from lastListedKey (with a limit -- up to 1000 keys, one LIST request) and collect all the files you see that are before endKey. If you hit a value past the endKey (or the actual end of the directory), the range is completely listed. Remove it from the list.
If the selected range was incomplete, cut the remaining space between lastListedKey and endKey in half. (See below for how.) Then split the range into two new ranges at that key. Replace the range in the data structure by the two new ranges.
Repeat 2-3 until the data structure is empty.

You can do steps 2-3 in parallel for many ranges.
So how do you cut the ranges in half? That's the trickiest bit.
E.g. if you have lastListedKey = "ab-def", endKey = "ab-h", then you first have to cut off the common prefix "ab-". The next characters are "d" and "h". You can then average the character codes, so d, e, f, g, h -> f is in the middle. So then the middle key is "ab-f".
If you have lastListedKey = "ab-de", endKey = "ab-def" then the common prefix is "ab-de" and the splitting is between suffix "" and "f". In that case it's the middle character between the starting point and f. Since f has ASCII code 102, and the first usable code is 32, that would be something like ASCII 66 (capital B).
If you have lastListedKey = "ab-dee", endKey = ab-def", then the common prefix is "ab-de" and the splitting is between suffix "e" and "f". These characters are adjacent. So instead you have to append a character to the suffix, so "eP" or something (P is ASCII 80 which is midway between space (32) and the max regular character (127).
Would that work? It might take some extra LIST calls to figure out which ranges are empty vs which ranges are interesting... We could seed it with some interesting values, like a sample of the partition directories, or implementations that use random prefixes could seed them with some of the random prefixes.
Here's a much more stupid but practical idea: what if we list all the files that are currently in the log, and assume that the to-be-deleted files are distributed similarly? Then we sample 1 in every 100K files, we sort the file paths, and then we cut the space into ranges at that boundary. We start listing in parallel from each of these starting points (and from the start), and for each of them we list until we hit the next key or the end of the directory.
WDYT?

@bart-samwel, thank you for your quick response. I do agree that the above change relies only on currently available features and mostly revolves around tables generated through the spark client. In our situation, we are also kind of not 100% satisfied with the need to keep track of logical/physical partitions when they would be updated, and when they should be cleaned up. The above tests without filtering predicate and just improving the listing parallelism reduces the run time significantly.
The listing of files will be significant for us because at our scale (we are still in the early days of lakehouse adoption), we have 100s if not 1000s of delta tables (and growing) with 100s Millions of objects. We plan to run automated maintenance of tables for our users and Inventory reports can be reused across the runs and across the tables helping save a lot every month.

@arunravimv

The listing of files will be significant for us because at our scale (we are still in the early days of lakehouse adoption), we have 100s if not 1000s of delta tables (and growing) with 100s Millions of objects. We plan to run automated maintenance of tables for our users and Inventory reports can be reused across the runs and across the tables helping save a lot every month. However, the challenges are not all clouds have them nor there are any standard across the clouds. So this could be a cloud-specific feature, what do you think? Or should we improve the parallelism logic first and then restart the conversation around the use of inventory reports?

If you're going for 100 Ms of objects per table and then 100+ Delta tables, then it could indeed add up. I hadn't thought about the idea of reusing inventory reports -- would it work? You still pay per 1M objects listed, and you can only choose between daily and weekly. If you do daily, and you reuse it the next day for VACUUMing the same table, then the files that are supposed to be deleted are definitely in that list -- anything that was added in the last day isn't going to be deleted. Actually, a weekly inventory would do the trick, because the to-be-deleted data is always at least 7 days old (assuming the default 7 day retention threshold).
I don't mind adding a cloud-specific feature, but I don't know what the rest of the community thinks -- I don't know if there's precedent for this kind of thing. Maybe some other folks can chime in here as well...
But I would also definitely love to see some kind of LIST speed improvements, especially if they're not dependent on particular partitions or file layouts! That should be totally not contentious. :)

@arunravimv

Actually, a weekly inventory would do the trick, because the to-be-deleted data is always at least 7 days old (assuming the default 7 day retention threshold).

Wait, but how would you track which files you had already deleted, if you VACUUM daily? You might be billed for a lot of failed duplicate DELETE requests.

@arunravimv Don't we parallelize listing already? Take a look at the recursiveListDirs implementation. It first does listUsingLogStore with recursive set to false so that it gets top level directories and create a dataset object. And then second iteration of list is done with recursive set to true. The second part is parallelized through mapPartitions interface.

@arunravimv Don't we parallelize listing already? Take a look at the recursiveListDirs implementation. It first does listUsingLogStore with recursive set to false so that it gets top level directories and creates a dataset object. And then second iteration of list is done with recursive set to true. The second part is parallelized through mapPartitions interface.

Hi @rajeshparangi , we have cases where top-level partition folder have fewer cardinality eg yyyy/mm/dd/hh pattern causing performance issues for large tables that have just one month retention of data.

@arunravimv

Actually, a weekly inventory would do the trick, because the to-be-deleted data is always at least 7 days old (assuming the default 7 day retention threshold).

Wait, but how would you track which files you had already deleted, if you VACUUM daily? You might be billed for a lot of failed duplicate DELETE requests.

@bart-samwel we prepare an inventory of delta/data lake objects on a daily basis and hence each day's management jobs will be run against the latest snapshot of inventory.

@arunravimv Don't we parallelize listing already? Take a look at the recursiveListDirs implementation. It first does listUsingLogStore with recursive set to false so that it gets top level directories and creates a dataset object. And then second iteration of list is done with recursive set to true. The second part is parallelized through mapPartitions interface.

Hi @rajeshparangi , we have cases where top-level partition folder have fewer cardinality eg yyyy/mm/dd/hh pattern causing performance issues for large tables that have just one month retention of data.

I see. In that case, can we continue to collect top-level directories until we reach a threshold (say 200 as that is the parallelism we use) and then do file level listing. This should cover your scenario, right?

@arunravimv Don't we parallelize listing already? Take a look at the recursiveListDirs implementation. It first does listUsingLogStore with recursive set to false so that it gets top level directories and creates a dataset object. And then second iteration of list is done with recursive set to true. The second part is parallelized through mapPartitions interface.

Hi @rajeshparangi , we have cases where top-level partition folder have fewer cardinality eg yyyy/mm/dd/hh pattern causing performance issues for large tables that have just one month retention of data.

I see. In that case, can we continue to collect top-level directories until we reach a threshold (say 200 as that is the parallelism we use) and then do file level listing. This should cover your scenario, right?

This could improve the speed/performance. @rajeshparangi How do you suggest we do the collection if it's not partition spec-based (without introducing any skewness)? Also, this does not address the cost concerns which is why we were inclined lately to use inventory reports to get around file listing.

@arunravimv

Actually, a weekly inventory would do the trick, because the to-be-deleted data is always at least 7 days old (assuming the default 7 day retention threshold).

Wait, but how would you track which files you had already deleted, if you VACUUM daily? You might be billed for a lot of failed duplicate DELETE requests.

@bart-samwel we prepare an inventory of delta/data lake objects on a daily basis and hence each day's management jobs will be run against the latest snapshot of inventory.

That should work. I think that it would be good to add this feature!
I also think that we should keep it cloud agnostic. So here's what I would propose:

Add syntax to be able to supply a files list, as a hint that the command may use.
This would ideally be just a simple list of file paths, one file per line. Could be relative or absolute paths. Converting from the cloud-specific format to something that VACUUM can use could be a code snippet that we publish somewhere else, but we shouldn't be trying to add specific support for all clouds. I saw that Azure and Google Cloud Platform also have an equivalent of this, and if we just support file lists, then we can support everything!
The implementation uses the file list, but filters it down using the actual prefix that it uses for listing. We have to prevent situations where users pass in a file that is outside the table, or crucial for the table metadata, and it accidentally gets deleted. :)

Would that work?

@bart-samwel yeap, I agree that we should keep the cloud-specific inventory format to list-of-files format (that we expect) conversion outside of the delta codebase (Maybe in Docs) and then provide the list of files as input to the vacuum command.  I think instead of giving the list of files directly in the vacuum command (because there may be too many files) line by line, probably we can pass it as a Parquet file or as a subquery data frame. Something like
vacuum from db.table retain X hours using inventory select filename, modification_time from inventory
or
vacuum from db.table retain X hours using inventory <path-to-parquet-file>
What do you think? If you agree I will try to create a design doc for review and raise a feature MR.

@bart-samwel yeap, I agree that we should keep the cloud-specific inventory format to list-of-files format (that we expect) conversion outside of the delta codebase (Maybe in Docs) and then provide the list of files as input to the vacuum command. I think instead of giving the list of files directly in the vacuum command (because there may be too many files) line by line, probably we can pass it as a Parquet file or as a subquery data frame. Then we apply safety checks on this data frame to avoid accidental deletion of files (using base path check, hidden file/folder check)
Something like
vacuum from db.table retain X hours using inventory select filename, modification_time from list_of_files
or
vacuum from db.table retain X hours using inventory <path-to-parquet-file>
What do you think? If you agree I will try to create a design doc for review and raise a feature MR.

I like it! Using a dataframe would be the most idiomatic Spark -- then the user can use an arbitrary source. (Or if we do want a file, it should of course be a Delta table, not a Parquet file. :) )
As for syntax, how about:
VACUUM FROM db.table RETAIN x HOURS USING INVENTORY FROM (query)
I'd like the parentheses around the query so that we can still add extra clauses to VACUUM at the end later. We may want to debate the "INVENTORY" keyword if it's too AWS specific.

@bart-samwel , sorry for the delay I have opened a new MR to pass list of files as a delta table or subquery please take a look.
https://github.com/delta-io/delta/pull/2257/files


Custom TitleBar is not work well in RTL

Describe the bug
Custom title bar doesn't work well in Right To Left layout
I used the code from the documentation
Below you can see the reaction of the title bar when layout is right to left with different methods:

Windows.Globalization.ApplicationLanguages.PrimaryLanguageOverride = "fa-IR";

As you can see, the title does not stick to the right side and is as far as the CaptionButtons

2.FlowDirection="RightToLeft"
As you can see, CaptionButtons and title are placed in each other

using NativeMethods WS_EX_LAYOUTRTL and SetWindowLong(hWnd, GWL_EXSTYLE, exstyle | WS_EX_LAYOUTRTL)

The drag regions are not correct and the title is still on the left side

using NativeMethods WS_EX_LAYOUTRTL and SetWindowLong(hWnd, GWL_EXSTYLE, exstyle | WS_EX_LAYOUTRTL) and FlowDirection="RightToLeft"

Steps to reproduce the bug
create a winui 3 project + full titlebar customization
change your app to RTL layout with 3 methods
Expected behavior
Caption buttons should be on the left side, title and icon should be on the right side, Also, the title and icon should be placed on the right without spaces [see first image]
Screenshots
No response
NuGet package version
Windows App SDK 1.3.2: 1.3.230602002
Packaging type
Packaged (MSIX), Unpackaged
Windows version
Windows 11 version 22H2 (22621, 2022 Update)
IDE
Visual Studio 2022
Additional context
No response

First of all, does setting WS_EX_LAYOUTRTL and FlowDirection="RightToLeft" at the same time work?
There is some fun quirkiness here, but WS_EX_LAYOUTRTL means that the frame is mirrored but the content doesn't have to be and FlowDirection="RightToLeft" means that the content is mirrored but the frame doesn't have to be. This is especially true since one of your previous bug reports showed that Xaml wasn't properly working with the underlying window. FrameworkElement.FlowDirection does state that the default is LeftToRight after all.

@DarranRowe
No, as you can see, title is far from the right side

The bug I reported earlier was in a very bad state, The title bar and caption buttons were not working. They fixed this bug .
But here we have a space on the right!

I would suggest using winappsdk 1.4 preview and see if it covers some of it. RTL in custom titlebar is still a in-progress issue but with merger with appwindow titlebar, 1.4 custom titlebar should be doing most of the heavy lifting of RTL display. can you test it out and let me know the experience ? Thanks


AWS Kubernetes Integration Doc

Complete guide for configuring an existing AWS-Kube cluster to use Calico
Modify existing Kube docs for clarity, accuracy

Just have to run through the commands on the minion and confirm it is working.

Done with this round of comments.  I'll take one more quick look when you're done addressing.

Back to you

One more comment you still need to address.

Ok - waiting on the calico-docker release now.

LGTM


No longer require a Procfile for Godeps and Govendor

This makes the go buildpack run the generated main package binary name to start the web worker if no  procfile is present and no start command is specified

Thanks for your interest!
Please see my comments.
Perhaps it would be better to look into bin and if there is only a single executable, use that instead, assuming there is no Procfile.

Also from our docs:
Of course, rather than using default_process_types, it’s simpler to just write a default Procfile if one isn’t provided.
I'd rather do that TBH.

Okay, I agree with the generating a default Procfile rather than changing bin/release point. I'll update the PR to reflect that.

Updated this PR and addressed the comments.

I am the primary maintainer and I was out on vacation. I'll look at this more soon.

:+1:

Thanks for the contribution!


pretrained model

Hi, where can I find siammaske_r50_l3/model.pth?
10x

Hi Shiragit,
Please find it from the pysot project.


Adding blogs in the blogs section

I want to add some blogs in the form of cards, user can click and read that particular blogs, please assign this to me

Hey @trisharaj06 , thanks for stepping up. Actually we are creating the blogs using hashnode. SO we won't be able to give access to everyone. If there is anything else in which u would like to work on ,pls go ahead:)


Value generation error in case of inheritance

Hello,
I have two entities: Contract and ContractState, configured as follows:
Contract -> Has many(List[ContractState]).Has one(Contract).With foreign key(Guid ContractId)
ContractState is abstract and has three concrete type inheriting it: ContractStateCease, ContractStateDetach, ContractStateSuspend. OnModelCreating sets it like following:
modelBuilder.Entity()
.Discriminator()
.HasValue(typeof(ContractStateCease), "ContractStateCease")
.HasValue(typeof(ContractStateDetach), "ContractStateDetach")
.HasValue(typeof(ContractStateSuspend), "ContractStateSuspend");
But i receive error:
InvalidOperationException: The property 'ContractId' on entity type 'ContractState' has value generation on add enabled and it is included in the foreign key {'ContractId'}. Foreign key properties should not have value generation on add enabled unless they are part of a cycle where they are the only properties with value generation on add enabled.
The error appeared in beta8, and worked in beta 7.
I tried:
modelBuilder.Entity()
.Property(cs => cs.ContractId)
.ValueGeneratedNever();
but no effect.
Any ideas?
Thanks, Ion

@smitpatel can you verify this is fixed after @AndriySvyryd checks in the conventions work.

Is possible to use another inheritance pattern, at least until the bug is solved? this bug is blocking for me now...

There are no other inheritance patterns supported by EF7 at this moment. The fix is currently being reviewed and should soon be available in nightly build.

Thanks, Andriy! I will wait for fix!

Verified this with latest code that this generates a valid model.

Hello,
It works now, but I receive another error which seems to be related. I described here: #3584


mathics: Fix unit tests

Disabled two tests that require a preinstalled build of Mathics to work, which is incompatible with the way nix runs them.  They're also not the most important tests in the world.

Any objections? Does this look ok?

IMHO, it would be nicer if the "fix unit tests" commit just fixed the tests, instead of moving the package between files at the same time.

I figured if I was creating a directory to put the patch in, I might as put the derivation in there too. I'll put it back if you want.

Hm, didn't think of that. I think I'd still go for minimal diff needed to fix unit tests. That is, put the patch where you have it now and just add the "patches = [...]" line to the existing derivation. If you still want to move it around, do that in a second commit.

done - I'll just leave it in python-packages.nix and add the patch to get tests passing.

Pushed to master (ae9bd823edbf936b652825c251343a6845b9ae38).

I removed disabling on Python 3, because that's unrelated to the unit test fixing.

Welp.  It doesn't work on python 3, so now there are still hydra builds failing for it.

Where are the broken builds?
It seems to me that there is a dependency of mathics that already prevents building with python 3, so adding explicit disabling in mathics will not change anything regarding Hydra (right now).
If mathics itself is incompatible with py3, it is of course cleaner to explicitly disable.

mathics explicitly only supports python 2.6 and 2.7 in the current release: https://github.com/mathics/Mathics/blob/v0.8/setup.py#L30


Hacktoberfest 23 in Brussels

We have an upcoming event on the 19th of October! This is a meta ticket to keep track of everything related:
Sign up link: https://events.mlh.io/events/10408-hacktoberfest-2023-in-brussels
ToDo

[x] location
[x] presentation
[x] food

Event was done


Hide back layout when front layout is touched

Awesome library as we wait for the official material version. I can't however hide the back layout by touching the top layout. Is this possible?

`   private fun update(withAnimation : Boolean) {
when(state) {
State.OPEN -> {
getToolbar()?.setNavigationIcon(closeIcon)
val transitionHeight = getTransitionHeight()
if(withAnimation) startTranslateAnimation(transitionHeight)
else getFrontLayout().translationY = transitionHeight
}
        State.CLOSE -> {
            getToolbar()?.setNavigationIcon(menuIcon)
            if(withAnimation) startTranslateAnimation(0F)
            else getFrontLayout().translationY = 0F
        }
    }
}`

This is exactly the function I need access to

fun open()
fun close()
or u can use : fun switch() to change backdropLayout state
all this function are public you can use them
so inorder to close backdropLatout when touching the from u can do this
backdropLayout.frontLayout.setOnClickListener{      backdropLayout.close() }


Add back setter for scaffolded collection navigations

Type of issue
Outdated article
Description
Add back setter for scaffolded collection navigations
Page URL
https://learn.microsoft.com/en-us/ef/core/modeling/relationships/navigations#reference-navigations
Content source URL
https://github.com/dotnet/EntityFramework.Docs/blob/main/entity-framework/core/modeling/relationships/navigations.md
Document Version Independent Id
46d5e507-b441-4c2c-26c5-7e8a4b03b29a
Article author
@ajcvickers

@Varorbc you'll have to provide a bit more context on what you're asking here.


List displaying behind other components

Hi,
I have this component on a form with components directly under it.  Is there a way I can set the list to display above the other component below it?  It's currently displaying behind it.  Do I set the list container's ZIndex?
Thanks!
Steve

Same issue

Hey Guys, Wrap your <AutocompleteInput with a  and provide some good zIndex to your , If this does not work. Try defining you list inside flatListProps={{}}
Let me know if it does not work!

Same for me but I used a Android trick from manual.
//...

render() {
  return(
    <View>
      <View style={styles.autocompleteContainer}>
        <Autocomplete {/* your props */} />
      </View>
      <View>
        <Text>Some content</Text>
      </View>
    </View>
  );
}

//...

const styles = StyleSheet.create({
  autocompleteContainer: {
    flex: 1,
    left: 0,
    position: 'absolute',
    right: 0,
    top: 0,
    zIndex: 1
  }
});

It means that iOS doesn't support overflows too? Probably this style should be implemented on component level?

The problem of zIndex

Same issue with the suggestions list appearing behind any element below it. I've tried setting z-index on everything I could but the problem remains.
Any attempt to select a suggestion will pass through to the element it's overlapping.

guys don't take tension . I have use listStyle={{position:'relative'}} inside Autocomplete.

I used listContainerStyle and simply added z-index in there if the result list ends up behind  other components! :) hope it helps

Yep, this is a huge problem!

On Android, my problem seems to be the fact that the dropdown doesn't overflow the container view of the Autocomplete component. Even when absolute, it doesn't show, you can only see half the dropdown. Setting it to relative isn't a possibility in my project, so I'm still looking for a solution. zIndex does nothing, elevation doesn't seem to work on suggestionsListContainer nor any other component.
Any suggestion would be appreciated!

On Android, my problem seems to be the fact that the dropdown doesn't overflow the container view of the Autocomplete component. Even when absolute, it doesn't show, you can only see half the dropdown. Setting it to relative isn't a possibility in my project, so I'm still looking for a solution. zIndex does nothing, elevation doesn't seem to work on suggestionsListContainer nor any other component.
Any suggestion would be appreciated!

The solution here  worked for me, on both platforms, but it really depends on the styling on your parent component. Could it be interfering? I'd try it out on an empty screen to see if it works, then one by one can add your parent styles. Let me know if that helps, or possibly post a code snippet.

My workaround to this problem was wrapper the components just bellow the input with a View with negative zIndex:zIndex: -1. This worked both in IOS and Android.

This is still a huge issue, though we can manipulate the display over other elements using position: "absolute" and zIndex, this could affect styling principles across an application. This needs to be fixed.

Still a problem.  I have to display the autocomplete input in a modal.  The listItems appear behind the modal.  Please fix.


Add Unet KITS19 in PyTorch

Also, can you please add a test to AML's CI for this model? See tests/pytorch_models.py

@MarcelWilnicki when can we merge?


MAINT Test against nightly dependencies

Setup a cron job to test against nightly dependencies. It can also be manually triggered with [deps nightly] in the commit message.

I think that it's also useful to turn deprecation warnings into errors now that we test against nightly builds of dependencies. I think it's better to do that as a follow up PR because it does trigger some errors and fixing them would make the scope of this PR too large.


Some functions and classes to resolve memory usage problem

This PR contains docs and tests for new functionalities wrapped in https://github.com/hpsc-lab/openfhe-julia/pull/61.
PR should be merged after https://github.com/hpsc-lab/openfhe-julia/pull/61 gets merged and new openfhe-julia release gets published

I added docs in README.md to explain how to clear evaluation keys. I set the minimal required version of openfhe-julia to 0.3.4 because new functions were added and OpenFHE.jl crashes with older versions.


Support Prometheus metrics directly

Required by https://github.com/jaegertracing/jaeger/issues/360 and https://github.com/jaegertracing/jaeger/issues/273

Coverage decreased (-0.9%) to 99.065% when pulling bd425f15793523970683051922ce237ffc84ee9a on prometheus-direct into bc381f836083a0f7d5778d4216022388c4aeaf46 on master.

Coverage decreased (-0.6%) to 99.402% when pulling c7a5c6f87fdc515d1ee8b3ca4b7ec652af44abb2 on prometheus-direct into bc381f836083a0f7d5778d4216022388c4aeaf46 on master.

Coverage remained the same at 100.0% when pulling 7e61af61bd1009cc7df81d4e80f142ed2dcb5ea4 on prometheus-direct into bc381f836083a0f7d5778d4216022388c4aeaf46 on master.

Coverage remained the same at 100.0% when pulling 7c1739cdacce985d0e1ed687af9c645fc65fc8ba on prometheus-direct into bc381f836083a0f7d5778d4216022388c4aeaf46 on master.

TODO - sanitize metric names, e.g. agent is failing with:
panic: descriptor Desc{fqName: "tc-reporter:zipkin:batches.submitted", help: "tc-reporter:zipkin:batches.submitted", constLabels: {}, variableLabels: []} is invalid: "tc-reporter:zipkin:batches.submitted" is not a valid metric name

Coverage remained the same at 100.0% when pulling 7ccca79c60033f40500864584c9d1662b413172c on prometheus-direct into bc381f836083a0f7d5778d4216022388c4aeaf46 on master.

Coverage remained the same at 100.0% when pulling be91209fa72583bb254bf83a00ffcf8a194e40de on prometheus-direct into bc381f836083a0f7d5778d4216022388c4aeaf46 on master.


Normalisation in timeseries_weather_forecasting.py is actually Standardisation

Good afternoon.
Just wanted to see if it's possible to correct this. I believe what is being used is Standardisation rather than Normalisation as the comments suggest.
This quote from line 177 suggests that we'll get range between 0 and 1 which is correct for normalisation but it's not what the code does.

Since every feature has values with varying ranges, we do normalization to confine feature values to a range of [0, 1]

Line 201 is where the incorrect normalisation label is used.
Thanks.

Hi @Kem47 ,
Standardisation often referred to as z-score Normalization is a special case of Normalization while other normalizations called as scaling normalizations . The author used z-score normalization technique here. Hence its not a problem I guess.
Thanks!


Fix missing env var

name: Bug report
about: Create a report to help us improve

Prerequisites

[X] Put an X between the brackets on this line if you have done all of the following:

Checked the FAQs for common solutions: https://github.com/digicatapult/veritable-ui/blob/main/CONTRIBUTING.md/#FAQs
Checked that your issue isn't already filed: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Averitable-ui

Description
 ~/git/veritable-ui/ [main] npm run dev

><EMAIL_ADDRESS>dev
> npm run tsoa:watch & NODE_ENV=dev node --import=tsimp/import --watch src/index.ts

><EMAIL_ADDRESS>tsoa:watch
> node --watch-path=./src ./node_modules/.bin/tsoa -- spec-and-routes

Completed running './node_modules/.bin/tsoa -- spec-and-routes'
================================
 Missing environment variables:
    COMPANY_PROFILE_API_KEY: undefined
================================

 Exiting with error code 1
Failed running 'src/index.ts'

Steps to Reproduce

Follow instructions in readme.md

Expected behaviour:
Starts server
Actual behaviour:
Borks with COMPANY_PROFILE_API_KEY: undefined
Reproduces how often:
100%

I'd rather not have a non-functional env var in the application. There are two improvements I do think could be made here:

Instructions in the readme on where to get an api key from and how to use it in dev
A switch when running in dev mode that set's up an undici mock so that we always get valid values from companies house. We might even be able to do it with a --import so that the logic can only ever apply to the npm run dev command


ArrayOutOfBoundException when setting a Map field

I have an entity "Account" with a "props" field that is Map<String, String>. When I'm trying to set this field via:

UpdateOperations ops = createUpdateOperations();
ops.set("props", new HashMap<String, String>());

I'm getting:

at java.util.Arrays$ArrayList.get(Arrays.java:3841)
at org.mongodb.morphia.internal.PathTarget.next(PathTarget.java:96)
at org.mongodb.morphia.internal.PathTarget.resolve(PathTarget.java:115)
at org.mongodb.morphia.internal.PathTarget.getTarget(PathTarget.java:90)
at org.mongodb.morphia.query.UpdateOpsImpl.add(UpdateOpsImpl.java:279)
at org.mongodb.morphia.query.UpdateOpsImpl.set(UpdateOpsImpl.java:223)

The reason for it that PathTarget contains:

if (!field.isMap()) {
translate(field.getNameToStore());
} else {
next();  // consume the map key segment
}

So the field is actually a map and PathTarget contains only 1 segment which has been already consumed by the time when next() is called from the else clause.
The issues occurs with 1.3.0 and does not appear on 1.2.0 and earlier.

duplicate of #1089


Computress Mission

Level 22 World Mission:  Computress Support (Part 2 of 2)
The main task is to visit and protect Mandroid M-80 against a cyberattack from Fuse.
However, the mission actually directs you to Mandroid M-114.
It's a coin-flip call whether M-114 or M-80 is the correct bot to visit... M-114 is in the fountain area between Dexter's lab and Mandark's.  M-80 is standing beside Mandark.

@CakeLancelot @dongresource have you guys taken a look at this at all?

have you guys taken a look at this at all?

This is an issue with the game's tabledata rather than the actual server implementation. [1] [2]
This could in theory be fixed with a client patch, however it is out of scope of the project for now. Closing.


Allow User to Specify Install and Build Folders

Thanks Michelle for the PR. We are honored that you joined Github to create your first repo and PR ever for our project. Welcome on Github!
We will review it if we find time in the next days.
Regards Stefan

Thanks Michelle for the PR. We are honored that you joined Github to create your first repo and PR ever for our project. Welcome on Github!
We will review it if we find time in the next days.
Regards Stefan

Thanks! I’m not super new to github, my company just requires me to contribute to open source projects using my corporate email.  Thanks for taking a look! My team is excited to have stronger conan integration with vscode.
-Michelle
From: Stefan Eicher<EMAIL_ADDRESS>Sent: Wednesday, January 6, 2021 2:11 PM
To: disroop/vs-code-conan<EMAIL_ADDRESS>Cc: Hetzel Michelle H<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [disroop/vs-code-conan] Allow User to Specify Install and Build Folders (#19)
[EXTERNAL]
Thanks Michelle for the PR. We are honored that you joined Github to create your first repo and PR ever for our project. Welcome on Github!
We will review it if we find time in the next days.
Regards Stefan
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/disroop/vs-code-conan/pull/19#issuecomment-755621002, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASMAJDZ5BF2MS5JFVW3WZL3SYS7W3ANCNFSM4VX4EBZA.
[EXTERNAL EMAIL]: This message was generated from an external source. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Thanks! I’m not super new to github, my company just requires me to contribute to open source projects using my corporate email.  Thanks for taking a look! My team is excited to have stronger conan integration with vscode.
-Michelle
From: Stefan Eicher<EMAIL_ADDRESS>Sent: Wednesday, January 6, 2021 2:11 PM
To: disroop/vs-code-conan<EMAIL_ADDRESS>Cc: Hetzel Michelle H<EMAIL_ADDRESS>Author<EMAIL_ADDRESS>Subject: Re: [disroop/vs-code-conan] Allow User to Specify Install and Build Folders (#19)
[EXTERNAL]
Thanks Michelle for the PR. We are honored that you joined Github to create your first repo and PR ever for our project. Welcome on Github!
We will review it if we find time in the next days.
Regards Stefan
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/disroop/vs-code-conan/pull/19#issuecomment-755621002, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASMAJDZ5BF2MS5JFVW3WZL3SYS7W3ANCNFSM4VX4EBZA.
[EXTERNAL EMAIL]: This message was generated from an external source. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hi Michelle this is a great improvement thank you for your contribution. It totally makes sense to integrate this into the conan vs-code plugin. I will release it in the store by today, under the version 0.6.0.

Hi Michelle this is a great improvement thank you for your contribution. It totally makes sense to integrate this into the conan vs-code plugin. I will release it in the store by today, under the version 0.6.0.

Hi Michelle this is a great improvement thank you for your contribution. It totally makes sense to integrate this into the conan vs-code plugin. I will release it in the store by today, under the version 0.6.0.

Thank you! This was for the issue I raised a while back 😄

Hi Michelle this is a great improvement thank you for your contribution. It totally makes sense to integrate this into the conan vs-code plugin. I will release it in the store by today, under the version 0.6.0.

Thank you! This was for the issue I raised a while back 😄


Not seeing data in Google Analytics after install

Description
Installed and configured VueGtag, but no events show up in Google Analytics
Expected behavior
Analytics show up
Actual behavior
Nothing reported to Google Analytics
Environment
<EMAIL_ADDRESS>Vue 3
Vite
Windows 10 (Testing)
Ubuntu 20.4 (Deploy)
Brave and others
Reproducible Demo
import { createApp } from 'vue'
import store from './store'
import App from './App.vue'
import VueGtag from "vue-gtag";
import './index.css'

const app = createApp(App).use(store).use(VueGtag, {
  config: { id: "G-XXXXXXXXXX" }
}).mount('#app')

@officiallymarky
Can you please describe how you solved this issue? I'm having the same problem :(

Solved it by moving GTag id environment variable from private env variables to public variables in defineNuxtConfig as Gtag ID should be visible on the client side in order for this plugin to work.
How the runtimeConfig option looks now:
runtimeConfig: {
    public: {
      gtagId: "",
      backendUrl: "xxxxxx",
    },
  },

How it looked before when it wasn't working:
runtimeConfig: {
    gtagId: "",
    public: {
      backendUrl: "xxxxxx",
    },
  },


Specify memory reqs as string

Byte-based resource requirements can be specified as various byte-based units in addition to just bytes. Adds this functionality to Job's resource requirements as well as the max_log_size and default/max memory/disk toil options. Adds bd2k-python-lib 1.7 as a dependency, and uses the human2byte method provided within that library.

Can one of the admins verify this patch?

Jenkins, test this please.

Tests hung

Caused by what appears to be a bug in the human2bytes:

sys.maxint
9223372036854775807

str(human2bytes(str(sys.maxint)))
9223372036854775808L

int(str(human2bytes(str(sys.maxint)))
9223372036854775808

In the job constructor, the default argument value for any resource req is sys.max int. I run human2bytes on memory/disk arguments,  and we later test if our resources equal sys.maxint to determine whether to instead set the parameter to it's configurable default. In this bug, they do not equal each other so we pass this huge int as the disk requirement for the job wrapper which mesos obviously cannot allocate, causing the test to hang.

This seems to be caused by float imprecision in the human2bytes method

Jenkins, test this please.


Not ready for review: Experimenting with shippable

Not ready for review, just creating a pull request so I can experiment w/ shippable

Closing this pull request, will create a new one with cleaner commit history


Is this project dead?

Looks like there's been no significant activity for a year. Too bad since this was the only implementation I found that treated Sequences as a first class citizen (and hence all the associated power).
Do I have to switch to using a combination of multiple libraries - e.g. lodash as utility belt + RxJS or IxJS or something for first class Sequences?

It's not dead! I just work on it on an incredibly inconsistent and sporadic basis. If you're not comfortable depending on it for that reason (which would be quite reasonable), then you might prefer to look elsewhere.

For what it's worth, I'll be releasing a new version in the next week.


Migrate Controller/Class/View doesn't do anything

Describe the bug
I installed the extension. Then I opened a solution that has a .NET 4.7.2 Web API project and selected the Migrate option. I opted to create a new project. It ran for a little bit, configured YARP, etc and then showed me the summary page.
I now have the option to migrate controller/class/view. However the dropdowns are all empty. The text says to migrate from Solution Explorer so I go to SE, select one of my controllers and then select 'Migrate Controller'. It does nothing. No messages, no spinning, nothing.
Going back to the summary screen it shows my .NET Framework version as 0 (NaN) and my .NET Core version as 0 (NaN). The Routes section is just a vertical white line.
I created a new web API project (no MVC) and tried it on that version. It migrated the project as before. However when I go to migrate a controller the only ones that show up are those that derive from Controller which is MVC, not API. Hence none of the API controllers are available (which is what I am seeing in my original project).
To Reproduce

Create a new .NET 4.7.2 ASP.NET Web (.NET Framework) application. Only enable Web API (no MVC).
Run the migration tool on the project.
In the summary go to Migrate Controllers and notice the API controller doesn't show up.
In Solution Explorer right click the API controller and migrate it. Nothing happens.

Exceptions (if any)
None. It just doesn't do anything.
Further technical details
Please include the following if applicable:
ASP.NET Framework Application:
.NET 4.7.2 Web API (no MVC)
Windows 10
ASP.NET Core Application:
.NET 6.0 Web API

Thanks for reporting the issue! We can repro it now and working on the fix.

@mtaylorfsmb we could repro the case when controller is API controller and it is not doing anything. However you mentioned that when you right click on just a class it did not do anything also? Or when you open Migrate Class dialog from the Migrations page it did not have any classes in the list (or you meant no controllers in the list)?

@abpiskunov Sorry for the confusion. Yes, when you click Migrate Controller nothing shows up (presumably because it doesn't see any API controllers). Migrate Class shows all the types in the project correctly.
As an aside note that the migration UI says to use SE to select the controller/class/view you want to migrate. But when you do that on an API controller it just sends you back to the migration UI summary page. This is what I meant by it not doing anything from SE.

@mtaylorfsmb thanks for clarification. Yes, that is what I see here too. Api controllers support is coming very soon.

This was fixed and there is a new VSIX where we consolidated all upgrade related scenarios: https://marketplace.visualstudio.com/items?itemName=ms-dotnettools.upgradeassistant&ssr=false#overview


v2.2.3 fixes

Fixes #1003 MissingMethodException thrown on Django solution load
Updates Django's completions and classifier to work with the public API changes made in VS 2015 U1.
Also fixes some build issues with VS 2015 U1.

:+1:


rancher cluster monitoring, prometheus keeps getting oomkilled

What kind of request is this (question/bug/enhancement/feature request):
Bug
Steps to reproduce (least amount of steps as possible):
Enable monitoring on cluster, we tried both persistent and non-persistent and increased the memory limit to 1250Mb
Result:
Prometheus keeps getting oomkilled. We already have a monitoring stack deployed so we can see it going oom in that grafana/prometheus. We have other clusters where the combination is working, so they don't seem to be interfering with each other.

Other details that may be helpful:
A describe of the pod after oomkilled:
Name:               prometheus-cluster-monitoring-0
Namespace:          cattle-prometheus
Priority:           0
PriorityClassName:  <none>
Node:               k8s-development-6/<IP_ADDRESS>
Start Time:         Wed, 17 Apr 2019 15:15:50 +0200
Labels:             app=prometheus
                    chart=prometheus-0.0.1
                    controller-revision-hash=prometheus-cluster-monitoring-c6f68955b
                    monitoring.coreos.com=true
                    prometheus=cluster-monitoring
                    release=cluster-monitoring
                    statefulset.kubernetes.io/pod-name=prometheus-cluster-monitoring-0
Annotations:        cni.projectcalico.org/podIP: <IP_ADDRESS>/32
Status:             Running
IP:                 <IP_ADDRESS>
Controlled By:      StatefulSet/prometheus-cluster-monitoring
Containers:
  prometheus:
    Container ID:  docker://4ed302f33e8a0b5b8d81d7b2680ec85c0b79e1caa9c5233c1d0d417441b0fa15
    Image:         rancher/prom-prometheus:v2.7.1
    Image ID:      docker-pullable://rancher/prom-prometheus@sha256:b32864cfe5f7f5d146820ddc6dcc5f78f40d8ab6d55e80decd5fc222b16344dc
    Port:          <none>
    Host Port:     <none>
    Args:
      --web.console.templates=/etc/prometheus/consoles
      --web.console.libraries=/etc/prometheus/console_libraries
      --config.file=/etc/prometheus/config_out/prometheus.env.yaml
      --storage.tsdb.path=/prometheus
      --storage.tsdb.retention=12h
      --web.enable-lifecycle
      --storage.tsdb.no-lockfile
      --web.route-prefix=/
      --web.listen-address=<IP_ADDRESS>:9090
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Wed, 17 Apr 2019 15:48:43 +0200
      Finished:     Wed, 17 Apr 2019 15:50:50 +0200
    Ready:          False
    Restart Count:  7
    Limits:
      cpu:     1
      memory:  1250Mi
    Requests:
      cpu:        750m
      memory:     750Mi
    Environment:  <none>
    Mounts:
      /etc/prometheus/config_out from config-out (ro)
      /etc/prometheus/configmaps/prometheus-cluster-monitoring-nginx from configmap-prometheus-cluster-monitoring-nginx (ro)
      /etc/prometheus/rules/prometheus-cluster-monitoring-rulefiles-0 from prometheus-cluster-monitoring-rulefiles-0 (rw)
      /etc/prometheus/secrets/exporter-etcd-cert from secret-exporter-etcd-cert (ro)
      /prometheus from prometheus-cluster-monitoring-db (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from cluster-monitoring-token-hpwqw (ro)
  prometheus-config-reloader:
    Container ID:  docker://b89874489455ac0845324967575b74161445def8fabb5f7537ff46b7845e2159
    Image:         rancher/coreos-prometheus-config-reloader:v0.29.0
    Image ID:      docker-pullable://rancher/coreos-prometheus-config-reloader@sha256:9c47f0451a3c0a2277fb259367785226bf3be317dd5b98bacbb6aa9b112feba2
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/prometheus-config-reloader
    Args:
      --log-format=logfmt
      --reload-url=http://localhost:9090/-/reload
      --config-file=/etc/prometheus/config/prometheus.yaml.gz
      --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
    State:          Running
      Started:      Wed, 17 Apr 2019 15:15:59 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     50m
      memory:  50Mi
    Requests:
      cpu:     50m
      memory:  50Mi
    Environment:
      POD_NAME:  prometheus-cluster-monitoring-0 (v1:metadata.name)
    Mounts:
      /etc/prometheus/config from config (rw)
      /etc/prometheus/config_out from config-out (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from cluster-monitoring-token-hpwqw (ro)
  prometheus-proxy:
    Container ID:  docker://248c2b143d138068261cb1a23ae86e0f54d835bc7d51802f9e24bd2773d9e963
    Image:         rancher/nginx:1.15.8-alpine
    Image ID:      docker-pullable://rancher/nginx@sha256:c9a462bce16d85dde1861d45aebfae80c812d733187d045cbb7ffe3d71ac37bd
    Port:          80/TCP
    Host Port:     0/TCP
    Command:
      /bin/sh
      -c
      cp /nginx/run-sh.tmpl /var/run/nginx-start.sh; chmod +x /var/run/nginx-start.sh; /var/run/nginx-start.sh
    State:          Running
      Started:      Wed, 17 Apr 2019 15:16:03 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  100Mi
    Requests:
      cpu:     50m
      memory:  50Mi
    Environment:
      POD_IP:   (v1:status.podIP)
    Mounts:
      /nginx from configmap-prometheus-cluster-monitoring-nginx (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from cluster-monitoring-token-hpwqw (ro)
  prometheus-agent:
    Container ID:  docker://423af7cc63520471ed1cb510502723214b8a248637931946957ed844e9c60d5f
    Image:         rancher/prometheus-auth:v0.2.0
    Image ID:      docker-pullable://rancher/prometheus-auth@sha256:8f715bf8eb4ea41c6598ccd244da51743d14f5a5c7cb93091fdced531cf48dbe
    Port:          9090/TCP
    Host Port:     0/TCP
    Command:
      prometheus-auth
    Args:
      --proxy-url=http://localhost:9090
      --listen-address=$(POD_IP):9090
      --filter-reader-labels=prometheus
      --filter-reader-labels=prometheus_replica
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Wed, 17 Apr 2019 15:50:42 +0200
      Finished:     Wed, 17 Apr 2019 15:51:49 +0200
    Ready:          False
    Restart Count:  7
    Limits:
      cpu:     500m
      memory:  200Mi
    Requests:
      cpu:      100m
      memory:   100Mi
    Liveness:   http-get http://:web/-/healthy delay=0s timeout=10s period=10s #success=1 #failure=6
    Readiness:  http-get http://:web/-/ready delay=0s timeout=10s period=10s #success=1 #failure=10
    Environment:
      POD_IP:   (v1:status.podIP)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from cluster-monitoring-token-hpwqw (ro)
  rules-configmap-reloader:
    Container ID:  docker://53ad8293a571ae6a217248d9018fde7026d0cf8e1e8bf42fdd9a018564456d2b
    Image:         rancher/coreos-configmap-reload:v0.0.1
    Image ID:      docker-pullable://rancher/coreos-configmap-reload@sha256:f58cb3b14aea0fc5b5f1d8f978218036b48207a2eb53079e3af9ce14db5f0676
    Port:          <none>
    Host Port:     <none>
    Args:
      --webhook-url=http://localhost:9090/-/reload
      --volume-dir=/etc/prometheus/rules/prometheus-cluster-monitoring-rulefiles-0
    State:          Running
      Started:      Wed, 17 Apr 2019 15:16:09 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     100m
      memory:  25Mi
    Requests:
      cpu:        100m
      memory:     25Mi
    Environment:  <none>
    Mounts:
      /etc/prometheus/rules/prometheus-cluster-monitoring-rulefiles-0 from prometheus-cluster-monitoring-rulefiles-0 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from cluster-monitoring-token-hpwqw (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  prometheus-cluster-monitoring
    Optional:    false
  config-out:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  prometheus-cluster-monitoring-rulefiles-0:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-cluster-monitoring-rulefiles-0
    Optional:  false
  secret-exporter-etcd-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  exporter-etcd-cert
    Optional:    false
  configmap-prometheus-cluster-monitoring-nginx:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      prometheus-cluster-monitoring-nginx
    Optional:  false
  prometheus-cluster-monitoring-db:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  cluster-monitoring-token-hpwqw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cluster-monitoring-token-hpwqw
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  beta.kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                   From                        Message
  ----     ------     ----                  ----                        -------
  Normal   Pulling    38m                   kubelet, k8s-development-6  pulling image "rancher/prom-prometheus:v2.7.1"
  Normal   Scheduled  38m                   default-scheduler           Successfully assigned cattle-prometheus/prometheus-cluster-monitoring-0 to k8s-development-6
  Normal   Pulled     38m                   kubelet, k8s-development-6  Successfully pulled image "rancher/prom-prometheus:v2.7.1"
  Normal   Pulling    38m                   kubelet, k8s-development-6  pulling image "rancher/coreos-prometheus-config-reloader:v0.29.0"
  Normal   Pulled     38m                   kubelet, k8s-development-6  Successfully pulled image "rancher/coreos-prometheus-config-reloader:v0.29.0"
  Normal   Pulling    38m                   kubelet, k8s-development-6  pulling image "rancher/nginx:1.15.8-alpine"
  Normal   Created    38m                   kubelet, k8s-development-6  Created container
  Normal   Started    38m                   kubelet, k8s-development-6  Started container
  Normal   Created    38m                   kubelet, k8s-development-6  Created container
  Normal   Pulled     38m                   kubelet, k8s-development-6  Successfully pulled image "rancher/nginx:1.15.8-alpine"
  Normal   Pulling    38m                   kubelet, k8s-development-6  pulling image "rancher/prometheus-auth:v0.2.0"
  Normal   Started    38m                   kubelet, k8s-development-6  Started container
  Normal   Pulled     38m                   kubelet, k8s-development-6  Successfully pulled image "rancher/prometheus-auth:v0.2.0"
  Normal   Created    38m                   kubelet, k8s-development-6  Created container
  Normal   Started    38m                   kubelet, k8s-development-6  Started container
  Normal   Pulling    38m                   kubelet, k8s-development-6  pulling image "rancher/coreos-configmap-reload:v0.0.1"
  Normal   Created    37m                   kubelet, k8s-development-6  Created container
  Normal   Started    37m                   kubelet, k8s-development-6  Started container
  Normal   Pulled     37m                   kubelet, k8s-development-6  Successfully pulled image "rancher/coreos-configmap-reload:v0.0.1"
  Normal   Started    37m (x2 over 38m)     kubelet, k8s-development-6  Started container
  Normal   Created    37m (x2 over 38m)     kubelet, k8s-development-6  Created container
  Warning  Unhealthy  28m                   kubelet, k8s-development-6  Liveness probe failed: HTTP probe failed with statuscode: 502
  Normal   Pulled     28m (x2 over 37m)     kubelet, k8s-development-6  Container image "rancher/prom-prometheus:v2.7.1" already present on machine
  Warning  BackOff    8m9s (x49 over 28m)   kubelet, k8s-development-6  Back-off restarting failed container
  Warning  Unhealthy  3m11s (x52 over 28m)  kubelet, k8s-development-6  Readiness probe failed: HTTP probe failed with statuscode: 502

Environment information

Rancher version: 2.2.2 (latest stable at the time of writing)
Installation option (single install/HA):single install

Cluster information

Cluster type (Hosted/Infrastructure Provider/Custom/Imported): Custom
Machine type (cloud/VM/metal) and specifications (CPU/memory): 3 admin nodes 4 core/3gb, 5 worker nodes: 8 cores/16gb, all running ubuntu 16.04

NAME                 STATUS   ROLES               AGE    VERSION
k8s-dev-admin-4      Ready    controlplane,etcd   142d   v1.12.7
k8s-dev-admin-5      Ready    controlplane,etcd   142d   v1.12.7
k8s-dev-admin-6      Ready    controlplane,etcd   142d   v1.12.7
k8s-development-10   Ready    worker              126d   v1.12.7
k8s-development-6    Ready    worker              126d   v1.12.7
k8s-development-7    Ready    worker              126d   v1.12.7
k8s-development-8    Ready    worker              126d   v1.12.7
k8s-development-9    Ready    worker              126d   v1.12.7

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-03-01T23:34:27Z", GoVersion:"go1.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.7", GitCommit:"6f482974b76db3f1e0f5d24605a9d1d38fad9a2b", GitTreeState:"clean", BuildDate:"2019-03-25T02:41:57Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Docker version (use docker version): 17.3.2

Enhanced the prometheus memory usage a bit by removing unused rules and reduced the scrap frequency.
Chart PR merged
Waiting for https://github.com/rancher/rancher/pull/19766 https://github.com/rancher/types/pull/801 https://github.com/rancher/system-charts/pull/37 merged and https://github.com/rancher/rancher/pull/19766 https://github.com/rancher/types/pull/801 backport to 2.2. branch.

The bug fix is validated in Rancher: master-head c2d9dc8a
Steps:

run Rancher container single install
add an EC2 cluster with 3 etcd/control plane nodes, 3 workers nodes. Each node is t2.medium 2 vCPU, 4 GB memory
enable the cluster monitoring with all default settings
run some automation test on the cluster to make it "busy"
keep and observe the cluster monitoring apps for a while (its 3+ hours as the time writing the comment)

Results:

the pod prometheus-cluster-monitoring-0 in workload prometheus-cluster-monitoring does not get oomkilled anymore
the cluster monitoring feature works as expected

This is still happening in 2.4.5. Could you please share some details about the root cause of the issue? Do I need to configure something to stop this happening? Symptoms are exactly the same as it is described in the original post.

I'm having this problem also in  v2.4.2

I was getting OOM on the prometheus container in the prometheus-cluster-monitoring-0 pod.. I raised the memory from the UI and it's holding.

@PeterMaco Same with 2.4.5 also increased memory limit like @brylor

@PeterMaco I can also confirm this on 2.4.5.  As @brylor said increasing prometheus's default memory limits prevents this from happening.

I am also experiencing this on 2.5.1 with 2.4.5 created cluster

Still having this in 2.4.12!
Seems like the docs at https://rancher.com/docs/rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/ have RAM values that are too low.

Still having this in 2.5.7

I got this issue on image rancher/mirrored-prometheus-prometheus:v2.24.0. The prometheus-rancher-monitoring-prometheus-0 pod keeps restarting.

Using rancher-monitoring 9.4.203 I'm getting the same issue, I was digging into the graphs and it seems graphana has managed to hit almost 8G ram usage on a empty cluster with barely anything running and only 2 custom scrappers, galeradb and gitlab.
Comparing this to the 'recommended' 1g ram or whatever it was it quite a big jump, is this supposed? I mean if it consumes this much ram on an empty cluster how much ram are you guys giving this for a full prod cluster? 32G?

We are having the same OOM killed issue on rancher-monitoring 9.4.202 on a 8GB ram instance dedicated only for this service.
Before upgrading our instance to 16GB or 32GB it would be good to know if its a memory leak/bug or minimum requirements have been updated.

More info and temporal fix on this for anyone coming here.
I am running prometheus on a dedicate server instance with 2 vCPU and 8GB ram, it only runs basic kube/rancher needed services and prometheus, something like this:
NAMESPACE                  NAME
cattle-monitoring-system   prometheus-rancher-monitoring-prometheus-0
cattle-monitoring-system   pushprox-kube-proxy-client-rt874
cattle-monitoring-system   rancher-monitoring-prometheus-adapter-5fc7c8768d-zthz5
cattle-monitoring-system   rancher-monitoring-prometheus-node-exporter-6qs6q
cattle-system              cattle-node-agent-kqxzn
ingress-nginx              nginx-ingress-controller-tt8sq
iptables                   firewall-k8swall-hqnbr

And this is the current status:
CPU: 66%
Memory:   4805Mi  62%
So because it kept getting oomkilled I modified the prometheus resources to this (basically x2 on both memory attributes):
    resources:
      limits:
        cpu: 1000m
        memory: 3000Mi
      requests:
        cpu: 750m
        memory: 1500Mi

For now it runs fine, no restarts due oomkilled crashes. I also run prometheus on a staging cluster with less nodes and there the default resources configuration works fine.
So looks like that configuration basically doesnt scale up, probably with more nodes it will need more resources, maybe it should appear on the helm README if its intended or should get fixed if its some kind of memory leak but that helm chart wont work on most of the production clusters with a decent number of nodes.


Knife summon issues

Knife summoning breaks at high speeds
something else I'm too dumb to remember

2 null reference exceptions in first frame, likely due to rb or some other component not being instantiated yet

Fixed by #22


Teleport hack

There is a hack someone is using on my server and it's allowing them to teleport. I have no idea of how to fix. Script of hack: https://hastebin.com/ilemujumuh.js

The script sets one of the player's mouse coordinates to Infinity - this reproduces it:
window.addEventListener("keydown", function(e) {
    switch (e.keyCode) {
        case 37:
            $("canvas").trigger($.Event("mousemove", {
                clientX: -Infinity,
                clientY: 0
            }));
            break;
        case 38:
            $("canvas").trigger($.Event("mousemove", {
                clientX: 0,
                clientY: -Infinity
            }));
            break;
        case 39:
            $("canvas").trigger($.Event("mousemove", {
                clientX: Infinity,
                clientY: 0
            }));
            break;
        case 40:
            $("canvas").trigger($.Event("mousemove", {
                clientX: 0,
                clientY: Infinity
            }));
            break;
    }
});

And because your server is flawed, doing this teleports all nearby cells along with the executor for some reason. I've tested it on a fresh install of MultiOgar-Edited locally and it doesn't work.

How can I stop it?

your server is flawed

We don't spoonfeed people.

@Luka967 It's an issue.

@AlexHGaming But he said that it's already fixed.

@AlexHGaming

I've tested it on a fresh install of MultiOgar-Edited locally and it doesn't work.

It's not our issue.

Ah, my bad.

Thanks for the info. I patched in my main_out.js
mainCanvas.onmousemove = function(event) { 	if (event.clientX == Infinity || event.clientX == -Infinity || event.clientY == Infinity || event.clientY == -Infinity) { 		return alert('STOP HACKING!'); 	} 	rawMouseX = event.clientX; 	rawMouseY = event.clientY; 	mouseCoordinateChange() };

@TehRealShot "main_out.js"
Facepalm...

i also patched it in the gameserver.js