CLOUD-685 fix monitoring test

Test name
Status

scheduled-backup
failed

recreate
passed

upgrade-haproxy
passed

storage
passed

big-data
passed

scaling
passed

haproxy
passed

scaling-proxysql
passed

self-healing
passed

self-healing-chaos
passed

init-deploy
passed

security-context
passed

restore-to-encrypted-cluster
passed

self-healing-advanced
passed

self-healing-advanced-chaos
passed

upgrade-proxysql
passed

limits
passed

demand-backup
failed

monitoring-2-0
passed

operator-self-healing
passed

operator-self-healing-chaos
passed

smart-update
passed

affinity
passed

one-pod
passed

auto-tuning
passed

proxysql-sidecar-res-limits
passed

users
passed

upgrade-consistency
passed

tls-issue-self
passed

pitr
failed

tls-issue-cert-manager
passed

tls-issue-cert-manager-ref
failed

commit: https://github.com/percona/percona-xtradb-cluster-operator/pull/1110/commits/422a09f1f0a57bd4c036cf8a6c1411a79a32d853
image: perconalab/percona-xtradb-cluster-operator:PR-1110-422a09f1

Test name
Status

recreate
passed

big-data
passed

scaling
passed

upgrade-haproxy
passed

storage
passed

scheduled-backup
failed

haproxy
passed

scaling-proxysql
passed

self-healing
passed

self-healing-chaos
passed

init-deploy
passed

restore-to-encrypted-cluster
passed

security-context
passed

self-healing-advanced
passed

self-healing-advanced-chaos
passed

upgrade-proxysql
passed

limits
passed

demand-backup
failed

monitoring-2-0
passed

operator-self-healing
passed

operator-self-healing-chaos
passed

affinity
passed

one-pod
passed

smart-update
passed

auto-tuning
passed

proxysql-sidecar-res-limits
passed

users
passed

upgrade-consistency
failed

tls-issue-self
passed

tls-issue-cert-manager
failed

pitr
passed

tls-issue-cert-manager-ref
passed

validation-hook
passed

demand-backup-encrypted-with-tls
passed

commit: https://github.com/percona/percona-xtradb-cluster-operator/pull/1110/commits/9cc142667173f02f6ff5622f5d8c755bfcc3b7c2
image: perconalab/percona-xtradb-cluster-operator:PR-1110-9cc14266

Test name
Status

upgrade-haproxy
passed

haproxy
passed

big-data
passed

scheduled-backup
passed

scaling
passed

init-deploy
passed

scaling-proxysql
passed

recreate
passed

storage
passed

limits
passed

security-context
passed

self-healing
passed

restore-to-encrypted-cluster
passed

monitoring-2-0
passed

demand-backup
passed

self-healing-chaos
passed

affinity
passed

self-healing-advanced
passed

one-pod
passed

self-healing-advanced-chaos
passed

auto-tuning
passed

operator-self-healing
passed

proxysql-sidecar-res-limits
passed

operator-self-healing-chaos
passed

users
passed

tls-issue-self
passed

tls-issue-cert-manager
passed

tls-issue-cert-manager-ref
passed

validation-hook
passed

upgrade-proxysql
passed

smart-update
passed

upgrade-consistency
passed

pitr
passed

demand-backup-encrypted-with-tls
passed

commit: https://github.com/percona/percona-xtradb-cluster-operator/pull/1110/commits/aef8f8bbf55756351636d28f0acd3e84f86e1135
image: perconalab/percona-xtradb-cluster-operator:PR-1110-aef8f8bb


Limitations - are there more?

I know from bitter experience that you can't have more than 128 deep nested levels of XML elements. Can this be added to the limitations. Are there more limitations that could be added to the limitation section?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 10b80f2a-c27b-b55e-1c72-2a976cd18bff
Version Independent ID: 81c2a3df-6055-d504-2580-a80cccf8bb50
Content: XML Data Type and Columns (SQL Server) - SQL Server
Content Source: docs/relational-databases/xml/xml-data-type-and-columns-sql-server.md
Product: sql
Technology: xml
GitHub Login: @MightyPen
Microsoft Alias: genemi

Hi @GilesDMiddleton , thanks for contacting Microsoft! I've assigned this issue to the author to research further. Thanks!

Fix is in PR #10263, in the sister private repo.
https://github.com/MicrosoftDocs/sql-docs-pr/pull/10263

@GilesDMiddleton Hello Giles, The fix should be live before Monday 2019/May/20.  It is an added bullet to the Limitations list.
Thank you very much for bringing this to our attention.
(I am now Closing this Issue 1982. However, the @ notification mechanism continues to function even in Closed Issues.)


[Update] Fix ERC721 delayed reveal detection and refactor related components

#4503  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @jnsdls and the rest of your teammates on  Graphite


Extended task graph creation

Which project is this feature idea for?
Turborepo
Describe the feature you'd like to request
The generated task graphs can become quite hard to read, once enough packages and tasks are defined.
As a workaround, I created @yeger/turbo-graph, which provides an interactive visualization for Turborepo task graphs.
However, its implementation is rather hacky, because Turborepo can currently only output the graphs in JSON format as files.
I suggest extending the CLI, or maybe even adding a programmatic JS API, for creating task graphs in all supported formats without the need for files.
Further, it would be nice to include all tasks by passing a flag, instead of including them in the command.
Describe the solution you'd like
Programmatic API (or extended CLI) for creating task graphs in all supported formats without the need for intermediate files.
Add a flag/option to create graphs containing all tasks without specifying them.
E.g. turbo --graph --graph-format=json --full-graph or
import { createGraph } from 'turbo/graph'

const myGraph = createGraph({
  format: 'json',
  tasks: 'all', // 'all' or string[] with task names
})

Describe alternatives you've considered
Currently, I'm parsing the dot graph output manually.
While I don't need most features of the JSON format (as I'm using my own d3-based graph library), the pos field of graph nodes would be of great use for laying out the graphs.
For detecting available tasks, I'm parsing the turbo.json file's pipeline.

Hey @DerYeger, awesome project - we were playing around with it yesterday. Have you have tried using --dry=json? This should give you all the information you need to construct the graph, and will output to stdout in json format (skipping the file write)

Thank you!
--dry=json looks great and contains even more data for me to display.
I'll give it a try and close this issue if it works out (which I strongly assume it will).

I gave it a try and ended up with way too busy graphs, because --dry=json includes transitive dependencies as well.
Once I find the time, I'll evaluate whether filtering transitive dependencies is any cleaner or simpler at all at the current parsing approach.
But in any case, the additional information is great, and I'm looking forward to displaying it as well.

v1.0.0 of @yeger/turbo-graph now uses the --dry=json command to construct it's model!


Assistance with training parameters in train_GENIE_model.py

Hi,
I hope this message finds you well.
I am reaching out because I am trying to run the train_GENIE_model.py file for a specific zone of interest. However, I am struggling to understand how to modify the training parameters to ensure code functions correctly for my study area.
From your documentation, I understand that certain training parameters need to be adjusted based on the study area. However, I am finding it challenging to determine  the appropriate values for the parameters spc_random, spc_thresh_rand, and dist_range.
Could you kindly provide any advice or guidance on how to choose these parameters for a specific zone? Your insights would be greatly appreciated.
Thank you for your time and support.

Hi, my apologies for the delay. Could you provide a screenshot of your study region + station distribution? To choose those parameters I need a sense of the scale of the region

This is one of my study areas and corresponding station distributions (I plan to apply the model to 2-3 other areas as well), all located in Colombia:

The minimum and maximum distances between stations are 4 meters and 425 km, respectively. Given that seismicity in this area occurs at considerable depths (~150 km) and the area is larger than the example provided, I am using the following grid spacing in X, Y and Z:
dx: 1000.0
dx_depth: 1000.0
If you can provide further details about the reasoning behind your choices for parameters like spc_random, spc_thresh_rand, dist_range, and any other parameters you consider could be crucial for ensuring the model to work properly I'll be very grateful, as I'll need to adapt this model to other regions as well. Thank you!

Thanks for sharing this. You can also edit "d_deg: 0.01" in config (so that it is roughly the same value as dx and dx_depth - since 0.01 degrees is ~1 km).
Basically, the logic behind these parameters is explained in the "Synthethic Training Data" section of the paper. "dist_range" controls the range of possible moveout distances between sources and recievers - roughly, you want the lower end to be low (e.g., 15000; for 15 km), and the upper end to be the maximum possible (e.g., 400000 or 500000, for 400 or 500 km, seems appropriate for your scale). Each source will have a maximum moveout distance sampled from this range of values in dist_range. "spc_random" and "spc_thresh_rand" control how much "randomness" is added per-station on-top of this maximum per-source moveout. That is, rather than use a hard threshold on the maximum distance for each station, for each source it's the value sampled from dist_range, plus a random pertubation for each station sampled from the spc_random and spc_thresh_rand values. This basically emulates the fact that at close to the maximum distance from each source, stations will begin to stochastically miss or detect each event depending on noise levels and attenuation patterns. So these values should be somewhat proportional to the inter-station spacing to make their effect useful. So for your case I think 15000 to 20000 would be reasonable values for these parameters. I hope that helps!

Thanks for your answer, a very helpful one.

No problem!


add CDF logo to screwdriver.cd as incubating project

https://github.com/screwdriver-cd/homepage/pull/49
add CDF logo to screwdriver.cd as incubating project

Completed and screwdriver.cd team pushed up all MRs


ANC @ Health Center - Labs - HIV [5h]

Issue
The HIV RDT is performed in the first antenatal visit at the health center and at an EGA of 38 weeks. There are 4 inputs needed on the screen.

Were you able to perform this test? If yes, you then move on to validate if it was done today. If no, you select from a series of checkboxes why not. Check box options shown here
If yes, then the question "Did you perform this test today?" appears. If yes, today's date auto-populates in the Test Date section. If no, then a date input appears and is shown here:
Since this is a Rapid Test, results will be ready immediately. The nurse will then select the result:

Positive / Negative / Indeterminate

*Note: we do not have the information on what happens as the next steps based on these results yet. We will build that out at a later time.
Design
Please refer to ANC Labs - Full Prototype in Figma for proper flow.

Screen 1
Screen 2
Screen 3
Screen 4
Screen 5

Please note the top of the screen should read "Labs" or "Lab Results" which is appropriate for the screen. Noticed after I made all the issues that the title of the screens were off.
To Do

Create sub-activity
Create logic for date input and calendar
Style per design

Based on feedback from the team, we are making changes to the HIV lab flow in ANC. We are adding in an additional question "Is this patient know to be HIV positive?" If yes, then the user will click save and it will update the progress report accordingly in the backend and they will move to the next test. If no, then the questions that we already have programmed will appear. I have updated the design in Figma here.  Quick clickthrough can be found here.
Quick explainer video here.
// @anvmn @adamhstewart

@jeanette-pelizzon
This issue has been completed, tested already.
At this stage, any changes require a separate issue.


Provide APK files ?

Is there a way to provided APK files under releases  ?

Hey, its a library to use in other flutter projects

Are you aware of any ongoing implementations ?


fix task scheduling and state evaluation

Pull Request Test Coverage Report for Build 2743

22 of 23 (95.65%)  changed or added relevant lines in 3 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-0.02%) to 69.191%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

core/src/main/java/com/netflix/conductor/core/events/SimpleEventProcessor.java
1
2
50.0%

Files with Coverage Reduction
New Missed Lines
%

redis-persistence/src/main/java/com/netflix/conductor/jedis/JedisMock.java
2
11.44%

Totals

Change from base Build 2723:
-0.02%

Covered Lines:
9394

Relevant Lines:
13577

💛  - Coveralls


syntastic: error: checker python/python returned abnormal status 1

When I use syntastic I meet this problem
Flake8 found issues
syntastic: error: checker output:
Traceback (most recent call last):
File "/home/stone/.vim/bundle/syntastic/syntax_checkers/python/compile.py", li
ne 11, in
compile(open(argv[1]).read(), argv[1], 'exec', 0, 1)
IOError: [Errno 2] No such file or directory: ''
syntastic: error: checker python/python returned abnormal status 1
Could anyone help me solve this problem?

Please open your test file, set g:syntastic_debug to 33, run :SyntasticCheck python, run :mes, and post the output.  Please use cut & paste rather than screenshots.

I run :mes after :SyntasticCheck python
"my_Code/stuff/my_arg.py" 16L, 423C
syntastic: 1.631629: g:syntastic_version = '3.8.0-3 (Vim 704, Linux)'
syntastic: 1.631855: &shell = '/bin/bash', &shellcmdflag = '-c', &shellpipe = '2>&1| tee', &shellquote = '', &shellredir =
'>%s 2>&1', &shelltemp = 1, &shellxquote = '', &autochdir = 0, &shellxescape = ''
syntastic: 1.632579: UpdateErrors: python
syntastic: 1.632901: CacheErrors: python
syntastic: 1.633758: g:syntastic_aggregate_errors = 0
syntastic: 1.633936: $TERM = 'xterm'
syntastic: 1.634056: $PATH = '/home/stone/.pyenv/plugins/pyenv-virtualenv/shims:/home/stone/.pyenv/shims:/home/stone/.pyen
v/bin:/home/stone/perl5/bin:/usr/local/texlive/2015/bin/i386-linux:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin
:/bin:/usr/games:/usr/local/games:/usr/local/texlive/2015/bin/x86_64-linux'
syntastic: 1.634213: getcwd() = '/home/stone'
syntastic: 1.697550: system: command run in 0.051171s
syntastic: 1.697791: python/python: getVersion: 'python --version': ['Python 2.7.6', '']
syntastic: 1.698030: python/python: python version = [2, 7, 6]
syntastic: 1.698306: CacheErrors: Invoking checker: python/python
syntastic: 1.698535: python/python: using compiler script '/home/stone/.vim/bundle/syntastic/syntax_checkers/python/compil
e.py'
syntastic: 1.699505: SyntasticMake: called with options: {'errorformat': '%E%f:%l:%c: %m', 'makeprg': 'python /home/stone/
.vim/bundle/syntastic/syntax_checkers/python/compile.py my_Code/stuff/my_arg.py', 'env': {'TERM': 'dumb'}, 'returns': [0]}
syntastic: 1.771510: system: command run in 0.071405s
syntastic: 1.772121: getLocList: checker python/python returned 0
syntastic: 1.772450: getLocList: checker python/python run in 0.073988s
syntastic: 7.760464: g:syntastic_version = '3.8.0-3 (Vim 704, Linux)'
syntastic: 7.760699: &shell = '/bin/bash', &shellcmdflag = '-c', &shellpipe = '2>&1| tee', &shellquote = '', &shellredir =
'>%s 2>&1', &shelltemp = 1, &shellxquote = '', &autochdir = 0, &shellxescape = ''
syntastic: 7.761242: UpdateErrors: python
syntastic: 7.761594: CacheErrors: python
syntastic: 7.762381: g:syntastic_aggregate_errors = 0
syntastic: 7.762569: $TERM = 'xterm'
syntastic: 7.762699: $PATH = '/home/stone/.pyenv/plugins/pyenv-virtualenv/shims:/home/stone/.pyenv/shims:/home/stone/.pyen
v/bin:/home/stone/perl5/bin:/usr/local/texlive/2015/bin/i386-linux:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin
:/bin:/usr/games:/usr/local/games:/usr/local/texlive/2015/bin/x86_64-linux'
syntastic: 7.762867: getcwd() = '/home/stone'
syntastic: 7.763486: CacheErrors: Invoking checker: python/python
syntastic: 7.763710: python/python: using compiler script '/home/stone/.vim/bundle/syntastic/syntax_checkers/python/compil
e.py'
syntastic: 7.764540: SyntasticMake: called with options: {'errorformat': '%E%f:%l:%c: %m', 'makeprg': 'python /home/stone/
.vim/bundle/syntastic/syntax_checkers/python/compile.py my_Code/stuff/my_arg.py', 'env': {'TERM': 'dumb'}, 'returns': [0]}
syntastic: 7.836444: system: command run in 0.071423s
syntastic: 7.837021: getLocList: checker python/python returned 0
syntastic: 7.837351: getLocList: checker python/python run in 0.073704s

According to your logs all commands have finished normally.

Thanks for your answer

I'm having the same issue. The output per your request is
Flake8 found issues
syntastic: 414.169683: g:syntastic_version = '3.8.0-10 (Vim 704, Neovim, Darwin)'
syntastic: 414.169877: &shell = '/bin/zsh', &shellcmdflag = '-c', &shellpipe = '2>&1| tee', &shellquote = '', &shellredir = '>%s 2>&1', &shelltemp = 1, &shellxquote = '', &autochdir
 = 0, &shellxescape = ''
syntastic: 414.170442: UpdateErrors (auto): default checkers
syntastic: 414.170909: CacheErrors: default checkers
syntastic: 414.172459: g:syntastic_aggregate_errors = 0
syntastic: 414.172658: $TERM = 'screen-256color'
syntastic: 414.172776: $TMPDIR = '/var/folders/70/vftfsmm538z6qghqbxxt1n7w0000gn/T/'
syntastic: 414.172910: $PATH = '/Users/mmorrissey/google-cloud-sdk/bin:/Users/matthew.morrissey/.rvm/gems/ruby-2.1.5/bin:/Users/matthew.morrissey/.rvm/gems/ruby-2.1.5@global/bin:/Us
ers/matthew.morrissey/.rvm/rubies/ruby-2.1.5/bin:/usr/local/bin:.git/safe/../../bin:/Users/matthew.morrissey/.bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/Users/ma
tthew.morrissey/.rvm/bin:/Users/mmorrissey/gocode/bin:/Users/mmorrissey/.fzf/bin'
syntastic: 414.173234: getcwd() = '/Users/mmorrissey/code/dramafever/www'
syntastic: 414.174044: CacheErrors: Invoking checker: python/python
syntastic: 414.174925: python/python: using compiler script '/Users/mmorrissey/.vim/plugged/syntastic/syntax_checkers/python/compile.py'
syntastic: 414.176072: SyntasticMake: called with options: {'errorformat': '%E%f:%l:%c: %m', 'makeprg': 'python /Users/mmorrissey/.vim/plugged/syntastic/syntax_checkers/python/compi
le.py ''''', 'env': {'TERM': 'dumb'}, 'returns': [0]}
syntastic: 414.230920: system: command run in 0.054316s
syntastic: error: checker output:
Traceback (most recent call last):
  File "/Users/mmorrissey/.vim/plugged/syntastic/syntax_checkers/python/compile.py", line 11, in <module>
    compile(open(argv[1]).read(), argv[1], 'exec', 0, 1)
IOError: [Errno 2] No such file or directory: ''
syntastic: error: checker python/python returned abnormal status 1
syntastic: 414.232067: getLocList: checker python/python run in 0.057208s
syntastic: 414.232298: CacheErrors: Invoking checker: python/flake8
syntastic: 414.232711: SyntasticMake: called with options: {'errorformat': '%E%f:%l: could not compile,%-Z%p^,%A%f:%l:%c: %t%n %m,%A%f:%l: %t%n %m,%-G%.%#', 'makeprg': 'flake8 '''''
, 'env': {'TERM': 'dumb'}}
syntastic: 414.653831: system: command run in 0.420789s
syntastic: 414.654391: getLocList: checker python/flake8 returned 0
syntastic: 414.654680: getLocList: checker python/flake8 run in 0.422295s
syntastic: 414.655001: CacheErrors: Checker python/pylint is not available

'makeprg': 'python /Users/mmorrissey/.vim/plugged/syntastic/syntax_checkers/python/compi le.py '''''

@mattamizer Unless you set syntastic_python_python_fname to an empty string, this looks like a Neovim issue.  As I said elsewhere, syntastic should work with Neovim, but it isn't supported.  Can you reproduce the problem with plain Vim?  Can you reproduce it with all other plugins disabled?

It is reproducible with plain VIM:
Flake8 found issues
syntastic: 23.770264: g:syntastic_version = '3.8.0-10 (Vim 704, Darwin)'
syntastic: 23.770449: &shell = '/bin/zsh', &shellcmdflag = '-c', &shellpipe = '2>&1| tee', &shellquote = '', &shellredir = '>%s 2>&1', &shelltemp = 1, &shellxquote = '', &autochdir = 0, &shellxescape = ''
syntastic: 23.770785: UpdateErrors (auto): default checkers
syntastic: 23.771233: CacheErrors: default checkers
syntastic: 23.772190: g:syntastic_aggregate_errors = 0
syntastic: 23.772330: $TERM = 'screen-256color'
syntastic: 23.772408: $TMPDIR = '/var/folders/70/vftfsmm538z6qghqbxxt1n7w0000gn/T/'
syntastic: 23.772636: $PATH = '/Users/mmorrissey/google-cloud-sdk/bin:/Users/matthew.morrissey/.rvm/gems/ruby-2.1.5/bin:/Users/matthew.morrissey/.rvm/gems/ruby-2.1.5@global/bin:/Users/matthew.morrissey/.rvm/rubies/ruby-2.1.5/bin:/usr/local/bin:.git/safe/../../bin:/Users/matthew.morrissey/.bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin:/Users/matthew.morrissey/.rvm/bin:/Users/mmorrissey/gocode/bin:/Users/mmorrissey/.fzf/bin'
syntastic: 23.773862: getcwd() = '/Users/mmorrissey/code/dramafever/www'
syntastic: 23.775039: CacheErrors: Invoking checker: python/python
syntastic: 23.775488: python/python: using compiler script '/Users/mmorrissey/.vim/plugged/syntastic/syntax_checkers/python/compile.py'
syntastic: 23.777006: SyntasticMake: called with options: {'errorformat': '%E%f:%l:%c: %m', 'makeprg': 'python /Users/mmorrissey/.vim/plugged/syntastic/syntax_checkers/python/compile.py ''''', 'env': {'TERM': 'dumb'}, 'returns': [0]}
syntastic: 23.837205: system: command run in 0.059487s
syntastic: error: checker output:
Traceback (most recent call last):
  File "/Users/mmorrissey/.vim/plugged/syntastic/syntax_checkers/python/compile.py", line 11, in <module>
    compile(open(argv[1]).read(), argv[1], 'exec', 0, 1)
IOError: [Errno 2] No such file or directory: ''
syntastic: error: checker python/python returned abnormal status 1
syntastic: 23.856991: getLocList: checker python/python run in 0.081618s
syntastic: 23.859841: CacheErrors: Invoking checker: python/flake8
syntastic: 23.860902: SyntasticMake: called with options: {'errorformat': '%E%f:%l: could not compile,%-Z%p^,%A%f:%l:%c: %t%n %m,%A%f:%l: %t%n %m,%-G%.%#', 'makeprg': 'flake8 ''''', 'env': {'TERM': 'dumb'}}
syntastic: 24.598594: system: command run in 0.737179s
syntastic: 24.601009: getLocList: checker python/flake8 returned 1
syntastic: 24.601399: getLocList: checker python/flake8 run in 0.741338s
syntastic: 24.602085: CacheErrors: Checker python/pylint is not available

With all plugins save syntastic disabled everything seems to work fine, so I think this issue can safely be closed. I'll dig and figure out what the conflict is. Thanks!

@mattamizer You could try something like binary search to isolate the conflicting plugin.  It's always useful to understand how and why these problems happen.

I edited my comment above but I'll add the same here for clarity; the conflict was with vim-flake8. Not really sure why I had that installed as well as syntastic but there you have it. Thanks for the assistance @lcd047!

@mattamizer There seem to be several plugins named vim-flake8.  Which one did you have installed?

nvie/vim-flake8

@mattamizer Looking at the sources it seems vim-flake8 shouldn't be incompatible with syntastic, provided that you don't run both at the same time.  So you could in principle either remove the autocmd on BufWritePost for vim-flake8, or configure syntastic_mode_map to be passive for python.  But syntastic has a flake8 checker, so there doesn't seem to be much point in running both vim-flake8 and syntastic against the same file anyway.


Server stop when i add the plugin

Hello my serveur stop when i add the plugin with no error 1.13.2

We require more information for us to help you.
Please provide this additional information:

Server type (Spigot, Sponge, Paper, ...)
LuckPerms version ("Latest" is not a version)
The latest log of your server (upload it to https://hasteb.in please)
Other plugins you use.

In addition also explain what exact steps you did, as "Server stops when I add the plugin" doesn't give us any clue what exactly you did.


Adding Domain for Universitas Pahlawan

universitaspahlawan.txt
Univ Pahlawan domain

@DGusmans Pull request merged. Thank you!

@DGusmans Pull request merged. Thank you!


Option to accept TLS client certificates even if they lack correct Subject Alternative Names

I'd like to add a machine to an etcd cluster and give it a certificate signed by a ca but not worry about specifying its IP address or its domain name in the certificate. Is there or will there be a way to do this? It would make joining the cluster securely easier.
Also, to the extent that one might want to use etcd to keep track of IP addresses, there's a chicken-and-the-egg problem here.
But perhaps I'm missing some important detail here. Is there a reason why having a signed client certificate isn't sufficient? Why is SAN or DNS name required in the first place?

Bump, im really interested in this as well, if I bootstrap an etcd cluster id like to pass everything (certs included) in the cloud-config but I only know the IP of each machine after it's started (dynamic ip allocation....)
I've considered bootstrapping a dedicated cluster but this becomes unfeasible for a large deployment i think (i'd rather have each service cluster run it's own etcd with local certs than one master ca/cert shared by all clusters)

@ariddell This is the default requirement for go tls library, and we think it is reasonable. If you cannot provide SAN or DNS to your server certificate, it is easy to be abused or misused.
@sepiroth887 etcd member cluster should be in size of 3-9, which is acceptable for manual deployment.

Hmmm.. it is unless you're planing to deploy say 50-90 node clusters of say 10-20 services running their own fleet. we should be able to use DNS though. That way automation is indeed possible. via IP this becomes an allocation and management nightmare.

The cloudflare PKI post from today had some comments on SAN

https://blog.cloudflare.com/how-to-build-your-own-public-key-infrastructure/

Internal services need to trust each other. Browsers validate website certificates by checking the signature on the certificate and checking the hostname against the list of Subject Alternative Names (SANs) in the certificate. This type of explicit check is useful, but it has a record of not working as expected. Another way for services to trust each other is an implicit check based on per-service CAs.

Yup i wanted to go with ca based trust and avoid the need for annoying reverse dns lookups.
Scenario im thinking about here:
vm provisioning -> ip allocated -> cloud config with etcd discovery and then local etcd cluster bootstrap.
Now its easy enough to secure the discovery cluster and small enough for manual cert creation ( though anytime a machine gets a new ip the cert would need to be recreated and all machines updated.
The local cluster would also need its own secured etcd cluster to store state and some service meta data. If i want to be able to just throw more machines at the cluster to scale my services i need to rebuild a cert and reconfigure all other node accordingly. This is a major drawback making it very undesirable for large scale deployments. Imagine doing this for 1000+ VMs.
However i can easily bootstrap an etcd cluster with say a puppet master that hands out certificates either by waiting for user approval or an encrypted password injected via the cluster conf.
As long as the puppet master can identify what cluster a machine belings to it can hand out a fully ca rooted certificate to the requesting node and the cluster is secured easily and scalable.
It also means i can centralize invalidation and audit on a small cluster of puppet masters. So i still think SAN is ok but shouldnt be a requirement.

Sorry for the typos. My hands are bleeding a bit today ;)

I was thinking of a case where one had dynamically assigned ip addresses
in some sort of overlay network -- or a case where you wanted to move
machines around without having to worry about changing the cert because
the IP address changed.
On 06/24, Tobias Haag wrote:

Yup i wanted to go with ca based trust and avoid the need for annoying reverse dns lookups.
Scenario im thinking about here:
vm provisioning -> ip allocated -> cloud config with etcd discovery and then local etcd cluster bootstrap.
Now its easy enough to secure the discovery cluster and small enough for manual cert creation ( though anytime a machine gets a new ip the cert would need to be recreated and all machines updated.
The local cluster would also need its own secured etcd cluster to store state and some service meta data. If i want to be able to just throw more machines at the cluster to scale my services i need to rebuild a cert and reconfigure all other node accordingly. This is a major drawback making it very undesirable for large scale deployments. Imagine doing this for 1000+ VMs.
However i can easily bootstrap an etcd cluster with say a puppet master that hands out certificates either by waiting for user approval or an encrypted password injected via the cluster conf.
As long as the puppet master can identify what cluster a machine belings to it can hand out a fully ca rooted certificate to the requesting node and the cluster is secured easily and scalable.
It also means i can centralize invalidation and audit on a small cluster of puppet masters. So i still think SAN is ok but shouldnt be a requirement.

Reply to this email directly or view it on GitHub:
https://github.com/coreos/etcd/issues/2056#issuecomment-115109306

It looks like it really is impossible to change without forking crypto/tls (https://golang.org/pkg/crypto/tls/). What arrogance.
If I'm wrong about this, I'll reopen.


One DB for multiple clusters

Is there a way to configure k8sguard to write to one DB for many clusters. I want to deploy k8sguard to many clusters and have it inventory everything running for me and write back to a central location.

@joshkurz it wouldn't be a problem if you choose the same database across all environments. as long as there is network connectivity from all environment to that database.


Feat: Separate the CSS for book recommendation page from style.css

Describe the feature
There is only one CSS file which has almost styling for all the pages ,which is very hectic to find the styles for particular pages or components of a pages Because of which the contributors find it hard to search particulars for styling. like pages:
Recommendation page
Solution:
By separating the files just make code modular and loosly coupled which will easy to all.
Add ScreenShots

Record

[X] I agree to follow this project's Code of Conduct
[X] I'm a GSSOC'24 contributor
[X] I want to work on this issue

@RishabhDhawad @huamanraj please assign me ...

@RishabhDhawad @huamanraj please assign me ...

Assigned!
Make sure to attach before & after screenshots in your PR and include the issue number.
Also, please star the repo before working on it.
I am the project mentor so you can always reach out to me for any help!
Happy open-source contributing!


docs(v2): convert teams page to MDX

Motivation
It contains React code, using mdx extension is better understood by tooling (IDE, Prettier, Crowdin...)

added @lex111 in active team


Build angular-cli fork fails

Please provide us with the following information:

OS?

Windows 10.

Versions.

angular-cli: 1.0.0-beta.26
node: 5.11.0
os: win32 ia32
@angular/compiler: 2.4.5
@angular/compiler-cli: 2.4.5
@angular/core: 2.4.5
@angular/tsc-wrapped: 0.5.1

Repro steps.

I have downloaded beta26 angular-cli, from here: https://github.com/angular/angular-cli/releases/tag/v1.0.0-beta.26
npm install - success
npm run build - fails

The log given by the failure.
node ./scripts/publish/build.js

Deleting dist folder...
Creating schema.d.ts...
module.js:341
    throw err;
    ^

Error: Cannot find module 'C:\Users\Fuit'
    at Function.Module._resolveFilename (module.js:339:15)
    at Function.Module._load (module.js:290:25)
    at Function.Module.runMain (module.js:447:10)
    at startup (node.js:148:18)
    at node.js:405:3
Command failed: node C:\Users\Fuit Attila\Desktop\angular-cli-1.0.0-beta.26\scripts\build-schema-dts.js C:\Users\Fuit At
tila\Desktop\angular-cli-1.0.0-beta.26\packages\angular-cli\lib\config\schema.json C:\Users\Fuit Attila\Desktop\angular-
cli-1.0.0-beta.26\packages\angular-cli\lib\config\schema.d.ts
module.js:341
    throw err;
    ^

Error: Cannot find module 'C:\Users\Fuit'
    at Function.Module._resolveFilename (module.js:339:15)
    at Function.Module._load (module.js:290:25)
    at Function.Module.runMain (module.js:447:10)
    at startup (node.js:148:18)
    at node.js:405:3

npm ERR! Windows_NT 10.0.14393
npm ERR! argv "C:\\Program Files (x86)\\nodejs\\node.exe" "C:\\Program Files (x86)\\nodejs\\node_modules\\npm\\bin\\npm-
cli.js" "run" "build"
npm ERR! node v5.11.0
npm ERR! npm  v3.8.6
npm ERR! code ELIFECYCLE
npm ERR<EMAIL_ADDRESS>build: node ./scripts/publish/build.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the<EMAIL_ADDRESS>build script 'node ./scripts/publish/build.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the angular-cli package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node ./scripts/publish/build.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs angular-cli
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls angular-cli
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     C:\Users\Fuit Attila\Desktop\angular-cli-1.0.0-beta.26\npm-debug.log
PS C:\Users\Fuit Attila\Desktop\angular-cli-1.0.0-beta.26

What am I doing wrong?
Thank you for your help!

Going out on a limb, could you rename the folder "C:\Users\Fuit_Attila\Desktop\angular-cli-1.0.0-beta.26" and re-try ?

@catull
Good idea thank you! I will give it a try soon.

@catull
I have moved my project to here: C:\WORK.... It worked, thank you!
But now I have an other failure:
><EMAIL_ADDRESS>build C:\WORK\angular-cli-1.0.0-beta.26
> node ./scripts/publish/build.js

Deleting dist folder...
Creating schema.d.ts...
Compiling packages...
  @angular-cli\ast-tools
Compilation error.
node_modules/@types/lodash/index.d.ts(245,25): error TS2304: Cannot find name 'Partial'.

Any idea what am I doing wrong? Maybe I shoud install some typings globally - unit it will be fixed?

In package.json should upgrade typescript version:
"typescript": "~2.1.0"
Than it builds perfectly! I hope it helps to build an own fork from beta26 :/

This issue can be closed now.
The solution is to not have a path with a space in it.
@fuitattila The next issue you reported has nothing to do with your original issue.  Please open a new separate one.

@fuitattila not being able to build on windows when on a path with spaces is a bug.
The cannot find name partial bug was fixed with https://github.com/angular/angular-cli/pull/4260.


Release: hwc-buildpack (Feb)

Release reminder for hwc-buildpack

See diff from latest version and validate if a release is required.
Make sure the latest commit on  has passed tests on the CI
Refer release instructions. (private link)

V3.1.35 Released.


Migrate models to ndb.Model

See https://developers.google.com/appengine/docs/python/ndb/

The models in model.py should inherit from ndb.Model, not (as they currently
do) db.Model.

Original issue reported on code.google.com by<EMAIL_ADDRESS>on 30 Sep 2013 at 11:42

Closing - This repository is being archived. It is no longer maintained.


the key binding is removed, how to add it?

hi, it's a good plugin.
But it seems that the key binding is removed in the latest version, how to add it by myself?

@cqroot hi, I am using Mac, but it seems I could not find this configuration。


Add package redux

Include more tags that are not the name of the package next time as well, for example library, js :)

@dom96 ok, but it's not limited by JS only 😄


Scars do not work with physics engine

I tried using scars and the cars have some will anti gravity ability.

It's duplicated. See #107 😄


Using autoload directory instead of plugin directory

When I use tweekmonster/startuptime.vim to measure this plugin, it took around 0.005 ms. That was too slow.
So, using autoload directory instead of plugin directory increase the speed significantly by only load the plugin when the user needs it.

Nice, I was having unacceptable latency too, especially on my lower-specced (HDD) machines as well
I take performance very seriously and your contribution is very welcome, thanks a lot!

@michaelb glad I could help, it would be scary if this plugin didn't become the fastest vim plugin ever!


使用rdrop, it shows a tf.split error. 大佬帮忙看看

code from https://github.com/EdwardChan5000/m3tl_run
错误show below
2022-09-21 16:17:06.321 | INFO     | m3tl.utils:set_phase:478 - Setting phase to infer
2022-09-21 16:17:06.345 | CRITICAL | m3tl.model_fn:compile:271 - Initial lr: 2e-05
2022-09-21 16:17:06.345 | CRITICAL | m3tl.model_fn:compile:272 - Train steps: 408675
2022-09-21 16:17:06.345 | CRITICAL | m3tl.model_fn:compile:273 - Warmup steps: 40867
2022-09-21 16:17:06.361554: I tensorflow/core/profiler/lib/profiler_session.cc:136] Profiler session initializing.
2022-09-21 16:17:06.361588: I tensorflow/core/profiler/lib/profiler_session.cc:155] Profiler session started.
2022-09-21 16:17:06.361613: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1365] Profiler found 1 GPUs
2022-09-21 16:17:06.369724: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcupti.so.11.0
2022-09-21 16:17:06.655982: I tensorflow/core/profiler/lib/profiler_session.cc:172] Profiler session tear down.
2022-09-21 16:17:06.656157: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1487] CUPTI activity buffer flushed
2022-09-21 16:17:07.017 | INFO     | m3tl.utils:set_phase:478 - Setting phase to train
WARNING:tensorflow:The parameters output_attentions, output_hidden_states and use_cache cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: config=XConfig.from_pretrained('name', output_attentions=True)).
WARNING:tensorflow:The parameter return_dict cannot be set in graph mode and will always be set to True.
2022-09-21 16:17:19.175 | INFO     | m3tl.utils:set_phase:478 - Setting phase to train
Traceback (most recent call last):
File "m3tl_4room_rdrop.py", line 195, in
main(args)
File "m3tl_4room_rdrop.py", line 149, in main
create_tf_record_only=False, model_dir=model_dir, mirrored_strategy=mirrored_strategy)
File "/usr/local/lib/python3.6/site-packages/m3tl/run_bert_multitask.py", line 319, in train_bert_multitask
verbose=verbose
File "/usr/local/lib/python3.6/site-packages/m3tl/run_bert_multitask.py", line 163, in _train_bert_multitask_keras_model
validation_steps=validation_steps
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1100, in fit
tmp_logs = self.train_function(iterator)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 828, in call
result = self._call(*args, **kwds)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 855, in _call
return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 2943, in call
filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 1919, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/function.py", line 560, in call
ctx=ctx)
File "/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument:  Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 7) and num_split 2
[[node BertMultiTask/BertMultiTaskTop/rdrop_preprocess/rdrop_preprocess/split_1 (defined at data/yard/workspace/vega/m3tl_run/custom_top.py:250) ]]
(1) Invalid argument:  Number of ways to split should evenly divide the split dimension, but got split_dim 0 (size = 7) and num_split 2
[[node BertMultiTask/BertMultiTaskTop/rdrop_preprocess/rdrop_preprocess/split_1 (defined at data/yard/workspace/vega/m3tl_run/custom_top.py:250) ]]
[[BertMultiTask/BertMultiTaskTop/rdrop_preprocess/rdrop_preprocess/split_1/_44]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_423171]
Errors may have originated from an input operation.
Input Source operations connected to node BertMultiTask/BertMultiTaskTop/rdrop_preprocess/rdrop_preprocess/split_1:
BertMultiTask/basic_mtl/GatherNd (defined at usr/local/lib/python3.6/site-packages/m3tl/utils.py:412)
Input Source operations connected to node BertMultiTask/BertMultiTaskTop/rdrop_preprocess/rdrop_preprocess/split_1:
BertMultiTask/basic_mtl/GatherNd (defined at usr/local/lib/python3.6/site-packages/m3tl/utils.py:412)
Function call stack:
train_function -> train_function

好的，我下周看看
On Thu, Sep 22, 2022, 8:35 PM Edward Chan @.***> wrote:

code from https://github.com/EdwardChan5000/m3tl_run
错误show below
2022-09-21 16:17:06.321 | INFO | m3tl.utils:set_phase:478 - Setting phase
to infer
2022-09-21 16:17:06.345 | CRITICAL | m3tl.model_fn:compile:271 - Initial
lr: 2e-05
2022-09-21 16:17:06.345 | CRITICAL | m3tl.model_fn:compile:272 - Train
steps: 408675
2022-09-21 16:17:06.345 | CRITICAL | m3tl.model_fn:compile:273 - Warmup
steps: 40867
2022-09-21 16:17:06.361554: I
tensorflow/core/profiler/lib/profiler_session.cc:136] Profiler session
initializing.
2022-09-21 16:17:06.361588: I
tensorflow/core/profiler/lib/profiler_session.cc:155] Profiler session
started.
2022-09-21 16:17:06.361613: I
tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1365] Profiler found
1 GPUs
2022-09-21 16:17:06.369724: I
tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully
opened dynamic library libcupti.so.11.0
2022-09-21 16:17:06.655982: I
tensorflow/core/profiler/lib/profiler_session.cc:172] Profiler session tear
down.
2022-09-21 16:17:06.656157: I
tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1487] CUPTI activity
buffer flushed
2022-09-21 16:17:07.017 | INFO | m3tl.utils:set_phase:478 - Setting phase
to train
WARNING:tensorflow:The parameters output_attentions, output_hidden_states
and use_cache cannot be updated when calling a model.They have to be set
to True/False in the config object (i.e.: config=XConfig.from_pretrained('name',
output_attentions=True)).
WARNING:tensorflow:The parameter return_dict cannot be set in graph mode
and will always be set to True.
2022-09-21 16:17:19.175 | INFO | m3tl.utils:set_phase:478 - Setting phase
to train
Traceback (most recent call last):
File "m3tl_4room_rdrop.py", line 195, in
main(args)
File "m3tl_4room_rdrop.py", line 149, in main
create_tf_record_only=False, model_dir=model_dir,
mirrored_strategy=mirrored_strategy)
File "/usr/local/lib/python3.6/site-packages/m3tl/run_bert_multitask.py",
line 319, in train_bert_multitask
verbose=verbose
File "/usr/local/lib/python3.6/site-packages/m3tl/run_bert_multitask.py",
line 163, in _train_bert_multitask_keras_model
validation_steps=validation_steps
File
"/usr/local/lib64/python3.6/site-packages/tensorflow/python/keras/engine/training.py",
line 1100, in fit
tmp_logs = self.train_function(iterator)
File
"/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py",
line 828, in call
result = self._call(*args, **kwds)
File
"/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/def_function.py",
line 855, in _call
return self._stateless_fn(*args, **kwds) # pylint: disable=not-callable
File
"/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/function.py",
line 2943, in call
filtered_flat_args, captured_inputs=graph_function.captured_inputs) #
pylint: disable=protected-access
File
"/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/function.py",
line 1919, in _call_flat
ctx, args, cancellation_manager=cancellation_manager))
File
"/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/function.py",
line 560, in call
ctx=ctx)
File
"/usr/local/lib64/python3.6/site-packages/tensorflow/python/eager/execute.py",
line 60, in quick_execute
inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root
error(s) found.
(0) Invalid argument: Number of ways to split should evenly divide the
split dimension, but got split_dim 0 (size = 7) and num_split 2
[[node
BertMultiTask/BertMultiTaskTop/rdrop_preprocess/rdrop_preprocess/split_1
(defined at data/yard/workspace/vega/m3tl_run/custom_top.py:250) ]]
(1) Invalid argument: Number of ways to split should evenly divide the
split dimension, but got split_dim 0 (size = 7) and num_split 2
[[node
BertMultiTask/BertMultiTaskTop/rdrop_preprocess/rdrop_preprocess/split_1
(defined at data/yard/workspace/vega/m3tl_run/custom_top.py:250) ]]
[[BertMultiTask/BertMultiTaskTop/rdrop_preprocess/rdrop_preprocess/split_1/_44]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_423171]
Errors may have originated from an input operation.
Input Source operations connected to node
BertMultiTask/BertMultiTaskTop/rdrop_preprocess/rdrop_preprocess/split_1:
BertMultiTask/basic_mtl/GatherNd (defined at
usr/local/lib/python3.6/site-packages/m3tl/utils.py:412)
Input Source operations connected to node
BertMultiTask/BertMultiTaskTop/rdrop_preprocess/rdrop_preprocess/split_1:
BertMultiTask/basic_mtl/GatherNd (defined at
usr/local/lib/python3.6/site-packages/m3tl/utils.py:412)
Function call stack:
train_function -> train_function
—
Reply to this email directly, view it on GitHub
https://github.com/JayYip/m3tl/issues/112, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADS2OTDITUG23H7HZGCEYTDV7RHADANCNFSM6AAAAAAQTAIMCM
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>


[Bug] Slider: Investigate issue when slider is in a vertical scroll view

https://adevinta.slack.com/archives/C04QCSQGQUX/p1712671798336269

For all UIControls using beginTracking / endTracking, we have to add this to prevent cancelTracking from being called on vertical swipes
        // Fix conflict with vertical swipes (scrollViews, bottomSheets...)
        let panGesture = UIPanGestureRecognizer(target: nil, action: nil)
        panGesture.cancelsTouchesInView = false
        self.addGestureRecognizer(panGesture)


Dike out the attractive nuisance deploy button.

It's good that it works, but pressing it right now is an invitation to have your data destroyed later. I've also called that out explicitly in the README.

Wrong source branch, wrong source repo.


提交规则：关闭WfpDiag.ETL日志

命令行方式：
NETSH WFP SET OPTIONS NETEVENTS = OFF
注册表方式：
Windows Registry Editor Version 5.00
[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\BFE\Parameters\Policy\Options]
"CollectNetEvents"=dword:00000000

谢谢支持~~

谢谢支持，id：0fa08da1已经添加此规则

When adding new options like this in Dism++, please make sure that there's not just a Chinese text for the UI, but add at least a rough English translation (even if it gets corrected later by native English translators).
Please don't just push it in updates sent to everyone, most of your users CANNOT read Chinese at all and cannot interpret this Chinese "garbage" in their translated UI (and there's also still no way to copy these texts in the UI to the clipboard to find some rough translation online) ! Thanks.
Instead take the time to copy-paste the text and use at least some online translator to English, and push it at least in the English language XML bundle (with a dummy q="" attribute to mark it as fuzzy, so transaltors will easily spot these new additions, because there's no way to locate the Chinese text hidden in submitted patches in the opensourced folder, also because you do not add any readable version history in Dism updates, and do not even list them with their Github issue number which would facilitate locating where these new/modified texts were submitted)

So in summary,
This is a trace file created by Windows to collect diagnostics information around ipsec and firewall activity.
Such info could be later read by a troubleshooting tool to help diagnose connectivity failures. I would not expect it to be active unless you are receiving (or maybe sending) traffic which is being blocked by your security software.
The builtin firewall may collect these events in large files which may sometimes become very large, and the option here is supposed to stop this collection when the user is unable to analyse them and does not intend to submit these logs to a professional. These ETL log from the firewall are normally small, and can already be cleared with the Windows Events Viewer. But they can also be disabled in the Windows Event viewer interface itself, which provides a summary of events by source. In some cases, the log files may exhaust their maximum size and Windows will either start recyling them by dropping the oldest messages to keep only the most recent ones (so no maintenance is needed in this case), or will stop logging more events (you'll get an additional event in the separate system log saying that the event log file from the firewall is full and needs either archiving for later inspection, and cleanup before Windows can keep new loged events, or that it needs to be tuned to give it a larger maximum size).
It you have a properly concigured Internet router, it should already block most unwanted incoming traffic and the internal firewall will block very few messages, except if your PC is being harnessed by some malicious third party: when this log file fills too fast, you should inspect your security settings and system/software updates, and inspect network activities to detect unwanted apps/components, and run an antivirus/antispyware/antirootkit scan (and probably you should also consider rebooting Windows to stop some background processes that are constantly failing and generate lot of logs: these apps may need to be reset or may have their own logs).


Update the http version in pubspeck dependency

Can someone upgrade the http version ebcause it using 0.13.5, I cant use the latest google fonts 5.10 because it depends on http ^1.0.0

Will be fixed in this PR https://github.com/livekit/client-sdk-flutter/pull/313


[SPARK-6190][core] create LargeByteBuffer for eliminating 2GB block limit

https://issues.apache.org/jira/browse/SPARK-6190

Test build #29808 has started for   PR 5400 at commit 9f53203.

Test build #29808 has finished for   PR 5400 at commit 9f53203.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

class LargeByteBufferOutputStream(chunkSize: Int = 65536)

public class BufferTooLargeException extends IOException

public class LargeByteBufferHelper

public class WrappedLargeByteBuffer implements LargeByteBuffer

This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29808/
Test PASSed.

I didn't look at the tests in detail; I found some discrepancies between the code and the LargeByteBuffer interface that should probably be fixed one way or another (either the interface needs updating, or the code needs fixing).

Test build #29942 has started for   PR 5400 at commit a759242.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29942/
Test FAILed.

jenkins, test this please

Test build #29944 has started for   PR 5400 at commit a759242.

Test build #29944 has finished for   PR 5400 at commit a759242.

This patch fails Spark unit tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

class LargeByteBufferOutputStream(chunkSize: Int = 65536)

public class BufferTooLargeException extends IOException

public class LargeByteBufferHelper

public class WrappedLargeByteBuffer implements LargeByteBuffer

This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29944/
Test FAILed.

jenkins, test this please

Test build #29972 has started for   PR 5400 at commit a759242.

Test build #29972 has finished for   PR 5400 at commit a759242.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

class LargeByteBufferOutputStream(chunkSize: Int = 65536)

public class BufferTooLargeException extends IOException

public class LargeByteBufferHelper

public class WrappedLargeByteBuffer implements LargeByteBuffer

This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29972/
Test PASSed.

Test build #30190 has started for   PR 5400 at commit c3efa4c.

Test build #30190 has finished for   PR 5400 at commit c3efa4c.

This patch fails Scala style tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

class LargeByteBufferOutputStream(chunkSize: Int = 65536)

public class BufferTooLargeException extends IOException

public class LargeByteBufferHelper

public class WrappedLargeByteBuffer implements LargeByteBuffer

This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30190/
Test FAILed.

Test build #30249 has started for   PR 5400 at commit e1d8fa8.

Test build #30249 has finished for   PR 5400 at commit e1d8fa8.

This patch fails Scala style tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

class LargeByteBufferOutputStream(chunkSize: Int = 65536)

public class BufferTooLargeException extends IOException

public class LargeByteBufferHelper

public class WrappedLargeByteBuffer implements LargeByteBuffer

This patch adds the following new dependencies:

commons-math3-3.4.1.jar

This patch removes the following dependencies:

commons-math3-3.1.1.jar

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30249/
Test FAILed.

Test build #30251 has started for   PR 5400 at commit d641834.

Test build #30251 has finished for   PR 5400 at commit d641834.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

class LargeByteBufferOutputStream(chunkSize: Int = 65536)

public class BufferTooLargeException extends IOException

public class LargeByteBufferHelper

public class WrappedLargeByteBuffer implements LargeByteBuffer

This patch adds the following new dependencies:

commons-math3-3.4.1.jar

This patch removes the following dependencies:

commons-math3-3.1.1.jar

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30251/
Test PASSed.

Test build #30272 has started for   PR 5400 at commit 78e4043.

Test build #30272 has finished for   PR 5400 at commit 78e4043.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

public class BufferTooLargeException extends IOException

public class LargeByteBufferHelper

public class WrappedLargeByteBuffer implements LargeByteBuffer

This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30272/
Test PASSed.

Thanks for comments @vanzin.  I think I've addressed all of your concerns.  I went back and forth a few times about what the contract of asByteBuffer should be, thanks for pointing out the inconsistencies.  I think I've cleaned that up and added more unit tests.

Test build #30280 has started for   PR 5400 at commit 117fb4a.

Test build #30280 has finished for   PR 5400 at commit 117fb4a.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

public class BufferTooLargeException extends IOException

public class LargeByteBufferHelper

public class WrappedLargeByteBuffer implements LargeByteBuffer

This patch adds the following new dependencies:

snappy-java-<IP_ADDRESS>.jar

This patch removes the following dependencies:

snappy-java-<IP_ADDRESS>.jar

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30280/
Test PASSed.

I left another round of comments. This is looking a lot better, although I need to look at the latter tests in more detail. I'm a little concerned about writing large files during the test, it would be great to have an alternative to that (maybe use mockito?).

@squito  is this still being worked on?

@tgravescs I'd really like to keep this going ... I was hoping to get feedback from more reviewers on the design.  Also the testing strategy -- I dunno if anybody else can think of a better way to test behavior over 2GB without making such big tests?
I had gotten lazy about updating from marcelo's latest round of comments, I'll do that now

thanks @squito I've got multiple customers hitting this issue so would love to see it fixed.  Unfortunately bad timing on my part as I'm going to be out next week, then at summits.  I'll look at this in more detail and give feedback when I can.

Thanks for the renewed interest.  I'd like to also reviewers to consider one aspect of the design that I went back and forth on a lot.  Currently, LargeByteBuffer has a asByteBuffer method -- that's so that if some parts of the code don't support large blocks (eg., replication & shuffles), we can still fall back on the existing code that uses ByteBuffers when possible.
However, that leads to some inefficiencies -- eg., WrappedLargeByteBuffer is forced to always create chunks as large as possible.  That forces allocating big byte arrays, even though otherwise the implementation would be happy with multiple modestly sized ones.  Similarly, LargeByteBufferOutputStream is forced to do extra copying to go from its smaller buffers to one large one.  I don't see any good alternatives, but I'd like other opinions.
I need to refresh myself a little on how this fits into the other changes I have for using this to cache large blocks, but I will work on making all the updates suggested here.

Merged build triggered.

Merged build started.

Test build #33903 has started for   PR 5400 at commit 3c599b2.

Test build #33903 has finished for   PR 5400 at commit 3c599b2.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class LargeByteBufferInputStream extends InputStream
public class LargeByteBufferOutputStream extends OutputStream
public class BufferTooLargeException extends IOException
public class LargeByteBufferHelper
public class WrappedLargeByteBuffer implements LargeByteBuffer

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #33904 has started for   PR 5400 at commit 95588c2.

Test build #33904 has finished for   PR 5400 at commit 95588c2.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class LargeByteBufferInputStream extends InputStream
public class LargeByteBufferOutputStream extends OutputStream
public class BufferTooLargeException extends IOException
public class LargeByteBufferHelper
public class WrappedLargeByteBuffer implements LargeByteBuffer

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #33906 has started for   PR 5400 at commit b77bbe2.

Test build #33906 has finished for   PR 5400 at commit b77bbe2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class LargeByteBufferInputStream extends InputStream
public class LargeByteBufferOutputStream extends OutputStream
public class BufferTooLargeException extends IOException
public class LargeByteBufferHelper
public class WrappedLargeByteBuffer implements LargeByteBuffer

Merged build finished. Test FAILed.

Jenkins, retest this please

Merged build triggered.

Merged build started.

Test build #33911 has started for   PR 5400 at commit b77bbe2.

Test build #33911 has finished for   PR 5400 at commit b77bbe2.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class LargeByteBufferInputStream extends InputStream
public class LargeByteBufferOutputStream extends OutputStream
public class BufferTooLargeException extends IOException
public class LargeByteBufferHelper
public class WrappedLargeByteBuffer implements LargeByteBuffer

Merged build finished. Test PASSed.

Merged build triggered.

Merged build started.

I think I've addressed the comments.  I also cleaned up a couple of bits that were more complicated than needed and added a few more tests. @tgravescs @vanzin @JoshRosen

Test build #33956 has started for   PR 5400 at commit 6c2a115.

Test build #33956 has finished for   PR 5400 at commit 6c2a115.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class LargeByteBufferInputStream extends InputStream
public class LargeByteBufferOutputStream extends OutputStream
public class BufferTooLargeException extends IOException
public class LargeByteBufferHelper
public class WrappedLargeByteBuffer implements LargeByteBuffer

Merged build finished. Test PASSed.

Hey, one idea for avoiding tests that write huge files: can we use a data generator + streaming checksum function to be able to detect data corruption without having to materialize the entire file?

Merged build triggered.

Merged build started.

Test build #34017 has started for   PR 5400 at commit 112c49e.

Hey, one idea for avoiding tests that write huge files: can we use a data generator + streaming
checksum function to be able to detect data corruption without having to materialize the entire file?

I don't see an easy way to do that.  WrappedLargeByteBuffer doesn't have any hooks to generate the underlying byte buffers on the fly.  This was intentional, to keep the implementation as simple as possible in the initial version, and to leave that as a potential optimization in the future.  I could add another implementation of LargeByteBuffer just for testing, but that seems to defeat the purpose of testing the implementation we'd actually use in.  Or we could change the actual implementation to support that, but again, I'm reluctant to over-complicate the initial version -- we could certainly do fancier things but its not needed to just eliminate the hard barrier at 2GB we have now.
I'm sort of just playing devil's advocate -- I totally understand the hesitation on including the tests as is, but not sure what else we should do.  Definitely open to continue discussing it & other ideas

The latest update seems ok to me after a cursory check of the things I commented on.  Maybe you should revive that test tags patch of yours, @squito.
@JoshRosen IIUC the purpose of the large file test would be to test the actual mmap code, so using a checksumming stream doesn't seem to me like it would help much (since you still would need to create the file and read it back through the large buffer API).

Test build #34017 has finished for   PR 5400 at commit 112c49e.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class LargeByteBufferInputStream extends InputStream
public class LargeByteBufferOutputStream extends OutputStream
public class BufferTooLargeException extends IOException
public class LargeByteBufferHelper
public class WrappedLargeByteBuffer implements LargeByteBuffer

Merged build finished. Test PASSed.

Merged build triggered.

Merged build started.

Test build #34031 has started for   PR 5400 at commit 8ec2c5c.

Test build #34031 has finished for   PR 5400 at commit 8ec2c5c.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class LargeByteBufferInputStream extends InputStream
public class LargeByteBufferOutputStream extends OutputStream
public class BufferTooLargeException extends IOException
public class LargeByteBufferHelper
public class WrappedLargeByteBuffer implements LargeByteBuffer

Merged build finished. Test PASSed.

Merged build triggered.

Merged build started.

Test build #34101 has started for   PR 5400 at commit b6620d0.

Merged build triggered.

Merged build started.

Test build #34105 has started for   PR 5400 at commit 54d09af.

Test build #34101 has finished for   PR 5400 at commit b6620d0.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class LargeByteBufferInputStream extends InputStream
public class LargeByteBufferOutputStream extends OutputStream
public class BufferTooLargeException extends IOException
public class LargeByteBufferHelper
public class WrappedLargeByteBuffer implements LargeByteBuffer

Merged build finished. Test PASSed.

Test build #34105 has finished for   PR 5400 at commit 54d09af.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class LargeByteBufferInputStream extends InputStream
public class LargeByteBufferOutputStream extends OutputStream
public class BufferTooLargeException extends IOException
public class LargeByteBufferHelper
public class WrappedLargeByteBuffer implements LargeByteBuffer

Merged build finished. Test PASSed.

Merged build triggered.

Merged build started.

Test build #34120 has started for   PR 5400 at commit 31244c8.

Test build #34120 timed out     for PR 5400 at commit 31244c8     after a configured wait of 150m.

Merged build finished. Test FAILed.

If you haven't already, can you pass this code through FindBugs to see if it finds any possible issues, such as arithmetic overflows, etc?

@JoshRosen I just ran FindBugs, it didn't turn up anything.  Good idea in any case, I hadn't done it yet.

jenkins retest this please

Merged build triggered.

Merged build started.

Test build #34349 has started for   PR 5400 at commit 31244c8.

Test build #34349 has finished for   PR 5400 at commit 31244c8.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class LargeByteBufferInputStream extends InputStream
public class LargeByteBufferOutputStream extends OutputStream
public class BufferTooLargeException extends IOException
public class LargeByteBufferHelper
public class WrappedLargeByteBuffer implements LargeByteBuffer

Merged build finished. Test PASSed.

Merged build triggered.

Merged build started.

Test build #34637 has started for   PR 5400 at commit ca23763.

@chenghao-intel here's a rough cut at how this would be used to make caching work for blocks over 2GB.  https://github.com/squito/spark/pull/1
(there is a lot of garbage in there, so don't look too closely, its just to give an idea ...)

Test build #34637 has finished for   PR 5400 at commit ca23763.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class LargeByteBufferInputStream extends InputStream
public class LargeByteBufferOutputStream extends OutputStream
public class BufferTooLargeException extends IOException
public class LargeByteBufferHelper
public class WrappedLargeByteBuffer implements LargeByteBuffer

Merged build finished. Test PASSed.

ping @tgravescs
btw, another design issue that was brought up earlier was whether we should just limit each block to never exceed 2GB, and just have the BlockManager take care of figuring out how to make multiple blocks if you ever went over.  However, after prototyping that idea, I think it just pushes a lot more complexity into the BlockManager -- I wrote up some more notes on the jira here https://issues.apache.org/jira/browse/SPARK-6190?focusedCommentId=14387275&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14387275
but I'm open to more opinions on it.

starting to take a look. have to catch up on the discussions.

so I haven't looked at all the network layers yet but this approach seems reasonable to me.  If we end up wanting to do more logic at higher layer this doesn't prevent that.  It would become an extra unneeded abstraction but at that point you could remove it if it makes sense.   Based on the previous pull request doing it this way seems fairly non-intrusive.
@rxin I think you had commented on previous pr or jira about the 2 options.  Did you have specific concerns that haven't been addressed?
@squito  Assuming you have patches for caching, network transfer, and shuffle based on this have you done any performance testing?

@tgravescs sorry I missed your comment earlier.  I only did a proof of concept for network transfer & shuffles, so I never did any performance testing.  I did some light testing w/ caching, there was no noticeable change in performance -- I can bring that up to date and test it out again.
fyi, the major complication w/ changing the network layers is that we are now using netty's  LengthFieldBasedFrameDecoder which has a limit of 2G.  so we'd need to implement our own frame decoder for larger frames.

@squito having heard no strong objections from anyone would it be possible to bring the proof of concepts up to date?  I would be very interested in getting this all together and doing some testing on it.

Hey @tgravescs , sorry for the late response.  Absolutely, it would be great to get you to start testing this.  I'm a bit overloaded at the moment but probably next week.
I think this PR itself is ready to go, though I know its not that useful by itself.  I'll update the other one which actually uses it for caching.

Merged build triggered.

Merged build started.

Test build #41253 has started for   PR 5400 at commit 040a461.

There was an old thread about how LargeByteBufferHelperSuite.testMapFiles and using random number generator instead of repeating the same bytes over and over again -- I switched to an XORShiftRandom generator, which makes the test go from 5s to 11s on my laptop.
The only other remaining issue that I know of is that there is a test that memory maps a file > 2GB, but there wasn't any consensus on what to do.  I'll gladly introduce test tags separately if that will alleviate concerns.

Test build #41253 has finished for   PR 5400 at commit 040a461.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class LargeByteBufferInputStream extends InputStream
public class LargeByteBufferOutputStream extends OutputStream
public class BufferTooLargeException extends IOException
public class LargeByteBufferHelper
public class WrappedLargeByteBuffer implements LargeByteBuffer

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41253/
Test PASSed.

Merged build finished. Test PASSed.

Merged build triggered.

Merged build started.

Test build #41274 has started for   PR 5400 at commit 80c4032.

Test build #41274 timed out     for PR 5400 at commit 80c4032     after a configured wait of 175m.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41274/
Test FAILed.

how long does the >2GB test take to run?

How is it going? Is it still WIP? I can help to test.

I had already reviewed this and I don't see any changes, so my only worry is still the same as Tom's: how long does the large file test takes. I guess it's not horrible if it's a single test taking 10s, but if we could avoid and still be reasonably sure that things work, it would be better.

Merged build triggered.

Merged build started.

Thanks for taking another look @vanzin, fixed those last issues.  Sorry I never responded earlier @tgravescs  -- that one test takes about 10s on my laptop, looks like it took 15s on the last jenkins run.  Personally I feel better w/ the test in there.  But I agree its not adding a ton of value -- happy to scrap it if you prefer.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45029/
Test FAILed.

jenkins, retest this please

Merged build triggered.

Merged build started.

Test build #45032 has started for PR 5400 at commit 3447bb9.

Test build #45032 has finished for PR 5400 at commit 3447bb9.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n  * public class LargeByteBufferInputStream extends InputStream \n  * public class LargeByteBufferOutputStream extends OutputStream \n  * public class JavaAssociationRulesExample \n  * public class JavaPrefixSpanExample \n  * public class JavaSimpleFPGrowth \n  * public class BufferTooLargeException extends IOException \n  * public class LargeByteBufferHelper \n  * public class WrappedLargeByteBuffer implements LargeByteBuffer \n  * class StreamInterceptor implements TransportFrameDecoder.Interceptor \n  * public final class ChunkFetchSuccess extends ResponseWithBody \n  * public abstract class ResponseWithBody implements ResponseMessage \n  * public final class StreamFailure implements ResponseMessage \n  * public final class StreamRequest implements RequestMessage \n  * public final class StreamResponse extends ResponseWithBody \n  * public class TransportFrameDecoder extends ChannelInboundHandlerAdapter \n

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45032/
Test PASSed.

hi @squito, can you explain in which situation users will hit the 2g limit? will a job of processing very large data(such as PB level data) reach this limit?

@scwf ,

the shuffle output from one mapper to one reducer cannot be more than 2GB.
partitions of an RDD cannot exceed 2GB.

@snnn 2 is not true. Partitions can be as large as possible. The cached size cannot be greater than 2GB.

The cached size cannot be greater than 2GB.

@rxin how to understand the cached size? the partition size of a cached rdd?

Hey does this address the issue of spark.sql.autoBroadcastJoinThreshold cannot be more than 2GB?

@squito @SparkQA @vanzin @shaneknapp @tgravescs
I am using spark.sql on AWS Glue to generate a single large (it is the clients requirement to have a single file)  csv compressed file which is greater than 2GB for sure. I am running to into this issue
write(transformed_feed)
File "script_2019-02-12-15-57-55.py", line 161, in write
output_path_premium, header=True, compression="gzip")
File "/mnt/yarn/usercache/root/appcache/application_1549986900582_0001/container_1549986900582_0001_01_000001/pyspark.zip/pyspark/sql/readwriter.py", line 766, in csv
File "/mnt/yarn/usercache/root/appcache/application_1549986900582_0001/container_1549986900582_0001_01_000001/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in call
File "/mnt/yarn/usercache/root/appcache/application_1549986900582_0001/container_1549986900582_0001_01_000001/pyspark.zip/pyspark/sql/utils.py", line 63, in deco
File "/mnt/yarn/usercache/root/appcache/application_1549986900582_0001/container_1549986900582_0001_01_000001/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o210.csv.
.
.
.
.
.
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4, ip-172-32-189-222.ec2.internal, executor 1): java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE
Below is python code used to write the file
def write(dataframe):
# write two files premium listings and non premium listings (critera : listing_priority > 30 = premium)
dataframe.filter(dataframe["listing_priority"] >= 30).drop('listing_priority').drop('image_count').write.csv(
output_path_premium, header=True, compression="gzip")
shell_command = "hdfs dfs -mv " + output_path_premium + '/part-' + '  ' + output_path_premium + output_file_premium
os.system(shell_command)
dataframe.filter(dataframe["listing_priority"] < 30).drop('listing_priority').drop('image_count').write.csv(
output_path_nonpremium, header=True, compression="gzip")
shell_command = "hdfs dfs -mv " + output_path_nonpremium + '/part-' + '  ' + output_path_nonpremium + output_file_nonpremium
os.system(shell_command)
I am assuming its because the file is greater than 2GB . has this issue been fixed ?

@jkhalid What spark version were you on?  many fixes were not available till spark 2.4.
can you share the entire stack trace, particularly the java portion?
the limit still exists for single records which are over 2GB.  It shouldn't be a problem  for reading a whole file which is over 2GB.  but there may be some problem when going back and forth from python, I'm not very familiar w/ that part.

@squito
:( My bad i guess AWS is using 2.2.1
https://aws.amazon.com/about-aws/whats-new/2018/04/aws-glue-now-supports-apache-spark-221/

@squito
Hey any idea from the stacktrace what i might be doing wrong

Can we move discussions unrelated to a long-closed PR to either jira or the mailing list? Thanks.


[WV-237] Convert methods with warning to @staticmethod TEAM REVIEW

Found warnings in WeVoteServer/api_internal_cache/models.py and converted indicated methods to staticmethods, removing self parameter and moving parameter lists to one line when possible.

👍  Great work!


function_type_alias9_test failure in strong mode

the fix for https://github.com/dart-lang/sdk/issues/27410 exposed that strong mode's subtype does not handle CircularTypeImpl, causing function_type_alias9_test 00 to fail (https://travis-ci.org/dart-lang/sdk/builds/163631529):
typedef void F(List
                   <G>  /// 00: compile-time error
                       l);
typedef void G(List<F> l);

main() {
  F foo(G g) => g;
  foo(null);
}

Just a guess, but perhaps it's because we don't handle CircularTypeImpl explicitly in our subtype relation.

https://codereview.chromium.org/2378243004/


Controlling width of selection can be fiddly

Consider binding increase / decrease width to mouse wheel.

Probably remove Control (area) selection entirely in favor of this.

Hmm, but we can't use the mousewheel since that's bound to zoom 🤔 Could use Control + Mousewheel?

Could use Control + Mousewheel?

Shift + Mousewheel or Ctrl + Mousewheel is sometimes used by digital painting tools to control the brush size. So I think it could work.
However, we will also need to think about (alternative) controls that work on keyboard only for accessibility, and the controller controls for Steam Deack & co.