Error when running Swagger-tester for Heroku-pets example

I was trying to test heroku-pets example file(http://editor.swagger.io/#/) using swagger tester. I am getting an error-
**ERROR:connexion.api:Failed to add operation for PUT /pet/
ResolverError: <ResolverError: Cannot resolve operationId "None"!>
ERROR:connexion.api:Failed to add operation for POST /pet/
ResolverError: <ResolverError: Cannot resolve operationId "None"!>
ERROR:connexion.api:Failed to add operation for GET /pet/
ResolverError: <ResolverError: Cannot resolve operationId "None"!>
ERROR:connexion.api:Failed to add operation for GET /pet/{petId}
ResolverError: <ResolverError: Cannot resolve operationId "None"!>**
Starting testrun against hero.json or None using examples: True
Below is the code for my test.py file to run my configuration file for heroku-pets, which is i have saved locally in json format(config. file name- hero.json)
from swagger_tester import swagger_test
authorize_error = {
'get': {
'/pet/': [200,400, 404],
'/pet/{petId}': [200,400,404],
},
'post': {
'/pet/': [200,400, 404]
},
'put': {
'/pet/': [200,400, 404]
}
}
swagger_test('hero.json', authorize_error=authorize_error)
Below is my console output when i run my test.py file:
ujjwal@ujjwal:~/Desktop/dataweave$ python test.py
ERROR:connexion.api:Failed to add operation for PUT /pet/
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/connexion/api.py", line 237, in add_paths
self.add_operation(method, path, endpoint, path_parameters)
File "/usr/local/lib/python2.7/dist-packages/connexion/api.py", line 189, in add_operation
resolver=self.resolver)
File "/usr/local/lib/python2.7/dist-packages/connexion/operation.py", line 210, in init
resolution = resolver.resolve(self)
File "/usr/local/lib/python2.7/dist-packages/connexion/resolver.py", line 39, in resolve
return Resolution(self.resolve_function_from_operation_id(operation_id), operation_id)
File "/usr/local/lib/python2.7/dist-packages/connexion/resolver.py", line 67, in resolve_function_from_operation_id
raise ResolverError(msg)
ResolverError: <ResolverError: Cannot resolve operationId "None"!>
ERROR:connexion.api:Failed to add operation for POST /pet/
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/connexion/api.py", line 237, in add_paths
self.add_operation(method, path, endpoint, path_parameters)
File "/usr/local/lib/python2.7/dist-packages/connexion/api.py", line 189, in add_operation
resolver=self.resolver)
File "/usr/local/lib/python2.7/dist-packages/connexion/operation.py", line 210, in init
resolution = resolver.resolve(self)
File "/usr/local/lib/python2.7/dist-packages/connexion/resolver.py", line 39, in resolve
return Resolution(self.resolve_function_from_operation_id(operation_id), operation_id)
File "/usr/local/lib/python2.7/dist-packages/connexion/resolver.py", line 67, in resolve_function_from_operation_id
raise ResolverError(msg)
ResolverError: <ResolverError: Cannot resolve operationId "None"!>
ERROR:connexion.api:Failed to add operation for GET /pet/
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/connexion/api.py", line 237, in add_paths
self.add_operation(method, path, endpoint, path_parameters)
File "/usr/local/lib/python2.7/dist-packages/connexion/api.py", line 189, in add_operation
resolver=self.resolver)
File "/usr/local/lib/python2.7/dist-packages/connexion/operation.py", line 210, in init
resolution = resolver.resolve(self)
File "/usr/local/lib/python2.7/dist-packages/connexion/resolver.py", line 39, in resolve
return Resolution(self.resolve_function_from_operation_id(operation_id), operation_id)
File "/usr/local/lib/python2.7/dist-packages/connexion/resolver.py", line 67, in resolve_function_from_operation_id
raise ResolverError(msg)
ResolverError: <ResolverError: Cannot resolve operationId "None"!>
ERROR:connexion.api:Failed to add operation for GET /pet/{petId}
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/connexion/api.py", line 237, in add_paths
self.add_operation(method, path, endpoint, path_parameters)
File "/usr/local/lib/python2.7/dist-packages/connexion/api.py", line 189, in add_operation
resolver=self.resolver)
File "/usr/local/lib/python2.7/dist-packages/connexion/operation.py", line 210, in init
resolution = resolver.resolve(self)
File "/usr/local/lib/python2.7/dist-packages/connexion/resolver.py", line 39, in resolve
return Resolution(self.resolve_function_from_operation_id(operation_id), operation_id)
File "/usr/local/lib/python2.7/dist-packages/connexion/resolver.py", line 67, in resolve_function_from_operation_id
raise ResolverError(msg)
ResolverError: <ResolverError: Cannot resolve operationId "None"!>
Starting testrun against hero.json or None using examples: True
INFO:swagger_tester.swagger_tester:TESTING PUT /pet/?
INFO:swagger_tester.swagger_tester:Using FlaskClient, got status code 404 for ********** PUT /pet/?
INFO:swagger_tester.swagger_tester:Got expected authorized error on /pet/? with status 404
INFO:swagger_tester.swagger_tester:TESTING POST /pet/?
INFO:swagger_tester.swagger_tester:Using FlaskClient, got status code 404 for ********** POST /pet/?

INFO:swagger_tester.swagger_tester:Using FlaskClient, got status code 404 for ********** GET /pet/string?
INFO:swagger_tester.swagger_tester:Got expected authorized error on /pet/string? with status 404
I have installed connexion framework also. How to resolve above errors and What is the expected output for testing ? Actually, I am new to swagger and want to learn how testing is done with help of swagger tester so that i can use this in my organisation.
Below is the code for heroku-pets config file in json format(hero.json):
{
"swagger": "2.0",
"info": {
"version": "1.0.0",
"title": "PetStore on Heroku",
"description": "This example has a working backend hosted in Heroku\n\nYou can try all HTTP operation described in this Swagger spec.\n\nFind source code of this API here\n"
},
"host": "petstore-api.herokuapp.com",
"basePath": "/pet",
"schemes": [
"http",
"https"
],
"consumes": [
"application/json",
"text/xml"
],
"produces": [
"application/json",
"text/html"
],
"paths": {
"/": {
"get": {
"parameters": [
{
"name": "limit",
"in": "query",
"description": "number of pets to return",
"type": "integer",
"default": 11,
"minimum": 11,
"maximum": 10000
}
],
"responses": {
"200": {
"description": "List all pets",
"schema": {
"title": "Pets",
"type": "array",
"items": {
"$ref": "#/definitions/Pet"
}
}
}
}
},
"post": {
"parameters": [
{
"name": "pet",
"in": "body",
"description": "The pet JSON you want to post",
"schema": {
"$ref": "#/definitions/Pet"
},
"required": true
}
],
"responses": {
"200": {
"description": "Make a new pet"
}
}
},
"put": {
"parameters": [
{
"name": "pet",
"in": "body",
"description": "The pet JSON you want to post",
"schema": {
"$ref": "#/definitions/Pet"
},
"required": true
}
],
"responses": {
"200": {
"description": "Updates the pet"
}
}
}
},
"/{petId}": {
"get": {
"parameters": [
{
"name": "petId",
"in": "path",
"type": "string",
"description": "ID of the pet",
"required": true
}
],
"responses": {
"200": {
"description": "Sends the pet with pet Id"
}
}
}
}
},
"definitions": {
"Pet": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"birthday": {
"type": "integer",
"format": "int32"
}
}
}
}
}

I have experienced similar errors. Please help.
ERROR:connexion.api:Failed to add operation for GET /auth/list
Traceback (most recent call last):
  File "/Users/.pyenv/versions/sentience/lib/python2.7/site-packages/connexion/api.py", line 237, in add_paths
    self.add_operation(method, path, endpoint, path_parameters)
  File "/Users/.pyenv/versions/sentience/lib/python2.7/site-packages/connexion/api.py", line 189, in add_operation
    resolver=self.resolver)
  File "/Users/.pyenv/versions/sentience/lib/python2.7/site-packages/connexion/operation.py", line 210, in __init__
    resolution = resolver.resolve(self)
  File "/Users/.pyenv/versions/sentience/lib/python2.7/site-packages/connexion/resolver.py", line 39, in resolve
    return Resolution(self.resolve_function_from_operation_id(operation_id), operation_id)
  File "/Users/.pyenv/versions/sentience/lib/python2.7/site-packages/connexion/resolver.py", line 67, in resolve_function_from_operation_id
    raise ResolverError(msg)
ResolverError: <ResolverError: Cannot resolve operationId "None"!>
ERROR:connexion.api:Failed to add operation for GET /auth/status
Traceback (most recent call last):
  File "/Users/.pyenv/versions/sentience/lib/python2.7/site-packages/connexion/api.py", line 237, in add_paths
    self.add_operation(method, path, endpoint, path_parameters)
  File "/Users/.pyenv/versions/sentience/lib/python2.7/site-packages/connexion/api.py", line 189, in add_operation
    resolver=self.resolver)
  File "/Users/.pyenv/versions/sentience/lib/python2.7/site-packages/connexion/operation.py", line 210, in __init__
    resolution = resolver.resolve(self)
  File "/Users/.pyenv/versions/sentience/lib/python2.7/site-packages/connexion/resolver.py", line 39, in resolve
    return Resolution(self.resolve_function_from_operation_id(operation_id), operation_id)
  File "/Users/.pyenv/versions/sentience/lib/python2.7/site-packages/connexion/resolver.py", line 67, in resolve_function_from_operation_id
    raise ResolverError(msg)
ResolverError: <ResolverError: Cannot resolve operationId "None"!>

Hi all,
I'm trying the Connexion library with Swagger, too.
I had the same issue at first but I think it's not a bug, it's just a missing property in our swagger.yaml file.
Please take a look at this file, from the Connexion's petstore example:
[https://github.com/hjacobs/connexion-example/blob/master/swagger.yaml]
first you need to add the 'operationId' property to your swagger.yaml file:
paths:
  /pets:
    get:
      tags: [Pets]
      operationId: app.get_pets
      summary: Get all pets

then you need to implement the proper function (with the same signature as you defined in swagger.yaml) inside your app:
#!/usr/bin/env python3

import connexion

def get_pets():
  return []  # your code here

if __name__ == '__main__':
    app = connexion.App(__name__, 8080,
    specification_dir='./swagger/')
    app.add_api('swagger.yaml', arguments={'title': 'your API title'})
    app.run()

I hope it helps.

@Swalloow @ujjwal08 Does @leonardofoderaro's solution help? Could you turn this discussion into a stackoverflow post?


⚠ Bofrost.at has degraded performance

In baa100e, Bofrost.at (https://www.bofrost.at) experienced degraded performance:

HTTP code: 200
Response time: 934 ms

Resolved: Bofrost.at performance has improved in f0cc07e after 20 minutes.


Unable to compute the prediction using ML Program. It can be an invalid input data or broken/unsupported model. on Swift image generation

I've downloaded the ready-made Core ML Models from Hugging Face Hub and trying to run the example command:
ml-stable-diffusion % swift run StableDiffusionSample "a photo of an astronaut riding a horse on mars" --resource-path ../coreml-stable-diffusion-2-1-base-palettized/original/compiled --seed 93 --output-path .
Building for debugging...
Build complete! (0.09s)
Loading resources and creating pipeline
(Note: This can take a while the first time using these resources)
Sampling ...
Error: Unable to compute the prediction using ML Program. It can be an invalid input data or broken/unsupported model.

I've tried other ready-made models too with the same result. I'm running the newest MacOS 14.

Hello! You are missing --compute-units cpuAndGPU for the particular model you are using (Judging from the model folder names, you are using a model exported with --attention-implementation ORIGINAL)

Great, thank you!


Next generation netCDF design and implementation discussion thread

Feature request/discussion
There are several people thinking about scalable and more standardized netCDF in MITgcm. This issue thread provides a place to

discuss and iterate on various use case and/or design considerations/tradeoffs and
identify what approach(es) might work best for different scenarios

Obviously https://github.com/altMITgcm/MITgcm/pull/16 implements something but it was largely divorced from practical knowledge of how folks want files organized.  I think if someone wanted to preperly respec the files, they could either push onto that PR or I could adapt from a written spec.  Further, if someone wanted to just use that PR as a proof of concept and engineer a different solution that would be great.  However, I would be disappointed if parallel netcdf files took another number of years to be available to folks.

A key principle we should try to follow is that the files that come out of MITgcm should be "analysis ready". For very big simulations (those producing many TB of output), it is not practical to have to duplicate and reformat the "raw" output into something more useful (as we do, for example, with gluemnc on the tiled netCDF files).
The most useful format is netCDF, and the way to produce global netCDF files is using parallel netCDF writes, as in @jklymak's PR altMITgcm/MITgcm#16. So I think this is the way to go.
In terms of how the netCDF files should be formatted, I have written a rant about that here:
https://gist.github.com/rabernat/bf4c886c0555aa9a6e7820d9398d9bbf
I think the original MNC package is already pretty good...if we could make it write global files, that would probably be the ideal solution.
A separate but related issue is related to asynchronous I/O. The new high-res LLC runs have the asyncio package, which apparently improved performance a lot. As long as we are discussing improving I/O, it seems like that should be part of the conversation.

@rabernat I started a issue on async I/O #36

My netcdf stuff is now #31   More than happy to change the output format.  Decided to use record instead of time because the way I implemented the netcdf files, they don't get over-written if there is a new run, with the idea that a user may want to use the same file for a spinup phase and a write-intensive phase.  The problem with that is users (ahem, me at this point) will sometimes forget to delete the old file and write to the same file by accident, and things get confusing if time is the unbounded incrementing variable.
I dislike T for temperature because in xarray the letter T is also the transpose operator, so dataset.U works, but dataset.T doesn't.
Not sure what to do about co-ordinate variables.  It may be possible to detect if the co-ordinates are a nice grid before writing and output those as Lat/Lon or Northing/Easting.  But that seemed susceptible to failure, so I just used the generalized indexes.

xarray.DataArray.T is deprecated now in favor of .transpose(), see pydata/xarray#1643.

Sorry I’m being silly
I don’t name the variables in #31.  They are the diagnostic names.
OTOH I do make decisions about be dimensions (both what’s used and their names) and I’m happy to modify those based on community consensus.

@jklymak sorry for delay on this and the slightly stalled #31. We have been thinking about the topic.
Here are some bullets we (@jm-c, @jahn, @edoddridge, @jrscott ) wrote, summarizing where I think we want to try and end up. It might be good to do a chat soon about how/whether we can get there!

CF conventions used whenever possible

Minimize number of different interfaces

Flexible support for all different modes of use of MITgcm

cartesian, general lat-lon, random planets, cylinders etc...
arbitrary time steps (fractional secs to decades etc...)

As far as possible works (and is tested) robustly against any combination of netcdf-4, pnetcdf back end i.e. no gratuitous use of particular backend specific features.

Maybe still allows tiled files? Its not clear whether this is useful to preserve, but it should be thought about/discussed a little.

Has some reasonable approach to faceted grids (e.g. traditional cube-sphere, LLC).  Note - this is an area where CF hasn't really figured itself out.  Our current thinking is that the so-called mosaic meta-data (see https://www.unidata.ucar.edu/software/libcf/html/group__gs__mosaic__grp.html ) may have a role here. Its not clear that many tools do anything with mosaic meta-data. However, we aren't aware of anything else that has traction. We don't think ugrid (https://github.com/ugrid-conventions/ugrid-conventions) is suitable, although we could be wrong. I did chat briefly with @rsignell-usgs about this topic.

Is well tested with a range model setups and backend netcdf-4, pnetcdf, MPI, compiler combinations on a wide range of platforms (e.g, ideally Raspberry PI -- Coral supercomputer)

Works well with downstream tools that are in use (xarray+dask, Ferret (???), numpy, Thredds/Opendap servers, ocean parcels etc..)

Maybe has some awareness of upcoming efforts (e.g. zarr - https://www.youtube.com/watch?v=np_p4JBAIYI, https://github.com/zarr-developers/zarr and other more scalable approaches). Note - zarr may mesh with binary MDS + JSON more easily. Pangeo (https://pangeo-data.github.io) has some experiments in this area.

@christophernhill  Glad it is being discussed - I appreciate that it is probably a big project to do properly!
I appreciate this is a bigger discussion than #31, much of which, as a process modeller who largely uses Cartesian grids for personal consumption, I'm not the best person to voice an opinion on.   But just to state where I think we are wrt #31:

CF conventions used whenever possible

I think #31 is CF compliant, or almost so.

Minimize number of different interfaces

#31 adds an interface to pkg/diagnostics.  Probably not consistent w/ this part of the wishlist.  OTOH, I can think of reasons to keep the existing MNC interface around, not least of which is that it allows input!

Flexible support for all different modes of use of MITgcm

cartesian, general lat-lon, random planets, cylinders etc...
- arbitrary time steps (fractional secs to decades etc...)

I don't think there is anything in #31 that does anything to the time steps: I made record be the unlimited (growing) dimension so that time could be uneven, etc.
WRT non cartesian grids: #31 just puts everything into a file organized by global i, and i_g (etc) indices. I don't know how to do anything else, and I think at the end of the day you have to choose some index for your "x" and "y" dimensions, even though I appreciate that they may mean something completely different for complex geometries. My assumption is that writing one netcdf file is no worse than writing 10,000 tile files, in that a user may still need to do some processing to get the file in a shape that makes physical sense.  Of course if we can make it better in some way, I'm all for it, but I don't think about strange exchange file setups.  Certainly an easy tweak is to add tile number cooridnate variables to help make some of this easier.

As far as possible works (and is tested) robustly against any combination of netcdf-4, pnetcdf back end i.e. no gratuitous use of particular backend specific features.

I don't think there is anything in #31 that cares about the hdf5 versus pnetcdf backends.  The tests I tried to implement in genmake2 were meant to check for hdf5.  But I don't think that was because of any requirements in the routines I used - they are just high-level netcdf90.

Maybe still allows tiled files? Its not clear whether this is useful to preserve, but it should be thought about/discussed a little.

The old interface allows this.  I think it would complicate #31 considerably to allow this, but I'm sure its possible w/ a bit of work.

Has some reasonable approach to faceted grids (e.g. traditional cube-sphere, LLC). Note - this is an area where CF hasn't really figured itself out. Our current thinking is that the so-called mosaic meta-data (see https://www.unidata.ucar.edu/software/libcf/html/group__gs__mosaic__grp.html ) may have a role here. Its not clear that many tools do anything with mosaic meta-data. However, we aren't aware of anything else that has traction. We don't think ugrid (https://github.com/ugrid-conventions/ugrid-conventions) is suitable, although we could be wrong. I did chat briefly with @rsignell-usgs about this topic.

Right, see above about what #31 does now: currently just uses generalized co-ordinates.

Is well tested with a range model setups and backend netcdf-4, pnetcdf, MPI, compiler combinations on a wide range of platforms (e.g, ideally Raspberry PI -- Coral supercomputer)

I've not tested #31 too widely: on an IBM supercomputer, a Cray supercomputer, and my macs.

Works well with downstream tools that are in use (xarray+dask, Ferret (???), numpy, Thredds/Opendap servers, ocean parcels etc..)

#31 works great w/ xarray+dask (and hence numpy).   No idea about Ferret, GMT, and I've not tried to put the data on an Opendap server.

Maybe has some awareness of upcoming efforts (e.g. zarr - https://www.youtube.com/watch?v=np_p4JBAIYI, https://github.com/zarr-developers/zarr and other more scalable approaches). Note - zarr may mesh with binary MDS + JSON more easily. Pangeo (https://pangeo-data.github.io) has some experiments in this area.

Looks like @rabernat has implemented a zarr backend for xarray: https://github.com/pydata/xarray/pull/1528
general point about #31
I think #31 as written is extremely useful for large simulations.  Further, its basically self-contained, and can even be used in parallel with the other output formats.  I'd still advocate for its release sooner rather than later on at least an experimental basis, and then let users decide what they need to make it better.  Once that has settled down, you could consider what the "proper" solution is, and implement that for the more serious large projects.  If the experimental nature of nf90io means its name should change, I'd have no problem with that.

Hi, I know I'm new here but I have a quick point to chime in: for my configuration, MNC has a 14% overhead (in walltime) compared to MDS output. The other MITgcm users in my group see a similar effect. So from a computational cost perspective, we just can't justify NetCDF files produced online. This might be dependent on the computational environment, but I wouldn't be surprised if it was widespread. Are you expecting nf90io to change this?
FYI - everyone here uses MDS output followed by an offline NetCDF converter (built in house) which loses most of the metadata. Clearly this is not ideal. So I'm really interested to hear about xmitgcm - I'll give it a try today.

The issue with xmitgcm on large files is that it has to virtually preallocate the files in memory.  Which is virtual so it’s fine unless your process node has a memory cap and prevents your process from doing this.
It’s an interesting question if nf90io is a lot slower than mds.  Startup will be slower because the model writes the grid info twice (once mds once in the netcdf file). After that so long as you’ve turned mds dumps off I’d be surprised if there was a huge difference but I could be wrong.     I can easily imagine writing a bunch of small files ends up faster than writing to one store but I’m not sure.


Get-SCPItem : -p -r -d options not supported

I'm trying to SCP a file from a Cisco router but it seems like Get-SCPItem uses some switches that aren't supported by Cisco.  I can SCP the files just fine using regular powershell scp commands.  Those switches appear to be used for Recursive, Preserving Date/Times, etc... the files I'm trying to copy don't require any of those switches-- is there any way to disable those switches for Get-SCPItem?
Get-SCPItem $Device.IPAddress -Credential $Device.Credentials -KeyFile $Backup.Key -AcceptKey -Path ":running-config" -PathType File -Destination $Backup.Location -NewName $Device.BackupName
"Get-SCPItem : -p -r -d options not supported" is the error I get

No, this is hartdcoded in SSH.net
You can try to comment/upvote this issue
https://github.com/sshnet/SSH.NET/issues/745


feat(cli): support '-edit' for "gas draft"

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 83.026%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
4999

Relevant Lines:
6021

💛  - Coveralls


CLOUD-868: ability to get ambari requests with specific statuses

@keyki pls review

Jenkins build finished, all tests passed.
Refer to this link for build results: http://ci.sequenceiq.com/job/ambari-rest-client-pull-request/97/

Jenkins build finished, all tests passed.
Refer to this link for build results: http://ci.sequenceiq.com/job/ambari-rest-client-pull-request/98/


Support D2E Lost Legends

Can make use of data provided by rfisha:
https://boardgamegeek.com/article/32528846#32528846
Lost+Legends.zip

Notes:
FFG Version 1.5.2
3 EoD oggs added
Tiles:
018B, 040B, 042B, S001B
Images:
EmbersOfDread_Icon.dds EoD_Defeat.dds EoD_Prologue.dds EoD_Victory.dds Lightmap-1_comp_light.dds Logo_EmbersOfDread.dds Map-EmbersofDread.dds Quests_CaveofWonders.dds Quests_FortRodric.dds Quests_Riverwatch.dds Quests_Skydown.dds Quests_Sudanya.dds Quests_Vynelvale.dds Riverwatch_Icon.dds RiverwatchBackground.dds
Many other images show as different, test for changes.

I am restoring font import, with an exception catch, so it shouldn't stop devices that have the fault from #964

Icon split to #1184


[SPARK-8644] Include call site in SparkException stack traces thrown by job failures

Example exception (new part at bottom, clearly demarcated):
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.RuntimeException: uh-oh!
	at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37$$anonfun$38$$anonfun$apply$mcJ$sp$2.apply(DAGSchedulerSuite.scala:880)
	at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37$$anonfun$38$$anonfun$apply$mcJ$sp$2.apply(DAGSchedulerSuite.scala:880)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
	at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1640)
	at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1099)
	at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1099)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1777)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1777)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
	at org.apache.spark.scheduler.Task.run(Task.scala:70)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1298)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1289)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1288)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1288)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:755)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:755)
	at scala.Option.foreach(Option.scala:236)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:755)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1509)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1470)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1459)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:560)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1744)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1762)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1777)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1791)
	at org.apache.spark.rdd.RDD.count(RDD.scala:1099)
	at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37$$anonfun$38.apply$mcJ$sp(DAGSchedulerSuite.scala:880)
	at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37$$anonfun$38.apply(DAGSchedulerSuite.scala:880)
	at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37$$anonfun$38.apply(DAGSchedulerSuite.scala:880)
	at org.scalatest.Assertions$class.intercept(Assertions.scala:997)
	at org.scalatest.FunSuite.intercept(FunSuite.scala:1555)
	at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37.apply$mcV$sp(DAGSchedulerSuite.scala:879)
	at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37.apply(DAGSchedulerSuite.scala:878)
	at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37.apply(DAGSchedulerSuite.scala:878)
	at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
	at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
	at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:42)
	at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
	at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
	at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
	at org.apache.spark.scheduler.DAGSchedulerSuite.org$scalatest$BeforeAndAfter$$super$runTest(DAGSchedulerSuite.scala:70)
	at org.scalatest.BeforeAndAfter$class.runTest(BeforeAndAfter.scala:200)
	at org.apache.spark.scheduler.DAGSchedulerSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(DAGSchedulerSuite.scala:70)
	at org.scalatest.BeforeAndAfterEach$class.runTest(BeforeAndAfterEach.scala:255)
	at org.apache.spark.scheduler.DAGSchedulerSuite.runTest(DAGSchedulerSuite.scala:70)
	at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
	at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
	at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
	at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
	at scala.collection.immutable.List.foreach(List.scala:318)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
	at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
	at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
	at org.scalatest.Suite$class.run(Suite.scala:1424)
	at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
	at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
	at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
	at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
	at org.apache.spark.scheduler.DAGSchedulerSuite.org$scalatest$BeforeAndAfter$$super$run(DAGSchedulerSuite.scala:70)
	at org.scalatest.BeforeAndAfter$class.run(BeforeAndAfter.scala:241)
	at org.apache.spark.scheduler.DAGSchedulerSuite.org$scalatest$BeforeAndAfterAll$$super$run(DAGSchedulerSuite.scala:70)
	at org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)
	at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)
	at org.apache.spark.scheduler.DAGSchedulerSuite.run(DAGSchedulerSuite.scala:70)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:462)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:671)
	at sbt.ForkMain$Run$2.call(ForkMain.java:294)
	at sbt.ForkMain$Run$2.call(ForkMain.java:284)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)

cc @JoshRosen

Merged build triggered.

Merged build started.

NAVER - http://www.naver.com/
<EMAIL_ADDRESS>님께 보내신 메일 <[spark] [SPARK-8644] Include call site in SparkException stack traces thrown by job failures (#7028)> 이 다음과 같은 이유로 전송 실패했습니다.

받는 사람이 회원님의 메일을 수신차단 하였습니다.

NAVER - http://www.naver.com/
<EMAIL_ADDRESS>님께 보내신 메일 <Re: [spark] [SPARK-8644] Include call site in SparkException stack traces thrown by job failures (#7028)> 이 다음과 같은 이유로 전송 실패했습니다.

받는 사람이 회원님의 메일을 수신차단 하였습니다.

Test build #35812 has started for   PR 7028 at commit 714be86.

Test build #35812 has finished for   PR 7028 at commit 714be86.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

This will be great to add that to the exception msg, the current Driver stacktrace is not very useful.  But, I think just munging it with the existing stack trace might be really confusing to spark users.  Java stack traces have very standard interpretations (part of the reason they are so useful).  Eg., if I saw this line at ===== Job Submission =====.(Native Method) I would just probably naively assume spark was calling some magical native method, but that it was still a normal call stack (and I wouldn't think I need to look in the docs to interpret a stack trace).
How about instead just making it a separate section in the msg?  eg.
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.RuntimeException: uh-oh!
    at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$33$$anonfun$34$$anonfun$apply$mcJ$sp$1.apply(DAGSchedulerSuite.scala:851)
...
Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1285)
...
Job Submission stacktrace:
    at org.apache.spark.rdd.RDD.count(RDD.scala:1095)
...

Then you could also just do this in JobWaiter or sc.runJob, which would make it much simpler.

I think the most important thing is to include the user stack trace somewhere. Users don't really care what's going on as long as they can identify a line of code from their codebase in the stack. That said, I see your point about the current approach being overly magical.
Note that producing an Exception that matches your example would require wrapping/replacing whichever actual exception with a SparkException (otherwise we cannot rewrite the message). However, we do have two disjoint options:

Keep the munging logic in DAGScheduler.failJobAndIndependentStages and simply put the "Job Submission" stack trace above the driver stack trace, included in the Exception's message. This puts the exception a bit "out of order" but would avoid having to parse the longForm CallSite. We can do this because this method only throws SparkExceptions anyway.
Move the munging logic to DAGScheduler.runJob and keep it as a fake part of the stack trace. This would allow us to inject the user stack trace into all exceptions thrown as part of the DAGScheduler EventLoop rather than just ones that induce normal stage fails.

With number 2, we could additionally replace the use of callSite parsing with the current thread's stack trace, which would make it look a lot less magical (it's just bridging the two real stack traces across the event loop).
I'd personally prefer 2 for its improved scope, but I don't know enough about how the call site may differ from the physical stack trace to say if we should also just use the local stack.

Merged build triggered.

Merged build started.

Test build #35813 has started for   PR 7028 at commit 4725d14.

Test build #35813 has finished for   PR 7028 at commit 4725d14.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

OK, I took a closer look at the the way the exception handling works and I see what you mean.  I'm not particularly tied to that formatting I suggested, that was just an example, totally fine to do something else that is similar.  I agree that as long as the user can clearly identify their code in the driver, and their code in the executors, that is a big help.
I think either of your proposed solutions sound fine -- I'm also a little partial to (2) if its simpler.  I am not sure I understand it 100%, but sounds good :)

I've updated the code to move the injection to runJob. This means that exceptions thrown from any path inside the DAGScheduler that fail the job will now have their stack trace supplemented, if the job was submitted via runJob (i.e., not an approximate job or submitted via SparkContext.submitJob).
After looking over the usages of special call site, I found that streaming is a significant user and thus opted to keep the callsite parsing code. I could definitely see the benefit of removing the 100 lines of code or so that's required to do this for the sake of simplicity at the cost of streaming stack traces not being quite as helpful as they could be, so feel free to push back and I can change it.

Merged build triggered.

Merged build started.

Test build #35824 has started for   PR 7028 at commit 19ab12e.

Test build #35824 has finished for   PR 7028 at commit 19ab12e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Merged build finished. Test FAILed.

Merged build triggered.

Merged build started.

Test build #35864 has started for   PR 7028 at commit 5fdb1cc.

Test build #35864 has finished for   PR 7028 at commit 5fdb1cc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Looks good, but now it doesn't merge.

Merged build triggered.

Merged build started.

Test build #36193 has started for   PR 7028 at commit a914cb0.

Test build #36193 has finished for   PR 7028 at commit a914cb0.

This patch fails some tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

hey, sorry I haven't taken another look at this yet -- do you mind giving me a bit more time to look before merging?  I should be able to look this evening, if not first thing tomorrow.

Sure

retest this please

Merged build triggered.

Merged build started.

Test build #36197 has started for   PR 7028 at commit a914cb0.

Test build #36197 has finished for   PR 7028 at commit a914cb0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Hi @aarondav @andrewor14 , I would like to discuss this a little more.  I think I didn't understand your proposal earlier and I didn't really express my concerns properly -- sorry I am going to backpedal on what I said before.
My concern is not so much about the implementation details, but just how this looks to the end user.  Certainly this change is better than the status quo.  But I'd rather we add this the right way the first time, rather than put in something now, make users learn this new exception format, and then down the line we update it and users need to learn some new format.  Understanding exceptions from Spark is already a big source of user confusion imo, so its worth spending a bit of time on this.
There are 3 stack traces here: (a) from the executor where the original error occurred (b) where the exception was handled in DAGScheduler.handleTaskSetFailed as part of the event loop and (c) where the user code triggered the job with an action like rdd.count().  (b) is probably totally useless for most users, its just occasionally useful for a spark developer, but right now we give the user (a) + (b).
This pr puts (b) + (c) together as if they are one stack trace.  I definitely think that is an improvement -- users get to see (c), and mostly they'll just ignore (b) anyway so it mostly doesn't matter.  But there will definitely be times that a curious user tries to understand the rest of the stack trace.  Maybe they hit some spark bug and they want to try to understand it more before filing a jira, or they think that perhaps they are misusing spark and the stack trace will help them understand better, etc.  In those cases, I think it'll cause a lot of confusion to have it appear that its all one stack trace.
If we did have them separated as three different stack traces, it would be much easier for a user to see understand in that case, and I think they'd also be much more likely to look for an explanation of what the different parts are.  (As I said earlier, I would not think I need to consult documentation to understand one simple stack trace.)
I put together an alternative implementation here: https://github.com/apache/spark/pull/7156 (sadly the tests don't run b/c the scala compiler crashes running mima ... but its just meant for discussion, not to actually merge in any case, so just pretend the tests pass ...).  That is an alternate way to get all the stack traces -- it makes (a) + (b) the cause of (c), so you get a stack trace like:
org.apache.spark.SparkException: job failed
        at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:558)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1741)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1759)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1774)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1788)
        at org.apache.spark.rdd.RDD.count(RDD.scala:1095)
        at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37$$anonfun$38.apply$mcJ$sp(DAGSchedulerSuite.scala:883)
...[snip]
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.RuntimeException: uh-oh!
        at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37$$anonfun$38$$anonfun$apply$mcJ$sp$2.apply(DAGSchedulerSuite.scala:883)
        at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37$$anonfun$38$$anonfun$apply$mcJ$sp$2.apply(DAGSchedulerSuite.scala:883)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1627)
        at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1095)
        at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1095)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1774)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1774)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
        at org.apache.spark.scheduler.Task.run(Task.scala:70)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1294)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1285)
...[snip]

I don't love that version either -- we're still doing a bit of magic that could confuse somebody if they look too closely.  Though I do think that a separate causing exception is at least a more clear break, rather than something that just looks like part of the call stack.   I'd really prefer if we could keep all three separate.
Sorry to backtrack on my earlier comment.  I don't mean to create an unnecessary obstacle here.  I do think the patch as-is is still an improvement -- I suppose the curious user could just search the code for ==== Job Submission ====, we could just add a clear comment in the code explaining what was going on.  I haven't really thought about the implementation in any great detail, so maybe it isn't worth it to do something more complicated.  (Also, I really haven't thought at all about what is going on w/ streaming -- perhaps that also makes it impossible to make a bigger change.)
thanks for working on this and taking the time to discuss w/ me and read through my rambling :)

The problem with any approach that wraps the exception is that we no longer throw an exception of the original type; we instead always throw SparkExceptions (as in your PR). This could be considered an API-breaking change, and one that would only break at runtime.
The benefit of appending stack trace elements is that one can apply it to any exception without impact on the callers or DAGScheduler.
I think that it is very intuitive to join the stacks before and after an event loop, and it has the expected semantics of "code leaving the area I know and entering into scary Spark internals". However, I agree that the fact that we're joining a "user-readable" stack instead of the actual stack may be confusing.
A compromise would be to make the stack look like this:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.RuntimeException: uh-oh!
        at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37$$anonfun$38$$anonfun$apply$mcJ$sp$2.apply(DAGSchedulerSuite.scala:883)
        at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37$$anonfun$38$$anonfun$apply$mcJ$sp$2.apply(DAGSchedulerSuite.scala:883)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1627)
        at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1095)
        at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1095)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1774)
        at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1774)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
        at org.apache.spark.scheduler.Task.run(Task.scala:70)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1294)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1285)
        at ===== DAGScheduler EventLoop Submission =====.()
        at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:558)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1741)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1759)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1774)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1788)
        at org.apache.spark.rdd.RDD.count(RDD.scala:1095)
...

The difference from the current version being that this also includes the Spark internals leading up to the actual EventLoop itself, which makes the stack appear more natural (but also uglier due to the several indirections through runJob). The other difference is that it would return a more intuitive (but less useful) stack trace in situations where callSite is currently used in Spark (such as when starting a new job in Spark Streaming).
I believe that this change would satisfy your biggest concern (making the stack trace magical) because, as I said, I think this stack trace is actually very readable, and I'm willing to trade off some usefulness for less magic because it still gets us much farther than today.

Hi @aarondav , sorry I had missed the part about breaking the api.  It looks to me like the only place JobWaiter.jobFailed is called with a non-SparkException is when there is an error submitting the job.  This seems pretty minor -- would it technically be breaking the api contract to change to also be a SparkException?
Or do you mean we break the api by just adding another wrapping SparkException, so the cause & msg are different?  That is true ... but the cause & msg were pretty useless for programmatic use before, so as long as we're not violating some hard rule on compatibility, I also don't see that change as being a problem.
all this said, I do still think what you've proposed is an improvement ... just want to make sure we've explored alternatives.

Merged build triggered.

Merged build started.

@squito, I have updated the PR and simplified it greatly, thus removing most of the magic. Now we simply append the current stack trace inside DAGScheduler, with no special marker. Only someone looking very closely at the trace would observe something amiss, and they would see the comment pointing to the associated JIRA.
I think we should go with this approach over the wrapping approach for the simple reason that it is less likely to break callers. Some callers may fail to print the cause of the exception, which would have been fine before with no cause but now would not work. Some callers may inspect the message directly, which would be different (or we would have to duplicate). Some callers may display the exception to users, which would look significantly different if wrapped (rather than just made longer).
Please let me know if you have further concerns; I have run into this problem sufficiently often now to want to push for this to go in (and the workaround is really annoying -- adding prints or try-catches to every line which may have caused the issue!).

Merged build finished. Test FAILed.

fetch failure - jenkins test this please

Merged build triggered.

Merged build started.

Test build #37107 has started for   PR 7028 at commit 4714664.

Test build #37107 has finished for   PR 7028 at commit 4714664.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class GetArrayItem(child: Expression, ordinal: Expression) extends BinaryExpression
case class GetMapValue(child: Expression, key: Expression) extends BinaryExpression

Merged build finished. Test PASSed.

Merged build triggered.

lgtm !  thanks for exploring all these options, I like this solution, huge improvement over the current behavior

Thanks! Merged into master.


Plugin configurations

Work on a guide for plugin configurations. I made these on a plane so some sections may be lacking. Let me know of any additions needed on the PR or discord :)
closes #105

Good points, should now be addressed.

CI is bugged. This PR is ready for final review and does not actually fail deployment locally

fixed


generated app cannot open and keeping crash

i have tested my binary using terminal, it 's working perfect.
console show some information:
Oct 23 13:05:56 Station com.apple.xpc.launchd[1] (application.Appify by Machine Box.goo.73055501.73055504[58537]): Service exited with abnormal code: 2

Some binaries need additional files that they locate according to their own path. I had the same problem as you but solved it by adding the needed files in the MacOS directory inside the App (where the main binary is located).


[Question] Avoid line break for components attributes

It formats this
 <x-button label="Delete" icon="o-trash" />

To this
 <x-button
    label="Delete"
    icon="o-trash"
 />

Iis there any option to avoid this behavior ? I wanna keep blade attributes on same line
 <x-button label="Excluir" icon="o-trash" />

👋 Hey, there!
This can be controlled using the singleAttributePerLine prettier option: https://prettier.io/docs/en/options.html#single-attribute-per-line
Here is an example .prettierrc file with this configuration option:
{
    "tabWidth": 4,
    "singleAttributePerLine": true,
    "plugins": [
        "./node_modules/prettier-plugin-antlers/",
        "./node_modules/prettier-plugin-blade/"
    ],
    "tailwindConfig": "tailwind.config.js",
    "overrides": [
        {
            "files": ["*.antlers.html"],
            "options": {
                "parser": "antlers"
            }
        },
        {
            "files": [
                "*.blade.php"
            ],
            "options": {
                "parser": "blade"
            }
        }
    ]
}

In addition you also need a compatible "printWidth": 300, as much needed for your use case.
Thanks!


Number subindexes not detected

At the moment, the LDPL compiler doesn't find any errors in codes like
data:
myNum is number
otherNum is number

procedure:
store myNum:0 in otherNum

and thus the C++ compilation crashes after the IR is generated. This should be fixed :disappointed:

@dgarroDC as you were working with sub-indexes and all that in the recent days (and thus you should remember how this all works better), is there any chance you could look into this issue?

I can't debug this right now but after a quick look at the code, I think that this might be the problem, maybe we should also check that indexes if empty.
But anyway, can the is_num_var be simplified by just calling variable_type with the full token and verifying if the type is NUMBER? I mean, something like this:
bool is_num_var(string & token, compiler_state & state)
{
    return (variable_type(token, state) == vector<unsigned int>{1}):
}

This is related to the bug we discussed here (is_num_var was returning true incorrectly), but when I fixed it (by replacing token with var_name), I removed that false positive but introduced this new false positive. In conclusion, it was okay to use token, the problem was all the other logic (dealing with indexes). We can remove all that logic and use token again, and the resulting function is much simpler.
This all also applies to the text case, of course.

You are right, I do like the simpler function. We should take that approach.

Fixed!

Wonderful! Thank you very much!


[Bug]: collection.query not returning n_results nearest neighbor embeddings for provided query_embeddings

What happened?
I stuffed a whole bunch of vector embeddings for images using OpenAI's CLIP model into a chroma database.  I calculated and specified the vector embeddings myself, rather than using your newly added support.
Now I enter a query like "lawn mower", my code computes the vector embedding and makes the query returning the 50 n_results along with their cosine distance / filepaths.  I am not seeing some of the pictures of lawn mowers I know are in the database, so I modified my query to include a where metadata {"filepath": "filepath_to_confirmed_lawn_mower_image"} this gives me the lawn mower image I expect and its cosine distance, which is somewhere in the middle of the range of the 50 returned from my first query.
Specifically the n_results 50 query returned cosine distances between 0.7385766336553357 and 0.7571907043457031 in my top 50.  Looking at the cosine distance returned when restricting it to the one particular image it has a cosine distance of 0.7489711046218872.  So the first query is obviously not returning the 50 closest embeddings.
My chromadb has about 0.5 million entries in it.  The number of results returned is somewhat arbitrary.  How can I get it to return the actual n_results nearest neighbor embeddings for provided query_embeddings or query_texts.
Versions
chromadb version 0.4.15
Relevant log output
No response

@shortcipher3 you mentioned your db has 0.5 million entries - does the individual collection have that many entries?

I want to try some different hnsw parameters, but I don't want to regenerate all of my vector embeddings - is there a way to iterate through the rows in a collection so that I can add them to a different collection?

@shortcipher3 hey, could you pls add your tunings for HNSW…I’m using prepared embedding set and it always return me a different results after restarting chroma server..


Fix decompression functions not returning errcodes

Compression::decompress returns -1 on error scenarios. This was not being propagated to let the user know there was an error while reading compressed files.
Fixes https://github.com/godotengine/godot/issues/57049

Thanks!

Cherry-picked for 3.5.

Cherry-picked for 3.4.3.


[New API] : Twitch API

Brief Description 🤓
Allows developers to access Twitch’s streaming data, including live stream details, channel information, and user interactions.
@dishamodi0910 assign me this issue under GSSOC'24
How will you develop it?🤩
React or Vue.js for building dynamic user interfaces.
CSS (with frameworks like Bootstrap, Tailwind CSS, or Material-UI) for styling your application.
Twitch Embed API for embedding streams and videos directly into your application.
Twitch JS SDK for integrating Twitch features and functionalities.
Deployment
None
Select the open source program you are contributing to
Other
Code of Conduct

[X] I follow CONTRIBUTING GUIDELINES of this project.

Greetings @revanth1718🎀 ! , We are excited to have you dive into another issue with us!🚀
Your involvement in our project has been invaluable, and we're confident that your skills and insights will help us conquer this challenge😇.  Use command /gssoc24 to self assign the issue.  This issue represents an exciting opportunity for us to improve and enhance our project, and we are thrilled to have you on board.
We request you to follow CONTRIBUTING GUIDELINES.
Looking for your PR soon!
In case of any issues, you can contact us on Discord.❣

/gssoc24

Hey @revanth1718 ! I wanted to inform you that we have closed issue.Thank you for your understanding, and we look forward to your continued engagement with our repository🤗.
In case of any issues, you can contact us on Discord.
Thank you! ❣
Happy Coding! ✨
Will See you soon❣


Add support for T5 for deepspeed-inference

This is the PR to add support to T5. Currently it is still work in progress. A lot of the codes are adapted from https://github.com/microsoft/DeepSpeed/pull/2451

Hi @HeyangQin, is there any update on the timeline of supporting T5 using this work?

@HeyangQin @loadams I take it that this is not on your roadmap?

Hello @alexcoca. T5 support should already be live with https://github.com/microsoft/DeepSpeed/pull/2962. Please feel free to let us know if you need more information


Create makefile with "make verify"

We should have a verify task that runs go tests and golangci-lint. As part of this, we should copy over the following files from Gateway API:

https://github.com/kubernetes-sigs/gateway-api/blob/main/.golangci.yml
https://github.com/kubernetes-sigs/gateway-api/blob/main/hack/verify-golint.sh

It may be useful to look at the Makefile from Gateway API when working on this as this one will likely be a subset of that:

https://github.com/kubernetes-sigs/gateway-api/blob/main/Makefile

I can take this up

Thanks @shouri007! Let us know if you need any help.

/remove-lifecycle stale


Is it possible to add a message to each polygon inside a multipolygon?

Given the following, is it possible to add a different message attribute to each of the polygons inside of the multipolygon array?
multiPolygon: {
   type: "multiPolygon",
   latlngs: [
     [ europeCapitals.London, europeCapitals.Lisbon , europeCapitals.Madrid, europeCapitals.Paris ],
     [ europeCapitals.Berlin, europeCapitals.Rome, europeCapitals.Brussels ],
     [ europeCapitals.London, europeCapitals.Rome, europeCapitals.Paris ]
   ]
}`

Is there an existing directive to take geojson of multipolygons and draw polygons?

No feedback closing.


[Notification center] - Plugin IFTTT + dependencies

 are comments left to guide you in writing your issue, they will not appear in the final version of the document. Feel free to contribute to the file in order to improve it.

As the last section is optional, it is surrounded by the comments characters to hide it by default. If you complete this section, remember to remove the comments characters.

Please keep in mind that this is the initial post in this thread on the issue at hand. You are welcome to regularly update it with additional information that will be helpful to readers and to make additional posts to continue the conversation. Consider this thread as a forum and use it to its full potential.
-->
Header

Name of the task: Plugin IFTTT

Name of the module: Notification center

Difficulty: 3

Waiting for: #37 #38 #39 #40 #41
Body
Short description

Now that we have a server that is consumed by our "watcher module" and received notification orders, we need to do these orders. The aim of this task is to create an open-source plugin for our notification center that will be in charge to send a notification to an IFTTT endpoint. This plugin will be used by our users to programmatically do whatever they want when they receive the notification.
Full description

Develop a simple open-source project that can be imported into the server developed in #39 to send notifications to an IFTTT endpoint. This plugin will be called by the server each time to_type is equal to ifttt.
The plugin will receive these parameters:
type PluginParameters = {
  to: string, // define the endpoint where the notification will be sent
  message: string, // define the message to send
}

All the secrets that the plugin will need must be passed during the initialization of the plugin.
The plugin must have two exposed methods:

init(...) used to init the plugin. Parameters are configuration and secrets needed
async send(to: string, message: string) used to send a notification.

The send function must return the status of the external request made.
pnpm must be used as a dependency manager. A .nvmrc must be set to the LTS version of node. No library should be used to interact with the endpoint, keep the code simple. Everything must be tested, prettied, eslinted and the README of the repository should describe the plugin. Keep in mind the plugin will be an npm package.

In addition to that, the server defined in #39 must be updated to accept the new to_type value: ifttt. At this point, the toType type will look like this:
type ToType = "telegram" | "mail" | "ifttt";

Also, the server defined in #38 must be updated in order to add ifttt as a key for the Values type. The Values type will look like this:
type Values = {
    telegrams?: string[],
    emails?: string[],
    ifttt?: string[], // IFTTT endpoints
};

Finally, the schema of the database defined in #37 must be updated. It will look like this:
type Schema = {
    id: string, // wallet address
    telegrams: string[], // handles
    emails: string[],
    ifttts: string[],
 }

The migration must be developed, and existing rows must inherit an empty value for the new iftts key.
Additionals ressources

pnpm
eslint
prettier
.nvmrc
IFTTT

Shipped by @apoorveth like a boss here 💪


Fixes session storage for HHVM (and possibly strict PHP environments)

The SessionHandlerInterface expects boolean return values for all methods (except read). This must've been an oversight when implementing this interface.
Unfortunately, in HHVM, this produces this (extremely unhelpful) error:
Warning: Failed to write session data (user). Please verify that the current setting of session.save_path is correct (unix:///var/run/redis/redis.sock) in ...

Quick and easy win!

@snc can you merge this?

@snc Pretty please?

@snc Please merge this!

@snc please this is a matter of life and death!

@snc anything that blocks this from merging?

@digitalkaoz No, just the original author being AWOL :(

Fixed it by overriding the class in the symfony dic. Works like a charm. A
merge would be a breeze :)
Burkhard Reffeling<EMAIL_ADDRESS>schrieb am Di., 30. Juni 2015
16:48:

@digitalkaoz https://github.com/digitalkaoz No, just the original
author being AWOL :(
—
Reply to this email directly or view it on GitHub
https://github.com/snc/SncRedisBundle/pull/164#issuecomment-117214351.

Merged, thanks.

:kissing_cat: merci bien


UnboundLocalError: Caught UnboundLocalError in DataLoader worker process 0.

https://github.com/ThibaultGROUEIX/AtlasNet/blob/90fcd8a2650ef840ebc41319f435402fa1ba43ca/auxiliary/dataset.py#L110-L114
UnboundLocalError: Caught UnboundLocalError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/brando/anaconda3/envs/pytorch-atlasnet/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/brando/anaconda3/envs/pytorch-atlasnet/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/brando/anaconda3/envs/pytorch-atlasnet/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "./auxiliary/dataset.py", line 260, in __getitem__
    point_set = point_set[:, 0:3]
UnboundLocalError: local variable 'point_set' referenced before assignment

So what's your "weird error"? I tried to move the point_set assignment out of the for loop but it did not work.

Hi @BrandoZhang,
point_set is unassigned which means all the try cases fail : it shouldn't happen.
What exception do you get from the "except" part ?
The weird error I am referring to is that sometime my_get_n_random_lines fails because each line in the file doesn't have the same number of character. I hardcoded the number of consecutive characters to read here, for speed. Those cases are rare, and it can't happen 15 times in a row. i'll update this part in the next iteration of the code.
Best regards,
Thibault

Hi @ThibaultGROUEIX

The weird error I am referring to is that sometime my_get_n_random_lines fails because each line in the file doesn't have the same number of character. I hardcoded the number of consecutive characters to read here, for speed. Those cases are rare, and it can't happen 15 times in a row. i'll update this part in the next iteration of the code.

Yep, I notice this. I use the following codes to bypass the problem:
def get_n_random_vertices(path_to_file, n=5):
    mesh = pymesh.load_mesh(path_to_file)
    rv_raw = mesh.vertices[np.random.randint(mesh.vertices.shape[0], size=n), :].tolist()
    return [str(i).replace('[', '').replace(']', '').replace(',', '') for i in rv_raw]

To use this, you may need to install and import pymesh.
I just figured it out minutes ago to make sure the data structure is consistent. However, I think this process might be much slower than previous one (Especially for the return line, it is really dirty to format in this way).
Any advice?

Yes pymesh (or trimesh) makes this very slow, because the pointcloud sizes are large. that's why i used my_get_n_random_lines in the first place. it only reads 2000 consecutive random lines from the file, making things much faster.
I think you  have two options :

fix your usage of my_get_n_random_lines(it should work)
rewrite the dataloader to put all the pointclouds in RAM (using pymesh to create a dataset tensor at the beginning of training). I think it should fit in memory. Just index the dataset tensor during training.
Best regards


How multi server deployment works

Q
A

Issue Type
Question

Deployer Version
4.2.1

Local Machine OS
N/A

Remote Machine OS
N/A

I just wondering it does start with server1 and then executes all tasks and after that will start with second server? or it does task1 on server 1 and server2 then task2 on server1 and server2..
I understand that updates will come almost with no downtime but in my case I am doing long upgrades and want to put whole server under maintenance(because I know other server will take care of requests) and after is up to date I would like to bring it up and do updates with second server

Some useful information are available at the end of the following webpage: https://deployer.org/docs/tasks

oh I see now :)
but it looks it does one task in all servers and only starts second when all servers finish with first one :(
maybe there is some way to do all tasks to first server and then start second and so on?

I haven't seen this feature, but you can create stages and run the deployer for them one after another.
Assign server to stage: https://deployer.org/docs/servers
Run Deployer for the assigned stage: php vendor/bin/dep deploy <your_stage>

ok this answers my question just don't like idea to run deploy command for each server of course i could create a bash script for this but would be more happy with built in feature :D

@svycka you can do this by bash script:
dep deploy servers_a &
dep deploy servers_b &
dep deploy servers_c &

This does not included in code, i try to keep code simple. I think adding this to docs.

@elfet yes I can, just would like to have more control how to run tasks


Returning an Exception "Container Not Found"

Type of issue
Missing information
Description
var uploadContainerEndpoint = string.Format("https://{0}.blob.core.windows.net/{1}", storageName, containerName);
var credential = new VisualStudioCredential();
var blobServiceClientOptions = new BlobClientOptions()
{
Retry = {
// Adjusting the Network Timeout to 5 minutes
NetworkTimeout = TimeSpan.FromMinutes(15)
}
};
var blobUploadContainerClient = new BlobContainerClient(new Uri(uploadContainerEndpoint), credential, blobServiceClientOptions);
bool isExists = blobUploadContainerClient.Exists();
It is returning "true" when container exists (expected behaviour).
But when container doesn't exist, instead of returning "false", it is throwing exception "Container Not Found".

Please help me understanf if this is an expected behaviour or I need to make some changes.
Page URL
https://learn.microsoft.com/en-us/dotnet/api/azure.storage.blobs.blobcontainerclient.exists?view=azure-dotnet
Content source URL
https://github.com/Azure/azure-docs-sdk-dotnet/blob/master/xml/Azure.Storage.Blobs/BlobContainerClient.xml
Document Version Independent Id
86e0dc6a-aea6-5baa-c6a2-d016ffd7445a
Article author
@azure-sdk
Metadata

ID: ddcbc767-2ed7-b376-51fe-34c13fe7b391
Service: azure

Hey Team.
Any update on this?

Hello Team.
Following up on the above issue

Hi @dotcom-gosahi, based on how the code is written, it seems that if the container does not exist it will just return false with no exceptions thrown
https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.Blobs/src/BlobContainerClient.cs#L1656
I have also tried replicating your above code as a sample test and did not run into your issue (container did not exist and just returned false). Here is my code that replicated the code you provided
        [Test]         public async Task SampleTestForContainerExists()         {             await using DisposingContainer test = await GetTestContainerAsync();             var uploadContainerEndpoint = string.Format("https://{0}.blob.core.windows.net/{1}", test.Container.AccountName, GetNewContainerName());             var credential = new VisualStudioCredential();             var blobServiceClientOptions = new BlobClientOptions()             {                 Retry = {                 // Adjusting the Network Timeout to 5 minutes                 NetworkTimeout = TimeSpan.FromMinutes(15)                 }             };             var blobUploadContainerClient = new BlobContainerClient(new Uri(uploadContainerEndpoint), credential, blobServiceClientOptions);             bool isExists = blobUploadContainerClient.Exists();         }
Have you made sure your account name and credentials are set correctly?
cc: @amnguye


Fix exit with error if ctest failed

I noticed that a unit test for intel 16.0.4 on quartz was failing but this wasn’t caught by CI.
This is due to a mistake in exit status management in build_and_test.sh script.
This PR fixes that.

LGTM


Update Architecture Direct link

Please provide a description of this PR:
The document for the isio operator architecture has been moved to a different location, but it is not reflected in the link. This link should be modified because it causes a small inconvenience in reading the document.

Hi @rlaisqls. Thanks for your PR.
I'm waiting for a istio member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.
Once the patch is verified, the new status will be reflected by the ok-to-test label.
I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

/ok-to-test
can you sign CLA?

@rlaisqls: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ds_istio
0c86e3d55ffefb1da36662f9ee7b95f496b991f7
link
true
/test integ-ds

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@rlaisqls: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-ds_istio
0c86e3d55ffefb1da36662f9ee7b95f496b991f7
link
true
/test integ-ds

integ-pilot-istiodremote-mc_istio
0c86e3d55ffefb1da36662f9ee7b95f496b991f7
link
true
/test integ-pilot-istiodremote-mc

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest

/retest

@rlaisqls: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

integ-pilot-istiodremote-mc_istio
0c86e3d55ffefb1da36662f9ee7b95f496b991f7
link
true
/test integ-pilot-istiodremote-mc

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

/retest


istio-egressgateway crashloop

2019-08-14T02:48:29.131445Z     info    Starting proxy agent
2019-08-14T02:48:29.131829Z     info    Received new config, resetting budget
2019-08-14T02:48:29.131859Z     info    Reconciling retry (budget 10)
2019-08-14T02:48:29.132192Z     info    Epoch 0 starting
2019-08-14T02:48:29.131834Z     info    watching /etc/certs for changes
2019-08-14T02:48:29.147816Z     info    Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-egressgateway --service-node router~<IP_ADDRESS>~istio-egressgateway-dbdf59977-wzfvs.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --allow-unknown-fields -l warning --component-log-level misc:error]
[2019-08-14 02:48:29.206][22][warning][runtime] [external/envoy/source/common/runtime/runtime_impl.cc:486] Skipping unsupported runtime layer: name: "base"
static_layer {
}

[2019-08-14 02:48:29.215][22][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, no healthy upstream
[2019-08-14 02:48:29.215][22][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:50] Unable to establish new stream
2019-08-14T02:48:31.588622Z     info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-08-14T02:48:33.306094Z     info    Envoy proxy is ready
2019-08-14T02:48:36.335596Z     warn    Epoch 0 terminated with an error: signal: killed
2019-08-14T02:48:36.335631Z     warn    Aborted all epochs
2019-08-14T02:48:36.335810Z     info    Epoch 0: set retry delay to 200ms, budget to 9
2019-08-14T02:48:36.535926Z     info    Reconciling retry (budget 9)
2019-08-14T02:48:36.536008Z     info    Epoch 0 starting
2019-08-14T02:48:36.542704Z     info    Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-egressgateway --service-node router~<IP_ADDRESS>~istio-egressgateway-dbdf59977-wzfvs.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --allow-unknown-fields -l warning --component-log-level misc:error]
[2019-08-14 02:48:36.587][93][warning][runtime] [external/envoy/source/common/runtime/runtime_impl.cc:486] Skipping unsupported runtime layer: name: "base"
static_layer {
}

[2019-08-14 02:48:36.591][93][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, no healthy upstream
[2019-08-14 02:48:36.591][93][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:50] Unable to establish new stream
2019-08-14T02:48:39.000452Z     info    Envoy proxy is NOT ready: server is not live, current state is: PRE_INITIALIZING
2019-08-14T02:48:39.339410Z     info    Envoy proxy is NOT ready: server is not live, current state is: INITIALIZING
2019-08-14T02:48:42.525052Z     warn    Epoch 0 terminated with an error: signal: killed
2019-08-14T02:48:42.525113Z     warn    Aborted all epochs
2019-08-14T02:48:42.525133Z     info    Envoy proxy is NOT ready: failed to get server info: failed retrieving Envoy stats: Get http://<IP_ADDRESS>:15000/server_info: dial tcp <IP_ADDRESS>:15000: connect: connection refused
2019-08-14T02:48:42.525237Z     info    Epoch 0: set retry delay to 400ms, budget to 8
2019-08-14T02:48:42.925401Z     info    Reconciling retry (budget 8)
2019-08-14T02:48:42.925514Z     info    Epoch 0 starting
2019-08-14T02:48:42.931550Z     info    Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-egressgateway --service-node router~<IP_ADDRESS>~istio-egressgateway-dbdf59977-wzfvs.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --allow-unknown-fields -l warning --component-log-level misc:error]
[2019-08-14 02:48:42.975][164][warning][runtime] [external/envoy/source/common/runtime/runtime_impl.cc:486] Skipping unsupported runtime layer: name: "base"
static_layer {
}

[2019-08-14 02:48:42.979][164][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, no healthy upstream
[2019-08-14 02:48:42.979][164][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:50] Unable to establish new stream
2019-08-14T02:48:43.284379Z     info    Envoy proxy is NOT ready: server is not live, current state is: PRE_INITIALIZING
2019-08-14T02:48:45.392460Z     info    Envoy proxy is NOT ready: server is not live, current state is: PRE_INITIALIZING
2019-08-14T02:48:48.732908Z     warn    Epoch 0 terminated with an error: signal: killed
2019-08-14T02:48:48.732946Z     warn    Aborted all epochs
2019-08-14T02:48:48.732992Z     info    Envoy proxy is NOT ready: failed to get server info: failed retrieving Envoy stats: Get http://<IP_ADDRESS>:15000/server_info: dial tcp <IP_ADDRESS>:15000: connect: connection refused
2019-08-14T02:48:48.733114Z     info    Epoch 0: set retry delay to 800ms, budget to 7
2019-08-14T02:48:49.283156Z     info    Envoy proxy is NOT ready: failed to get server info: failed retrieving Envoy stats: Get http://<IP_ADDRESS>:15000/server_info: dial tcp <IP_ADDRESS>:15000: connect: connection refused
2019-08-14T02:48:49.533385Z     info    Reconciling retry (budget 7)
2019-08-14T02:48:49.533540Z     info    Epoch 0 starting
2019-08-14T02:48:49.539895Z     info    Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-egressgateway --service-node router~<IP_ADDRESS>~istio-egressgateway-dbdf59977-wzfvs.istio-system~istio-system.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --allow-unknown-fields -l warning --component-log-level misc:error]
[2019-08-14 02:48:49.595][235][warning][runtime] [external/envoy/source/common/runtime/runtime_impl.cc:486] Skipping unsupported runtime layer: name: "base"
static_layer {
}

[2019-08-14 02:48:49.602][235][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, no healthy upstream
[2019-08-14 02:48:49.602][235][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:50] Unable to establish new stream

This is with the release-1.3 build from 8/13

This happened in a large cluster, later I seemed to be getting OOMs as well -- maybe they were all from OOM and I didn't notice. it seems to not happen on a smaller cluster

ah it looks like our memory limit is only 256Mi but 1GB on ingress, probably the root cause

The large cluster you mean a cluster with high egress traffic or large number of serviceentry/gateway crd?

I have no egress traffic, just a fair amount of config - about 500 pods across 50ns or so. Egress was using about 256mb - not a crazy amount, but just enough to trigger oom


No tests for non-"time" ROM pivots

Issue Description
There are few coverage tests for time-dependent scikitlearn ROMs, and apparently none of them uses the pivotParameter node.  We should add testing to time-dependent ROMs to cover basic utility.

For Change Control Board: Issue Review
This review should occur before any development is performed as a response to this issue.

[x] 1. Is it tagged with a type: defect or improvement?
[x] 2. Is it tagged with a priority: critical, normal or minor?
[x] 3. If it will impact requirements or requirements tests, is it tagged with requirements?
[x] 4. If it is a defect, can it cause wrong results for users? If so an email needs to be sent to the users.
[x] 5. Is a rationale provided? (Such as explaining why the improvement is needed or why current code is wrong.)

For Change Control Board: Issue Closure
This review should occur when the issue is imminently going to be closed.

[x] 1. If the issue is a defect, is the defect fixed?
[x] 2. If the issue is a defect, is the defect tested for in the regression test system?  (If not explain why not.)
[x] 3. If the issue can impact users, has an email to the users group been written (the email should specify if the defect impacts stable or master)?
[x] 4. If the issue is a defect, does it impact the latest stable branch? If yes, is there any issue tagged with stable (create if needed)?
[x] 5. If the issue is being closed without a merge request, has an explanation of why it is being closed been provided?

The closure checklist is ok.


Cannot set locale in terminal

Showing bash: warning: setlocale: LC_ALL: cannot change locale (en-US) when opening a terminal window

VSCode Version: code 1.51.1
OS Version: Arch Linux

Steps to Reproduce:

Open terminal using ctrl + shift + `

Does this issue occur when all extensions are disabled?: Yes

Did you try to set the locale or do anything that could have caused this?
If not, you can set "terminal.integrated.detectLocale": "off"

The problem is at electron level: https://github.com/microsoft/vscode/issues/110322 and https://github.com/electron/electron/issues/26443

Downgrading Electron sorts it until an update for it is available
https://github.com/microsoft/vscode/issues/110322#issuecomment-726426703

I also have this issue running any command, look at the terminal, it just breaks:

This started happening after I upgraded electron9 without upgrading code. I received code version 1.51.1 (ArchLinux) and I still have the same problem. I  also saw this problem on reddit...

Showing bash: warning: setlocale: LC_ALL: cannot change locale (en-US) when opening a terminal window

VSCode Version: code 1.51.1
OS Version: Arch Linux

Steps to Reproduce:

Open terminal using ctrl + shift + `

Does this issue occur when all extensions are disabled?: Yes

I have the same problem

Merging to https://github.com/microsoft/vscode/issues/110322

I have the same problem.

I have this problem too

This solution
telmotrooper commented 9 days ago •
For anyone having this issue on Arch Linux, downgrade package electron9 from 9.3.4 to 9.3.3.
curl -O https://archive.archlinux.org/packages/e/electron9/electron9-9.3.3-1-x86_64.pkg.tar.zst
sudo pacman -U electron9-9.3.3-1-x86_64.pkg.tar.zst
You'll probably want to add electron9* to IgnorePkg in /etc/pacman.conf, so the next time you update your packages you won't accidentally update this one as well. When a fix has been deployed you can remove this setting...

Electron 9.3.5-1 is available for Arch.
pacman -Sy electron9 should do the trick

manjaro does not have this this version yet, what can i do in the interim?


en: add some notices about connectivity

What is changed, added or deleted? (Required)
add some notices about connectivity
Which DM version(s) do your changes apply to? (Required)

[x] master (the latest development version, including v3.0 changes)
[x] v2.0 (TiDB DM 2.0 versions)
[ ] v1.0 (TiDB DM 1.0 versions)

What is the related PR or file link(s)?

This PR is translated from: https://github.com/pingcap/docs-dm/pull/342
Other reference link(s):

Do your changes match any of the following descriptions?

[ ] Delete files
[ ] Change aliases
[ ] Have version specific changes
[ ] Might cause conflicts after cherry-picked

/merge


Files pane wont update if a file or folder added from outside of vscode

VSCode Version:1.1.1
OS Version:ubuntu 16.4

Steps to Reproduce:

open vscode
open files pane
open file manager (in my case nautilus)
create a file or folder
hit refresh button on right top corner of files pane in vscode
new created file or folder wont take place there untill restarting vscode

@NikosEfthias can you provide more details what file you create and where? can you attach a video showing the issue in detail?

i am compilig typescript and if the compilatiion occured first time the fil is not there untill i resrart vscode

But your "steps to reproduce" sound different from that?

@bpasero somehow it seems to be working now


Trembl file not formated correctly

Hello, I got this error:
Error: Error detecting input file format. First line seems to be blank.
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/guelou01/miniconda3/envs/microbeannotator/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/home/guelou01/miniconda3/envs/microbeannotator/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/home/guelou01/miniconda3/envs/microbeannotator/lib/python3.7/site-packages/microbeannotator/pipeline/protein_search.py", line 232, in similarity_search
    blast_filter_slow(method_output_file, filtered_file, id_perc, bitscore, evalue, aln_percent)
  File "/home/guelou01/miniconda3/envs/microbeannotator/lib/python3.7/site-packages/microbeannotator/pipeline/protein_search.py", line 138, in blast_filter_slow
    with open(input_tab) as blast_input:
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/projects_tn03/KMER_analysis/DATA/AcneData/annotation_unitigs/trembl_results/case_unitigs_1000.fasta.step_4.diamond'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/guelou01/miniconda3/envs/microbeannotator/bin/microbeannotator", line 981, in <module>
    main()
  File "/home/guelou01/miniconda3/envs/microbeannotator/bin/microbeannotator", line 755, in main
    multiple_arguments=arguments_to_pass), input_proteins)
  File "/home/guelou01/miniconda3/envs/microbeannotator/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/guelou01/miniconda3/envs/microbeannotator/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/projects_tn03/KMER_analysis/DATA/AcneData/annotation_unitigs/trembl_results/case_unitigs_1000.fasta.step_4.diamond'

The trembl table I found in my database seems fine..
UPDATE:  I rebuilt the database, ths is the result, notice the Trembl part:
2024-01-20 06:06:11,918 [INFO]: KO profiles:    MicrobeAnnotator_DB/kofam_data/profiles
2024-01-20 06:06:11,918 [INFO]: KO profile metadata:    MicrobeAnnotator_DB/kofam_data/ko_list
2024-01-20 06:06:11,918 [INFO]: Swissprot Fasta:        MicrobeAnnotator_DB/protein_db/uniprot_sprot.fasta.gz
2024-01-20 06:06:11,918 [INFO]: TrEMBL Fasta:   None
2024-01-20 06:06:11,918 [INFO]: RefSeq Fasta:   MicrobeAnnotator_DB/protein_db/refseq_protein.fasta
2024-01-20 06:06:11,919 [INFO]: SQLite database:        MicrobeAnnotator_DB/microbeannotator.db
2024-01-20 06:06:11,919 [INFO]: Conversion database:    MicrobeAnnotator_DB/conversion.db
2024-01-20 06:06:11,919 [INFO]: KO-to-EC table: MicrobeAnnotator_DB/KO_to_EC_identifiers.txt.gz
2024-01-20 06:06:11,919 [INFO]: InterPro table: MicrobeAnnotator_DB/interpro_metadata.xml.gz

My input was not formated correctly, once changed it went fine. My bad !


[Feature][e2e] add e2e test case time analysis

Purpose of this pull request
add e2e test case time analysis , it provides data basis for further optimization of E2E test cases
log format : [$className#$methodName] E2E test case cost $time s.

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[ ] If you are contributing the connector code, please check that the following files are updated:

Update plugin-mapping.properties and add new connector information in it
Update the pom file of seatunnel-dist
Add ci label in label-scope-conf
Add e2e testcase in seatunnel-e2e
Update connector plugin_config

[ ] Update the release-note.

good pr


Unable to find "select" (dropdown) component

Is there a dropdown component in material design lite? I searched but was unable to locate.

Duplicate of #854.
Please search existing issues before creating new ones.
Thank you.


Pipeline completion trigger doc doesn't mention parameters aren't allowed

I added a parameter with a default value in a pipeline that is triggered by another. It caused the pipeline not to be triggered. I can't find any documentation on this.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 86285f72-9e28-da97-59bb-c29eb60f627d
Version Independent ID: 18d5a591-a7d3-c261-6bff-8808ae433f54
Content: Configure pipeline triggers - Azure Pipelines
Content Source: docs/pipelines/process/pipeline-triggers.md
Product: devops
Technology: devops-cicd-process
GitHub Login: @steved0x
Microsoft Alias: sdanie

Finally it seems to work


Yabai keeps restarting using Xcode 11.5

Yabai keeps restarting whenever I am writing code in Xcode. Doesn't happen in any other application.
I have not setup auto-updates which might cause it to restart.
The only keys I have mapped to restart yabai are shift + alt + ctrl + r which are not pressed when I am writing code.

Duplicate of #580
Update to yabai v3.2.1 and the issue should be gone.

Thank you, it was indeed this issue.


Jeanette/scoot175 daemon api

Looking for early feedback on this approach for a python command line inteface to the local daemon.
daemon_cli_test.py runs in my eclipse environment, but does not run via command line ($python -m unittest daemon_cli_test) - it fails because it cannot start the daemon.
I've checked in the docopt.py code for now (so folks don't have to install it, till we agree we want to use this tool).
This is not ready to merge to master yet - looking for feedback on:

the use of docopt
the syntax for the daemon commands
'pythonic' coding practices that I'm not using

Current coverage is 65.54% (diff: 100%)

Merging #171 into master will increase coverage by 0.53%

@@             master       #171   diff @@
==========================================
  Files            63         63
  Lines          2918       3149   +231
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
+ Hits           1897       2064   +167
- Misses          896        957    +61
- Partials        125        128     +3

Powered by Codecov. Last update be9cbc2...0b5cee3

Since we got rid of most of the manual parsing I am fine with the way docopt is working, but if we're still going to keep it we should remove the real docopt.py module and include the dependency in the client setup.py (assuming the cli and lib will be in the same distribution).


sudo atom -a file open in new window (not in window already open)

Description
sudo atom -a file open in new window (not in window already open)
$ atom --help
  -a, --add  Open path as a new project in last used window.  [boolean]

Steps to Reproduce
Actual behavior: [What actually happened]
atom -a file1 ( first time open in new window of atom )
atom -a file2 ( second time open in window already open of atom )
sudo atom -a file3  ( Third time open in new window of sudo atom )
sudo atom -a file4  ( Fourt time open in window already open of sudo atom )
Expected behavior: [What you expected to happen]
atom -a file1 ( first time open in new window of atom )
atom -a file2 ( second time open in window already open of atom )
sudo atom -a file3  ( Third time open in window already open of atom )
sudo atom -a file4  ( Fourth time open in window already open of atom )
Versions
$ atom --version
Atom    : 1.7.3
Electron: 0.36.8
Chrome  : 47.0.2526.110
Node    : 5.1.1

usuario at wicope in ~
$ apm --version
apm  1.9.2
npm  2.13.3
node 0.10.40
python 2.7.6
git 1.9.1

usuario at wicope in ~
$ uname -a
Linux wicope 3.13.0-76-lowlatency #120+7.0trisquel2 SMP PREEMPT Tue Jan 19 19:55:06 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Trisquel
Description:	Trisquel GNU/Linux 7.0, Belenos
Release:	7.0
Codename:	belenos

Could you clarify on a little thing here, why would you want to open it in the same window. That way you will not know which "Tabs" have superuser, and which do not?
I propose this though, a feature which asks you to open the file with sudo when you try to save it and it fails (due to user restrictions).
Kudos


Added AWS HA RDS deploy guidance

Added AWS RDS HA config guidance to the following product docs pages:

HA Cluster > Database Configuration
Data Source, Data Source Replicas, and Data Source Search Replicas configuration setting docs

Are there additional docs sections that would benefit from links into this new guidance?

@adityadav1987 - Are you comfortable with this docs PR being merged and the content published to production?


Feature request: format code

Maybe by letting the user specify a binary for an extension, so you won't have to support all the programming language in the world.

this is an interesting idea though i'm not quite sure how to implement this - do you have any ideas?

Do you know if we can execute binaires on the user's computer? Or if it's like a web browser.

sorry, i have no clue

this feature is a standalone plugin called Code Formatter and can be found in the Logseq marketplace