G2 arc code freeze

Hello. i think this programs is great!
I use it for a year and I like it more than the web of chilipeperr.com,It is comfortable and visually pleasing
I use now tinyg2, de 100.x branch.
i use vectric aspire and when i generate gcode with arcs, g2 or g3, the tinyg2 freeze or make bad arcs.
First i  think was a tinyg2 issue, may be the configuration. i try other earlier versions but not resolve the problem.
But yesterday i try to send the same gcode through chilipepperr.com, and woilaa, it works correct.
When i use this program to send gcode like this:

G17
G21
G0 Z2.0010
G0 X0.0000 Y0.0000
G0 X5.0000 Y0.0000 Z2.0000
G1   Z-1.0000 F10000.0
G2 X0.0000 Y5.0000 I0.0000 J5.0000 F1000.0
G2 X5.0000 Y10.0000 I5.0000 J0.0000
G2 X10.0000 Y5.0000 I0.0000 J-5.0000
G2 X5.0000 Y0.0000 I-5.0000 J0.0000
G0   Z2.0000
G0 Z2.0010
G0 X0.0000 Y0.0000
M30

the program stops o make bad desings (like straigth instead of arcs).
some times in one point. sometimes in other.
may be the problem is with the J besause,
when i send code like this:

G17
G21
G2 I50 F2000
it makes the correct circle.

Can it be a problem sending the gcode through the json server?
thanks

It works well if I send commands (in JSON format) line by line from the Console widget. You can try it manually:
{gc: "G17"}
{gc: "G21"}
{gc: "G0 Z2.0010"}
{gc: "G0 X0.0000 Y0.0000"}
{gc: "G0 X5.0000 Y0.0000 Z2.0000"}
{gc: "G1 Z-1.0000 F10000.0"}
{gc: "G2 X0.0000 Y5.0000 I0.0000 J5.0000 F1000.0"}
{gc: "G2 X5.0000 Y10.0000 I5.0000 J0.0000"}
{gc: "G2 X10.0000 Y5.0000 I0.0000 J-5.0000"}
{gc: "G2 X5.0000 Y0.0000 I-5.0000 J0.0000"}
{gc: "G0 Z2.0000"}
{gc: "G0 Z2.0010"}
{gc: "G0 X0.0000 Y0.0000"}
{gc: "M30"}

However, if I send the whole in a batch, things goes wrong just like you said - straight instead arcs. I will do a test on Chillipeppr and check for differences.

Here is my output with cnc -vvv, I removed the last line M30 in this test:
2016-12-09T02:44:19.961Z - silly: [TinyG2] < {"r":{},"f":[1,0,12]}
2016-12-09T02:44:19.962Z - silly: [TinyG2] < {"r":{},"f":[1,0,12]}
2016-12-09T02:44:19.966Z - silly: [TinyG2] < {"r":{},"f":[1,0,18]}
2016-12-09T02:44:19.966Z - silly: [TinyG2] < {"qr":27,"qi":1,"qo":0}
2016-12-09T02:44:19.967Z - silly: [TinyG2] < {"r":{},"f":[1,0,17]}
2016-12-09T02:44:19.968Z - silly: [TinyG2] < {"r":{},"f":[1,0,20]}
2016-12-09T02:44:19.969Z - silly: [TinyG2] < {"qr":26,"qi":1,"qo":0}
2016-12-09T02:44:19.970Z - silly: [TinyG2] < {"r":{},"f":[1,0,22]}
2016-12-09T02:44:19.970Z - silly: [TinyG2] < {"qr":25,"qi":1,"qo":0}
2016-12-09T02:44:19.972Z - silly: [TinyG2] < {"r":{},"f":[1,0,23]}
2016-12-09T02:44:19.973Z - silly: [TinyG2] < {"r":{},"f":[1,0,24]}
2016-12-09T02:44:19.975Z - silly: [TinyG2] < {"r":{},"f":[1,0,25]}
2016-12-09T02:44:19.978Z - silly: [TinyG2] < {"r":{},"f":[1,0,24]}
2016-12-09T02:44:19.980Z - silly: [TinyG2] < {"r":{},"f":[1,0,14]}
2016-12-09T02:44:19.982Z - silly: [TinyG2] < {"r":{},"f":[1,0,18]}
2016-12-09T02:44:19.984Z - silly: [TinyG2] < {"r":{},"f":[1,0,17]}
2016-12-09T02:44:19.988Z - silly: [TinyG2] < {"qr":5,"qi":20,"qo":0}
2016-12-09T02:44:20.213Z - silly: [TinyG2] < {"sr":{"vel":70.18,"stat":5,"cycs":1,"mots":1,"momo":0,"posz":0.069,"mpoz":0.069}}
2016-12-09T02:44:20.463Z - silly: [TinyG2] < {"sr":{"vel":228.65,"posz":0.784,"mpoz":0.784}}
2016-12-09T02:44:20.718Z - silly: [TinyG2] < {"sr":{"vel":156.70,"posz":1.665,"mpoz":1.665}}
2016-12-09T02:44:20.877Z - silly: [TinyG2] < {"qr":6,"qi":0,"qo":1}
2016-12-09T02:44:20.971Z - silly: [TinyG2] < {"sr":{"vel":118.62,"posx":0.168,"posz":2.001,"mpox":0.168,"mpoz":2.001}}
2016-12-09T02:44:21.224Z - silly: [TinyG2] < {"sr":{"vel":321.49,"posx":1.041,"posz":2.001,"mpox":1.041,"mpoz":2.001}}
2016-12-09T02:44:21.478Z - silly: [TinyG2] < {"sr":{"vel":418.91,"posx":2.708,"posz":2.000,"mpox":2.708,"mpoz":2.000}}
2016-12-09T02:44:21.733Z - silly: [TinyG2] < {"sr":{"vel":267.10,"posx":4.252,"posz":2.000,"mpox":4.252,"mpoz":2.000}}
2016-12-09T02:44:21.986Z - silly: [TinyG2] < {"sr":{"vel":110.41,"posx":4.945,"posz":2.000,"mpox":4.945,"mpoz":2.000}}
2016-12-09T02:44:22.011Z - silly: [TinyG2] < {"qr":7,"qi":0,"qo":1}
2016-12-09T02:44:22.241Z - silly: [TinyG2] < {"sr":{"vel":203.76,"feed":10000.00,"momo":1,"posx":5.000,"posz":1.479,"mpox":5.000,"mpoz":1.479}}
2016-12-09T02:44:22.491Z - silly: [TinyG2] < {"sr":{"vel":296.97,"posz":0.328,"mpoz":0.328}}
2016-12-09T02:44:22.746Z - silly: [TinyG2] < {"sr":{"vel":155.18,"posz":-0.686,"mpoz":-0.686}}
2016-12-09T02:44:22.896Z - silly: [TinyG2] < {"qr":8,"qi":0,"qo":1}
2016-12-09T02:44:23.002Z - silly: [TinyG2] < {"sr":{"vel":154.17,"momo":2,"posx":4.795,"posy":0.013,"posz":-1.000,"mpox":4.795,"mpoy":0.013,"mpoz":-1.000}}
2016-12-09T02:44:23.125Z - silly: [TinyG2] < {"qr":9,"qi":0,"qo":1}
2016-12-09T02:44:23.253Z - silly: [TinyG2] < {"sr":{"vel":254.47,"posx":4.052,"posy":0.428,"mpox":4.052,"mpoy":0.428}}
2016-12-09T02:44:23.508Z - silly: [TinyG2] < {"sr":{"vel":516.69,"posx":3.066,"posy":1.713,"mpox":3.066,"mpoy":1.713}}
2016-12-09T02:44:23.762Z - silly: [TinyG2] < {"sr":{"vel":554.75,"posx":1.623,"posy":3.593,"mpox":1.623,"mpoy":3.593}}
2016-12-09T02:44:24.016Z - silly: [TinyG2] < {"sr":{"vel":302.31,"posx":0.492,"posy":5.067,"mpox":0.492,"mpoy":5.067}}
2016-12-09T02:44:24.191Z - silly: [TinyG2] < {"qr":10,"qi":0,"qo":1}
2016-12-09T02:44:24.270Z - silly: [TinyG2] < {"sr":{"vel":236.26,"posx":0.098,"posy":5.932,"mpox":0.098,"mpoy":5.932}}
2016-12-09T02:44:24.524Z - silly: [TinyG2] < {"sr":{"vel":302.96,"posx":0.770,"posy":6.695,"mpox":0.770,"mpoy":6.695}}
2016-12-09T02:44:24.779Z - silly: [TinyG2] < {"sr":{"vel":528.01,"posx":2.281,"posy":7.705,"mpox":2.281,"mpoy":7.705}}
2016-12-09T02:44:25.033Z - silly: [TinyG2] < {"sr":{"vel":478.75,"posx":4.136,"posy":8.944,"mpox":4.136,"mpoy":8.944}}
2016-12-09T02:44:25.287Z - silly: [TinyG2] < {"sr":{"vel":256.20,"posx":5.378,"posy":9.774,"mpox":5.378,"mpoy":9.774}}
2016-12-09T02:44:25.363Z - silly: [TinyG2] < {"qr":12,"qi":0,"qo":2}
2016-12-09T02:44:25.541Z - silly: [TinyG2] < {"sr":{"vel":246.39,"posx":6.328,"posy":9.779,"mpox":6.328,"mpoy":9.779}}
2016-12-09T02:44:25.794Z - silly: [TinyG2] < {"sr":{"vel":409.25,"posx":7.040,"posy":8.713,"mpox":7.040,"mpoy":8.713}}
2016-12-09T02:44:26.048Z - silly: [TinyG2] < {"sr":{"vel":539.46,"posx":8.225,"posy":6.939,"mpox":8.225,"mpoy":6.939}}
2016-12-09T02:44:26.303Z - silly: [TinyG2] < {"sr":{"vel":371.28,"posx":9.355,"posy":5.248,"mpox":9.355,"mpoy":5.248}}
2016-12-09T02:44:26.528Z - silly: [TinyG2] < {"qr":14,"qi":0,"qo":2}
2016-12-09T02:44:26.557Z - silly: [TinyG2] < {"sr":{"vel":247.19,"posx":9.938,"posy":4.248,"mpox":9.938,"mpoy":4.248}}
2016-12-09T02:44:26.813Z - silly: [TinyG2] < {"sr":{"vel":258.41,"momo":0,"posx":9.503,"posy":3.456,"posz":-0.797,"mpox":9.503,"mpoy":3.444,"mpoz":-0.788}}
2016-12-09T02:44:27.068Z - silly: [TinyG2] < {"sr":{"vel":531.56,"posx":8.317,"posy":2.545,"posz":-0.060,"mpox":8.317,"mpoy":2.545,"mpoz":-0.040}}
2016-12-09T02:44:27.323Z - silly: [TinyG2] < {"sr":{"vel":555.64,"posx":6.574,"posy":1.208,"posz":1.022,"mpox":6.574,"mpoy":1.208,"mpoz":1.022}}
2016-12-09T02:44:27.578Z - silly: [TinyG2] < {"sr":{"vel":253.73,"posx":5.298,"posy":0.229,"posz":1.815,"mpox":5.298,"mpoy":0.229,"mpoz":1.815}}
2016-12-09T02:44:27.832Z - silly: [TinyG2] < {"sr":{"vel":6.32,"posx":5.001,"posy":0.001,"posz":1.999,"mpox":5.001,"mpoy":0.001,"mpoz":1.999}}
2016-12-09T02:44:27.901Z - silly: [TinyG2] < {"qr":16,"qi":0,"qo":2}
2016-12-09T02:44:28.087Z - silly: [TinyG2] < {"sr":{"vel":70.88,"momo":2,"posx":5.044,"posy":0.030,"posz":1.971,"mpox":5.044,"mpoy":0.030,"mpoz":1.971}}
2016-12-09T02:44:28.341Z - silly: [TinyG2] < {"sr":{"vel":436.09,"posx":5.819,"posy":0.547,"posz":1.468,"mpox":5.819,"mpoy":0.547,"mpoz":1.468}}
2016-12-09T02:44:28.596Z - silly: [TinyG2] < {"sr":{"vel":558.06,"posx":7.456,"posy":1.641,"posz":0.405,"mpox":7.456,"mpoy":1.641,"mpoz":0.405}}
2016-12-09T02:44:28.850Z - silly: [TinyG2] < {"sr":{"vel":359.38,"posx":8.966,"posy":2.650,"posz":-0.575,"mpox":8.966,"mpoy":2.650,"mpoz":-0.575}}
2016-12-09T02:44:29.102Z - silly: [TinyG2] < {"sr":{"vel":68.95,"posx":9.549,"posy":3.039,"posz":-0.954,"mpox":9.549,"mpoy":3.039,"mpoz":-0.954}}
2016-12-09T02:44:29.200Z - silly: [TinyG2] < {"qr":17,"qi":0,"qo":1}
2016-12-09T02:44:29.356Z - silly: [TinyG2] < {"sr":{"vel":169.14,"posx":9.519,"posy":2.883,"posz":-1.000,"mpox":9.519,"mpoy":2.883,"mpoz":-1.000}}
2016-12-09T02:44:29.465Z - silly: [TinyG2] < {"qr":18,"qi":0,"qo":1}
2016-12-09T02:44:29.610Z - silly: [TinyG2] < {"sr":{"vel":278.90,"momo":0,"posx":8.896,"posy":2.249,"posz":-0.699,"mpox":8.896,"mpoy":2.249,"mpoz":-0.699}}
2016-12-09T02:44:29.865Z - silly: [TinyG2] < {"sr":{"vel":508.53,"posx":7.609,"posy":1.506,"posz":0.193,"mpox":7.609,"mpoy":1.506,"mpoz":0.193}}
2016-12-09T02:44:30.118Z - silly: [TinyG2] < {"sr":{"vel":452.75,"posx":6.029,"posy":0.594,"posz":1.288,"mpox":6.029,"mpoy":0.594,"mpoz":1.288}}
2016-12-09T02:44:30.373Z - silly: [TinyG2] < {"sr":{"vel":126.88,"posx":5.105,"posy":0.061,"posz":1.928,"mpox":5.105,"mpoy":0.061,"mpoz":1.928}}
2016-12-09T02:44:30.606Z - silly: [TinyG2] < {"qr":19,"qi":0,"qo":1}
2016-12-09T02:44:30.627Z - silly: [TinyG2] < {"sr":{"vel":0.04,"momo":2,"posx":5.000,"posy":0.000,"posz":2.001,"mpox":5.000,"mpoy":0.000,"mpoz":2.001}}
2016-12-09T02:44:30.881Z - silly: [TinyG2] < {"sr":{"vel":185.13,"posx":5.189,"posy":0.093,"posz":1.858,"mpox":5.189,"mpoy":0.093,"mpoz":1.858}}
2016-12-09T02:44:31.136Z - silly: [TinyG2] < {"sr":{"vel":478.99,"posx":6.321,"posy":0.651,"posz":1.002,"mpox":6.321,"mpoy":0.651,"mpoz":1.002}}
2016-12-09T02:44:31.392Z - silly: [TinyG2] < {"sr":{"vel":447.24,"posx":7.847,"posy":1.404,"posz":-0.153,"mpox":7.847,"mpoy":1.404,"mpoz":-0.153}}
2016-12-09T02:44:31.645Z - silly: [TinyG2] < {"sr":{"vel":141.86,"posx":8.780,"posy":1.864,"posz":-0.859,"mpox":8.780,"mpoy":1.864,"mpoz":-0.859}}
2016-12-09T02:44:31.837Z - silly: [TinyG2] < {"qr":20,"qi":0,"qo":1}
2016-12-09T02:44:31.898Z - silly: [TinyG2] < {"sr":{"vel":64.03,"posx":8.934,"posy":1.918,"posz":-1.000,"mpox":8.934,"mpoy":1.918,"mpoz":-1.000}}
2016-12-09T02:44:32.092Z - silly: [TinyG2] < {"qr":21,"qi":0,"qo":1}
2016-12-09T02:44:32.155Z - silly: [TinyG2] < {"sr":{"vel":252.22,"momo":0,"posx":8.323,"posy":1.428,"posz":-0.925,"mpox":8.323,"mpoy":1.428,"mpoz":-0.925}}
2016-12-09T02:44:32.407Z - silly: [TinyG2] < {"sr":{"vel":443.30,"posx":7.046,"posy":1.209,"posz":-0.476,"mpox":7.046,"mpoy":1.209,"mpoz":-0.476}}
2016-12-09T02:44:32.661Z - silly: [TinyG2] < {"sr":{"vel":639.69,"posx":4.812,"posy":0.826,"posz":0.309,"mpox":4.812,"mpoy":0.826,"mpoz":0.309}}
2016-12-09T02:44:32.914Z - silly: [TinyG2] < {"sr":{"vel":588.93,"posx":2.329,"posy":0.400,"posz":1.182,"mpox":2.329,"mpoy":0.400,"mpoz":1.182}}
2016-12-09T02:44:33.168Z - silly: [TinyG2] < {"sr":{"vel":289.74,"posx":0.550,"posy":0.094,"posz":1.808,"mpox":0.550,"mpoy":0.094,"mpoz":1.808}}
2016-12-09T02:44:33.422Z - silly: [TinyG2] < {"sr":{"vel":26.97,"posx":0.016,"posy":0.003,"posz":1.995,"mpox":0.016,"mpoy":0.003,"mpoz":1.995}}
2016-12-09T02:44:33.566Z - silly: [TinyG2] < {"qr":22,"qi":0,"qo":1}
2016-12-09T02:44:33.677Z - silly: [TinyG2] < {"sr":{"vel":11.39,"momo":2,"posx":0.005,"posy":0.001,"posz":1.999,"mpox":0.005,"mpoy":0.001,"mpoz":1.999}}
2016-12-09T02:44:33.930Z - silly: [TinyG2] < {"sr":{"vel":263.21,"posx":0.451,"posy":0.058,"posz":1.833,"mpox":0.451,"mpoy":0.058,"mpoz":1.833}}
2016-12-09T02:44:34.185Z - silly: [TinyG2] < {"sr":{"vel":589.30,"posx":2.207,"posy":0.283,"posz":1.178,"mpox":2.207,"mpoy":0.283,"mpoz":1.178}}
2016-12-09T02:44:34.439Z - silly: [TinyG2] < {"sr":{"vel":634.68,"posx":4.687,"posy":0.602,"posz":0.252,"mpox":4.687,"mpoy":0.602,"mpoz":0.252}}
2016-12-09T02:44:34.693Z - silly: [TinyG2] < {"sr":{"vel":425.98,"posx":6.871,"posy":0.883,"posz":-0.563,"mpox":6.871,"mpoy":0.883,"mpoz":-0.563}}
2016-12-09T02:44:34.947Z - silly: [TinyG2] < {"sr":{"vel":107.04,"posx":7.860,"posy":1.010,"posz":-0.931,"mpox":7.860,"mpoy":1.010,"mpoz":-0.931}}
2016-12-09T02:44:35.136Z - silly: [TinyG2] < {"qr":23,"qi":0,"qo":1}
2016-12-09T02:44:35.206Z - silly: [TinyG2] < {"sr":{"vel":57.46,"posx":8.004,"posy":1.007,"posz":-1.000,"mpox":8.004,"mpoy":1.007,"mpoz":-1.000}}
2016-12-09T02:44:35.408Z - silly: [TinyG2] < {"qr":24,"qi":0,"qo":1}
2016-12-09T02:44:35.456Z - silly: [TinyG2] < {"sr":{"vel":271.07,"posx":7.354,"posy":0.598,"mpox":7.354,"mpoy":0.598}}
2016-12-09T02:44:35.622Z - silly: [TinyG2] < {"qr":26,"qi":0,"qo":2}
2016-12-09T02:44:35.706Z - silly: [TinyG2] < {"sr":{"vel":335.92,"posx":5.835,"posy":0.079,"mpox":5.835,"mpoy":0.079}}
2016-12-09T02:44:35.959Z - silly: [TinyG2] < {"sr":{"vel":104.65,"posx":5.055,"posy":0.004,"mpox":5.055,"mpoy":0.004}}
2016-12-09T02:44:36.058Z - silly: [TinyG2] < {"qr":28,"qi":0,"qo":2}
2016-12-09T02:44:36.210Z - silly: [TinyG2] < {"sr":{"vel":0.00,"stat":3,"cycs":0,"mots":0,"momo":0,"posx":0.000,"posy":0.000,"posz":2.001,"mpox":0.000,"mpoy":0.000,"mpoz":2.001}}
2016-12-09T02:44:36.398Z - debug: [cncengine] socket.command("/dev/cu.usbmodemFD121", "stop"): id=mqe6Iz2J_8JpQRO8AAAB, args=[]     at Socket.<anonymous> (/Users/cheton/.nvm/versions/node/v4.5.0/lib/node_modules/cncjs/dist/cnc/app/index.js:8240:32)
2016-12-09T02:44:36.399Z - silly: [TinyG2] > !%
2016-12-09T02:44:36.650Z - silly: [TinyG2] > {clear:null}
2016-12-09T02:44:36.650Z - silly: [TinyG2] > {"qr":""}
2016-12-09T02:44:36.651Z - silly: [TinyG2] < {"r":{"clear":null},"f":[1,0,12]}
2016-12-09T02:44:36.652Z - silly: [TinyG2] < {"r":{"qr":28},"f":[1,0,9]}

My firmware build is 78.02. It didn't work for me when running this G-code program on Chilipeppr's TinyG workspace as well. Sometimes G2 goes straight line with random moves and unexpected result, not sure if it's related to these issues.
Hi @jarretluft,
Do you have chances to test the G-code program with TinyG2 on Chilipeppr?
G17
G21
G0 Z2.0010
G0 X0.0000 Y0.0000
G0 X5.0000 Y0.0000 Z2.0000
G1 Z-1.0000 F10000.0
G2 X0.0000 Y5.0000 I0.0000 J5.0000 F1000.0
G2 X5.0000 Y10.0000 I5.0000 J0.0000
G2 X10.0000 Y5.0000 I0.0000 J-5.0000
G2 X5.0000 Y0.0000 I-5.0000 J0.0000
G0 Z2.0000
G0 Z2.0010
G0 X0.0000 Y0.0000
M30

I made a firmware build 100.12 from G2 edge branch, and it still cannot work correctly on both cnc and Chilipeppr.

??¡ I with Chilipeppr have not problems.
Summarizing:
I have g2 firmware 100.12 too. I have cnc 1.8.7, and Chilipeppr from web.
Now i try this gcode. Its a circle with radius 10mm. I use laser gcode to see the results.
G17
G21
G0 X0.0000 Y0.0000 M03 S0
G0 X10.0000 Y0.0000 M03 S0
G1   F10000.0 M03 S50
G2 X0.0000 Y10.0000 I0.0000 J10.0000 F2000.0 M03 S50
G2 X10.0000 Y20.0000 I10.0000 J0.0000
G2 X20.0000 Y10.0000 I0.0000 J-10.0000
G2 X10.0000 Y0.0000 I-10.0000 J0.0000
S0
M05
G0 X0.0000 Y0.0000
M30

With Chilipeppr it make a fluid circle, not problem. It do a good round corners in squares too.
not problem using g2 or g3 code with I and J parameter.
With CNC 1.8.7 some software freeze: CNC, g2core?  may be can laser a quartes of circle and then stop
(danger, laser stays ON) I need to reset arduino due, reset serial json server and reconect CNC sofware.
May be a problem with serial comunication CNC<->JSON ??

Dear Cheton, dear Gendebeat,
I observed the same behavior using arcs (G2 and G3 commands), even with the latest firmware 100.15. The problem seems to be the handshake protocol of the TinyG2 controller. G2 and G3 commands may lead to a buffer overflow of the planner buffer, because these commands fill the planner buffer directly. I tried to modify the code due to the description of the communication protocol. 1) send a command to the controler 2) wait for a receive (ack -  send as r: 3) send next command. This didn't work with arc commands, because these commands fill the planner buffer with a lot more than one value and the qr response is to slow. If you send more commands when the planner buffer gets filled by the G2/G3 commands, the controler crashes (??) I tried to program around this issue, but I don't know if it is the best implementation using the status report of the controler when a motion command is finished. (sr: line: #) The commands will be checked before sending and generate line numbers. If a G2/G3 command is sent, the sender will wait until the planner buffer has some free space and motion command is completed. I tested it with some different programs and looks quite good. You will find the modified files as attachment.
Best
Axel
TinyG2.zip

@amx1 I will do some tests for your code. Thank you so much for your help.

Hi,
I am still working on the TinyG2 problem. (Even the latest version of CNC has problems with arc commands on TinyG2). The update of the firmware V100.17 for TinyG2 arduino due based hardware  changed the handshake behavoir of the controller pretty much. Alden Hart is working on it. The handshake is very close to the proposed protocol now. If you read the TinyG2 forum, he is still fighting with the input buffer overflow issue.  I changed my old approach - which tried to overcome the problems of V100.16 and earlier. As attachment you will find my actual version, which is pretty much straight forward now. Dont forget to update the controller to V100.17. It doesnt run on 100.16 or earlier. It seems to run pretty smooth and you can really take advantage of the large planner queue.
I tested the codes of the example directory, e.g. jsdc.gcode, g2-arc.gcode, ... Some examples like the helical and the 3D print codes don't work, because the commands (E... and 3D arc) are not supported by TinyG2.
Best
Axel
TinyG2_V100.17.zip

Thank you for the information. I did experience some problems with v100.15 and cannot figure out what went wrong. I will try your solution with v100.17 and keep you updated.

Dear Cheton,
i contacted the developers of TinyG2.  It is very simple now with V100.17 and „simple linemode“ it works like a charm. I attached the latest version. BTW it is not working with V100.15.
Best
Axel
Von: Rob Giseburt<EMAIL_ADDRESS>Gesendet: Sonntag, 8. Januar 2017 02:02
An: synthetos/g2<EMAIL_ADDRESS>Cc: amx1<EMAIL_ADDRESS>Comment<EMAIL_ADDRESS>Betreff: Re: [synthetos/g2] Random GCode Lines Skipped & Stopping in the middle of a Cycle. (#209)
I think we're seeing a lot over over-complication on the sender side. A lot
of this is due to historical protocols getting mixed up with the current
protocol. Let me explain.
The simplest way to send to a g2core machine is to simply send lines. You
have to honor when to OS blocks, but that's usually just a setting when you
open the buffer. We'll call this the "blasting" technique.
The main down side of that "blasting data" technique is that the buffer on
the host side becomes large and you can't send a feed hold. Also, some
hosts (small Linux machines in particular) don't handle large send buffers
well and use more processor than they should (probably in paging the buffer
to/from RAM).
So, the next simplest way to send to a g2core machine, and the simplest
with a very responsive feed hold, is to:

send a line
wait for the {r} response
go to 1 if there are more lines to send
done

We'll call this "simple line mode."
There is one problem with that technique in that some moves are extremely
brief, and may take longer to transmit than to execute. So to avoid jerky
motion in this case we "prime" the channel with a few lines. Four seems to
work well.

Count how many lines you have to send (IOW, in the gcode file, etc) and
put that in lines_in_file. Set lines_to_send=4
while (lines_to_send && lines_in_file) { send_next_line();
lines_to_send--; lines_in_file--; }
read lines from serial, counting {r}s. For each {r}, set
lines_to_send++. (Error checking and status report processing goes here.)
if lines_in_file > 0 the goto (1)
done

It's best if 1 and 2 are run asynchronously, either in different threads or
event driven, but that's not strictly necessary.
We'll call this "complete line mode."
Notice that in none of these do we care about queue reports (which are
deprecated, btw), or do we care about characters sent, etc. I did gloss
over the error handling, which is parsing the lines coming back paying
attention to the {sr:{stat:...}}, error reports in {er:{...}}, and non-zero
status in the response footer {r:..., f:[...,status,...]}. Non-zero status
is generally non-fatal and can be ignored.
As you can see, you can't "blow out" the g2core receive buffer. It'll just
back up on the host side.
We deprecated queue reports (qr) just for what amx1 was saying about how
some gcode lines (like an arc) can fill an empty queue all by themselves.
There's no clear relationship between a "line" and a "queue entry," and it
was way too hard to keep track of them.
On Sat, Jan 7, 2017 at 6:11 PM amx1<EMAIL_ADDRESS><EMAIL_ADDRESS>> wrote:

@aldenhart https://github.com/aldenhart, @giseburt
https://github.com/giseburt and all the others working on this,
it is the first time I am posting and therefore I would like to thank you
all for your amazing work first.
Maybe I can also help a bit by providing some experience I made. I have
some problems to use chilipeppr because I dont have a permanent internet
connection in my workshop and thats why I played a bit with the CNC
Software of Cheton Wu (https://github.com/cheton/cnc). The implementation
of the TinyG2 protocol in chetons code is not working without problems and
I tried to fix it.
The update to V100.17 changed the behaviour of the controller very much.
It is now working much closer to the described protocol ( "r" handshake aka
linemode), but I also saw the errors of dropping lines or even crashing the
controller especially with ARC (G2 G3) or S.. or M commands also. You will
find my modified subroutines of chetons code in the attachment. Take it as
an example for the implemented state engine. The problem for me was, that
that the controller sends an {r: ..} always after receiving the command
independent of the status of the planner queue which leads to a buffer
overflow if you dont check QR. Simple commands and linear moves only
generate one entry in the planner queue, but arcs and other commands could
generate more entries automatically. Therefore checking the qr state before
sending against a "low water mark" only is not sufficient. I implemented a
wait state for the "complex" commands which stops the sender until the
controller issues the next sr: line report. From my timing log, it seems
that the controller sends the sr: line statement when starting the
execution of the specific command. It has then enough space for the next
arc or linear move, if the planner queue is not filling (qi == 0 or qo >
qi). Up to now it is running pretty smooth with all my test cases.
The challenge for me is, that the complex buffer chain and the variable
reporting makes it very difficult to synch with the controller, if you want
to fill the planner queue as much as possible. Chilipeppr was the benchmark
for me. The "handshake" behaviour changed significantly from V100.16 to
V100.17. The attached version only runs with V100.17 only.
An ideal solution for me would be, if the controller send the {r:..} as an
acknowledge stating that it is ready to receive the next command
independent of the type of the command and the status of the planner queue,
because the simple handshake protocol: send a command wait until {r:..} and
then send next command floods all buffers and crashes the controller even
in V100.17.
All test were made with Arduino due, V100.17, a custom grbl Shield and a
CNC3040
Best
Axel
TinyG2_V100.17.zip
https://github.com/synthetos/g2/files/691778/TinyG2_V100.17.zip
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/synthetos/g2/issues/209#issuecomment-271119585, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAXj0UigGdNcJK7aZ_8HOBB2zx-WccpNks5rQCnJgaJpZM4LV7xj
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/synthetos/g2/issues/209#issuecomment-271121598 , or mute the thread https://github.com/notifications/unsubscribe-auth/AXqtvMV9gS8ledvPnx7YUIuQGd8MWb1lks5rQDWQgaJpZM4LV7xj .  https://github.com/notifications/beacon/AXqtvLv364IXGiOYF03-7iseyq8IO88Kks5rQDWQgaJpZM4LV7xj.gif

Note that I made some code changes to the controller interface in v1.8.13. If you need to merge new changes, you need to update your code with new initialization parameter and the 'load' command callback.
https://github.com/cheton/cnc/blob/83377954b77ac84cd5c8b1aad6e11f47967ceaec/src/app/controllers/TinyG2/TinyG2Controller.js#L105
this.sender = new Sender(STREAMING_PROTOCOL_SEND_RESPONSE);

https://github.com/cheton/cnc/blob/83377954b77ac84cd5c8b1aad6e11f47967ceaec/src/app/controllers/TinyG2/TinyG2Controller.js#L491
'load': () => {
    const [name, gcode, callback = noop] = args;

    const ok = this.sender.load(name, gcode);
    if (!ok) {
        callback(new Error(`Invalid G-code: name=${name}`));
        return;
    }

    log.debug(`[TinyG2] Load G-code: name="${this.sender.name}", size=${this.sender.gcode.length}, total=${this.sender.total}`);

    this.workflowState = WORKFLOW_STATE_IDLE;
    callback(null, { name: name, gcode: gcode });
},

I upgraded my Arduino Due firmware to g2core v100.19 and tested several gcode files (e.g. "jsdc.gcode") with your code, but it didn't report machine position and all stopped from the beginning.

May I have your terminal output to compare the differences? You can run cnc -vvv with verbose output and try loading jsdc.gcode as shown below:

I created a g2core branch (https://github.com/cheton/cnc/tree/g2core). You can try it with v100.17 on an Arduino due board.

Ok, we worked in parallel. There were a lot of left overs in my old
version. I cleaned it up and made an update to 1.8.13. You might try it. I
included a diff as patch input to see what I changed. The jsdc example runs
smooth.  I attached the log. I am on nvm version 7.3.0 and npm 4.0.5.
After some exchange with aldenhart the main issue was using JSON stringify
for sending gcodes. This is not forseen and leads to an error (104) and
bypasses the planner queue - but executes the commands.
There is one open point left: running codes in imperial coordinates, e.g.
the circle-inch.gcode The controller reports the work coordinates in inch
but the machine position still in mm. I did not find the position in the
cnc code where to fix it. The log is attached.
2017-01-09 10:34 GMT+01:00 Cheton Wu<EMAIL_ADDRESS>
I created a g2core branch (https://github.com/cheton/cnc/tree/g2core).
You can try it with v100.17 on an Arduino due board.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/cheton/cnc/issues/106#issuecomment-271241707, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AXqtvNvU_MvaQqekds6eCX22B2UupHIVks5rQf8sgaJpZM4LITGq
.

The machine position is always reported in MM, and in absolute (machine) coordinates with no offsets. This is to provide UIs with the ability to always display a graphic position without having to track the units more or state of the offsets (which can get complicated).

Hi amx1,
I cannot see your attachment, could you upload files again?
BTW, please let me know if you are interested in becoming a project collaborator, I can add you to the collaborator list, and you can push commits to the g2core branch for verification.

Hi Cheton,
ok,
2017-01-09 15:43 GMT+01:00 Cheton Wu<EMAIL_ADDRESS>
Hi amx1,
I cannot see your attachment, could you upload files again?
BTW, please let me know if you are interested in becoming a project
collaborator, I can add you to the collaborator list, and you can push
commits to the g2core branch for verification.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/cheton/cnc/issues/106#issuecomment-271301032, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AXqtvACZJ8jw1jWEKPIiq7bSZmXdGVDzks5rQkd4gaJpZM4LITGq
.

Invitation sent

The firmware TinyG2 for arduino due had some issues suddenly stopping the job, but the problems should be solved now. Aldenhart and Giseburt will include the changes in the next few days in the edge branch. You might test the FW branch dev-209b...  https://github.com/synthetos/g2/issues/209. I had no errors since then.
I also updated the TinyG controller part of cnc. Thank you cheton for inviting me and setting up a separate branch, but I hope it is fixed now and I will leave to you to include the changes into your dev branches. I think we don't need a separate branch for this now. There is one point left: The performance of the GUI depends very much on the  power and the OS of the machine running the browser. My setup: the controller is connected to a X86 PC running Ubuntu 16.04 64bit. The front end is a dual boot laptop dell x86 Windows 8.1 or Ubuntu 16.04 64bit, Wifi. The GUI is quite slow and the updates of the positions of the machine are stuttering on Windows 8.1 either with chrome or firefox. On linux it is very smooth in both cases even on the same Laptop! The controller is not loosing any commands or steps in all cases. It is mainly a GUI problem.
Attached you will find the updated version based on v1.8.15. It implements the simple linemode protocol with a low / high water threshold to throttle the sender in case of very fast jobs. Otherwise you might get not updates of the position. See discussions with fw developers https://github.com/synthetos/g2/issues/209
CNC_TinyG2_g2core_V1.zip

I just patched the 1.9.0 alpha version to support the latest g2core FW on Arduino Due
@cheton I had to change the init sequence, because you can't split the sr sequence in two parts. The first one will be overwritten on Arduino Due. There is a buffer limit of 254 chars, so I implemented a compression hack.
TinyG2_g2core_1.9.0.zip

Thank you for your efforts! I will make a test by tomorrow.

@amx1,
Thanks for the information. I took a look at https://github.com/synthetos/g2/issues/209 and made several tests, the result of my experiment confirms the same thing as @giseburt said in https://github.com/synthetos/g2/issues/209#issuecomment-271165948, sending {"gc":...} will bypass queue throttling and caused unexpected behaviour with G2 or G3 arcs. Change the sending routine to  send simple command string instead of {gc:...} can resolve this issue with TinyG firmware version 0.97 and 0.98. 🥂
In addition, I'll also remove the help query (?) from my init commands, as it will turn subsequent status reports to standard text output, not in JSON format. That might be the actual reason why I chose to use {"gc":...} to send g-code lines from the beginning, since sending {"gc":...} will enforce JSON reporting.
Hi @giseburt,
After upgrading to g2core firmware build 100.19 for my Arduino Due, I faced two issues as listed below:

Both work and machine positions are not updated in status reports. You can check the output, after sending initialization commands, I manually send "G0X50Y50", "G0X20Y20Z50", "{sr:null}", and "G0X0Y0" as shown below, but couldn't get updated result for machine and work positions.

{"r":{"fv":0.99,"fb":100.19,"fbs":"100.19-12-gb6995","fbc":"settings_default.h","hp":3,"hv":0,"id":"0084-7bd6-29c6-7bd","msg":"SYSTEM READY"},"f":[1,0,1]}
> {"ej":1}
{"r":{"ej":1},"f":[1,0,10]}
> {"jv":4}
{"r":{"jv":4},"f":[1,0,10]}
> {"qv":2}
{"r":{"qv":2},"f":[1,0,10]}
> {"sv":1}
{"r":{"sv":1},"f":[1,0,10]}
> {"si":250}
{"r":{"si":250},"f":[1,0,12]}
{"sr":{"stat":3}}
> {sr:{line:t,vel:t,feed:t,stat:t,cycs:t,mots:t,hold:t,momo:t,coor:t,plan:t,unit:t,dist:t,frmo:t,path:t,posx:t,posy:t,posz:t,posa:t,mpox:t,mpoy:t,mpoz:t,mpoa:t}}
{"r":{"sr":{"line":7,"vel":0,"feed":500,"stat":3,"cycs":0,"mots":0,"hold":0,"momo":0,"coor":1,"plan":0,"unit":1,"dist":0,"frmo":1,"path":2,"posx":0,"posy":0,"posz":0,"posa":0,"mpox":0,"mpoy":0,"mpoz":0,"mpoa":0}},"f":[1,0,161]}
> {"hp":null}
> {"fb":null}
{"r":{"hp":3},"f":[1,0,13]}
{"r":{"fb":100.19},"f":[1,0,13]}
> {"mt":null}
> G0X50Y50
{"r":{},"f":[1,0,9]}
{"sr":{"line":0,"stat":5,"cycs":1}}
> G0X20Y20Z50
{"r":{},"f":[1,0,12]}
> {sr:null}
{"r":{"sr":{"line":0,"vel":0,"feed":500,"stat":5,"cycs":1,"mots":0,"hold":0,"momo":0,"coor":1,"plan":0,"unit":1,"dist":0,"frmo":1,"path":2,"posx":0,"posy":0,"posz":0,"posa":0,"mpox":0,"mpoy":0,"mpoz":0,"mpoa":0}},"f":[1,0,11]}
> G0X0Y0
{"r":{},"f":[1,0,7]}

With g2core firmware build 100.19, this gcode file "circle.nc"

G0 X-5 Y0 Z0 F200
G2 X0 Y5 I5 J0 F200
G2 X5 Y0 I0 J-5
G2 X0 Y-5 I-5 J0
G2 X-5 Y0 I0 J5
G1 Z1 F500
G0 X0 Y0 Z5

reported an error of "159: Arc specification error - endpoint is starting point"
> G2 X5 Y0 I0 J-5
{"r":{"n":2},"f":[1,159,17]}

This error doesn't happen with firmware version 0.98 (firmware build: 87.01).

@cheton
I made a fast check with my modifications. I see no errors with your short example. It is running without problems. I attached the full log (-vvv) of my actual version.
I saw also the effect of the missing reports of line, feed, etc. The problem is, that  you split the configuration of the report in 2 parts from v1.8.15 to 1.9.0 during init. All switches not set to true, will be set to false by default. Therefore the first part is ignored. You can overcome this and the limitation of 254 character by compressing the commands using t instead of true. It will give you some extra space. With this option the whole configuration could be send in one line. If you take a look at my version, you can see that I put everything in one command and used a .replace after JSON.stringify. It was the fastest way to implement it. Take a look at the sent commands in the log marked as: "[TinyG2] > Init: ...."
Test.zip

I've reduced the init command as below within the limit of 254 characters, so it should not be a  problem of serial buffer overrun:
{sr:{line:t,vel:t,feed:t,stat:t,cycs:t,mots:t,hold:t,momo:t,coor:t,plan:t,unit:t,dist:t,frmo:t,path:t,posx:t,posy:t,posz:t,posa:t,mpox:t,mpoy:t,mpoz:t,mpoa:t}}

The strange thing is that my g2core firmware build 100.19 reported the error code 159 for the line 2:
> G2 X5 Y0 I0 J-5
{"r":{"n":2},"f":[1,159,17]}

I also noticed that we have one minor difference in the fbs string: "100.19-16-g80bf-dirty" vs. "100.19-12-gb6995"
May I tried using your built firmware? Either .elf or .bin is fine. Thus we can make a test on the same basis. Thank you!

@cheton
for shure, btw did checkout the Motate branch too?
Test_g2core.zip

@cheton
looking at {"r":{"n":2},"f":[1,159,17]}, the controller takes this command as line number 2  because of "n":2.  Which sender routine are you using? The original one or mine, because the original one checks only the f statement which means no error, my version waits for the correct line number. The  {"r":{"n":... This is sent when the command is processed.

I just tested your firmware build and it ran without problems. I remembered that I checkout edge branch for g2core, but forgot to checkout the edge branch for the Motate submodule. Would it be the problem?

@cheton
I used the marked branches: dev-209b-random-pauses and dev-5-usb-interrupts these are the latest modifications of aldenhart and giseburt and will be merged to edge/master in a few days. check your versions (git branch -l marked with *)
am@PC-LAB:~/workspace/git/TinyG2/g2core$ git branch -l

dev-209b-random-pauses
edge
am@PC-LAB:~/workspace/git/TinyG2/g2core$ cd ../Motate
am@PC-LAB:~/workspace/git/TinyG2/Motate$ git branch -l
dev-5-usb-interrupts
master
am@PC-LAB:~/workspace/git/TinyG2/Motate$

May I confirm how did you create your build?
My steps: cd g2core; make BOARD=gShield

I think you should also specify the CONFIG parameter, I defined my own values based on the Othermill settings. I started with:
cd g2core; make clean; make CONFIG=Othermill BOARD=gshield
You may take a look at boards.mk to find the right combination for you, because the CONFIG parameter may overwrite the BOARD definition and selects the wrong ARM platform

The CONFIG parameter works! You saved my day. Thank you so much. 👍

Perfect!
There is one thing left for jobs with imperial coordinates. The controller return the wpos, vel and feed  in inch but the machine pos always in  mmi dont know where to implement this conversion

I also noticed that yesterday, it's fixed in this commit:
https://github.com/cncjs/cncjs/commit/c64edd0b1e7223433de3c2f48433abb8d6f01109

"pos (IOW posx, posy, etc) are always in mm.
"mpo" (mpox, mpoy, etc) are in the current units.
Also, the current unit is in the "unit" value. 0 is inches (G20), and 1 is
mm (G21).
These can all be put into the SR. (Note you can't add a group to the SR, so
you have to add mpox, mpoy, not mpo.)
-Rob
On Fri, Feb 3, 2017 at 12:49 AM amx1<EMAIL_ADDRESS>wrote:

Perfect!
There is one thing left for jobs with imperial coordinates. The controller
return the wpos, vel and feed in inch but the machine pos always in mmi
dont know where to implement this conversion
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/cncjs/cncjs/issues/106#issuecomment-277177048, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAXj0ewMczRulNlxzzrscvfA2_2x7OWCks5rYs3ygaJpZM4LITGq
.

Actually, it's the other way around. pos_ is returned in in work coordinates - current units with all offsets applied, and mpo_ is returned in internal machine coordinates - absolute coordinates (G53 with no offsets) and mm units. (MPO means machine position).
The reason for this is that DROs usually need pos responses, and graphics displays usually need mpo responses.

@cheton
I will check this at the weekend

@amx1
For the GUI performance issue you mentioned earlier, I will remove the list view from the G-code widget in 1.9. Each time when receiving new state update from server, it will re-render the list view, and it will cause significant performance drop on auto scrolling.

@cheton
I checked the latest master branch, sorry to say, the G2 error is back again. I tried both parameters: complete line mode and simple line mode. see test and log files.
I think the complete linemode is not needed, because the USB connection is fast enough. The idea of CLM is to "burst" the first 4 lines to fill the planner buffer and then loop in simple linemode, but it has the risk that even the first 4 lines scramble the input buffer.
I will check the code. I think I saw the problem already.
I had a different issue with GUI performance. It is related to the update of the position of the tool in the 3D graph. It is not due to your 3D implementation. It is due the concept of the software of the controller for very fast moves. It is clear that the controller has to prioritize the different tasks. The control of the steppers has to have the absolute priority. Therefore status reports are delayed or postponed in the case of very fast jobs (series of very fast moves). The only feedback is the ACK of a received command, {r: You can see this effect even in the attached log files. You send a lot of commands without receiving a status update of pos, vel, feed, ... You will have no information to update the position of the tool in the 3D widget. The tool is "jumping" with very large steps. It doesn't influence the correct function of the controller. The cnc machine is running properly, but the display doesn't look nice.
My approach was to implement the simple linemode, with a throttling when the planner buffer is full. I stop the sender if qr =4 and restart the sender when e.g. qr=6. This will give some headroom to receive an update of pos, vel and also line. You can see the different behaviour in the log files. I attached also the log for my actual version for the same test case
Test_20170204.zip
. (test_amx1.log)

Thank you for your detailed and clear explanation.
I'm not aware the effect of delayed status reports for a very fast job. The reason why I tried not to watch queue reports since it will be deprecated as mentioned earlier by @giseburt.
G2 and G3 arcs did work when I tested yesterday, but now it happened again after I made several code changes. I think the G2 error might be caused by refactoring the workflow state (https://github.com/cncjs/cncjs/commit/877b674955a7e85e8bbf3c667ecf3fdb0223fbd3). I will take a closer look at your attached logs for further investigation.

I tried to implement my first idea. It is running, but I did not finish
checking yet.
amx1
2017-02-04 17:38 GMT+01:00 Cheton Wu<EMAIL_ADDRESS>
Thank you for your detailed and clear explanation.
I'm not aware the effect of delayed status reports for a very fast job.
The reason why I tried not to watch queue reports since it will be
deprecated as mentioned earlier by @giseburt https://github.com/giseburt
.
G2 and G3 arcs did work when I tested yesterday, but now it happened again
after I made several code changes. I think the G2 error might be caused by
refactoring the workflow state (877b674
https://github.com/cncjs/cncjs/commit/877b674955a7e85e8bbf3c667ecf3fdb0223fbd3).
I will take a closer look at your attached logs for further investigation.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/cncjs/cncjs/issues/106#issuecomment-277457925, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AXqtvCl7ZIvK9lrJvwo6XL2AUMC5CLXJks5rZKmdgaJpZM4LITGq
.

@cheton
the sending routine seems to work, but the control of the workflow: start, pause, stop, resume is still buggy, I implemented the renumbering of gc lines also, because I like the display of lines sent and compare it with lines processed. you will find errors of the gcode a bit faster
Test_2017.02.041.zip

@cheton
quick and dirty solution with line numbers and throttling for fast jobs
Test_20170204_V1.zip

@cheton
thank you for fixing the imperial coordinates for 3D view, it is ok now. There is still a difference in the axes widget. The machine position is ok, the work position is to small by the conversion factor

I changed a bit to your work, most were some name changes and refining if-else statements, you can checkout the master branch and do tests again. BTW, the work position was corrected in this commit 2bac497ecce3f187537efc11538eef0286862f7f.

I did a quick check. It looks really good now. I didn't find any critical issues yet. Thank you very much! We should thank also @aldenhart  and @giseburt for their quick responses.

@cheton
the "pulldown" options of the TinyG widget are really cool, what do think of implementing this also for the G-code widget to show the gcodes, because in some error cases it might be helpful to see the actual gcode line.

How about adding a button that opens a new window to show the gcode?

Dear cheton,
yes, perfect, this is the best idea, because it will be needed only for debugging.
BTW: it runs perfect now. I am thinking about a soft reset sequence, because in case of an gcode error, it is quite inconvinient to find the right sequence of commands to stop the workflow, sender and clear the controller.
Von: Cheton Wu<EMAIL_ADDRESS>Gesendet: Montag, 6. Februar 2017 12:32
An: cncjs/cncjs<EMAIL_ADDRESS>Cc: amx1<EMAIL_ADDRESS>Mention<EMAIL_ADDRESS>Betreff: Re: [cncjs/cncjs] G2 arc code freeze (#106)
How about adding a button that opens a new window to show the gcode?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/cncjs/cncjs/issues/106#issuecomment-277656396 , or mute the thread https://github.com/notifications/unsubscribe-auth/AXqtvFiUAG44EuBVQmPmr67V5ndXX5OVks5rZwSzgaJpZM4LITGq .  https://github.com/notifications/beacon/AXqtvGy9Thk_akTuknhoH77vmQgvQhIeks5rZwSzgaJpZM4LITGq.gif

@cheton
I checked the actual master branch, with the following errors
/home/am/workspace/git/cnc-master/showdown.js
1:1   warning  Unexpected var, use let or const instead       no-var
2:1   warning  Unexpected var, use let or const instead       no-var
3:1   warning  Unexpected var, use let or const instead       no-var
4:1   warning  Unexpected var, use let or const instead       no-var
5:1   warning  Unexpected var, use let or const instead       no-var
81:70  error    Trailing spaces not allowed                    no-trailing-spaces
88:28  error    Calls to require() should use string literals  import/no-dynamic-require

I applied https://cnc.js.org yesterday and integrated showdown into the post build process to generate a html page under /docs. It's fixed in the master branch.


Selected analysis areas are opaque, which can prevent users from seeing details

Once you click on an agglomeration, it turns bright orange, making it impossible to still see the street map/satellite imagery beneath it. This can be annoying for the user.

Two possible fixes:
-Instead of highlighting the polygon and making it opaque, highlight only the outline (maybe in white), leaving the polygon with the same transparency
-Let the user click outside of any polygon and de-select the current selection

I have only kept the outline in black - I think it looks good.

@kauebraga Does it show up looking nice on 'dark' and 'satellite' basemaps? Maybe a white outline on 'dark'? Reopening for now until you confirm.

maybe grey works for all of them?


Set default value?

Hi and thank you!
Is there any way to pass a value to the select2 when creating it which will be the default value?

Hi @yehudahkay, right now you could not  pass the default value, but in next release, I've added this possibility

Thank you! Is it released?

@yehudahkay, not now, in Release v1.16 will be added and I've updated the documentation

Wow thank you, ive seen some repositories which have releases on github and then you can subscribe to them

@yehudahkay already in Release v1.16


Livewire v3: ray($component) shows the rendered HTML

Describe the bug
In Livewire v3, outputting a component doesn't show the class info but instead renders the component's HTML and shows that in ray.
Versions
Ray version (you can see this in "About Ray"): 2.6.5
You can use composer show to get the version numbers of:

spatie/ray package version: 1.37.2
spatie/laravel-ray package version (if applicable): 1.32.6

PHP version: 8.2.9
Laravel version: 10.20.0
Livewire version: 3.0.1
To Reproduce
Steps to reproduce the behavior:

Create a new project with Livewire v3
E.g. inside a component, in a function called by a button click in the browser, put ray($this)
See the rendered HTML inside ray

Expected behavior
See the normal class overview, e.g. state of the component etc
Desktop (please complete the following information):

OS: Arch Linux

This is fixed in the underlying spatie/ray package: https://github.com/spatie/ray/releases/tag/1.37.4


Update netdb-white.txt for additional signal2noi.se pool (Asia)

@pnlambert
Please follow the steps to apply for white-list.
https://github.com/XDagger/xdag/wiki/White-List


[Pinot 1.2.0 UI] No support for "database" construct from UI

Problem statement
Pinot 1.2.0 supports database constructs as mentioned here.
However, the UI does not allow creating a table using a database or querying a table within a database.
Steps to reproduce

Launch the Pinot 1.2.0 controller UI.
On the table screen there is no option to create table within a database.
Create a table using REST API passing the database http header.
Now, try to query above table using the UI passing the database as below:

SET database='testDb';
select * from myTable limit 10;

Above query gives following error:

Database name 'testDb' from table prefix does not match database name 'default' from header

Thanks for reporting this. cc @jayeshchoudhary

Proposal to Contribute a Fix for Database Support in Pinot 1.2.0 UI
I came across this issue where in Pinot 1.2.0 the UI does not support creating or querying tables apart from the 'default' database. I propose to fix this issue by enhancing the REST API and updating the UI accordingly. (I am a backend developer with little UI experience, but the changes seem manageable.)
Problem Statement

Pinot 1.2.0 supports database constructs, but the UI does not allow creating tables within a database or querying tables within a database.
The REST API currently returns only resources attached to the default database unless explicitly specified.

Proposed Solution

Enhance REST API to support fetching resources for all databases for all relevant APIs:

Modify the REST API to support fetching all tables/schemas/etc. irrespective of the database name.
Modify the existing endpoint to accept a wildcard (*) for the database header to return resources from all databases.

Example:
curl -X GET \
  http://localhost:9000/tables \
  -H 'Accept: application/json' \
  -H 'database: *'

Update the Pinot controller UI:

Allow creating tables within any database by introducing a new, optional, input field 'Database'.
Allow creating schemas within any database by introducing a new, optional, input field 'Database'.
Modify the UI to fetch tables/schemas/etc. for all databases by setting the database to * in the UI context.

Support for db.table Queries:

Implement support for queries in the format db.table in the UI.
Ensure that the UI sets the appropriate database context before executing queries.

Additional Tasks:

Add more tasks as needed, based on deeper insights into the codebase.

PoC:
I just tried doing a PoC of this by changing the behavior of the table API to return all tables when no database is specified. See results:
Create 2 tables with same name in different DBs, here's the output.

Conclusion
By implementing these changes, we can enhance the functionality of the Pinot UI and REST API to support database constructs more effectively. This will simplify the process for users working with multiple databases within the same cluster. I am eager to contribute to this project and look forward to your feedback on this proposal.
Thank you for considering my proposal.
Best regards,
Nihal Jain

Hi @ajeydudhe, @Jackie-Jiang, @shounakmk219, @npawar  Any thoughts?

Hey @NihalJain thanks a lot for taking this up!
Few questions

What is the usecase for showing tables/schemas across all databases in one view?
Can we have a place on UI to set the database name context globally which gets passed in headers for all the relevant APIs? We already have an API that lists all the database names.

I am more inclined to a user flow where user is required to set the database context first and then proceed to do any operations, same flow that's followed by other standard SQL CLIs where we are required to set the database/catalog context before doing any operation.
This will also ensure that we are not required to prefix the table name with database name while building query on UI.
Let me know your thoughts.
Thanks!


Setup help

Hello,
I have built package in Maven and I do have .jar file with dependencies, as shown below. But I don't know where am I supposed to execute export command to set WAVES_SEED variable.

Hello! As I can see you have a wrong version of JVM, take current LTS version from here for example (Java 17).
If you run app from windows cmd you shoud set env by command set WAVES_SEED=seed before java -jar command.


TOS-328 To add Verify NLP in IF conditional step

TOS-328 To add Verify NLP in IF conditional step

@shanmukha2000 examples missing


Create the top bar

The top bar, according to the design (#19) should have the logo (can be found in src/images/logo_small.png) on the left, the burger menu to open the side bar and the social media icons on the right.

We can use http://fontawesome.io/ for that. What you think?

Sure! That should work. Just make sure the font actually only contains the required icons.


Bluetooth AVRCP browsing support (IDFGH-6757)

Is your feature request related to a problem? Please describe.
Currently, song navigation is limited to next/previous commands.
Describe the solution you'd like
When using ESP as A2DP sink / AVRCP controller, I would like to navigate and view media content on my phone using AVRCP browsing channel as described in Bluetooth AVRCP specification 1.4 and newer.

Hi @sgolob  thanks for the question. AVRCP browsing channel is listed in our current development plan. AVRCP browsing channel uses L2CAP Enhanced Retransmission Mode(ERTM in short), L2CAP ERTM is supposed to be supported in ESP-IDF 5.0, and we will consider to support of AVRCP browsing channel after that.

Any updates on this? I would also be interested in using the AVRCP Browsing Functionality on an ESP32

Hi @Craft4Cube, according to our plan, AVRCP browsing function(for AVRCP Controller role) can be delivered by the end of this year.
Currently we are developing another feature: AVRCP cover art, and it is expected to be delivered a couple of weeks later, before esp-idf v5.4 is released.


Couldn't roll back to original configs

Hi,
I'm getting this issue that no matter what I did, tmux can't roll back to original look.
I've tried these things:

rm -rf ~/.tmux
rm ~/.tmux.conf
rm ~/.tmux.conf.local
Copy back original .tmux.conf
In tmux: tmux source-file ~/.tmux.conf
brew uninstall -f tmux
brew install tmux

Yet none of these works.
Did I miss something big?

Hello @solomonxie,
I'm convinced what you're experiencing is that you didn't kill all tmux processes (client and server) before starting over.
A thing to remember. When tmux sources ~/.tmux.conf, it doesn't

reset all settings to default
apply settings from ~/.tmux.conf

So make sure you kill all tmux processes and you should be fine

Hi @gpakosz,
thanks a lot! Saved me a chunk of time :)
tmux kill-server -a works fine for me.

btw, I was about to reboot my laptop...

Are we good?

All good :)
Sent from my iPhone
On 18 Jul 2018, at 4:46 am, Gregory Pakosz <notifications<EMAIL_ADDRESS>wrote:
Are we good?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/gpakosz/.tmux/issues/169#issuecomment-405722091, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ANZCFg16tMoZqymm0CS--v5LL_-8wYCqks5uHk0zgaJpZM4VTDUq.

Alright, closing 👋


Camel-case rest endpoint path /createPrivacyGroup

Not sure what can be done immediately but this camel-case convention isn't used with the existing endpoints (at least not those that spring to mind). I have no idea why it was deemed okay to start with this endpoint.
Actions: Decide on a path convention and use that.
Additional cases: /retrievePrivacyGroup, /findPrivacyGroup

Seems the convention is there to provide support for Orion calls. To make besu compatible with tessera tessera needs to provide the same endpoints.
There might be a case for introducing a servlet filter or something to mapp the Orion paths to internal ones could be possible.

Compatibility with Besu - These paths were made similar to Orion to support legacy Orion users


[analyzer] data-driven: TransformSetErrorCode error code names

On working with some tests, there was an error:
To accept the current state, expect:
  error(TransformSetErrorCode.missing_one_of_multiple_keys, 25, 43),

However, no such field exists, as the error code doesn't match the field name, e.g.
https://github.com/dart-lang/sdk/blob/e3d6f448f21cb649a1eda4faa27866082a46ce91/pkg/analysis_server/lib/src/services/correction/fix/data_driven/transform_set_error_code.dart#L105-L106
I guess the names of the error codes in TransformSetErrorCode should be changed to camel-case e.g. missingOneOfMultipleKeys.
Or all can be upper-cased.

No, it should go the other way around. All of our diagnostic names are snake_case, and most of the constants are named the same way. These constants should be made consistent.
Either that or we should change both the error code generator and the test harness to generate lowerCamelCase identifiers from the snake_case names. (But that's too big a change to make without discussion among a larger team.)


Prometheus TLS

Hello,
Is there a way to pass cacert/cert/key via the CLI if I want to authenticate to Prometheus via TLS?
When trying to pass in https://prom-url
I get:
SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)'))))

I would like to pass in the cert and key to the cli
Thanks

hi @rohank07
Yes, you can pass the self sign certificate using an env var named CERTIFICATE
see here

Thanks @arikalon1
The cacert I am trying to already injected inside my container. It is an istio cert for mTLS. I see that the docs say the cert has to be base-encoded. Is there a way I can pass in the location of the cert (already inside the pod).

Hey @rohank07
Passing the cacert location on the container isn't supported yet.
The code that handles loading the custom certificate is here
I think supporting loading a custom cert from a file, based on a different env var is a valid suggestion.
Would you be open to contribute a PR for this?

I will take a look at this once time permits. Thanks again!


Adds healthcheck to watch and refine containers; makes refine "depend" on watch

Pre-Flight checklist

[X] Testing

[X] Code is tested
[X] Changes have been manually tested

What are the relevant tickets?
n/a
What's this PR do?
This makes a couple small changes to the docker-compose.yml file to make the services start up a bit more reliably locally:

Adds a healthcheck for the watch and refine containers that just pings the existing watchdog script. (This is required for the next bit for the watch container; added it to refine just for consistency.)
Adds a dependency on refine to watch so that they don't start up at the same time.

After doing some troubleshooting with @annagav regarding her local instance not starting properly, I noticed that the watch container and the refine container seem to conflict with each other when they're starting up. (There's some package resolution that happens in each, and since they share the same node_modules folder one has a tendency to remove stuff that the other is expecting.) The end result of this is that either the watch container or the refine container fails to start. Restarting it fixes the problem but you have to figure out and restart whichever one failed, and if you don't catch it in time, the health check in the web container will also timeout and cause it to stop. So, this makes docker-compose check to see if the watch container has started up before starting the refine one.
As an added benefit, the Docker healthcheck will restart the containers if they stop (but only a few times).
How should this be manually tested?
You shouldn't have to completely tear down your environment (docker compose down) but it won't hurt to do it for testing. In addition, the problem here seems to manifest itself more on macOS than anywhere else; I can definitely reproduce this on macOS Monterey but not so much on my WSL/Ubuntu 20 environment.

In a terminal, run this:while [ 1 ] ; do docker compose ps ; sleep 5 ; done

This will loop the ps command so it's easier to see which containers are starting when.
Start the environment using docker compose up.
Monitor the ps output. The watch container should come up first, and the refine container shouldn't move from "created" status until watch is "running (healthy)".
When the app has come up, check to make sure everything's working as per usual.

Other
The timings on the healthchecks might need some adjustment.

Thanks Jame. This PR fixed the issue I had when running locally.


chore(main): release 2.13.1

:robot: I have created a release beep boop
2.13.1 (2024-08-22)
Bug Fixes

cloud init problems with ubuntu (#1150) (c35b7d4)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/mesosphere/konvoy-image-builder/releases/tag/v2.13.1 :sunflower:


Upgrade to Go 1.13

Upgrades Go to 1.13, which affects the go.mod file and the Travis config. I also ran go mod tidy which pulled in those changes to the vendored dependencies.

Going to merge this, as it's very low risk.


Cannot import module in subfolder when connected through remote

Applies To

[X] Notebooks (.ipynb files)
[ ] Interactive Window and/or Cell Scripts (.py files with #%% markers)

What happened?
When trying to import a file located into a subdirectory of the folder where the notebook is running I get this error

I know the import is supposed to be working because:

when running the same code into a Python file everything is working as expected
it used to work until a few weeks ago, so probably something has been introduced in the new version of either the Jupyter extension or Vscode.

I've been trying to narrow down the cause of this and I'm at a lost. It seems the PATH environmental variable is correctly sourced and the correct Python version is used. I tried to select multiple times the Python interpreter, restart kernel, restart server...
Curiously enough, if I connect to the Jupyter server that VsCode is itself starting (as shown in the Output:Jupyter tab), I'm indeed able to run the code without issues (VsCode has automatically forwarded the port 8888 locally from the remote host)

This tells me that the Jupyter server spawned by VsCode is indeed correct, but the kernel that VsCode is spawning have something wrong. I tried to investigate a simple !printenv on both Jupyter running on localhost:8888 and Jupyter running in Vscode, and I've indeed seen some differences, but nothing stands out as problematic. Furthermore, I wouldn't really know how to tinker with the way VsCode starts the kernel (because I don't know where this is happening).
I can only reproduce this issue with a remote Machine running CentOS, I didn't have it on more recent machines running Ubuntu.
VS Code Version
1.90.1
Jupyter Extension Version
v2024.5.0
Jupyter logs
Visual Studio Code (1.90.1, ssh-remote, desktop)
Jupyter Extension Version: 2024.5.0.
Python Extension Version: 2024.8.1.
Pylance Extension Version: 2024.6.1.
Platform: linux (x64).
Workspace folder /var/models/wnetwork_new, Home = /home/cioni
19:07:08.576 [warn] Exception while attempting zmq : /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /home/~/.vscode-server/extensions/ms-toolsai.jupyter-2024.5.0-linux-x64/dist/node_modules/zeromq/prebuilds/linux-x64/node.napi.glibc.node)
19:07:08.577 [warn] Exception while attempting zmq (fallback) : /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /home/~/.vscode-server/extensions/ms-toolsai.jupyter-2024.5.0-linux-x64/dist/node_modules/zeromqold/prebuilds/linux-x64/node.napi.glibc.node)
19:07:09.185 [info] Attempting to start a server because of preload conditions ...
19:07:09.191 [debug] Start refreshing Kernel Picker<PHONE_NUMBER>191)
19:07:11.451 [warn] No interpreter with path ~/miniconda3/envs/stations_new/bin/python found in Python API, will convert Uri path to string as Id ~/miniconda3/envs/stations_new/bin/python
19:07:11.454 [debug] Get Custom Env Variables, Class name = Bm, completed in 2264ms, has a truthy return value, Arg 1: undefined, Arg 2: "RunPythonCode"
19:07:11.454 [debug] Jupyter Paths /kernels:
19:07:11.455 [debug] Kernel Spec Root Paths, /usr/share/jupyter/kernels, /usr/local/share/jupyter/kernels, ~/.local/share/jupyter/kernels
19:07:11.770 [debug] End refreshing Kernel Picker<PHONE_NUMBER>191)
19:07:13.532 [debug] Start refreshing Kernel Picker<PHONE_NUMBER>532)
19:07:13.639 [debug] Get Custom Env Variables, Class name = Bm, completed in 6ms, has a truthy return value, Arg 1: "/var/models/wnetwork_new", Arg 2: "RunPythonCode"
19:07:13.898 [debug] End refreshing Kernel Picker<PHONE_NUMBER>532)
19:07:13.928 [debug] KernelProvider switched kernel to id = .jvsc74a57bd0fdb8be4ba885e374374c2797cae172193a724b22d3e0bf323b23803166b5f94c./home/~/miniconda3/envs/stations_new/python./home/~/miniconda3/envs/stations_new/python.-m#ipykernel_launcher
19:07:13.929 [debug] start the kernel, options.disableUI=true for /var/models/wnetwork_new/scripts/test.ipynb
19:07:13.994 [debug] Controller selection change completed
19:07:14.000 [info] Starting Kernel (Python Path: ~/miniconda3/envs/stations_new/bin/python, Conda, 3.11.7) for '/var/models/wnetwork_new/scripts/test.ipynb' (disableUI=true)
19:07:15.291 [debug] Got env vars from Python Ext in 1658ms for ~/miniconda3/envs/stations_new/bin/python, with env var count 83.
19:07:15.293 [debug] Getting activated env variables, Class name = Pp, completed in 1660ms, has a truthy return value, Arg 1: "/var/models/wnetwork_new", Arg 2: "~/miniconda3/envs/stations_new/bin/python", Arg 3: undefined
19:07:15.309 [info] Process Execution: ~/miniconda3/envs/stations_new/bin/python -c "import jupyter;print('6af208d0-cb9c-427f-b937-ff563e17efdf')"
19:07:15.321 [info] Process Execution: ~/miniconda3/envs/stations_new/bin/python -c "import notebook;print('6af208d0-cb9c-427f-b937-ff563e17efdf')"
19:07:15.332 [info] Process Execution: ~/miniconda3/envs/stations_new/bin/python -m pip list
19:07:15.370 [debug] Checking if product is installed, Class name = ND, completed in 1817ms, has a truthy return value, Arg 1: 18, Arg 2: undefined, Return Value: true
19:07:15.378 [debug] Checking if product is installed, Class name = ND, completed in 1823ms, has a truthy return value, Arg 1: 20, Arg 2: undefined, Return Value: true
19:07:15.388 [info] Process Execution: ~/miniconda3/envs/stations_new/bin/python -m jupyter kernelspec --version
19:07:15.689 [debug] Interpreter for Pylance for Notebook URI "/var/models/wnetwork_new/scripts/test.ipynb" is ~/miniconda3/envs/stations_new/bin/python
19:07:16.010 [debug] Starting notebook server.
19:07:16.011 [debug] Launching server
19:07:16.014 [info] Starting Jupyter Server
19:07:16.017 [debug] Get Custom Env Variables, Class name = Bm, completed in 1ms, has a truthy return value, Arg 1: "/var/models/wnetwork_new", Arg 2: "RunPythonCode"
19:07:16.018 [debug] Got env vars from Python Ext in 2ms for ~/miniconda3/envs/stations_new/bin/python, with env var count 83.
19:07:16.019 [debug] Getting activated env variables, Class name = Pp, completed in 3ms, has a truthy return value, Arg 1: undefined, Arg 2: "~/miniconda3/envs/stations_new/bin/python", Arg 3: undefined
19:07:16.026 [debug] Generating custom default config at /tmp/09c5957b-dfbc-419b-9c46-0c9915c96e7f/jupyter_notebook_config.py
19:07:16.027 [debug] Starting Jupyter from ~/miniconda3/envs/stations_new/bin/python with arguments --no-browser --notebook-dir="/var/models/wnetwork_new" --config=/tmp/09c5957b-dfbc-419b-9c46-0c9915c96e7f/jupyter_notebook_config.py --NotebookApp.iopub_data_rate_limit=10000000000.0
19:07:16.046 [info] Process Execution: ~/miniconda3/envs/stations_new/bin/python -m jupyter notebook --no-browser --notebook-dir="/var/models/wnetwork_new" --config=/tmp/09c5957b-dfbc-419b-9c46-0c9915c96e7f/jupyter_notebook_config.py --NotebookApp.iopub_data_rate_limit=10000000000.0
19:07:16.047 [debug] Waiting for Jupyter Notebook
19:07:18.943 [debug] [I 2024-06-17 17:07:18.941 ServerApp] jupyter_lsp | extension was successfully linked.
19:07:18.952 [debug] [I 2024-06-17 17:07:18.951 ServerApp] jupyter_server_terminals | extension was successfully linked.
19:07:18.956 [debug] [W 2024-06-17 17:07:18.955 LabApp] 'iopub_data_rate_limit' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
19:07:18.968 [debug] [W 2024-06-17 17:07:18.967 ServerApp] ServerApp.iopub_data_rate_limit config is deprecated in 2.0. Use ZMQChannelsWebsocketConnection.iopub_data_rate_limit.
19:07:18.969 [debug] [I 2024-06-17 17:07:18.967 ServerApp] jupyterlab | extension was successfully linked.
19:07:18.979 [debug] [I 2024-06-17 17:07:18.978 ServerApp] notebook | extension was successfully linked.
19:07:19.583 [debug] [I 2024-06-17 17:07:19.582 ServerApp] notebook_shim | extension was successfully linked.
19:07:19.610 [debug] [I 2024-06-17 17:07:19.609 ServerApp] notebook_shim | extension was successfully loaded.
19:07:19.620 [debug] [I 2024-06-17 17:07:19.618 ServerApp] jupyter_lsp | extension was successfully loaded.
19:07:19.629 [debug] [I 2024-06-17 17:07:19.628 ServerApp] jupyter_server_terminals | extension was successfully loaded.
19:07:19.642 [debug] [I 2024-06-17 17:07:19.641 LabApp] JupyterLab extension loaded from /home/~/miniconda3/envs/stations_new/lib/python3.11/site-packages/jupyterlab
19:07:19.647 [debug] [I 2024-06-17 17:07:19.646 LabApp] JupyterLab application directory is /home/~/miniconda3/envs/stations_new/share/jupyter/lab
19:07:19.648 [debug] [I 2024-06-17 17:07:19.647 LabApp] Extension Manager is 'pypi'.
19:07:19.722 [debug] [I 2024-06-17 17:07:19.721 ServerApp] jupyterlab | extension was successfully loaded.
19:07:19.734 [debug] [I 2024-06-17 17:07:19.732 ServerApp] notebook | extension was successfully loaded.
19:07:19.739 [debug] [I 2024-06-17 17:07:19.738 ServerApp] Serving notebooks from local directory: /var/models/wnetwork_new
19:07:19.741 [debug] [I 2024-06-17 17:07:19.741 ServerApp] Jupyter Server 2.13.0 is running at:
19:07:19.743 [debug] [I 2024-06-17 17:07:19.742 ServerApp] http://localhost:8888/tree?token=2c0cfb94c6732ca87677ffcda15f1c6857325d69e42a9461
19:07:19.790 [info] Process Execution: ~/miniconda3/envs/stations_new/bin/python /home/~/.vscode-server/extensions/ms-toolsai.jupyter-2024.5.0-linux-x64/pythonFiles/vscode_datascience_helpers/getServerInfo.py
19:07:19.798 [debug] [I 2024-06-17 17:07:19.758 ServerApp]     http://<IP_ADDRESS>:8888/tree?token=2c0cfb94c6732ca87677ffcda15f1c6857325d69e42a9461
    [I 2024-06-17 17:07:19.758 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
    [C 2024-06-17 17:07:19.767 ServerApp]
    To access the server, open this file in a browser:
    file:///home/~/.local/share/jupyter/runtime/jpserver-22643-open.html
    Or copy and paste one of these URLs:
    http://localhost:8888/tree?token=2c0cfb94c6732ca87677ffcda15f1c6857325d69e42a9461
    http://<IP_ADDRESS>:8888/tree?token=2c0cfb94c6732ca87677ffcda15f1c6857325d69e42a9461
19:07:19.855 [info] Process Execution: ~/miniconda3/envs/stations_new/bin/python /home/~/.vscode-server/extensions/ms-toolsai.jupyter-2024.5.0-linux-x64/pythonFiles/vscode_datascience_helpers/getServerInfo.py
19:07:19.947 [debug] [I 2024-06-17 17:07:19.896 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
19:07:20.001 [info] Process Execution: ~/miniconda3/envs/stations_new/bin/python /home/~/.vscode-server/extensions/ms-toolsai.jupyter-2024.5.0-linux-x64/pythonFiles/vscode_datascience_helpers/getServerInfo.py
19:07:20.425 [debug] Handle Execution of Cells 58 for /var/models/wnetwork_new/scripts/test.ipynb
19:07:21.389 [debug] Check if a kernel .jvsc74a57bd0fdb8be4ba885e374374c2797cae172193a724b22d3e0bf323b23803166b5f94c./home/~/miniconda3/envs/stations_new/python./home/~/miniconda3/envs/stations_new/python.-m#ipykernel_launcher is usable
19:07:21.391 [info] Check & install missing Kernel dependencies for ~/miniconda3/envs/stations_new/bin/python, ui.disabled=false for resource '/var/models/wnetwork_new/scripts/test.ipynb'
19:07:21.423 [info] Process Execution: ~/miniconda3/envs/stations_new/bin/python -c "import ipykernel;print('6af208d0-cb9c-427f-b937-ff563e17efdf')"
19:07:22.131 [debug] Checking if product is installed, Class name = ND, completed in 740ms, has a truthy return value, Arg 1: 19, Arg 2: undefined, Return Value: true
19:07:22.132 [debug] Are Dependencies Installed, Class name = dd, completed in 741ms, has a truthy return value, Arg 1: "~/miniconda3/envs/stations_new/python./home/~/miniconda3/envs/stations_new/python.-m#ipykernel_launcher", Arg 2: {"a":false,"b":{"w":1,"u":{"i":-1,"end":0},"t":[{"id":1603}]}}, Arg 3: true, Return Value: true
19:07:22.133 [debug] Install Missing Dependencies, Class name = dd, completed in 743ms, has a falsy return value, Arg 1: {"resource":{"$mid":1,"fsPath":"/var/models/wnetwork_new/scripts/test.ipynb","external":"file:///var/models/wnetwork_new/scripts/test.ipynb","path":"/var/models/wnetwork_new/scripts/test.ipynb","scheme":"file"},"kernelConnection":{"id":".jvsc74a57bd0fdb8be4ba885e374374c2797cae172193a724b22d3e0bf323b23803166b5f94c./home/~/miniconda3/envs/stations_new/python./home/~/miniconda3/envs/stations_new/python.-m#ipykernel_launcher","kernelSpec":{"specFile":"/home/~/.vscode-server/extensions/ms-toolsai.jupyter-2024.5.0-linux-x64/temp/jupyter/kernels/python3117jvsc74a57bd0fdb8be4ba885e374374c2797cae172193a724b22d3e0bf323b23803166b5f94c/kernel.json","interpreterPath":"/home/~/miniconda3/envs/stations_new/bin/python","isRegisteredByVSC":"registeredByNewVersionOfExt","name":"python3117jvsc74a57bd0fdb8be4ba885e374374c2797cae172193a724b22d3e0bf323b23803166b5f94c","argv":["python","-m","ipykernel_launcher","-f","{connection_file}"],"language":"python","executable":"python","display_name":"stations_new","metadata":{"interpreter":{"path":"/home/~/miniconda3/envs/stations_new/bin/python"}},"env":{}},"interpreter":{"id":"/home/~/miniconda3/envs/stations_new/bin/python","path":"/home/~/miniconda3/envs/stations_new/bin/python"},"kind":"startUsingPythonInterpreter"},"ui":{"_event":{"w":4,"u":{"i":4,"end":4},"t":[{"id":1302},{"id":1316},{"id":1330},{"id":1331}]},"_disableUI":false},"token":{"a":false,"b":{"w":1,"u":{"i":-1,"end":0},"t":[{"id":1603}]}},"ignoreCache":true,"cannotChangeKernels":false}, Return Value: 0
19:07:22.145 [debug] Updating Kernel Environment for .jvsc74a57bd0fdb8be4ba885e374374c2797cae172193a724b22d3e0bf323b23803166b5f94c./home/~/miniconda3/envs/stations_new/python./home/~/miniconda3/envs/stations_new/python.-m#ipykernel_launcher for interpreter /home/~/miniconda3/envs/stations_new/bin/python
19:07:22.161 [debug] Python KernelSpec argv[0] updated for .jvsc74a57bd0fdb8be4ba885e374374c2797cae172193a724b22d3e0bf323b23803166b5f94c./home/~/miniconda3/envs/stations_new/python./home/~/miniconda3/envs/stations_new/python.-m#ipykernel_launcher from '/home/~/miniconda3/envs/stations_new/bin/python' to '~/miniconda3/envs/stations_new/bin/python'
19:07:22.164 [debug] Get Custom Env Variables, Class name = Bm, completed in 1ms, has a truthy return value, Arg 1: "/var/models/wnetwork_new/scripts/test.ipynb", Arg 2: "RunPythonCode"
19:07:22.166 [debug] Got env vars from Python Ext in 2ms for ~/miniconda3/envs/stations_new/bin/python, with env var count 83.
19:07:22.166 [debug] Getting activated env variables, Class name = Pp, completed in 3ms, has a truthy return value, Arg 1: "/var/models/wnetwork_new/scripts/test.ipynb", Arg 2: "~/miniconda3/envs/stations_new/bin/python", Arg 3: undefined
19:07:22.286 [debug] Updated kernel spec for .jvsc74a57bd0fdb8be4ba885e374374c2797cae172193a724b22d3e0bf323b23803166b5f94c./home/~/miniconda3/envs/stations_new/python./home/~/miniconda3/envs/stations_new/python.-m#ipykernel_launcher with environment variables for ~/.vscode-server/extensions/ms-toolsai.jupyter-2024.5.0-linux-x64/temp/jupyter/kernels/python3117jvsc74a57bd0fdb8be4ba885e374374c2797cae172193a724b22d3e0bf323b23803166b5f94c/kernel.json
19:07:22.470 [debug] [I 2024-06-17 17:07:22.469 ServerApp] Kernel started: 7e075c4c-98c4-4787-a803-29a8ca4440b6
19:07:22.499 [info] http://localhost:8888: Kernel started: 7e075c4c-98c4-4787-a803-29a8ca4440b6
19:07:24.122 [debug] [I 2024-06-17 17:07:24.110 ServerApp] Connecting to kernel 7e075c4c-98c4-4787-a803-29a8ca4440b6.
19:07:24.275 [info] Started session for kernel startUsingPythonInterpreter:.jvsc74a57bd0fdb8be4ba885e374374c2797cae172193a724b22d3e0bf323b23803166b5f94c./home/~/miniconda3/envs/stations_new/python./home/~/miniconda3/envs/stations_new/python.-m#ipykernel_launcher
19:07:24.424 [debug] Requesting Kernel info
19:07:24.432 [info] Kernel successfully started
19:07:24.436 [debug] getDataDirsImpl, Class name = jr (started execution), Arg 1: "/var/models/wnetwork_new/scripts/test.ipynb", Arg 2: "~/miniconda3/envs/stations_new/bin/python"
19:07:24.437 [debug] Get Custom Env Variables, Class name = Bm, completed in 1ms, has a truthy return value, Arg 1: undefined, Arg 2: "RunPythonCode"
19:07:24.438 [debug] Jupyter Paths :
19:07:24.453 [info] Process Execution: ~/miniconda3/envs/stations_new/bin/python /home/~/.vscode-server/extensions/ms-toolsai.jupyter-2024.5.0-linux-x64/pythonFiles/printJupyterDataDir.py
19:07:24.489 [debug] Kernel acknowledged execution of cell 58 @<PHONE_NUMBER>487
19:07:24.492 [debug] getDataDirsImpl, Class name = jr, completed in 55ms, has a truthy return value, Arg 1: "/var/models/wnetwork_new/scripts/test.ipynb", Arg 2: "~/miniconda3/envs/stations_new/bin/python", Return Value: <Uri:/home/~/.local/share/jupyter>, <Uri:/home/~/miniconda3/envs/stations_new/share/jupyter>, <Uri:/usr/local/share/jupyter>, <Uri:/usr/share/jupyter>
19:07:24.873 [warn] Cell completed with errors [hd [Error]: No module named 'lib'
	at r.execute (/home/~/.vscode-server/extensions/ms-toolsai.jupyter-2024.5.0-linux-x64/dist/extension.node.js:307:4958)
	at process.processTicksAndRejections (node:internal/process/task_queues:95:5)] {
  ename: 'ModuleNotFoundError',
  evalue: "No module named 'lib'",
  traceback: [
    '\x1B[0;31m---------------------------------------------------------------------------\x1B[0m',
    '\x1B[0;31mModuleNotFoundError\x1B[0m                       Traceback (most recent call last)',
    'Cell \x1B[0;32mIn[1], line 1\x1B[0m\n' +
      '\x1B[0;32m----> 1\x1B[0m \x1B[38;5;28;01mfrom\x1B[39;00m \x1B[38;5;21;01mlib\x1B[39;00m\x1B[38;5;21;01m.\x1B[39;00m\x1B[38;5;21;01mdata_utils\x1B[39;00m \x1B[38;5;28;01mimport\x1B[39;00m get_daily\n',
    "\x1B[0;31mModuleNotFoundError\x1B[0m: No module named 'lib'"
  ]
}

Coding Language and Runtime Version
Python 3.11
Language Extension Version (if applicable)
No response
Anaconda Version (if applicable)
No response
Running Jupyter locally or remotely?
Remote

I'm also seeing this issue on Windows 11 running VSCode connected to a dev container. I had to roll back to v2024.3.1 as both v2024.4.0 and v2024.5.0 would not let me import local files.

I'm also seeing this issue on Windows 11 running VSCode connected to a dev container. I had to roll back to v2024.3.1 as both v2024.4.0 and v2024.5.0 would not let me import local files.

Did you manage to understand what is causing it? I have other projects on the same remote machine where the local import does not fail. Plus, I don't have the issue on other machines...
I really cannot narrow it down, but I have the feeling vscode is unsetting something when starting the kernel

I'll be honest, I didn't do much more digging once I unblocked myself. I'm happy to debug if needed though.

Cannot believe ChatGPT pointed me in the right direction...by making it analyzing the two different files I indeed realized that the --notebook-dir argument passed when starting jupyter was different.

Working Log: --notebook-dir="/var/models/wnetwork_new/scripts".
Issue Log: --notebook-dir="/var/models/wnetwork_new".

It was then clear to understand how this could lead to issue as, in my case, the module I'm trying to import is located in /var/models/wnetwork_new/scripts/lib.
I added this line to the settings.json of my workspace only and now everything is working.
    "jupyter.jupyterCommandLineArguments": [
        "--notebook-dir='/var/models/wnetwork_new/scripts'"
    ],

Of course this is only a workaround.
The problem seems so stem from the fact that the notebook file root is not set to ${fileDirname} so the issue may be related to https://github.com/microsoft/vscode-jupyter/issues/8771

Thank you for filing this issue and sorry got not getting back sooner.

19:07:08.576 [warn] Exception while attempting zmq : /lib64/libstdc++.so.6: version GLIBCXX_3.4.20' not found (required by /home/~/.vscode-server/extensions/ms-toolsai.jupyter-2024.5.0-linux-x64/dist/node_modules/zeromq/prebuilds/linux-x64/node.napi.glibc.node) 19:07:08.577 [warn] Exception while attempting zmq (fallback) : /lib64/libstdc++.so.6: version GLIBCXX_3.4.20' not found (required by /home/~/.vscode-server/extensions/ms-toolsai.jupyter-2024.5.0-linux-x64/dist/node_modules/zeromqold/prebuilds/linux-x64/node.napi.glibc.node)
19:07:09.185

I understand there's an issue with the file directory. however I believe the above is part of the problem
Is it possible for you to somehow install GBLIC on your machine.

Thank you for filing this issue and sorry got not getting back sooner.

19:07:08.576 [warn] Exception while attempting zmq : /lib64/libstdc++.so.6: version GLIBCXX_3.4.20' not found (required by /home/~/.vscode-server/extensions/ms-toolsai.jupyter-2024.5.0-linux-x64/dist/node_modules/zeromq/prebuilds/linux-x64/node.napi.glibc.node) 19:07:08.577 [warn] Exception while attempting zmq (fallback) : /lib64/libstdc++.so.6: version GLIBCXX_3.4.20' not found (required by /home/~/.vscode-server/extensions/ms-toolsai.jupyter-2024.5.0-linux-x64/dist/node_modules/zeromqold/prebuilds/linux-x64/node.napi.glibc.node)
19:07:09.185

I understand there's an issue with the file directory. however I believe the above is part of the problem Is it possible for you to somehow install GBLIC on your machine.

Hi,
my previous comment shows that glibc is not part of the problem. This is related to the issues I linked.
I cannot install glibc because this is a.system running on an old centos version.

The reason I suggested installing glib is because there is a big difference in the way the kennels are started and the cwd is setup


Changes to add user scratch pad for matmul primitive to fix OOM issue in Transformer LT

Adding changes for the matmul primitive to use user scratch pad. This reduces memory footprint of the primitive. It fixes an out of memory issue when running Transformer LT with multiple instances and total thread count is large. Managing scratch pad for the primitive from the framework, fixes the out of memory issue, reduces memory footprint and does not affect performance. The changes :
Creates a new struct that hold the Tensor for scratch pad arg.
Allocates memory based on scratch pad size queried from  primitive description.
Sets user scratch pad in post ops for the primitive.

(2) scratchpad buffer is allocated with Compute(), just before invoking oneDNN execution.
The buffer size is decided by oneDNN. Please check the "struct UserScratchPad" implementation in core/util/mkl_util.h

Thank you for the quick reply! I understand that the user scratchpad is allocated in Compute(). But I don't understand how this differs from letting oneDNN allocate the buffer itself, since the matmul primitive here is not cached in LRU and is created every time Compute() is called too (and oneDNN allocates memory at primitive creation time). Both approaches seem to allocate scratchpad memory with similar lifetime, and the sizes shouldn't be too different since the user-allocated approach also gets the size from oneDNN. So I'm wondering what makes the difference here (in this PR).

@penpornk
When user scratch pad is not used by framework, oneDNN primitive adds its own scratch pad and hold it as long as the primitive is alive. TF also keeps all primitives alive till the LRU cache is full. In cases where there are many threads, each thread can create primitives for same dimensions, since the cache is thread local (this is essential). This results is many (similar) primitives created across multiple threads. For example, in the case of transformer (for which the issue was reported), the matmul dimensions varies based on input. This results in 100s of primitives getting created in each thread (when the thread count is greater than 1), each holding its scratch pad mem alive. And transformer with greater than 4 threads was going out of memory in machines with less memory (<182 GB)
With UserScratchPad, the TF framework controls the scratchpad by creating and releasing the scratchpad memory as needed : -ie (as of now) create it before execution and release it after execution of the primitive. This keeps the memory consumption low for the entire model


ssd-resnet34 on coco

current run here https://wandb.ai/mosaic-ml/ssd/runs/3p7840t1/overview?workspace=user-laura.
@Landanjs @A-Jacobson

Hey Laura nice job, sorry for the holiday delay. That said. I need a few questions answered before I can really review this.

What's the target performance for this and are we logging it anywhere? The ssd runs i see in w&b are only logging loss. @florescl

How do we deal with adapted code that doesn't fit our style guidelines? @ravi-mosaicml

It depends on how much code:

If its an entire file, and we are not modifying it at all, then it's fine to exclude it from our formatting rules by adding this to the top:

# isort: skip_file
# type: ignore
# yapf: disable

If its anything less than an entire file, or we are modifying it, then it should be brought up to date with out guidelines.
In both cases, there should be a citation (url is fine) to where the code came from.

Wow this is a lot! Like Austin said, it is very impressive you were able to get this put together. I wanted to make an initial pass over a few documents and will do another pass within the next few days.
@florescl How is the performance as of now? Depending on current performance, do you think it would be easier to import a model from another library? I'm wondering if this would be quicker and cleaner than trying to get the MLperf model to work well.

I looked at torchvision, detectron2, and mmdet when I was working on the first version of my kaggle notebook.
main blockers I noticed:

the standard batch format for coco is a list of dicts which isn't supported by composer yet.
the models are structured to take images/labels and return a dict of losses in train mode and image -> prediction in eval mode. Makes wrapping them in ComposerModel a bit tricky.

For 1, I think dictionary outputs from dataloaders is typically handled by split_batch functions. Laura has one for the current COCO dataset, so it should be modifiable for other datasets (but we can hold off on that consideration for now)
For 2, HuggingFace models also output a dictionary of logits and loss values, so there is some precedent to handle this. I would expect the detection models to be more complicated, but should still be doable. If anything, it would motivate making the training loop more flexible for arbitrary model outputs or help us define restrictions on what can be returned.

closing and reopening another PR with all the fixes and all your comments taken into account.


Support creation of lean application "runtime" images

Clubbing
https://github.com/redhat-developer/build/issues/40
https://github.com/redhat-developer/build/issues/39
Goal
As a developer, I want to separate build tools used to build the application from runtimes, so that I can create lean application images.
A Build would require three image designations in this case:

Builder image: containing the build tools related to the particular strategy and runtime (e.g. s2i-java-11)
Base runtime image: lean base image containing the runtime for the app (e.g. openjdk-slim)
Output image: the application image layered on the base runtime image (a.g. myapp)
Where possible, the build strategy should have appropriate defaults for these images but allow user to override them.

Problem
S2I images on OpenShift contain both build tools and runtime dependencies which produce bulky images. The build tools are only needed when building the application and image and not needed when the image is deployed but they add a unnecessarily overhead to the size of the image.
Why is this important?
To enable developers to build lean runtime images for their application that only contain the application and the minimal dependencies it has at runtime
Solution requirements
The solution needs to be generic so that it usable by any strategy if the user provides the information on

runtime image base ( openjdk-slim, ubi, etc ) .
while files to copy from source built image.
where to copy the files to, in the lean output image.

HI Shoubhik,
We also have the requirement, for buildpacks buildstrategy, we would like to use RedHat UBI base image to build the output image. I am not sure if this requirement can be addressed in this issue.
I think like the builder image, we should not ask end-user to set the builder image and base runtime image explicitly. But we should have the capability to choose the different base image we used for end user to build the output image.
Pls let me know if I misunderstand something.
Thanks!

Yes, a non mandatory spec.runtime section should be added.
The controller must generate a TaskRun step on-the-fly to COPY X in image A to Y in image B.

@sbose78 from the problem you are trying to solve, is the problem related to bulky images = slower builds? just to get an idea on what to expect from the solution.
Some general questions:

For Base runtime image what would be the mechanism to auto-detect the specific runtime to use? based on the source-code. Will this be a new feature?
If I understand correctly, we need to pass some layers from Builder image into Base runtime image. For this, would a multi-stage build approach would make more sense? similar to https://docs.docker.com/develop/develop-images/multistage-build/

I would like to share this proof-of-concept implementation with you, please consider. The idea is to be able to "reduce" a image footprint by copying over parts of a given image, to compose a new target-image.
Trying to illustrate this process in a single image:
Therefore, this application works as build-operator helper tool, and receives parameters as all the other applications employed by BuildStrategy steps.
In the context of build-operator, we would have the following BuildStrategy step generated by the operator, and filling up the parameters the helper application will need. Also, Build CR would need extra attributes, to be passed along to helper tool.

@qu1queee ,
(1)  The user needs to specify the base runtime image to be used in a spec.runtime block. So, yes this is a new features.
(2) Yes, this is actually a multistage build where the first stage isn't necessarily a Dockerfile build.
This is equivalent to a:
COPY --from=previously-build-image:latest /home/path/to/build/war /opt/war

It is called chained builds in Build v1
https://docs.openshift.com/container-platform/4.3/builds/advanced-build-operations.html#builds-chaining-builds_advanced-build-operations

I would like to start small & simple here:
Irrespective of which strategy is being used, here's my suggestion of what an initial PR should look like:

Our controller should generate a simple Dockerfile.runtime on-the-fly which only copies files from the 'output' into the new to-be-built image.
 FROM ubi-rhel8-runtime
 ...
 COPY --from=EXISTINGIMAGE:TAG /path/in/source/image /path/in/destination/image
 ...

Our controller should then add a TaskRun step on-the-fly to do a buildah bud or a kaniko step to build using the Dockerfile.runtime.

At the moment, we don't need copying over of labels, we could address that as part of https://github.com/redhat-developer/build/issues/50 .
I wouldn't introduce a new CLI at the beginning because it has an upstream & downstream delivery/maintenance overhead, other than it is API versioning ( alpha, beta, GA ).

I would like to start small & simple here:
Irrespective of which strategy is being used, here's my suggestion of what an initial PR should look like:
* Our controller should generate a simple `Dockerfile.runtime` on-the-fly which only copies files from the 'output' into the new to-be-built image.
  ```
   FROM ubi-rhel8-runtime
   ...
   COPY --from=EXISTINGIMAGE:TAG /path/in/source/image /path/in/destination/image
   ...
  ```

I think to explain this from a different perspective, we would like to have the controller to orchestrate how the build will be executed. And among the build steps, we would like to intervene, in order to create a "runtime" leaner image.
And I think you have a good point here. By generating a Dockerfile.runtime from the controller directly might be the shortest path to have a learner image.

* Our controller should then add a `TaskRun` step on-the-fly to do a `buildah bud` or a `kaniko` step to build using the `Dockerfile.runtime`.

For this step to work, it will be dependant on the strategy in use. With buildpacks, it can export images to a local Docker daemon, as per exporter (CNB) parameter:
$ docker run heroku/buildpacks:18 /cnb/lifecycle/exporter --help
[...]
  -daemon
    	export to docker daemon
[...]

So, exporter command could materialize the image in a local Docker instance. However, we won't have it available, as we know already. Leaving us with a question, how to expose images that are being build to buildah or kaniko?
In the current scenario, that would only be possible after uploading the image to a container registry first, and then buildah/kaniko would be able to reach it.
To note, on strategies where the image is being build directly from a Dockerfile, then the approach suggested would work. My concern now lies on how to reach all strategies.

At the moment, we don't need copying over of labels, we could address that as part of #50 .

Okay, lets use #50 to dive in the labels use-case. 👍

I wouldn't introduce a new CLI at the beginning because it has an upstream & downstream delivery/maintenance overhead in addition to API versioning costs ( alpha, beta, GA ).

We don't necessarily need to introduce a new CLI. If we decide to adopt a helper application to deal with use-cases like image format conversion, and reading cache for supported builders, we can use the same repository than the operator itself. And consequently, it would be GA at the same time than the operator.
But to assume, I might be rushing on the solution before we can explore the use-case as a whole. So, I'll keep this suggestion for a later time, if I may. :-)

So, exporter command could materialize the image in a local Docker instance. However, we won't have it available, as we know already. Leaving us with a question, how to expose images that are being build to buildah or kaniko?

I was kinda thinking that we could let the image push to the registry happen as is, the way it happens today, and then

we pull it back ( COPY --from=quay.io/sbose78/myoutputimage... )
rebuild it on top of the runtime input base
and then push it back again to the same resgistry/tag

This should make things strategy agnostic ? What do you think?

We don't necessarily need to introduce a new CLI. If we decide to adopt a helper application to deal with use-cases like image format conversion, and reading cache for supported builders, we can use the same repository than the operator itself. And consequently, it would be GA at the same time than the operator.

+1, I've given this a thought too, don't get me wrong, I'm not ruling out helper applications :) Just want to see how far we can get without introducing that. The successes and failures might lead us to design the helper app better in future if/when needed.

I was kinda thinking that we could let the image push to the registry happen as is, the way it happens today, and then
* we pull it back ( `COPY --from=quay.io/sbose78/myoutputimage...` )

* rebuild it on top of the runtime input base

* and then push it back again to the same resgistry/tag

This should make things strategy agnostic ? What do you think?

Lets give it a try. I'm going to open a draft PR with this idea :-)

following this discussion since a while. Looking forward for an initial simplistic PR as a first step.

FYI, in BuildConfig, it was expressed as
apiVersion: v1
kind: BuildConfig
metadata:
  name: image-build
spec:
  output:
    to:
      kind: ImageStreamTag
      name: image-build:latest
  source:
    dockerfile: |-
      FROM jee-runtime:latest
      COPY ROOT.war /deployments/ROOT.war
    images:
    - from:
        kind: ImageStreamTag
        name: artifact-image:latest
      paths:
      - sourcePath: /wildfly/standalone/deployments/ROOT.war
        destinationDir: "."
  strategy:
    dockerStrategy:
      from:
        kind: ImageStreamTag
        name: jee-runtime:latest
  triggers:
  - imageChange: {}
    type: ImageChange

Please consider this gist, it contains the generated Dockerfile.runtime inline, and it does work. So, we can use the container registry as a intermediary point, and upload and re-upload the build.output.image.
I'm chancing the operator API resources, and improving the generation of Dockerfile.runtime and other resources you see in the example gist, for a PR later on.

Having PR #263 merged, could we also close this issue as well?


Reduce memSeries memory usage by decoupling metadata

Metadata was added recently but doesn't seem to be used much, at least as far as I could identify.
Yet it's part of memSeries struct and so even when empty takes 48 bytes,
which is a lot given that without it memSeries requires 224 bytes.
This change turns it into a pointer on the struct, that get set only when metadata is actually set of given series.
Signed-off-by: Łukasz Mierzwa<EMAIL_ADDRESS>

@tpaschalis

Metadata seems to be mostly strings and a lot of them are likely to be repeated.
Would be great if that didn’t end up creating a lot of unique strings that later would require some smarter encoding or interning - #11133 #10991 #10231