hg_test_read_bw triggers segfault on hg_test_server with verbs;ofi_rxm

The server crashes with a segfault before any performance output is generated.  This is with libfabric 1.6 and mlx5_0 interfaces.  #231 has no impact on behavior.

In a debug build (-DCMAKE_BUILD_TYPE=Debug) I get the following stack trace in gdb when the server crashes:
#0  0x00007ffff6b30664 in rxm_ep_rma_common ()
   from /gpfs/mira-home/carns/working/src/spack/opt/spack/linux-rhel7-x86_64/gcc-7.1.0/libfabric-1.6.0-jqtjpocuo7v45k7vfslhk3pzyz3a32wp/lib/libfabric.so.1
#1  0x00007ffff6b30f51 in rxm_ep_writemsg ()
   from /gpfs/mira-home/carns/working/src/spack/opt/spack/linux-rhel7-x86_64/gcc-7.1.0/libfabric-1.6.0-jqtjpocuo7v45k7vfslhk3pzyz3a32wp/lib/libfabric.so.1
#2  0x00007ffff7395251 in fi_writemsg (ep=0x664800, msg=0x7fffe8ff8970,
    flags=268435456)
    at /home/carns/working/src/spack/opt/spack/linux-rhel7-x86_64/gcc-7.1.0/libfabric-1.6.0-jqtjpocuo7v45k7vfslhk3pzyz3a32wp/include/rdma/fi_rma.h:136
#3  0x00007ffff739bbcf in na_ofi_put (na_class=0x61f3c0, context=0x772720,
    callback=0x7ffff77bc1d1 <hg_bulk_transfer_cb>, arg=0x7fffd8000970,
    local_mem_handle=0x820a30, local_offset=0,
    remote_mem_handle=0x7fffd8000940, remote_offset=0, length=1,
    remote_addr=0x7fffe00008c0, remote_id=0 '\000', op_id=0x7fffd80009f0)
    at /home/carns/working/src/mochi/mercury/src/na/na_ofi.c:3758
#4  0x00007ffff7393816 in NA_Put (na_class=0x61f3c0, context=0x772720,
    callback=0x7ffff77bc1d1 <hg_bulk_transfer_cb>, arg=0x7fffd8000970,
    local_mem_handle=0x820a30, local_offset=0,
    remote_mem_handle=0x7fffd8000940, remote_offset=0, data_size=1,
    remote_addr=0x7fffe00008c0, remote_id=0 '\000', op_id=0x7fffd80009f0)
    at /home/carns/working/src/mochi/mercury/src/na/na.c:1661
#5  0x00007ffff77bb536 in hg_bulk_na_put (na_class=0x61f3c0, context=0x772720,
    callback=0x7ffff77bc1d1 <hg_bulk_transfer_cb>, arg=0x7fffd8000970,
    local_mem_handle=0x820a30, local_address=140737068888080, local_offset=0,
    remote_mem_handle=0x7fffd8000940, remote_address=39698688,
    remote_offset=0, data_size=1, remote_addr=0x7fffe00008c0,
    remote_id=0 '\000', op_id=0x7fffd80009f0)
    at /home/carns/working/src/mochi/mercury/src/mercury_bulk.c:252
#6  0x00007ffff77bc4be in hg_bulk_transfer_pieces (
    na_bulk_op=0x7ffff77bb4b9 <hg_bulk_na_put>, origin_addr=0x7fffe00008c0,
    origin_id=0 '\000', use_sm=0 '\000', hg_bulk_origin=0x7fffd80008c0,
    origin_segment_start_index=0, origin_segment_start_offset=0,
    hg_bulk_local=0x8209b0, local_segment_start_index=0,
    local_segment_start_offset=0, size=1, scatter_gather=0 '\000',
    hg_bulk_op_id=0x7fffd8000970, na_op_count=0x0)
    at /home/carns/working/src/mochi/mercury/src/mercury_bulk.c:784
#7  0x00007ffff77bcac7 in hg_bulk_transfer (context=0x61f1c0,
    callback=0x4071dc <hg_test_perf_bulk_transfer_cb>, arg=0x772dc0,
    op=HG_BULK_PUSH, origin_addr=0x680140, origin_id=0 '\000',
    hg_bulk_origin=0x7fffd80008c0, origin_offset=0, hg_bulk_local=0x8209b0,
    local_offset=0, size=1, op_id=0x1)
    at /home/carns/working/src/mochi/mercury/src/mercury_bulk.c:955
#8  0x00007ffff77be3f6 in HG_Bulk_transfer_id (context=0x61f1c0,
    callback=0x4071dc <hg_test_perf_bulk_transfer_cb>, arg=0x772dc0,
    op=HG_BULK_PUSH, origin_addr=0x680140, origin_id=0 '\000',
    origin_handle=0x7fffd80008c0, origin_offset=0, local_handle=0x8209b0,
    local_offset=0, size=1, op_id=0x1)
    at /home/carns/working/src/mochi/mercury/src/mercury_bulk.c:1721
#9  0x0000000000407199 in hg_test_perf_bulk_read_thread_cb (handle=0x772dc0)
    at /home/carns/working/src/mochi/mercury/Testing/mercury_rpc_cb.c:861
#10 0x0000000000407bb8 in hg_test_perf_bulk_read_thread (arg=0x772dc0)
    at /home/carns/working/src/mochi/mercury/Testing/mercury_rpc_cb.c:1046
#11 0x00007ffff7188bfa in hg_thread_pool_worker (args=0x81f5d0)
    at /home/carns/working/src/mochi/mercury/src/util/mercury_thread_pool.c:62
#12 0x00007ffff6f6fdd5 in start_thread (arg=0x7fffe8ff9700)
    at pthread_create.c:308
#13 0x00007ffff680fb3d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Updating to libfabric master (as of August 6) and compiling libfabric with -g, the top of the stack now looks like this:
#0  0x00007ffff6af37cf in fi_mr_desc (mr=0xc)
    at ../include/rdma/fi_domain.h:296
#1  0x00007ffff6af4311 in rxm_ep_rma_reg_iov (rxm_ep=0x666370,
    msg_iov=0x7fffe8ff89d0, desc=0x8080a0, desc_storage=0x7fffe8ff8820,
    iov_count=1, comp_flags=512, tx_entry=0x7fffec001ca8)
    at ../prov/rxm/src/rxm_rma.c:59
#2  0x00007ffff6af4a32 in rxm_ep_rma_common (rxm_ep=0x666370,
    msg=0x7fffe8ff8970, flags=268435456, rma_msg=0x7ffff6af3a4f <fi_writemsg>,
    comp_flags=512) at ../prov/rxm/src/rxm_rma.c:249
#3  0x00007ffff6af4fc1 in rxm_ep_writemsg (ep_fid=0x666370,
    msg=0x7fffe8ff8970, flags=268435456) at ../prov/rxm/src/rxm_rma.c:391
#4  0x00007ffff7395271 in fi_writemsg (ep=0x666370, msg=0x7fffe8ff8970,
    flags=268435456)
    at /home/carns/working/install-cooley/include/rdma/fi_rma.h:136
#5  0x00007ffff739bbef in na_ofi_put (na_class=0x61f3c0, context=0x77bd40,
    callback=0x7ffff77bc201 <hg_bulk_transfer_cb>, arg=0x7fffd8000970,
    local_mem_handle=0x808030, local_offset=0,
    remote_mem_handle=0x7fffd8000940, remote_offset=0, length=1,
    remote_addr=0x7fffe00008c0, remote_id=0 '\000', op_id=0x7fffd80009f0)
    at /home/carns/working/src/mochi/mercury/src/na/na_ofi.c:3758

We changed that in the OFI plugin to FI_DELIVERY_COMPLETE and I'm wondering if that's not the reason of the segfault. Let me try it with a standard fi_write instead.

I'm not familiar with this api, but it looks like maybe the fi_msg_rma.desc field should be a pointer to a pointer in na_ofi_put()?
I tried the patch listed below as a quick hack. When I run this, it does not segfault, but it hangs elsewhere instead.  So I don't actually know if this made any progress.
``
diff --git a/src/na/na_ofi.c b/src/na/na_ofi.c
index 2df7a0e..3ca8a4c 100644
--- a/src/na/na_ofi.c
+++ b/src/na/na_ofi.c
@@ -3693,6 +3693,7 @@ na_ofi_put(na_class_t *na_class, na_context_t *context, na_cb_t callback,
struct na_ofi_mem_handle *ofi_remote_mem_handle =
(struct na_ofi_mem_handle *) remote_mem_handle;
struct na_ofi_addr *na_ofi_addr = (struct na_ofi_addr *) remote_addr;

void *local_desc;
struct iovec local_iov = {
.iov_base = (char *)ofi_local_mem_handle->nom_base + local_offset,
.iov_len = length
@@ -3703,10 +3704,11 @@ na_ofi_put(na_class_t *na_class, na_context_t *context, na_cb_t callback,
.key = (domain->nod_mr_mode == NA_OFI_MR_SCALABLE) ? NA_OFI_RMA_KEY :
ofi_remote_mem_handle->nom_mr_key
};
local_desc = (domain->nod_mr_mode == NA_OFI_MR_SCALABLE) ? NULL :

   fi_mr_desc(ofi_local_mem_handle->nom_mr_hdl);

struct fi_msg_rma msg_rma = {
.msg_iov = &local_iov,

   .desc = (domain->nod_mr_mode == NA_OFI_MR_SCALABLE) ? NULL :

       fi_mr_desc(ofi_local_mem_handle->nom_mr_hdl),

   .desc = &local_desc,
   .iov_count = 1,
   .addr = na_ofi_with_sep(na_class) ?
       fi_rx_addr(na_ofi_addr->noa_addr, remote_id, NA_OFI_SEP_RX_CTX_BITS) :

``

If I do not apply the above patch, and instead just revert https://github.com/mercury-hpc/mercury/commit/799a33da0f694e6fe5b3199b4a3e6801bc26a164 , then the read_bw test is able to make progress without a segfault.  Something doesn't seem to be set up right in the fi_writemsg() call.  Maybe the logic that is there works correctly for transports using MR_SCALABLE, but not for endpoints that use MR_BASIC?  That would explain why we didn't see a problem with PSM2.
Any idea @liuxuezhao ?
I haven't thought about it very hard, but the FI_DELIVERY_COMPLETE might be the semantics we want, but it just isn't set up right on the verbs;ofi_rxm transport for some reason?

Looking closer at FI_DELIVERY_COMPLETE, I wonder if the problem that prompted this change could be solved with different FI_ORDER* flags?  Generally it isn't clear to me that Mercury needs remote completion notification on puts; the Mercury RPC messages usually act as implied completion notification, as long as we can make sure that they don't overtake the RMA operations.

Thanks for trying it, sorry I had not gotten to it yet. The problem is that the other flags do not have the same granularity (at least the last time I checked them) and we really just need that for puts. Right I am not really sure how we can guarantee that they don't overtake so I think that we need the DELIVERY_COMPLETE flag...

Ok.  I wouldn't be surprised if the ordering flags (even if applied to too many cases) perform better, though, simply because the FI_DELIVERY_COMPLETE semantic by definition requires waiting for an extra network message for remote notification, while the ordering could plausibly be done without remote communication in most cases.
That said it's kind of a moot point; we can just benchmark and see if push and pull performance is roughly symmetric once we have the logic around this fi_writemsg() call fixed.  If it performs fine there is no need to change.

If the fi_verbs man page is accurate, we may not have much choice for the verbs transport.  According to https://ofiwg.github.io/libfabric/v1.6.1/man/fi_verbs.7.html that provider does not support FI_DELIVERY_COMPLETE.  It does, however, support several ordering modes.
That's annoying if the failure mode for passing FI_DELIVERY_COMPLETE to fi_writemsg() on a provider that doesn't support it is for it to silently not generate completion events.
We need to gather a little more information.

Hmm ok looking at the code in src/verbs_ep.c it seems to be only supported in the case of the read?

I haven't checked the libfabric code, but I experimented with the writemsg() flags a little just now (this is all in combination with the "local_desc" patch above).  If I replace FI_DELIVERY_COMPLETE with FI_DELIVERY_COMPLETE|FI_COMPLETION then it proceeds to run the benchmark (at least until hitting a concurrency bug documented in a different issue).
I'll clean this up and submit a PR.  It would help me if we can get this working for now and then revisit what the ideal flag combination would be.

Hmm ok that's interesting... I thought that we had the FI_COMPLETION flag set globally but it may not cover the RMA operations... Thanks!

If I look at the manpage:

FI_COMPLETION  :  Indicates that a completion entry should be generated
for the specified operation.  The endpoint must be bound to  a  comple‐
tion  queue with FI_SELECTIVE_COMPLETION that corresponds to the speci‐
fied operation, or this flag is ignored.

We clearly do not have FI_SELECTIVE_COMPLETION set though...

:) yes that's also true that it should be ignored


Flight mode updates

This replaces #2250 and is the last known change required for the v1.0.0 release.

@sjwilks @AndreasAntener This fixes the yaw control on multicopters in manual flight and should improve the heading hold experience for POSCTL on fixed wing.

@sjwilks @AndreasAntener @LorenzMeier The yaw fix works in SITL ;-)

@LorenzMeier @AndreasAntener Fixed and rebased, please review!

FYI, the relevant commit is fd099b5

I'm going for a quick flight test tonight

I'm just fixing it, can test it after.

How does it look? Ready for a flight test? I have wind to test :)

@AndreasAntener I did the fix but the HIL on my home pc is lagging like crazy. Think it's because of the graphics driver but for some reason I cannot update.

Not sure about my HIL state, but I just did a test and it doesn't seem to behave ;)
When I go into posctl it banks forth and back a little like it's trying to fly a circle

@AndreasAntener Ok, thanks. I'm in the lab tomorrow where I have a working HIL setup.

Ok, you probably spot the error sooner than me :) The heading it's trying to keep is somehow off.
I tested the throttle cap yesterday on my QAV250. If I just go full throttle, the yawing is gone now. But full throttle and full pitch still results in yaw.  What happened in this case before the mixer prioritized things? I'm guessing that just capping throttle more isn't the solution.
http://dash.oznet.ch/view/CH6qJ58MAPiFpa55h3xJuN

@AndreasAntener Yep, something is wrong, let me check.

@AndreasAntener I think I have e 1e6 missing

@AndreasAntener Fixed and tested in HIL

Thanks! Works for me too in HIl. I noticed that if I switch between alt/posctl, the heading converges either towards N or S (which is nearer)
I might be able to give it a flight test today, else tomorrow.

Glancing quickly over it I can't seem to find a reason for a north / south convergence - are the angles consistent before that happens, e.g. if you're looking east and engage the lock does it fly correctly east?

Just did a HILSTAR/Xplane9 test of altctl/posctl, and I observe about 10 degrees of left turn on switching from altctl to posctl when heading is north of East/West, also saw 10 degrees of right turn with heading south of E/W line.

yaw setpoint changing with each transition from altctl to posctl

It's fascinating :) I flew it before, and I had the same "stuttering" in POSCTL as I had in HIL yesterday, which should have been fixed. So I'm flying HIL again, and noticed that this issue Mark and I are having gets also triggered if you just switch in and out of heading hold (always in POSCTL but because of roll > 0.01).
I can reproduce this with my radio when I trim it carefully a little bit to to the right. So it always rolls right just a little bit, but because the PWM signal is never perfect, it just so happens to flick over and under 0.01, which results it which @tumbili just posted before I could finish this sentence ;)

I assume the problem is that we're calculating the waypoint vector from the previous vector and shifting it around on a 2d plane.

I haven't found the issue yet, but the plot is very helpful, thanks!

please try PR #2273
I think it fixes the problem

I think it might need to be COG instead

No, for the described reason its needs to be heading. The FPV pilot expects it to lock onto the current view, not onto the current COG.

@kd0aij Do you have new plots? I'm wondering how it looks like if you fly a few meters with your changes.

still turns a little bit on toggling from altctl to posctl, and on releasing roll stick

@kd0aij I see where you're going at - because we're sideways to compensate the cross track error just flicking back and forth introduces a drift. So we can debate now if that's a logic error or the desired behaviour. To me it seems rather desired, as the user only switches out of this mode if he wants to adjust the heading. Using COG would be very confusing I think.

I agree; plain heading hold would be more intuitive. But then you can't use the waypoint navigation code. OTOH, if you want a plane with rudder to fly down runway 17 then this with COG is what you want.

Plain heading hold is trivial to implement. It's not what FPV pilots want though. They don't want to fly a heading, they want to "go there". So when they center a heading and altitude, they mean to go exactly there.

@kd0aij How does the yaw setpoint look right now? Does it still do the steps or does it lock more or less to the current heading?

the steps are gone, and I think it performs as intended now. What are the units for HDG_HOLD_SET_BACK_DIST? And perhaps increasing it would be desirable.

@tumbili @LorenzMeier Testing an easterly heading with a North wind in HIL, it looks like the plane establishes the correct COG to fly to the WP designated by releasing the roll stick. But it is difficult to establish a heading of precisely 0 degrees (North). Don't know why this should be the case, but I have to fly a heading closer to 350 to wind up heading north. The plane turns about 10 degrees to the right after I center the ailerons.

@kd0aij It would be great if you could play with it a bit further. I suspect its because of pad limits and/or magic numbers. They are used in two places:
https://github.com/PX4/Firmware/pull/2266/files#diff-1623c0ae5ae66f14c639e7c1ac03a77bR1421
and the 2nd entry 3 lines down here, again the 0.2f magic number:
https://github.com/PX4/Firmware/pull/2266/files#diff-e286114cb4359c03dcd9e90c57b99214L856
I would propose to check if things improve if you stop checking if wings are level (0.2f roll), but look for a low yaw-rate. Because what we're looking for is the point where the pilot stopped giving roll input AND the plane has a stable heading angle.
If you look at the rate live plot in QGroundControl during HIL it should be easy to find suitable yaw rate thresholds.

I see 2 issues in wind, when the pilot is already correcting with roll:

the magic numbers won't work because the pilot won't let go of the sticks
he's actually trying to hold COG

Looking for a low yaw-rate makes sense I think, but how do you flick out of that again?

Mark, how is it when it flicks on in crosswind? do you notice the change of attitude?
I also agree that heading is more intuitive for casual fpv fliers, but they might wonder why it starts to fly "different" in that case, because its starting to compensate for wind

@AndreasAntener We probably need to show the true course in the HUD then - given that HUDs currently are moving to digital that should not be a major deal

@AndreasAntener @LorenzMeier My HIL tests are on a fixed-wing platform, so it is not necessary to hold aileron to maintain a crosswind heading (possibly excepting the effects of turbulence, which I turned off). And the same is actually true for a quad (if that is what you were thinking about) being flown FPV, assuming the pilot is crabbing into the wind. But either platform flown LOS in a strong crosswind would not behave intuitively with the way FW POSCTL is currently defined.

@LorenzMeier @kd0aij @AndreasAntener So the condition for locking yaw are now:

user is off the roll stick
yawrate is below threshold

Personally I wouldn't make the threshold for the roll stick too low because the pilot may want to change hight during the flight without loosing the yaw if he accidentally applies a bit of roll.

We can increase the threshold of you like

I think testing will show

I would like to settle this on HIL and merge, if needed we can adjust after.

Rebased and merged to release branch (so normal users can't access it yet, need to update docs). Thanks everybody! Please keep testing it!


Longformer ort model raises Non-zero status code returned while running Expand node. Name:'Expand_1657' for sequence length longer than `attention_window``

System Info
optimum dev
transformers dev
python 3.9

Who can help?
@michaelbenayoun
Information

[ ] The official example scripts
[X] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[X] My own task or dataset (give details below)

Reproduction
from transformers import AutoTokenizer
import torch
import onnxruntime as ort

tokenizer = AutoTokenizer.from_pretrained("fxmarty/tiny-random-longformer")
session = ort.InferenceSession("/path/to/model.onnx", providers=["CPUExecutionProvider"])

seq_len = 520
batch_size = 1

input_ids = torch.randint(tokenizer.vocab_size, (batch_size, seq_len))
attention_mask = torch.ones((batch_size, seq_len), dtype=torch.int64)

onnx_inputs = {
    "input_ids": input_ids.cpu().detach().numpy(),
    "attention_mask": attention_mask.cpu().detach().numpy(),
}

res = session.run(None, onnx_inputs)

Error:
  File "/home/fxmarty/anaconda3/envs/hf-inf/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 200, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running Expand node. Name:'Expand_1657' Status Message: invalid expand shape

Using a shorter sequence length ( < 512 = attention_window) there is no issue. The tokenizer.model_max_length is 4096.
Note: I had to hack a bit https://github.com/huggingface/transformers/blob/94b3f544a1f5e04b78d87a2ae32a7ac252e22e31/src/transformers/models/longformer/configuration_longformer.py#L170 to remove global_attention_mask as a required input, which is optional in transformers, in order to have it work with ORTModelForFeatureExtraction.
There are a bunch of warnings during the conversion as well:
Some weights of the model checkpoint at allenai/longformer-base-4096 were not used when initializing LongformerModel: ['lm_head.dense.weight', 'lm_head.bias', 'lm_head.dense.bias', 'lm_head.decoder.weight', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight']
- This IS expected if you are initializing LongformerModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing LongformerModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
/home/fxmarty/hf_internship/transformers/src/transformers/models/longformer/modeling_longformer.py:1615: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if padding_len > 0:
/home/fxmarty/hf_internship/transformers/src/transformers/models/longformer/modeling_longformer.py:1297: TracerWarning: Converting a tensor to a Python number might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  is_global_attn = is_index_global_attn.flatten().any().item()
/home/fxmarty/hf_internship/transformers/src/transformers/models/longformer/modeling_longformer.py:565: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert (
/home/fxmarty/hf_internship/transformers/src/transformers/models/longformer/modeling_longformer.py:832: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert (
/home/fxmarty/hf_internship/transformers/src/transformers/models/longformer/modeling_longformer.py:835: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert query.size() == key.size()
/home/fxmarty/hf_internship/transformers/src/transformers/models/longformer/modeling_longformer.py:785: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if hidden_states.size(1) == window_overlap * 2:
/home/fxmarty/hf_internship/transformers/src/transformers/models/longformer/modeling_longformer.py:594: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert list(attn_scores.size()) == [
/home/fxmarty/hf_internship/transformers/src/transformers/models/longformer/modeling_longformer.py:900: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert seq_len % (window_overlap * 2) == 0
/home/fxmarty/hf_internship/transformers/src/transformers/models/longformer/modeling_longformer.py:901: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert attn_probs.size()[:3] == value.size()[:3]
/home/fxmarty/hf_internship/transformers/src/transformers/models/longformer/modeling_longformer.py:902: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert attn_probs.size(3) == 2 * window_overlap + 1
/home/fxmarty/hf_internship/transformers/src/transformers/models/longformer/modeling_longformer.py:668: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert attn_output.size() == (batch_size, seq_len, self.num_heads, self.head_dim), "Unexpected size"
/home/fxmarty/hf_internship/transformers/src/transformers/models/longformer/modeling_longformer.py:1353: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if padding_len > 0:
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.

Expected behavior
no error

python -m transformers.onnx --model=hf-internal-testing/tiny-random-LongformerModel --feature default tiny_random_longformer_tf_export failing
although goes through with allenai/longformer-base-4096

cc @deutschmn did you ever ran into this issue? It is probably not tested against in transformers.

No, I haven't come across this particular problem yet.

Thanks Patrick!
The issue comes from https://github.com/huggingface/transformers/blob/main/src/transformers/models/longformer/modeling_longformer.py#L866-L881 , when the sequence length is longer than window_attention, we have chunks_count > 0, and I expect the onnx export to silently fail due to https://pytorch.org/docs/stable/onnx.html#unsupported-tensor-indexing-patterns
@JingyaHuang @michaelbenayoun Did you ever encounter similar issue? Do you think it can be the problem?

Hey @deutschmn , just so to let you know, the issue was coming from elsewhere: https://github.com/huggingface/transformers/pull/20292
Lesson for myself from hours of debugging: read the warning messages from the onnx export, control flow is your biggest enemy :)

Fixed in https://github.com/huggingface/transformers/pull/20292

Does the latest release fix this issue. I am running into the same issue


DepCheck Error: (empty) when trying to run any model on MACOS

Environment:
RWKV-Runner_macos_universal v1.5.3
Apple M2 16GB
MacOS Ventura 13.2.1
Python 3.10.11 (installed using brew)

I've followed the instructions on Readme_Install.txt
I've set Custom Python Path to python3.10
When trying to run a model in Chat I'm always getting the message "Python dependencies are incomplete, would you like to install them?" (DepCheckError: ).
When I click in install, the dependences installs just fine (logs below), but when I try to run the model again I get again DepCheckError: (empty).
Any idea what is the issue or how I can debug this?
Logs:
Last login: Tue Nov 21 13:51:18 on ttys015
cd /Users/frandagostino/dev/AI/RWKV/RWKV-Runner/RWKV-Runner.app/Contents/MacOS/../../../ && python3.10 -m pip install -r ./backend-python/requirements_without_cyac.txt
frandagostino@MacBook-Air-de-Francisco ~ % cd /Users/frandagostino/dev/AI/RWKV/RWKV-Runner/RWKV-Runner.app/Contents/MacOS/../../../ && python3.10 -m pip install -r ./backend-python/requirements_without_cyac.txt
Requirement already satisfied: torch in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 1)) (2.0.1)
Requirement already satisfied: torchvision in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 2)) (0.15.2)
Requirement already satisfied: torchaudio in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 3)) (2.0.2)
Requirement already satisfied: rwkv==0.8.16 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 4)) (0.8.16)
Requirement already satisfied: langchain==0.0.322 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 5)) (0.0.322)
Requirement already satisfied: fastapi==0.104.0 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 6)) (0.104.0)
Requirement already satisfied: uvicorn==0.23.2 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 7)) (0.23.2)
Requirement already satisfied: sse-starlette==1.6.5 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 8)) (1.6.5)
Requirement already satisfied: pydantic==2.4.2 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 9)) (2.4.2)
Requirement already satisfied: psutil==5.9.6 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 10)) (5.9.6)
Requirement already satisfied: gputil==1.4.0 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 11)) (1.4.0)
Requirement already satisfied: tiktoken==0.5.1 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 12)) (0.5.1)
Requirement already satisfied: ftfy==6.1.1 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 13)) (6.1.1)
Requirement already satisfied: lm-dataformat==0.0.20 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 14)) (0.0.20)
Requirement already satisfied: numpy==1.24.4 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 15)) (1.24.4)
Requirement already satisfied: tokenizers==0.14.1 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 16)) (0.14.1)
Requirement already satisfied: tqdm==4.66.1 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 17)) (4.66.1)
Requirement already satisfied: midi2audio==0.1.1 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 18)) (0.1.1)
Requirement already satisfied: mido==1.3.0 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 19)) (1.3.0)
Requirement already satisfied: safetensors==0.4.0 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 20)) (0.4.0)
Requirement already satisfied: PyMuPDF==1.23.5 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 21)) (1.23.5)
Requirement already satisfied: python-multipart==0.0.6 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 22)) (0.0.6)
Requirement already satisfied: Cython==3.0.4 in /opt/homebrew/lib/python3.10/site-packages (from -r ./backend-python/requirements_without_cyac.txt (line 23)) (3.0.4)
Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in /opt/homebrew/lib/python3.10/site-packages (from langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (8.2.3)
Requirement already satisfied: anyio<4.0 in /opt/homebrew/lib/python3.10/site-packages (from langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (3.7.1)
Requirement already satisfied: dataclasses-json<0.7,>=0.5.7 in /opt/homebrew/lib/python3.10/site-packages (from langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (0.6.2)
Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /opt/homebrew/lib/python3.10/site-packages (from langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (3.8.4)
Requirement already satisfied: jsonpatch<2.0,>=1.33 in /opt/homebrew/lib/python3.10/site-packages (from langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (1.33)
Requirement already satisfied: langsmith<0.1.0,>=0.0.43 in /opt/homebrew/lib/python3.10/site-packages (from langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (0.0.66)
Requirement already satisfied: PyYAML>=5.3 in /opt/homebrew/lib/python3.10/site-packages (from langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (6.0)
Requirement already satisfied: requests<3,>=2 in /opt/homebrew/lib/python3.10/site-packages (from langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (2.31.0)
Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /opt/homebrew/lib/python3.10/site-packages (from langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (4.0.2)
Requirement already satisfied: SQLAlchemy<3,>=1.4 in /opt/homebrew/lib/python3.10/site-packages (from langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (2.0.23)
Requirement already satisfied: typing-extensions>=4.8.0 in /opt/homebrew/lib/python3.10/site-packages (from fastapi==0.104.0->-r ./backend-python/requirements_without_cyac.txt (line 6)) (4.8.0)
Requirement already satisfied: starlette<0.28.0,>=0.27.0 in /opt/homebrew/lib/python3.10/site-packages (from fastapi==0.104.0->-r ./backend-python/requirements_without_cyac.txt (line 6)) (0.27.0)
Requirement already satisfied: click>=7.0 in /opt/homebrew/lib/python3.10/site-packages (from uvicorn==0.23.2->-r ./backend-python/requirements_without_cyac.txt (line 7)) (8.1.3)
Requirement already satisfied: h11>=0.8 in /opt/homebrew/lib/python3.10/site-packages (from uvicorn==0.23.2->-r ./backend-python/requirements_without_cyac.txt (line 7)) (0.12.0)
Requirement already satisfied: pydantic-core==2.10.1 in /opt/homebrew/lib/python3.10/site-packages (from pydantic==2.4.2->-r ./backend-python/requirements_without_cyac.txt (line 9)) (2.10.1)
Requirement already satisfied: annotated-types>=0.4.0 in /opt/homebrew/lib/python3.10/site-packages (from pydantic==2.4.2->-r ./backend-python/requirements_without_cyac.txt (line 9)) (0.6.0)
Requirement already satisfied: regex>=2022.1.18 in /opt/homebrew/lib/python3.10/site-packages (from tiktoken==0.5.1->-r ./backend-python/requirements_without_cyac.txt (line 12)) (2023.6.3)
Requirement already satisfied: wcwidth>=0.2.5 in /opt/homebrew/lib/python3.10/site-packages (from ftfy==6.1.1->-r ./backend-python/requirements_without_cyac.txt (line 13)) (0.2.6)
Requirement already satisfied: jsonlines in /opt/homebrew/lib/python3.10/site-packages (from lm-dataformat==0.0.20->-r ./backend-python/requirements_without_cyac.txt (line 14)) (4.0.0)
Requirement already satisfied: ujson in /opt/homebrew/lib/python3.10/site-packages (from lm-dataformat==0.0.20->-r ./backend-python/requirements_without_cyac.txt (line 14)) (5.8.0)
Requirement already satisfied: zstandard in /opt/homebrew/lib/python3.10/site-packages (from lm-dataformat==0.0.20->-r ./backend-python/requirements_without_cyac.txt (line 14)) (0.22.0)
Requirement already satisfied: huggingface_hub<0.18,>=0.16.4 in /opt/homebrew/lib/python3.10/site-packages (from tokenizers==0.14.1->-r ./backend-python/requirements_without_cyac.txt (line 16)) (0.17.3)
Requirement already satisfied: packaging~=23.1 in /opt/homebrew/lib/python3.10/site-packages (from mido==1.3.0->-r ./backend-python/requirements_without_cyac.txt (line 19)) (23.1)
Requirement already satisfied: PyMuPDFb==1.23.5 in /opt/homebrew/lib/python3.10/site-packages (from PyMuPDF==1.23.5->-r ./backend-python/requirements_without_cyac.txt (line 21)) (1.23.5)
Requirement already satisfied: filelock in /opt/homebrew/lib/python3.10/site-packages (from torch->-r ./backend-python/requirements_without_cyac.txt (line 1)) (3.8.0)
Requirement already satisfied: networkx in /opt/homebrew/lib/python3.10/site-packages (from torch->-r ./backend-python/requirements_without_cyac.txt (line 1)) (3.1)
Requirement already satisfied: jinja2 in /opt/homebrew/lib/python3.10/site-packages (from torch->-r ./backend-python/requirements_without_cyac.txt (line 1)) (3.1.2)
Requirement already satisfied: sympy in /opt/homebrew/lib/python3.10/site-packages (from torch->-r ./backend-python/requirements_without_cyac.txt (line 1)) (1.12)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/homebrew/lib/python3.10/site-packages (from torchvision->-r ./backend-python/requirements_without_cyac.txt (line 2)) (9.5.0)
Requirement already satisfied: aiosignal>=1.1.2 in /opt/homebrew/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (1.3.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /opt/homebrew/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (6.0.4)
Requirement already satisfied: frozenlist>=1.1.1 in /opt/homebrew/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (1.3.3)
Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /opt/homebrew/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (3.1.0)
Requirement already satisfied: attrs>=17.3.0 in /opt/homebrew/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (23.1.0)
Requirement already satisfied: yarl<2.0,>=1.0 in /opt/homebrew/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (1.9.2)
Requirement already satisfied: exceptiongroup in /opt/homebrew/lib/python3.10/site-packages (from anyio<4.0->langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (1.1.1)
Requirement already satisfied: idna>=2.8 in /opt/homebrew/lib/python3.10/site-packages (from anyio<4.0->langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (3.4)
Requirement already satisfied: sniffio>=1.1 in /opt/homebrew/lib/python3.10/site-packages (from anyio<4.0->langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (1.3.0)
Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /opt/homebrew/lib/python3.10/site-packages (from dataclasses-json<0.7,>=0.5.7->langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (3.20.1)
Requirement already satisfied: typing-inspect<1,>=0.4.0 in /opt/homebrew/lib/python3.10/site-packages (from dataclasses-json<0.7,>=0.5.7->langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (0.9.0)
Requirement already satisfied: fsspec in /opt/homebrew/lib/python3.10/site-packages (from huggingface_hub<0.18,>=0.16.4->tokenizers==0.14.1->-r ./backend-python/requirements_without_cyac.txt (line 16)) (2023.5.0)
Requirement already satisfied: jsonpointer>=1.9 in /opt/homebrew/lib/python3.10/site-packages (from jsonpatch<2.0,>=1.33->langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (2.4)
Requirement already satisfied: certifi>=2017.4.17 in /opt/homebrew/lib/python3.10/site-packages (from requests<3,>=2->langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (2023.5.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/homebrew/lib/python3.10/site-packages (from requests<3,>=2->langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (1.26.16)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/homebrew/lib/python3.10/site-packages (from jinja2->torch->-r ./backend-python/requirements_without_cyac.txt (line 1)) (2.1.3)
Requirement already satisfied: mpmath>=0.19 in /opt/homebrew/lib/python3.10/site-packages (from sympy->torch->-r ./backend-python/requirements_without_cyac.txt (line 1)) (1.3.0)
Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/homebrew/lib/python3.10/site-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain==0.0.322->-r ./backend-python/requirements_without_cyac.txt (line 5)) (1.0.0)

[notice] A new release of pip is available: 23.0.1 -> 23.3.1
[notice] To update, run: python3.10 -m pip install --upgrade pip
frandagostino@MacBook-Air-de-Francisco RWKV-Runner %

Custom Python Path should be filled with the output of which python3, and the path should not contain any spaces. Please provide a screenshot of the Settings page.


Fervo example fails with fracture height/width = 320x320

Reservoir Model, 1
Reservoir Volume Option, 1
Reservoir Density, 2800, -- per the paper phyllite + quartzite + diorite + granodiorite,
Reservoir Depth, 7700 feet, -- per the paper
Reservoir Heat Capacity, 1000
Reservoir Thermal Conductivity, 2.7
Number of Fractures, 100, -- per the paper
Fracture Shape, 4, -- per the paper
Fracture Height, 320, -- per the paper
Fracture Width, 320, -- per the paper (distance between wellbores assumes a dipole flow field)
Fracture Separation, 10 meter
Number of Segments, 1
Gradient 1, 76.74, -- per the paper
Plant Outlet Pressure, 1500 psi, -- per the paper
Number of Injection Wells, 1
Number of Production Wells, 1
Well Geometry Configuration, 4
Has Nonvertical Section, True
Multilaterals Cased, True
Number of Multilateral Sections, 2, -- Two parallel horizontal sections
Total Nonvertical Length, 3250 feet, -- per the paper
Well Drilling Cost Correlation, 10, -- per the drill cost paper - works out to $400/ft
Horizontal Well Drilling Cost Correlation, 10, -- per the drill cost paper - works out to $400/ft
Production Flow Rate per Well, 41.02, -- =650 gpm per the paper - per the paper the maximum flow rate was 63 L/s but the range was 550-750 gpm
Production Well Diameter, 7, -- per the paper
Injection Well Diameter, 7, -- per the paper
Well Separation, 365 feet, -- per the paper
Injection Temperature, 38 degC, -- per the paper 75 to 125 degF
Injection Wellbore Temperature Gain, 3
Reservoir Impedance, 0.33, -- per paper, matching pumping power report 500-1000 kW
Ramey Production Wellbore Model, True
Utilization Factor, .9
Water Loss Fraction, 0.1
Maximum Drawdown, 1, -- no redrilling considered
Ambient Temperature, 8 degC, -- per the paper. Published ambient is 10.27 degC
Surface Temperature, 8 degC, -- per the paper. Published ambient is 10.27 degC
End-Use Option, 1, -- electricity as the main product
Plant Lifetime, 10
Power Plant Type, 1, -- subcritical binary cycle
End-Use Efficiency Factor, .9
Circulation Pump Efficiency, 0.80, -- per the paper
Economic Model, 3, -- BICYCLE Levelized Cost Model
Starting Electricity Sale Price, 0.15
Ending Electricity Sale Price, 1.00, -- essentially no limit as the rate can rise continuously with inflation
Electricity Escalation Rate Per Year, 0.004053223, -- End up being about 2% inflation
Electricity Escalation Start Year, 1
Fraction of Investment in Bonds, .5, -- Required for BICYCLE model
Combined Income Tax Rate, .3, -- Required for BICYCLE model
Gross Revenue Tax Rate, 0, -- Required for BICYCLE model
Inflated Bond Interest Rate, .05, -- Required for BICYCLE model
Inflated Equity Interest Rate, .08, -- Required for BICYCLE model
Inflation Rate, .02, -- Required for BICYCLE model
Investment Tax Credit Rate, 0, -- Required for BICYCLE model
Inflation Rate During Construction, 0.05
Property Tax Rate, 0, -- Required for BICYCLE model
Print Output to Console, 1
Time steps per year, 10

Gives error:
⚠GEOPHIRES encountered an exception: Input temperature & pressure (-5.428032407456756e+24, 23.015815283999995 megapascal) are out of range or otherwise could not be used to calculate

I think we will have to wait for our fracture model expert, @kfbeckers to weigh in, but to reduce his effort, I did some sensitivity testing and determined that if the Fracture Height&Fracture Width=266, everything behaves normally. If Fracture Height&Fracture Width=267 or 268, the code misbehaves (produces some irrational numbers but does not fail), and if Fracture Height&Fracture Width>268, if fails as described.

The issue was that the Inverse Laplace method in the Gringarten model became unstable for large fracture areas. Submitted a pull request to fix this issue: https://github.com/NREL/GEOPHIRES-X/pull/266


'relative path' field for uploaded images

Hi,
I am starting to use Netlify CMS with Gatsby 1.0.
I need a relative path to fetch pictures with the graphQL layer. So in my frontmatter I would like to have a field like:
fieldName: ../images/myImage.jpg
Of course the last part ("myImage.jpg") would be different for each entry depending on the file I upload.
Is there a way to auto generate a string field in the frontmatter that would concatenate a string of my choice (for example '../images/') with the name of a picture I may upload in an image field of the same entry ?
I also tried to set:
public_folder: "../images"
This would theoretically change all my image fields in all collections to get that leading string in the file path but when I create a new file with the CMS I get an auto opening '/'... So in my frontmatter:
image: /../images/myImage.jpg
instead of
image: ../images/myImage.jpg
Of course this breaks my groove as well.
What is the best way to get around this issue?

@MarcCoet a simple, non-breaking approach would be to accept a public_folder_relative setting in the config - set it to true to avoid the prepended slash. If you're up for giving this a shot, implementation would probably look like this:

Update the resolvePath helper to accept a third option allowRelative.
Update the two places in that function where slashes are prepended to only do so if the allowRelative param is not truthy.
Update the assetProxy function to pass in the new config value as a third param to resolvePath.

This issue should be at least helped by #350.

@MarcCoet FWIW a quick workaround is to just edit the markdown of your post to the correct relative path from within netlify cms (ie, just toggle the 'Markdown' switch and edit the path)
It'll break the preview, but so far it appears to push fine. here's my config:
...
media_folder: content/blog-posts/media
public_folder: ../media/
collections:
  - name: blog
    label: Blog
    folder: content/blog-posts/posts
...

I still think this use case should be supported, but it merits further discussion.

Hey! 👋  I took a first quick shot at what @erquhart outlined in https://github.com/netlify/netlify-cms/issues/325#issuecomment-290162337 over the weekend.
Previews don't work yet (haven't looked into the code much) but this does enough to let a friend of mine that I'm helping with his portfolio site author his stuff – relative paths are correctly inserted in Markdown. My context is exactly like discussed in #843 (using Gatsby).
I would love to provide a PR for this eventually but most probably will need some guidance.

@fk that's awesome!
In the time since I wrote that comment, I've come to avoid "switches" that affect other settings in the config. A better approach that's still non-breaking would be to just check if the path begins with one or two dots and a slash and don't prepend a slash if so. We'd want to test it out a bit for unexpected consequences, though. Are you up for giving that a shot?

Hey @erquhart! Yes! Up for anything! ;-) :-D
I also have been thinking a little about how to approach this feature in the meantime. I didn't have much time, so I only came as far as to look into assetProxy in regards on how to restore media previews in the editor and preview pane.
I also started to wonder if approaching this change from what I believe is a quite specific, "Gatsby" point of view, is the right thing to do to actually land a PR that is beneficial to anyone else – in other words: "What are other use cases for a relative public path (which couldn't be resolved by just changing the location of the media folder)?" is a question I figured I should ask here before going any further.
A quick recap of my specific use case: For Gatsby to be able to process URLs in Markdown (process images, copy linked files), it requires them to be relative (I still have to check back with @KyleAMathews on that, but I think "it's a Webpack thing") – basically what surfaced in #843.
My initial approach was to add the public_folder_relative option as you suggested (and then somehow make previews work with that), and to put all my markdown content at the same "distance" to the media folder – e.g. if my public_folder_relative was ../../img, all my Markdown content would have to live two folders down from img per convention.
Ideally I thought netlify-cms could resolve the path to be inserted in Markdown relative to the Markdown source and the location of the media folder. But somehow I feel like I'm missing something obvious still…

I've come to avoid "switches" that affect other settings in the config. A better approach that's still non-breaking would be to just check if the path begins with one or two dots and a slash and don't prepend a slash if so.

Yes, totally can follow your thoughts. FWIW, I've been seeing sindresorhus/is-relative-url being used in Gatsby.

We'd want to test it out a bit for unexpected consequences, though. Are you up for giving that a shot?

Yes! How about I adjust my fork to look at the path/remove the new public_folder_relative option and report back?

@fk apologies for the delay - that would be awesome. The problem and solution can really approached quite abstractly: for myriad reasons, folks sometimes want to set their media folder via relative path. I can't think of other specific use cases off the top of my head.

Hugo is going to be supporting image processing very soon, and if I understand correctly, it will only work with relatively placed images.

@erquhart I don't quite understand how the media library will handle relatively placed media. Will it just walk through all the directories to find files that match the media_folder string (via regex?)?
Also, what do we do if an image is added to one entry, and then it is added to a different post via the media library? Do we just add the path relatively to the old location, or would we copy the actual image to the second location as well?

I found another workaround. Maybe it will be useful for somebody else. The path added to the markdown files by Netlify CMS can be adjusted and set (as Gatsby requires) to the relative path from the particular node (markdown file) the image is used in.  Adding the following code to gatsby-node.js did the trick for me:
exports.onCreateNode = ({
  node,
  getNode,
  loadNodeContent,
  boundActionCreators,
}) => {
  const { frontmatter } = node
  if (frontmatter) {
    const { image } = frontmatter
    if (image) {
      if (image.indexOf('/img') === 0) {
        frontmatter.image = path.relative(
          path.dirname(node.fileAbsolutePath),
          path.join(__dirname, '/static/', image)
        )
      }
    }
  }
}

This assumes that config.yml has the media file paths set as follows:
media_folder: "static/img"
public_folder: "/img"

You shouldn't mutate node data. The data should be treated as immutable (I'm actually surprised that worked as I thought we were already passing a copy of data to APIs to prevent exactly this sort of thing. Instead use https://www.gatsbyjs.org/docs/bound-action-creators/#createNodeField

Ok, makes sense. I actually tried to use the spread operator to avoid mutation, but somehow it didn't work. Will try with createNodeField

how would the graphQL query look like with createNodeField?
This works:

markdownRemark{
     frontmatter {
        image
      }
      fields {
        imageRel
      }
}

But nicer would be:
markdownRemark{
     frontmatter {
        image {
          relativePath
            // and then here the childImageSharp stuff
        }
      }
}

which fails with: GraphQL Error Field "image" must not have a selection since type "String" has no subfields.

If you want to link to a file node, you need to add ___NODE to the end of the field name so image___NODE

I'm trying to get gatsby-image working with netlify cms and I gave this a try.  I used @ilyabo's code above and modified it with the createNodeField.  I tried to access it in my query with imageLocalFile___NODE but I got this error: GraphQL Error Unknown field imageLocalFile___NODEon typefields``

Does anyone know how to use NetlifyCMS to manage media images stored as "Page Resources" (relative to page collections they are used on in Page Bundles)?
This is needed to use Hugo's new native image processing.

@01ivr3 It actually is possible to use Hugo's image processing with a single media folder, see https://stackoverflow.com/questions/48213883/image-processing-outside-bundles.

@tech4him1 very cool! Thank you for the heads up.
Do I have it right that this method can be made to work with NetlifyCMS' media management as long as the images are added to pages using the image widget? Meaning, it will break the ability to use add images using the Media Library from within the markdown widget's wysiwyg editor?

@01ivr3 you can use my method if you are not worried about page bundling your assets in the same paths as your pages. You would have to store all original images in one media folder location. The issue is the location of the preview when you add the image into the markdown because there is not a way to translate back the image to the preview correctly from Hugo. We would have to write a custom widget to be able to know to use the original image path for preview and let Hugo manage the path on build. Hope that makes sense.
NetlifyCMS does not use multiple locations for media at this time, but will be addressing in the future.

@talves & @tech4him1 thank you both your help.
I've got a site I'm working on setup to use Hugo's built-in image resizing with images stored to a single folder, as per @talves StackOverflow examples. But I am a bit confused on the optimum way to configure this with NetlifyCMS.
If I stored images that I want to be both CMS manageable and accessible to Hugo's image resizing to a single /site/content/media folder, I assume the following parameters should be set in NetlifyCMS' config.yml.
backend:
  name: git-gateway
media_folder: "site/content/media"
public_folder: "media"

I believe, the Media Library in the cms loads images directly from the media_folder stored in the GitHub project repo.
On selecting an image using the image widget, filenames are prefixed with public_folder, (in my case /media/), and the widget's preview image is set to the public folder url (I think)
This works fine for a basic src link e.g.
But with how I've setup image resizing in my page templates, I believe having the public_folder auto-prefixed to image values is an unwanted.
For example, if I have the following custom param set in a pages front matter nav_image: SOME_IMAGE.jpg
And the following template chunk
{{ $media := .Site.GetPage "section" "media" }}
{{ $currentPage := . }}
{{ range .Site.Menus.main -}}
  {{- if .Page.Params.nav_image -}}

  {{ end -}}
{{ end }}

I will receive a can't evaluate field Resize in type resource.Resource error once the cms prefixes media_folder to the SOME_IMAGE.jpg value.
I realize I could strip off the auto-prefixed folder value with something like {{ strings.TrimLeft .Page.nav_image "/media/" }} but I have yet to test this.
When first started rambling this post, I was thinking config.yml should be set as follows
media_folder: "site/content/media"
public_folder: ""

But this I assume would break the image widget preview images.
Any help greatly appreciated.
This is my first Go with Hugo's templating and NetlifyCMS, so good chance I've done something foolish. 🤓

Netlify CMS prefixes the public_folder so the resulting can be used as the image src. For your case, where would the live CMS find your published images to show in the preview pane?

I get that. In my case, the live CMS should find the published images to show in the preview pane under /media/. That said, I am not outputting these values directly to an img src. This is because the value is being used with Hugo's built in image resizing via @talves workaround to use store images to a single folder.
In my above example, the value of the media folder has already being passed to the image resize function as follows {{ (.Site.GetPage "section" "media").Resources.GetMatch .Page.nav_image }}.
In this case, I believe also prefixing media_folder to the image filename is an undesirable behaviour. Hugo will throw an error if the auto-prefixed value is not first stripped out before it is used in an image processing function.

@01ivr3 since the solution I proposed in that SO, there was a change, because my example is not working the way it was first setup in 0.32 of Hugo.  You will have to run a build and see how the folder layout will be published for the correct paths, but it could be this solution will not be viable in the new versions of Hugo.

@talves do you know what version of Hugo breaks the folder structure?
I'm currently using hugo-bin ^0.22.0 as part of Netlify's victor-hugo starter. I believe this use Hugo v0.35. Images stored under site/content/media still appear to be copied to /media when building site.

Sorry, I am not sure.

Is it possible to hide files from NetlifyCMS' Media Library?
If I store images in a single /content folder to be able to use Hugo's built in image processing, NetlifyCMS will show the required _index.md file in this directory, and give the end user the option to delete this file. How can I set NetlifyCMS to not show this file in its Media Library UI??

@fk @KyleAMathews Hey all, just wanted to let you know I created a simple gatsby plugin to help gatsby-remark-images match the relative path from markdown images without needing to modify netlify-cms. You just add this plugin before gatsby-remark-images in your gatsby-config.js. Check it out here: https://www.npmjs.com/package/gatsby-remark-relative-images
Hopefully, this won't be needed once NetlifyCMS supports relative paths.

@erquhart @talves Is it worth adding @danielmahon's plugin to gatsby-plugin-netlify-cms in some way?

@tech4him1 Not sure if that would work as it is. The plugin basically has to "intercept" the node returned from gatsby-transformer-remark before it gets to gatsby-remark-images.
It bascially just sets the node.url to path.relative(node.parent.dir, file.url).
So, given a node like /src/pages/blog/my-first-post.md, with an image src of /uploads/fat-racoon.jpg, and the referenced image in /static/uploads it changes the node.url to .../.../../static/uploads/fat-racoon.jpg. Which lets gatsby-remark-images resolve it correctly when it does path.join(node.parent.dir, node.url).
You can "maybe" see it in action here:
https://github.com/danielmahon/gatsby-starter-procyon/tree/netlifycms.
I say maybe because I'm still working on that starter, expecially the netlifycms branch.
So get ready to step in some :poop:.

@danielmahon tried your package but nothing appeared to happen. Upon debugging it, for whatever reason the select(markdownAST, "image") and select(markdownAST, "html") return nothing, despite there being plenty of html content. Wondering if you or anyone had any ideas why that might be? For what it's worth, my config is pretty much set up the same as your example.

@brendan-hurley Hmm, not sure. markdownAST is passed to the plugin array via gatsby-transformer-remark. Those calls are mirroring how gatsby-remark-images grabs the images and html nodes so as long as you have the plugin in the correct place (right before gatsby-remark-images) then it should work . Double check your config. gatsby-remark-relative-images should be inside the plugins array of gatsby-transformer-remark and before gatsby-remark-images. If markdownAST isnt returning anything then something else is going on upstream with the remark parsing.
I've tested this with md ![](/path/to/fat-racoon.jpg), and html . Also works with nested img tags.
...
    {
      resolve: 'gatsby-transformer-remark',
      options: {
        plugins: [
          {
            resolve: 'gatsby-remark-relative-images',
            options: {
              name: 'uploads',
            },
          },
          {
            resolve: 'gatsby-remark-images',
            options: {
              // It's important to specify the maxWidth (in pixels) of
              // the content container as this plugin uses this as the
              // base for generating different widths of each image.
              maxWidth: 720,
            },
          },
        ],
      },
    },
...

@danielmahon Aye I have it set up as such:
plugins: [
    'gatsby-plugin-react-next',
    'gatsby-plugin-react-helmet',
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        path: `${__dirname}/static/content`,
        name: 'content',
      },
    },
    {
      resolve: 'gatsby-source-filesystem',
      options: {
        path: `${__dirname}/static/assets`,
        name: 'assets',
      },
    },
    'gatsby-transformer-sharp',
    {
      resolve: 'gatsby-transformer-remark',
      options: {
        plugins: [
          {
            resolve: `gatsby-remark-relative-images`,
            options: {
              name: 'assets',
            },
          },
          {
            resolve: `gatsby-remark-images`,
            options: {
              maxWidth: 1200,
            },
          },
        ],
      },
    },
    '@jacobmischka/gatsby-plugin-react-svg',
    {
      resolve: 'gatsby-plugin-postcss-sass',
      options: {
        postCssPlugins: [
          pixrem(),
          autoprefixer({
            browsers: ['last 2 versions']
          }),
        ],
        precision: 8,
      },
    },
    {
      resolve: 'gatsby-plugin-netlify-cms',
    },
    'gatsby-plugin-sharp',
    'gatsby-plugin-netlify',
  ],

@brendan-hurley Open an issue and include a sample markdown string you're trying to parse, and I'll take a look. Trying not to hijack this thread :wink:

fwiw, a 'public_folder' config containing an absolute url is also munged with a leading /WHACK ->  -- after having much discussion in gitter, and reading (grepping really...) the source i think the global, and correct, fix is to never touch 'public_folder' , eg.  never prepend a /...
understanding that this could break existing customers but, also, from a naming perspective my suggestion is similar to others in that i'd add another key, eg 'media_prefix' and prefer that when expanding links in content.  this value should be a promise that would always resolve a media item whether static or dynamic.  it would have a very, very simple contract: media_prefix + '/' + media_filename  is valid.   what that might be so is the customer's problem - not netlify's

Hi everyone!
I had the problem of using Hugos image processing while at the same time maintaining the easy image adding that netlify cms provides.
I used the method talves provided to store all media in a folder in the content directory. Then I created a shortcode which resizes the image and inserts the link. Now I created a custom editor widget for the netlify cms which inserts the shortcode by using the media library and edits the paths, so preview and output work.
I can provide all the files if anyone's interested
(This is my first post in a github discussion - I hope I'm in the right place ˆˆ)

Hi @PaulEibensteiner - welcome! And that's awesome! Can you put your code in a gist and provide a link?

@PaulEibensteiner great job. I was planning to eventually get to creating a widget for a Hugo shortcode, but have been really busy with some other fun stuff that will be coming soon.

Sure!
So first you have to create the directory "images" in your content directory, and add the _index.md with following front matter:
title: Media Folder
headless: true
Then the shortcode in the layouts/shortocodes directory:
https://gist.github.com/PaulEibensteiner/c88e14439abc1111f7388520ade2cf36
Now the custom editor widget: I have a 1-year knowledge of self taught javascript so this might be a complete mess - also I didn't quite understand the documentation about building a custom widget. So I just added the following code at the bottom of my static/admin/index.html (after the   tag):
https://gist.github.com/PaulEibensteiner/c2f1d107deef3a0d7d30fa25ef0a198e
I know that the method to remove and add the path sequence for the folder is really messy. Ideal would probably be to access the public_folder and media_folder configuration and change the path accordingly, I just don't have a clue how to to that - and it works so whatever...
(You could also remove the path like others did in the shortcode via go...but there you wouldn't even have the option to change it based on the folder name I guess)

now you only have to move all your images to the new location and change all links to the new shortcodes...and it should work

ps: I might have messed up the naming of elements because I already changed some stuff in the original project but omitted it here for 'simplicity' - I hope it works. Pls let me know if anything is flawed or if my explanations are confusing
Have fun coding - Hugo's pretty awesome ˆˆ

@PaulEibensteiner This is good stuff. Don't worry about the widget, I will eventually formalize yours into an npm module if you don't get to it first. Again, great job. Also, yes Hugo is pretty Awesome!

@talves Thank you, yeah, I don't really know about npm and that stuff ˆˆ But that would be awesome...
Quick question off-topic: Are there custom widgets provided by other users somewhere on the internet? I could imagine there are a lot of examples (like the yt widget in the docs) which a lot of people would find useful? Googled and didn't find anything...

We have been discussing creating a place for people to publish information about their custom widgets on the NetlifyCMS site somewhere. You are right, and a plan to have a full information format will help quite a bit. I am sure we will get there. We just need someone in the community to take it on when they get time. I might tackle it with some other stuff I am doing, but it is lower on the priority at the moment.

@zionis137 Responding to an old comment of yours https://github.com/netlify/netlify-cms/issues/325#issuecomment-354538798
If you use createNodeField to convert frontmatter.image to a relative path in gatsby-node.js:
createNodeField({
  node,
  name: 'imageRel',
  value: someConversionToFileSystemPath(node.frontmatter.image)
});

then you should be able to query your image node like this:
markdownRemark{
    frontmatter {
        image // this will still be original string that couldn't resolve to file system path
    }
    fields {
        imageRel { // this will contain a file node
            relativePath
            // and then here the childImageSharp stuff
        }
    }
}

As I understand it, any paths that Gatsby can identify as a file system path will be converted to file nodes in the query results in which case you can access relativePath, and childImageSharp, etc.

Thanks, that works great, I used a trick like this to let me leave the images in src/static whilst letting me process them in remark-images. Doing that means the netlify CMS previews work as expected as well.

I've found this working damn great!
Image Loading with Gatsby v2 and Netlify CMS v2
Can also find the link to the dude repos in the article if you wanna see the full config.

Has anyone tried this with an animated gif file? I previously used @danielmahon plugin and had that working...but when my client just decided they wanted a gif...I'm now getting errors that childImageSharp is null. Does it behave differently with a gif vs jpg?  I had looked at this...but on first try didn't have much luck https://www.gatsbyjs.org/packages/gatsby-remark-copy-linked-files/?=copy-

@joshuarule I'm assuming this is still the case?  https://github.com/gatsbyjs/gatsby/issues/7317#issuecomment-412984851


Puerto Rico Flag

Contribution a bit from me and a lot from @jorluiseptor

@aallfredo Hey! This looks great! :smile: Thank you for contributing! I have left a few comments. Please fix them. Apart from that, please change puertorico-flag.png to PuertoRico.png and make same change in README.md.
Also, please resolve conflicts so that it can be merged. :smile:

I made these changes and submitted a pull request.

Closing this PR for #87