VRAM increasing slightly every step until I run out.

import torch
from PIL import Image
from pyramid_dit import PyramidDiTForVideoGeneration
from diffusers.utils import load_image, export_to_video

torch.cuda.set_device(0)
model_dtype, torch_dtype = 'bf16', torch.bfloat16   # Use bf16 (not support fp16 yet)

model = PyramidDiTForVideoGeneration(
    'PATH',                                         # The downloaded checkpoint dir
    model_dtype,
    model_variant='diffusion_transformer_768p',     # 'diffusion_transformer_384p'
)

model.vae.enable_tiling()
#model.vae.to("cuda")
#model.dit.to("cuda")
#model.text_encoder.to("cuda")

prompt = "A dog walking on the beach."

with torch.no_grad(), torch.cuda.amp.autocast(enabled=True, dtype=torch_dtype):
    frames = model.generate(
        prompt=prompt,
        num_inference_steps=[20, 20, 20],
        video_num_inference_steps=[10, 10, 10],
        height=768,
        width=1280,
        temp=31,                    # temp=16: 5s, temp=31: 10s
        guidance_scale=9.0,         # The guidance for the first frame, set it to 7 for 384p variant
        video_guidance_scale=5.0,   # The guidance for the other video latent
        output_type="pil",
        cpu_offloading=True,
        save_memory=True,           # If you have enough GPU memory, set it to `False` to improve vae decoding speed
    )

export_to_video(frames, "./text_to_video_sample.mp4", fps=24)

And the vram starts off low but gradually increases every step until it hits the limit. I have 24 GB of vram.

Even using app.py, it hits step 14 and starts flooding into shared ram, halting progress.

I'm unable to reproduce using your shared script and the latest commit. Make sure your repo is up to date as the fix for this just got merged yesterday. Disabling system memory fallback (I believe this is something you can do in NVIDIA control panel, though I don't use windows) will also fix this, as the gpu will recognize its time to deallocate when it reaches the max

Okay thank you!

I think that was my issue. I disabled the sysmem fallback and it seems to be helping.

nevermind. Now it says:
Traceback (most recent call last):
  File "text.py", line 23, in <module>
    frames = model.generate(
  File "C:\Users\cross\miniconda3\envs\pyramid\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\cross\Downloads\Pyramid-Flow\pyramid_dit\pyramid_dit_for_video_gen_pipeline.py", line 703, in generate
    intermed_latents = self.generate_one_unit(
  File "C:\Users\cross\miniconda3\envs\pyramid\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\cross\Downloads\Pyramid-Flow\pyramid_dit\pyramid_dit_for_video_gen_pipeline.py", line 285, in generate_one_unit
    noise_pred = self.dit(
  File "C:\Users\cross\miniconda3\envs\pyramid\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\cross\miniconda3\envs\pyramid\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\cross\Downloads\Pyramid-Flow\pyramid_dit\modeling_pyramid_mmdit.py", line 479, in forward
    encoder_hidden_states, hidden_states = block(
  File "C:\Users\cross\miniconda3\envs\pyramid\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\cross\miniconda3\envs\pyramid\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\cross\Downloads\Pyramid-Flow\pyramid_dit\modeling_mmdit_block.py", line 640, in forward
    attn_output, context_attn_output = self.attn(
  File "C:\Users\cross\miniconda3\envs\pyramid\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\cross\miniconda3\envs\pyramid\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\cross\Downloads\Pyramid-Flow\pyramid_dit\modeling_mmdit_block.py", line 548, in forward
    hidden_states, encoder_hidden_states = self.var_len_attn(
  File "C:\Users\cross\Downloads\Pyramid-Flow\pyramid_dit\modeling_mmdit_block.py", line 308, in __call__
    stage_hidden_states = F.scaled_dot_product_attention(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 13.68 GiB. GPU 0 has a total capacty of 23.99 GiB of which 12.48 GiB is free. Of the allocated memory 6.10 GiB is allocated by PyTorch, and 3.78 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I dont understand why it is doing this still

hmm. Do you have the temp, res, or something else set super high? It shouldn't be allocating 14 gb with the script you provided in the original post

I tried app.py as well and it got to step 17 and crashed. I have not modified anything.

I set the steps to 31 because I want a 10 second video, but everything else is the default.

I get a crash after step 6. Radeon 7900XTX (24 GB VRAM). 80 GB system RAM.
torch.OutOfMemoryError: HIP out of memory. Tried to allocate 8.46 GiB. GPU 0 has a total capacity of 23.98 GiB of which 7.98 GiB is free. Of the allocated memory 14.00 GiB is allocated by PyTorch, and 1.64 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_HIP_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

i've got the same situation as reported above: memory usage goes up and down in line with the gc collecting and cuda cache cleaning, but on every step it goes a bit higher until OOM at step 17 (so it's not a VAE issue).
i've got 3090 24gb, windows 10.
my configuration is also the same: diffusion_transformer_768p, 10 sec (temp=31) generation, dtype bf16, cpu_offloading=True (in generate function), sequential_cpu_offload disabled.
seriously wondering what's the difference with your settings that you managed to fit it within 12gb vram

one guess for such difference in behaviour was that gc.collect() or torch.cuda.empty_cache() worked differently on linux and windows, not cleaning up vram as effectively on the latter. however, i haven't faced such issues for quite a few years of active dealing with this optimization technique.
yet i see that the generation loop populates generated_latents_list with a new item from intermed_latents on every step, so this vram occupation growth looks quite natural; so i'm more curious how it may stay the same on your side, where do those new generated latents go then?

This seems about in line with what I expect. The latent growth is more apparent when using sequential CPU offload, but even then doesn't seem to get nearly that large. I expect windows has some sort of implementation of the cache that acts different. On Linux, I'm able to run a 10s (temp 31) video with only 12gb of vram available. I'm not quite sure where the drastically increasing ram is coming from though, as a [2,16,1,96,160] tensor at fp32 should only take up ~2 MB. Perhaps something is getting deep copied a bunch on windows

This seems about in line with what I expect. The latent growth is more apparent when using sequential CPU offload, but even then doesn't seem to get nearly that large. I expect windows has some sort of implementation of the cache that acts different. On Linux, I'm able to run a 10s (temp 31) video with only 12gb of vram available. I'm not quite sure where the drastically increasing ram is coming from though, as a [2,16,1,96,160] tensor at fp32 should only take up ~2 MB. Perhaps something is getting deep copied a bunch on windows

I can't get a 768p video to generate on Linux, at bf16. I run out of 24 GB VRAM around step 6. Could this have something to do with the Pytorch version? What Python and Pytorch version did you use?

I'm using 3.12 and torch 2.5.0, are you using an amd GPU?

I'm using 3.12 and torch 2.5.0, are you using an amd GPU?

Yes, Radeon 7900XTX. I wasn't aware of the PyTorch 2.5.0 release yet.

Also, try reducing VAE tile decode size (and make sure save_memory is enabled). This is currently on line 759 of the DiT pipeline file
            image = self.vae.decode(latents, temporal_chunk=True, window_size=1, tile_sample_min_size=256).sample

Set tile_sample_min_size to multiples of 64 (multiples of 8 also seem to work, but slow down decoding significantly in my experience), lower will be less vram during the decoding step
I don't believe this is the source of the issue, but it's possible

I'm using 3.12 and torch 2.5.0, are you using an amd GPU?

Yes, Radeon 7900XTX. I wasn't aware of the PyTorch 2.5.0 release yet.

I can't really verify any behavior with AMD gpus as I don't have one and I'm not too familiar with how it works with pytorch. It may be the source of the issue

@Ednaordinary just to let you know that the issue was solved on my side by a simple upgrade. it appeared i tried it only on python 3.9 + pytorch 2.1.2; the newer setup with python 3.11 + pytorch 2.4 behaves exactly as yours, fitting within 10.5gb vram (windows 10, 3090 24gb).


clean up lg descriptions

They should all read like this:
Sundays 5:30pm 12308 Flintlock Way Herriman
(day) (time) (street address) (city)
We don't need "on" or "at".
We don't need Utah or zipcodes.
Thanks for letting me be picky on this! :)

Cool. The part about not having Utah or the zip code is in Contentful.
The other part I can do in the code.

I cleaned up the addresses in Contentful.


Fix the bug with rendering a component as first child

This PR fixes this bug:
html! {
    <div>
        <Component: />
        <button>{ "Push" }</button> // It will be rendered as a first child :(
    </div>
}

The example had rendered the button first.

bors r+


fix(webpack): update dependency file-loader to v5

Well, the filer-loader's minimum required Node version is 10.13, while untool@next requires 10. So this won't really harmonize. 🤷♂


Give warning in console when forward name not in the list of possible names

Related Problem or Business Case
Find problems sooner
Describe the Feature
I used the following config for some time without having an unhappy flow test (should have done that off course). I've been using the config in demo's for quite some time but had never discovered my copy/paste typo of having false as forward name (unhappy flow was never triggered). FixedResultPipe only supports filenotfound, success and exception as forward names but it doesn't complain when you use something else. You can also add a correct forward and leave the wrong one, this will not give a warning so you can have dead config so to say without ever getting a warning or error (even when this pipe is triggered).
		<FixedResultPipe
			name="Return error message"
			returnString="Temperature not found in response of api.openweathermap.org"
			>
			<Forward name="false" path="InternalServerError" />
		</FixedResultPipe>

Collaboration with External Companies
No, community resolution only
Anything else?
No response

[ ] Introduce a new Java annotation which the frank-doc can also parse, see https://github.com/frankframework/frank-doc/issues/203.
[ ] Framework Javadoc-tag to annotation migration.
[ ] Create ConfigurationWarning in console.

This would also make https://github.com/frankframework/frank-doc/issues/137 trivial to implement, because an annotation can easily be extended with more meta-data, and can easily be read.
I like the idea of making the language more strict, but I would personally prefer a solution such as https://github.com/frankframework/frank-doc/issues/70. Feedback in the editor is better than on configuration reload.  Pipes which have a list of predetermined forwards could use this more strict xsd, which are most pipes if I am not mistaken. This just an idea I had while experimenting with the xsd

Introducing a wildcard attribute on the Forward annotation would make that problem irrelevant. It is important that a pipe specifies a wildcard forward for documentation purposes and to prevent an adapter warning.


How to reset a slot in Story

Description of Problem:
Sometimes we have situations that need to be reset for efficient conversation, for example if agency and account are filled in a slot, the bot can make a transaction that the user doesn't want to do. So the Slot must be reset after the interaction. In the Rasa documentation I didn't find an example of how to reset a Slot, but I learned how to do it.
Overview of the Solution:
# story_01
* greet
  - action_fetch_profile
  - slot{"account_type" : "premium"}
  - slot{"account_type" : null}
  - utter_welcome_premium

Definition of Done:

[ ] Create a exemple of how to reset a slot in Rasa documentation

Thanks for submitting this feature request 🚀@JustinaPetr will get back to you about it soon!✨

Is this still valid in Rasa 2.x? How to achieve this?


TypeError: Path must be a string. Received undefined

 ERROR in   TypeError: Path must be a string. Received undefined

      - index.js:110 HtmlWebpackInlineSourcePlugin.processTag
        [street]/[html-webpack-inline-source-plugin]/index.js:110:32

      - index.js:45
        [street]/[html-webpack-inline-source-plugin]/index.js:45:20

      - Array.forEach

      - index.js:44 HtmlWebpackInlineSourcePlugin.processTags
        [street]/[html-webpack-inline-source-plugin]/index.js:44:19

      - index.js:25 Compilation.<anonymous>
        [street]/[html-webpack-inline-source-plugin]/index.js:25:25

      - Tapable.js:135 Compilation.applyPluginsAsyncWaterfall
        [street]/[tapable]/lib/Tapable.js:135:13

      - util.js:16 Compilation.tryCatcher
        [street]/[bluebird]/js/release/util.js:16:23

      - index.js:627
        [street]/[html-webpack-plugin]/index.js:627:12

      - index.js:150
        [street]/[html-webpack-plugin]/index.js:150:16

      - util.js:16 tryCatcher
        [street]/[bluebird]/js/release/util.js:16:23

      - promise.js:510 Promise._settlePromiseFromHandler
        [street]/[bluebird]/js/release/promise.js:510:31

      - promise.js:567 Promise._settlePromise
        [street]/[bluebird]/js/release/promise.js:567:18

      - promise.js:612 Promise._settlePromise0
        [street]/[bluebird]/js/release/promise.js:612:10

      - promise.js:691 Promise._settlePromises
        [street]/[bluebird]/js/release/promise.js:691:18

      - async.js:138 Async._drainQueue
        [street]/[bluebird]/js/release/async.js:138:16

      - async.js:148 Async._drainQueues
        [street]/[bluebird]/js/release/async.js:148:10

      - async.js:17 Immediate.Async.drainQueues
        [street]/[bluebird]/js/release/async.js:17:14

I discovered this is due to me not setting a publicPath in my webpack config. I've never had this issue before. Should you maybe default to '/' if one isn't defined?

I ran into this issue as well. Thanks @ShimShamSam for indicating what was causing it. Setting a output.publicPath helped resolve it, but I agree that maybe the plugin should have a default, or defer to the output.path if no public path is define?

Should be fixed by 22a7af61235b4c8b20005aacbd791f5da918cd7c.

Nice, thanks @DustinJackson :)


fix RuntimeError for Pool

If you don't put Pool in main that would cause it to execute multiple times and ran into this problem.
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its .

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

This problem occurs on Windows platform, and this pull request should fix it according to my experience. @LynnHo

@sheiun @CastellanLiu It has been merged, thanks a lot.


i18n - a few french translations corrected

Description of changes:
Just a few corrections of translations in french.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Thank you for your contribution!


[BUG] While using Mistral models Flowise answers "forbidden"

When using Mistral models in a conversational retrieval QnA-Chain, Flowise answers "forbidden".
error message as pictured in the attached images. No langsmith link because the error occur before any information is send to langsmith.

sorry!
had false qdrant credentials.
problem solved.


⚠ Bofrost.de has degraded performance

In 257d26e, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1605 ms

Resolved: Bofrost.de performance has improved in b623557 after 1 hour, 53 minutes.


Some weird rare error in requestQueues.getHead()

@VaclavRut reported this error, not sure how it happens
2019-03-28T08:33:57.612Z TypeError: Cannot destructure property `items` of 'undefined' or 'null'.
2019-03-28T08:33:57.614Z     at queryQueueHeadPromise.requestQueues.getHead.then (/usr/src/app/node_modules/apify/build/request_queue.js:500:15)
2019-03-28T08:33:57.615Z     at process._tickCallback (internal/process/next_tick.js:68:7)```

@gippy you were looking into this, did you find anything?

@gippy @jancurn here is one of the runs with this bug (from today though)
https://my.apify.com/admin/users/ppBFjmkrMaGdENQWp/actors/cXj8yrGqdZWHqtSLG#/runs/b3ew8pqtZfMjA2XnS
The actor is using<EMAIL_ADDRESS>

This is fixed in BETA. It's caused by incomplete JSON body returned from API which is them parsed to null. We have added exponential backoff for this case.


feat: How do we want to do logging?

Opinion:

imap-proxy should log
imap-flow and imap-tasks shouldn't log but emit all events with its data to the user

Agree, and agree. log crate or tracing for imap-{flow,tasks}. (My love for tracing overshadows all objectivity.)

I suggest using println :p

https://crates.io/crates/println 🤔

Jakob doesn't care if log or tracing

Prefers to only log in imap-{flow,tasks} when unavoidable
Log generously in proxy

Damian loves tracing


Get-DbaLastGoodCheckDb rounds down DaysSinceLastGoodCheckDb

Bug Report
Get-DbaLastGoodCheckDb DaysSinceLastGoodCheckDb returns floor/integer value. Displaying a Calendar Datetime that is 7 days, 23 hours in the past next to a label of DaysSinceLastGoodCheckDb equals 7 can be initially misleading.
General Troubleshooting steps

[x] Verified running the latest release of dbatools?
[x] Verified errors are not related to permissions?

Steps to Reproduce
$ServerInstance="localhost"
Get-DbaLastGoodCheckDb -SqlServer $ServerInstance | Where-Object {
    ($_.Status -eq "CheckDB should be performed") -and `
    ($_.DaysSinceLastGoodCheckDb -le 7)
} | ft

...or more generally...
New-TimeSpan "2018-01-22 00:00" "2018-01-29 00:00" | Select Days,TotalDays
New-TimeSpan "2018-01-22 00:00" "2018-01-29 23:59" | Select Days,TotalDays
New-TimeSpan "2018-01-22 00:00" "2018-01-30 00:00" | Select Days,TotalDays

Feature Request
The general query "Has there been a good CheckDB in the last 7 days." Returns true from days 0-7 up through day 8.
Problem to solve
Instead of using (New-TimeSpan).Days at line 137, perhaps use (New-TimeSpan).TotalDays or call out in documentation that it is a zero-indexed integer.
To be fair, a TotalDaysSinceLastGoodCheckDb of 7.9 becoming a DaysSinceLastGoodCheckDb of 7 still results in the Status of CheckDB should be performed. However: when querying "Show me all DBs without a good CheckDB in the last 7 days.", this behavior requires that the user know to compare -ge 7 instead of -gt 7.

Wow! Thanks for including the fix in there. I just submitted a PR. Please let me know if it helps:
https://github.com/sqlcollaborative/dbatools/pull/3316


On-board Data Factory service

Hi,
PowerShell seems to have seamless integration to Data Factory (e.g: command Set-AzureRmDataFactoryV2).
It would be great if azure-cli also had support to it.

Hello @tjprescott,
Is there any estimate for this feature to be implemented?

Hi @eltermann, I'm not aware of any active CLI work on this feature.
cc/ @mayurid @twitchax

+1 on Azure CLI support for ADF v2

+1 as well for both ADF and ADFv2.  Powershell support is OS specific.

issue triaged.

Thanks for reporting the issue, we noticed the coverage gap with Powershell, and working on support Data Factory in CLI. The work is tracked in https://github.com/Azure/azure-cli/issues/11527. close this one for now. pls get latest status from issue https://github.com/Azure/azure-cli/issues/11527.


Model 802: SOC Incorrect Type

Per my comment on the file. I believe this should be an Option<u16>.
    /// State of Charge
    ///
    /// State of charge, expressed as a percentage.
    ///
    /// Notes: Measurement.
    pub soc: u16,

The SoC field is marked as mandatory in the JSON model:
            {
                "desc": "State of charge, expressed as a percentage.",
                "label": "State of Charge",
                "mandatory": "M",
                "name": "SoC",
                "sf": "SoC_SF",
                "size": 1,
                "type": "uint16",
                "units": "%WHRtg"
            },

https://github.com/sunspec/models/blob/241057a11d8cd82ca1c2c7e69a7eb4b210169bb1/json/model_802.json#L112-L121
I just checked the SMDX (XML) model file and it's been mandatory ever since:
      <point id="SoC" offset="9" type="uint16" sf="SoC_SF" units="%WHRtg" mandatory="true" />

https://github.com/sunspec/models/blob/a1184356743f7a2673b2a58d633c3d884b6a04ba/smdx/smdx_00802.xml#L14C7-L14C94
Just to be sure I also checked in the SunSpec_Information_Model_Referenc_20211209.xls file:

Unless the spec is wrong this really should be a u16 and not an Option<u16>.

I just opened a new issue about parsing invalid model data:

#16

Is this maybe what you were looking for?

I'll close this issue a invalid then. :+1:
Ticket #16 now tracks the feature of parsing invalid model data. While I think it's going to cause a lot of troubles when parsing invalid data it shouldn't be impossible. I just don't have a great idea on what's necessary here.


Consider having a NULL_IF_FAIL mode when parsing values

Currently there are 3 modes: PERMISSIVE, DROPMALFORMED, and FAIL_FAST.
I use PERMISSIVE, but it still fails if there is a date that does not match the specified date format, or a string in a column that is specified as numeric in the schema. It might be good to have a mode like NULL_IF_FAIL which reads such problematic values as null. Another possibility is allowing multiple date formats so that an alternative can be used if the first fails.

I also have thought this before altough I felt a bit hesitate to open an issue because there are currently too many options for this library.
Nevertheless,  maybe PERMISSIVE mode should include this. Since this is being failed to read, I personally think it might be okay to add on this behaviour in terms of backward compatibility.

Another thought on this. How about being able to specify a "invalidValue" property that is used if parsing was not successful. So instead of throwing a parse exception, the "invalidValue" will be used. That value of the invalidValue may be something like "INVALID" or perhaps the nullValue. If it s the null value, then values that do not parse would be loaded as null. If invalidValue is not set, then behavior would be as it is today.


Set cool temp

👋 Hello, thanks for you work
At parameter Z1_Cool_Request_Temp using direct.
In order to set the cool temperature I use the parameter Z1_Cool_Request_Temp.
For that in file definitions line 441 I have to change the max to 26. With this change I can set ten cool temperature at home to 25 and it works perfect. native_max_value=25.
On line 447 : range(5, 26),
Line 449 true

Hello @guillermohor ,
thanks for the report. I've changed the ranges as you suggested.
I'll let entity_registry_enabled_default=False. It's used to hide by default all settings related to less common parameters (cooling for instance) to avoid overloading the interface.
I think you can enable them from the interface.

Thankyou!


Change

Set
public $depends = [
'yii\web\AssetBundle',
];
Set
use yii2mod\google\maps\markers\GoogleMaps;

Can you describe your issue?


Also regen gobject if needed

It's all because gobject has a GIR file named Gir_gobject.toml, which didn't match what generator.py was expecting. :)

Ah yes, that would explain why I had to manually change gobject/auto in 659b2f5e45eff3efac7ec69459577004d3a560b3.

Now you know. :3


Add JustList: {JustList name}

{
"name": "TokenListExample", #List name
"logoURI": "https://coin.top/production/logo/usdj.png", #List logo uri
"timestamp":<PHONE_NUMBER>000, #Update time in millisecond
"tokens": [{
"chainId": 1, #ChainId, which is currently 1
"address": "TN3W4H6rK2ce4vX9YnFQHwKENnHjoxb3m9", #Token address
"name": "BitCoin", #Token name
"symbol": "BTC", #Token symbol
"decimals": 8, #Token decimals
"logoURI": "https://coin.top/production/logo/TN3W4H6rK2ce4vX9YnFQHwKENnHjoxb3m9.png" #Token logo uri
},
{
"chainId": 1,
"address": "TMwFHYXLJaRUPeW6421aqXL4ZEzPRFGkGT",
"name": "USDJ",
"symbol": "USDJ",
"decimals": 18,
"logoURI": "https://coin.top/production/logo/usdj.png"
}
],
"version": { #Version info
"major": 1, #Major version
"minor": 0, #Minor version
"patch": 0  #Patch version
}
}

Please submit your information in the correct format.

Sorry, your issue will be closed as you did not submit your information in the correct format.Please resubmit.


Add a requirement for link to /coverage

A requirement is missing for a link to the (prescribed path) /coverage resource with the relation type http://www.opengis.net/def/rel/ogc/1.0/coverage.

2022-10-05: Fixed by #144.


panic: runtime error: invalid memory address or nil pointer dereference

What did you do?
Upgraded from prometheus-1.3.1.linux-amd64 to /opt/prometheus-1.5.0.linux-amd64/ .
Service will run for slightly less than 24hrs before encountering this issue.  Little else was changed in going from 1.3.1 to 1.5.0.  Problem was not experienced with 1.3.1.
What did you expect to see?
What did you see instead? Under which circumstances?
Environment

System information:

AWS EC2 c4.4xlarge
Linux 3.13.0-48-generic x86_64

Prometheus version:

./prometheus -version
prometheus, version 1.5.0 (branch: master, revision: d840f2c400629a846b210cf58d65b9fbae0f1d5c)
build user:       root@a04ed5b536e3
build date:       20170123-13:56:24
go version:       go1.7.4

Alertmanager version:

(not likely relevant, but for completeness)
./alertmanager -version
alertmanager, version 0.4.2 (branch: master, revision: 9a5ab2fa63dd7951f4f202b0846d4f4d8e9615b0)
build user:       root@2811d2f42616
build date:       20160902-15:33:13
go version:       go1.6.3

Prometheus configuration file:

global:
  scrape_interval: 30s

rule_files:
  - /etc/prometheus/prometheus.rules
  - /etc/prometheus/aerospike.rules
  - /etc/prometheus/kafka.rules
  - /etc/prometheus/zookeeper.rules

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

  - job_name: aerospike_nodeexporter
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        action: keep
        regex: (.*aerospike.*)
      - source_labels: [__meta_ec2_tag_environment]
        action: keep
        regex: prod
      - action: labelmap
        regex: __meta_(.+)

  - job_name: aerospike
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        action: keep
        regex: (.*aerospike.*)
      - source_labels: [__meta_ec2_tag_environment]
        action: keep
        regex: prod
      - action: labelmap
        regex: __meta_(.+)
  - job_name: zookeeper_nodeexporter
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        action: keep
        regex: (.*[zZ]ookeeper.*)

      - source_labels: [__meta_ec2_tag_environment]
        action: keep
        regex: prod
      - action: labelmap
        regex: __meta_(.+)

  - job_name: zookeeper
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        action: keep
        regex: (.*[zZ]ookeeper.*)

      - source_labels: [__meta_ec2_tag_environment]
        action: keep
        regex: prod
      - action: labelmap
        regex: __meta_(.+)
  - job_name: druid_nodeexporter
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        action: keep
        regex: (.*druid.*)
      - source_labels: [__meta_ec2_tag_environment]
        action: keep
        regex: prod
      - action: labelmap
        regex: __meta_(.+)

  - job_name: kafka_nodeexporter
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        action: keep
        regex: (.*kafka.*)
      - source_labels: [__meta_ec2_tag_environment]
        action: keep
        regex: prod
      - action: labelmap
        regex: __meta_(.+)

  - job_name: kafka
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        action: keep
        regex: (.*kafka.*)
      - source_labels: [__meta_ec2_tag_environment]
        action: keep
        regex: prod
      - action: labelmap
        regex: __meta_(.+)
  - job_name: prometheus_nodeexporter
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        action: keep
        regex: (.*prometheus.*)
      - source_labels: [__meta_ec2_tag_environment]
        action: keep
        regex: prod
      - action: labelmap
        regex: __meta_(.+)

  - job_name: prometheus
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        action: keep
        regex: (.*prometheus.*)
      - source_labels: [__meta_ec2_tag_environment]
        action: keep
        regex: prod
      - action: labelmap
        regex: __meta_(.+)
  - job_name: spark_nodeexporter
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        action: keep
        regex: (.*spark.*)
      - source_labels: [__meta_ec2_tag_environment]
        action: keep
        regex: prod
      - action: labelmap
        regex: __meta_(.+)

  - job_name: webhook-application_nodeexporter
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        action: keep
        regex: (.*webhook-application.*)
      - source_labels: [__meta_ec2_tag_environment]
        action: keep
        regex: prod
      - action: labelmap
        regex: __meta_(.+)

  - job_name: webhook-application
    metrics_path: '/metrics/prometheus'
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        action: keep
        regex: (.*webhook-application.*)
      - action: labelmap
        regex: __meta_(.+)

  - job_name: 'kubernetes_nodes'
    basic_auth:
      username: [REDACTED]
      password: [REDACTED]
    tls_config:
      insecure_skip_verify: [REDACTED]
      ca_file: [REDACTED]
    scheme: https
    kubernetes_sd_configs:
      - api_server: https://[REDACTED_BUT_SANE]/
        basic_auth:
          username: [REDACTED]
          password: [REDACTED]
        tls_config:
          insecure_skip_verify: [REDACTED]
        role: node
    scrape_interval: 30s

  - job_name: 'kubernetes_nodeexporter'
    basic_auth:
      username: [REDACTED]
      password: [REDACTED]
    tls_config:
      insecure_skip_verify: [REDACTED]
      ca_file: [REDACTED]
    scheme: http
    kubernetes_sd_configs:
      - api_server: https://[REDACTED_BUT_SANE]/
        basic_auth:
          username: [REDACTED]
          password: [REDACTED]
        tls_config:
          insecure_skip_verify: [REDACTED]
        role: node
    scrape_interval: 30s
    relabel_configs:
      - source_labels: [__address__]
        action: replace
        regex: (.+):(?:\d+)
        replacement: ${1}:9100
        target_label: __address__

  - job_name: 'kafka_offsets'
    static_configs:
    - targets: ['localhost:[REDACTED]']

  - job_name: 'push_gateway'
    honor_labels: true
    static_configs:
    - targets: ['localhost:[REDACTED]']

  - job_name: 'k8s_state_new'
    static_configs:
    - targets: ['[REDACTED_BUT_SANE:PORT]']

  - job_name: 'k8s_state_old'
    static_configs:
    - targets: ['[REDACTED_BUT_SANE:REDACTED]']

  - job_name: 'k8s_federate'
    honor_labels: true
    scrape_interval: 15s
    params:
      'match[]':
        - '{job=~"skopos"}'
    static_configs:
    - targets: ['[REDACTED_BUT_SANE:REDACTED]']
    relabel_configs:
      - source_labels: [__metrics_path__]
        replacement: /federate
        target_label: __metrics_path__

  - job_name: 'k8s_federate2'
    honor_labels: true
    params:
      'match[]':
        - '{job=~"pods"}'
    static_configs:
    - targets: ['[REDACTED_BUT_SANE:REDACTED]']
    relabel_configs:
      - source_labels: [__metrics_path__]
        replacement: /federate
        target_label: __metrics_path__

  - job_name: nodejs_api1
    metrics_path: '/metrics/prometheus'
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        action: keep
        regex: api
      - source_labels: [__meta_ec2_tag_Stage]
        action: keep
        regex: Live
      - action: labelmap
        regex: __meta_(.+)
      - action: labeldrop
        regex: ec2(.*)

  - job_name: nodejs_api2
    metrics_path: '/metrics/prometheus'
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        action: keep
        regex: api
      - source_labels: [__meta_ec2_tag_Stage]
        action: keep
        regex: Live
      - action: labelmap
        regex: __meta_(.+)
    #  - action: labelmap
     #   regex: ec2_(.*)

  - job_name: nodejs_api3
    metrics_path: '/metrics/prometheus'
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        action: keep
        regex: api
      - source_labels: [__meta_ec2_tag_Stage]
        action: keep
        regex: Live
      - action: labelmap
        regex: __meta_(.+)

  - job_name: nodejs_api4
    metrics_path: '/metrics/prometheus'
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        action: keep
        regex: api
      - source_labels: [__meta_ec2_tag_Stage]
        action: keep
        regex: Live
      - action: labelmap
        regex: __meta_(.+)

  - job_name: nodejs_link1
    metrics_path: '/metrics/prometheus'
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        target_label: role
        action: keep
        regex: link
      - source_labels: [__meta_ec2_tag_Stage]
        action: keep
        regex: Live
      - action: labelmap
        regex: __meta_(.+)

  - job_name: nodejs_link2
    metrics_path: '/metrics/prometheus'
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        target_label: role
        action: keep
        regex: link
      - source_labels: [__meta_ec2_tag_Stage]
        action: keep
        regex: Live
      - action: labelmap
        regex: __meta_(.+)

  - job_name: nodejs_link3
    metrics_path: '/metrics/prometheus'
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        target_label: role
        action: keep
        regex: link
      - source_labels: [__meta_ec2_tag_Stage]
        action: keep
        regex: Live
      - action: labelmap
        regex: __meta_(.+)

  - job_name: nodejs_link4
    metrics_path: '/metrics/prometheus'
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        target_label: role
        action: keep
        regex: link
      - source_labels: [__meta_ec2_tag_Stage]
        action: keep
        regex: Live
      - action: labelmap
        regex: __meta_(.+)

  - job_name: nodejs_link5
    metrics_path: '/metrics/prometheus'
    ec2_sd_configs:
      - region: us-west-1
        access_key: [REDACTED]
        secret_key: [REDACTED]
        port: [REDACTED_BUT_SANE]
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - source_labels: [__meta_ec2_tag_Role]
        target_label: role
        action: keep
        regex: link
      - source_labels: [__meta_ec2_tag_Stage]
        action: keep
        regex: Live
      - action: labelmap
        regex: __meta_(.+)

Alertmanager configuration file:

insert configuration here (if relevant to the issue)

Logs:

We're aware of the sample discard messaging - we plan to use relabeling to deal with some of that.  (Those were occurring under 1.3.1, as well, without this issue.)  There are log captures from two events below.
EVENT ONE:
time="2017-01-30T16:35:32Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:611"
time="2017-01-30T16:35:54Z" level=warning msg="Error on ingesting out-of-order samples" numDropped=206 source="scrape.go:517"
time="2017-01-30T16:35:54Z" level=warning msg="Scrape health sample discarded" error="sample timestamp out of order" sample=up{instance="api-asg", job="nodejs_api2", role="api"} => 1 @[1485794154.879] source="scrape.go:570"
time="2017-01-30T16:35:54Z" level=warning msg="Scrape duration sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{instance="api-asg", job="nodejs_api2", role="api"} => 0.043731663000000004 @[1485794154.879] source="scrape.go:573"
time="2017-01-30T16:35:54Z" level=warning msg="Scrape sample count sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{instance="api-asg", job="nodejs_api2", role="api"} => 0.043731663000000004 @[1485794154.879] source="scrape.go:576"
time="2017-01-30T16:35:54Z" level=warning msg="Scrape sample count post-relabeling sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{instance="api-asg", job="nodejs_api2", role="api"} => 0.043731663000000004 @[1485794154.879] source="scrape.go:579"
time="2017-01-30T16:35:56Z" level=info msg="Done checkpointing in-memory metrics and chunks in 23.40928269s." source="persistence.go:638"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x82f818]

goroutine 191 [running]:
panic(0x18b1120, 0xc420012040)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/prometheus/prometheus/discovery/ec2.(*EC2Discovery).refresh.func2(0xc4baa981a0, 0x1, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/ec2/ec2.go:179 +0x9b8
github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2.(*EC2).DescribeInstancesPages.func1(0x1907100, 0xc4baa981a0, 0x1, 0xc4baa981c0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2/api.go:6785 +0x49
github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/aws/request.(*Request).EachPage(0xc57a8b6e00, 0xc5fb79bb38, 0x2, 0x2)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/aws/request/request_pagination.go:98 +0x90
github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2.(*EC2).DescribeInstancesPages(0xc4dea956b8, 0x0, 0xc5fb79bc58, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2/api.go:6786 +0x116
github.com/prometheus/prometheus/discovery/ec2.(*EC2Discovery).refresh(0xc567c3c9f0, 0xc4e42fbc20, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/ec2/ec2.go:199 +0x347
github.com/prometheus/prometheus/discovery/ec2.(*EC2Discovery).Run(0xc567c3c9f0, 0x7f88bbf72030, 0xc5435ac880, 0xc538bd2d20)
	/go/src/github.com/prometheus/prometheus/discovery/ec2/ec2.go:114 +0x1b8
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4

EVENT TWO:

time="2017-01-31T13:33:17Z" level=warning msg="Error on ingesting out-of-order samples" numDropped=273 source="scrape.go:517"
time="2017-01-31T13:33:17Z" level=warning msg="Scrape health sample discarded" error="sample timestamp out of order" sample=up{instance="api-asg", job="nodejs_api1"} => 1 @[1485869597.057] source="scrape.go:570"
time="2017-01-31T13:33:17Z" level=warning msg="Scrape duration sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{instance="api-asg", job="nodejs_api1"} => 0.049201368 @[1485869597.057] source="scrape.go:573"
time="2017-01-31T13:33:17Z" level=warning msg="Scrape sample count sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{instance="api-asg", job="nodejs_api1"} => 0.049201368 @[1485869597.057] source="scrape.go:576"
time="2017-01-31T13:33:17Z" level=warning msg="Scrape sample count post-relabeling sample discarded" error="sample timestamp out of order" sample=scrape_duration_seconds{instance="api-asg", job="nodejs_api1"} => 0.049201368 @[1485869597.057] source="scrape.go:579"
time="2017-01-31T13:33:19Z" level=info msg="Checkpointing in-memory metrics and chunks..." source="persistence.go:611"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x82f4b5]

goroutine 692839 [running]:
panic(0x18b1120, 0xc420012040)
	/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/prometheus/prometheus/discovery/ec2.(*EC2Discovery).refresh.func2(0xc48961c480, 0x1, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/ec2/ec2.go:192 +0x655
github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2.(*EC2).DescribeInstancesPages.func1(0x1907100, 0xc48961c480, 0x1, 0xc48961c4a0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2/api.go:6785 +0x49
github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/aws/request.(*Request).EachPage(0xc5bb36b880, 0xc590f59b38, 0x2, 0x2)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/aws/request/request_pagination.go:98 +0x90
github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2.(*EC2).DescribeInstancesPages(0xc5a97e33b0, 0x0, 0xc590f59c58, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/vendor/github.com/aws/aws-sdk-go/service/ec2/api.go:6786 +0x116
github.com/prometheus/prometheus/discovery/ec2.(*EC2Discovery).refresh(0xc51d631b30, 0xc6060e30e0, 0x0, 0x0)
	/go/src/github.com/prometheus/prometheus/discovery/ec2/ec2.go:199 +0x347
github.com/prometheus/prometheus/discovery/ec2.(*EC2Discovery).Run(0xc51d631b30, 0x7f0591e3ab80, 0xc587f65400, 0xc5d291a420)
	/go/src/github.com/prometheus/prometheus/discovery/ec2/ec2.go:114 +0x1b8
created by github.com/prometheus/prometheus/discovery.(*TargetSet).updateProviders
	/go/src/github.com/prometheus/prometheus/discovery/discovery.go:242 +0x2b4

Thanks!

Thanks for reporting.
Both instances seem rather odd as the API response seems to contain nil interfaces and tags.
The code didn't change at all w.r.t version 1.3.1.
That can only really mean that AWS suddenly changed the contents of their API responses to have these fields zero'd out. Any chance you can run two small Prometheus servers with 1.3.1 and 1.5.0 in parallel and check whether one crashes while the other keeps going?

yeah.. I have a staging env where I can run 1.3.1.  I haven't spent much time with golang, unfortunately.  Would it make more sense to have the API call dump its payload to file (this assuming all the data is pulled before parsing, and not parsed per tokenized page)?  And naturally.. the problem hasn't reproduced since I posted this.  ;)

Might have been a one-time artifact. Probably should still be handled by the discovery.

1.3.1 instance hasn't been crashing, but it's also not pulling the same set of data from AWS, unless Prom does a full decribe-instances pull, instead of individuals assets.

Can you confirm this is no longer happening?

I presume by the non-response that this is resolved.


Add im2col function for Chainer Variable

This PR adds im2col that acts on chainer.Variable.

Thank you very much for reviewing.
I fixed what you pointed as well as some grammatical issues in the documentation.

LGTM


Replace disk path operations to use fileNames instead of path

#55427 reported that the new files are not detected and per @gluxon investigation it turned out that the case was when tsserver was on case-insensitive volume while the user code was on the case sensistive volume. While changing complete architecture of useCaseSensitive and getCanonicalName is huge change that needs to percolate to everywhere, trying to use file names in all the host operations and using canonical path for comparing (though thats not perfect and bar.ts and Bar.ts will be treated as same file) is minimal change that can help with some of the scenarios. While you may get wrong casing in say some of the auto imports but atleast you should get completions etc just like when you open tsserver.

@gluxon can you please try this build and see if this helps

@typescript-bot pack this

@typescript-bot test this
@typescript-bot test top100
@typescript-bot test tsserver top100
@typescript-bot user test this
@typescript-bot user test tsserver
@typescript-bot run dt
@typescript-bot perf test this

@typescript-bot perf test this

@typescript-bot perf test this

I’m sure it would be incredibly disruptive, but it kind of feels like we would benefit from a FileName branded string type like Path so that a Path can’t be passed to a function that needs the name of an existing file with original casing.

We have this somewhat in server with normalized path but not in absolute path and we probably want that distinction as well .. but that’s too much


[SPARK-35320][SQL] Improve error message for unsupported key types in MapType in from_json expression

What changes were proposed in this pull request?

Currently, when a map is parsed in a from_json function, only StringType key is supported. If you try to parse other type, it results on a cast exception.
For example:
Seq((s"""{"2021-05-05T20:05:08": "sampleValue"}"""))
  .toDF("value")
  .withColumn("value1", from_json(col("value"),  MapType(TimestampType, StringType)))
  .show

Exception in thread "main" java.lang.ClassCastException: class org.apache.spark.unsafe.types.UTF8String cannot be cast to class java.lang.Long (org.apache.spark.unsafe.types.UTF8String is in unnamed module of loader 'app'; java.lang.Long is in module java.base of loader 'bootstrap')
	at scala.runtime.BoxesRunTime.unboxToLong(BoxesRunTime.java:107)
	at org.apache.spark.sql.catalyst.expressions.CastBase.$anonfun$castToString$8$adapted(Cast.scala:297)
	at org.apache.spark.sql.catalyst.expressions.CastBase.buildCast(Cast.scala:285)
	at org.apache.spark.sql.catalyst.expressions.CastBase.$anonfun$castToString$7(Cast.scala:297)

This PR proposes to improve the error message.
org.apache.spark.sql.AnalysisException: cannot resolve 'entries' due to data type mismatch: Input schema map<timestamp,string> can only contain StringType as a key type for a MapType.;
'Project [unresolvedalias(from_json(MapType(TimestampType,StringType,true), value#1, Some(America/Los_Angeles)), Some(org.apache.spark.sql.Column$$Lambda$1496/546936087@10e5bf9c))]
+- LocalRelation [value#1]
	at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:197)
	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:182)
	at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUpWithPruning$2(TreeNode.scala:535)
	at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)
...

In https://github.com/apache/spark/pull/32599 we decide to improve the error message instead of support this.
Why are the changes needed?

Avoid confusion in the interpretation of the error
Does this PR introduce any user-facing change?

Yes, the error message returned in this case
How was this patch tested?

Unit testing and manual testing

CC @viirya @cloud-fan
Thanks!

Can one of the admins verify this patch?

@viirya Thanks for the comments, I have updated the PR.

ok to test

Test build #141691 has started for PR 33525 at commit b2ed6b9.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46205/

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46205/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46205/

Test build #141691 has finished for PR 33525 at commit b2ed6b9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141691/

Can one of the admins verify this patch?

Test build #141733 has started for PR 33525 at commit 496b869.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46246/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46246/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46246/

Test build #141733 has finished for PR 33525 at commit 496b869.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public class RetryingBlockTransferor
public final class Aggregation implements Serializable
public final class Count implements AggregateFunc
public final class CountStar implements AggregateFunc
public final class Max implements AggregateFunc
public final class Min implements AggregateFunc
public final class Sum implements AggregateFunc
case class CoalescedMapperPartitionSpec(
trait AQEShuffleReadRule extends Rule[SparkPlan]
case class CoalesceShufflePartitions(session: SparkSession) extends AQEShuffleReadRule
case class ScanBuilderHolder(

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141733/

thanks, merging to master!

looks good 👍


Invalid TDs are accepted by Frontend and Backend

I would suggest validating TDs using the validation schema provided by the TD spec and some more stuff. At some point, they can be left to the backend but Playground is able to do all the required validation in the frontend.
Simple JSON Schema validation (using ajv) to check against the schema (frontend?). Do assertion test to find what is supported by the TD that can be also used to generate tags for searching.

Fixed in #32


Can't pay any invoice

### Background
I already tried paying several invoices from different nodes but none seems to go through. I even have opened direct channel to the node I got invoice from, but it still shows the same error that no route was found.
$ lncli payinvoice lnbc1500n1p0caagvpp53wgq4hpzqn2y6vsms4jzsvuv6xyf467gwn4cahv72njuls9755rqdpa2fjkzep6ypkxjmntypcxzunpypjkcgrxv96kxet5yqcjuvpsxqs8xct5wvsxxcqzpgxqr23ssp574pz8n8mtrrdjf66e3ttkehu0pjxvasrjw4z8tuntgeex08vkkhs9qy9qsq700am04xg6he3wysjmx263yremh75p26c2wm39n877ewf6vued5qt2y2gdp62lujvzwskmgdepfckmc7yxtx7a3ctefcxderv00jr5gqzaqmsu
Payment hash: 8b900adc2204d44d321b856428338cd1889aebc874eb8edd9e54e5cfc0bea506
Description: Read: link para el faucet 1.000 sats c
Amount (in satoshis): 150
Fee limit (in satoshis): 150
Destination: 03d06758583bb5154774a6eb221b1276c9e82d65bbaceca806d90e20c108f4b1c7
Confirm payment (yes/no): yes
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
| HTLC_STATE | ATTEMPT_TIME | RESOLVE_TIME | RECEIVER_AMT | FEE | TIMELOCK | CHAN_OUT | ROUTE |
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
+------------+--------------+--------------+--------------+-----+----------+----------+-------+
Amount + fee:   0 + 0 sat
Payment hash:   8b900adc2204d44d321b856428338cd1889aebc874eb8edd9e54e5cfc0bea506
Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE
[lncli] FAILED
### Your environment

"version": "0.11.99-beta commit=v0.11.0-beta-312-gd12f76fd6d48f5d4df8d830b5ec49e9277ffa5c0"
MacOS
BTC neutrino
Node runs behind Tor

### Steps to reproduce
Not clear.
### Expected behaviour
Payment should go through.
### Actual behaviour
Error: Payment status: FAILED, reason: FAILURE_REASON_NO_ROUTE

Do you have any logs for this?

Never mind. Just found out that my node is only badly connected in the network. And regarding the direct channel, I checked the destination from invoice and it's actually a different pubkey than the recipient has on its website. So I thought I have different channel to him but in fact I did not. My bad. Closing issue.


Question about the temporal span I1 and I2

Thanks for sharing your excellent work as open source. I have a question about your code. I couldn't find the specific parameters for the temporal span I1 and I2 that you mentioned in your paper. Could you please explain how to pass them and how they work in the attention mechanism?

I apologize for any confusion caused by the terminology used in our paper.   The parameters referred to as "temporal span I1 and I2" in the paper are named pos_duration and pred_duration in the code.   These parameters control the window size for past positions and past predictions, respectively.

Thank you for your prompt response.  Just one more question, I see the pos_duration I1 defaults 20, which is already the maximum value for history length. In other words, the window size for past positions is [t − I1, t]==>[t − 20, t]. In such case, t − I1− I2<t − 20, and it is out of the interval [t − 20, t]. How do you handle such case?

You are absolutely right. During normal training, although the visible window is larger, we are limited to only the last 20 frames of data due to data constraints. However, in the experimental section of our work, I set a longer history length to test the advantages of a larger visible window. For more details, you can refer to Figure 3 in the paper.


security question

this project is vulnerable to this problem?
https://bitcointalk.org/index.php?topic=977070.0

Yes and no!
It depends on your operating system, your hardware, and all sorts of things. This library uses /dev/urandom or the windows equivalent.
So, if your machine does produce cryptographically strong random data, you are not at risk.

I don't completely understand...

The particular problem here is that ECDSA says you must use a nonce (random number) in the signature. As you know, if 'r reuse' occurs for a given private key then someone can solve an equation for that private key.
The problem here is the quality of your random numbers. There is no algorithm to produce 100% random bytes, instead computers usually use what's called a cryptographically strong pseudo random bytes generator, which while not entirely random, is random enough. This is usually provided by the operating system, because it can tap into data that usually is different from computer to computer.
Stuff like your computers temperature, the wind in your hard drive, are useful things to seed the random number generator. I mentioned the raspberry pi, because it doesn't have a lot of entropy sources (it has a solid state drive).
The thing is, ECDSA, and hence this library, are only secure if random numbers actually are random.. So what people have started doing is using an algorithm that takes a private key and the messages hash and produces a value for r which is unique, which means you no longer have to worry about whether your random number generator is good enough.


Markdown test

Headers
# H1
## H2
### H3
#### H4
##### H5
###### H6

Alternatively, for H1 and H2, an underline-ish style:

Alt-H1
======

Alt-H2
------

H1
H2
H3
H4
H5
H6
Alternatively, for H1 and H2, an underline-ish style:
Alt-H1
Alt-H2
Emphasis
Emphasis, aka italics, with *asterisks* or _underscores_.

Strong emphasis, aka bold, with **asterisks** or __underscores__.

Combined emphasis with **asterisks and _underscores_**.

Strikethrough uses two tildes. ~~Scratch this.~~

Emphasis, aka italics, with asterisks or underscores.
Strong emphasis, aka bold, with asterisks or underscores.
Combined emphasis with asterisks and underscores.
Strikethrough uses two tildes. Scratch this.
Lists
1. First ordered list item
2. Another item
    * Unordered sub-list.
    * Another unordered item
        1. Embedded-embedded list
        2. And another
1. Actual numbers don't matter, just that it's a number
    1. Ordered sub-list
    2. Ordered sub-list
4. And another item.

Text before paragraph
* Unordered list can use asterisks
- Or minuses
+ Or pluses

First ordered list item
Another item

Unordered sub-list.
Another unordered item

Embedded-embedded list
And another

Actual numbers don't matter, just that it's a number

Ordered sub-list
Ordered sub-list

And another item.

Text before paragraph

Unordered list can use asterisks

Or minuses

Or pluses

Links
There are two ways to create links.
[I'm an inline-style link](https://www.google.com)

[I'm an inline-style link with title](https://www.google.com "Google's Homepage")

[I'm a reference-style link][Arbitrary case-insensitive reference text]

[I'm a relative reference to a repository file](../blob/master/LICENSE)

[You can use numbers for reference-style link definitions][1]

Or leave it empty and use the [link text itself].

URLs and URLs in angle brackets will automatically get turned into links.
http://www.example.com or <http://www.example.com> and sometimes
example.com (but not on Github, for example).

Some text to show that the reference links can follow later.

[arbitrary case-insensitive reference text]: https://www.mozilla.org
[1]: http://slashdot.org
[link text itself]: http://www.reddit.com

I'm an inline-style link
I'm an inline-style link with title
I'm a reference-style link
I'm a relative reference to a repository file
You can use numbers for reference-style link definitions
Or leave it empty and use the link text itself.
URLs and URLs in angle brackets will automatically get turned into links.
http://www.example.com or http://www.example.com and sometimes
example.com (but not on Github, for example).
Some text to show that the reference links can follow later.
Images
Here's our logo (hover to see the title text):

Inline-style:
![alt text](https://github.com/adam-p/markdown-here/raw/master/src/common/images/icon48.png "Logo Title Text 1")

Reference-style:
![alt text][logo]

[logo]: https://github.com/adam-p/markdown-here/raw/master/src/common/images/icon48.png "Logo Title Text 2"

Here's our logo (hover to see the title text):
Inline-style:

Reference-style:

Code and Syntax Highlighting
Inline `code` has `back-ticks around` it.

Inline code has back-ticks around it.
Blocks of code are either fenced by lines with three back-ticks ```, or are indented with four spaces. I recommend only using the fenced code blocks -- they're easier and only they support syntax highlighting.
```javascript
var s = "JavaScript syntax highlighting";
alert(s);
```

```python
s = "Python syntax highlighting"
print s
```

```
No language indicated, so no syntax highlighting.
But let's throw in a tag.
```

var s = "JavaScript syntax highlighting";
alert(s);

s = "Python syntax highlighting"
print s

No language indicated, so no syntax highlighting in Markdown Here (varies on Github).
But let's throw in a tag.

Tables
Tables aren't part of the core Markdown spec, but they are part of GFM and Markdown Here supports them. They are an easy way of adding tables to your email -- a task that would otherwise require copy-pasting from another application.
Colons can be used to align columns.

| Tables        | Are           | Cool  |
| ------------- |:-------------:| -----:|
| col 3 is      | right-aligned | $1600 |
| col 2 is      | centered      |   $12 |
| zebra stripes | are neat      |    $1 |

There must be at least 3 dashes separating each header cell.
The outer pipes (|) are optional, and you don't need to make the
raw Markdown line up prettily. You can also use inline Markdown.

Markdown | Less | Pretty
--- | --- | ---
*Still* | `renders` | **nicely**
1 | 2 | 3

Colons can be used to align columns.

Tables
Are
Cool

col 3 is
right-aligned
$1600

col 2 is
centered
$12

zebra stripes
are neat
$1

There must be at least 3 dashes separating each header cell. The outer pipes (|) are optional, and you don't need to make the raw Markdown line up prettily. You can also use inline Markdown.

Markdown
Less
Pretty

Still
renders
nicely

1
2
3

Blockquotes
> Blockquotes are very handy in email to emulate reply text.
> This line is part of the same quote.

Quote break.

> This is a very long line that will still be quoted properly when it wraps. Oh boy let's keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can *put* **Markdown** into a blockquote.

>This is a quote with no space

> Quote level one
> > Quote level two
> > Another block
> > > And three

Blockquotes are very handy in email to emulate reply text.
This line is part of the same quote.

Quote break.

This is a very long line that will still be quoted properly when it wraps. Oh boy let's keep writing to make sure this is long enough to actually wrap for everyone. Oh, you can put Markdown into a blockquote.

This is a quote with no space

Quote level one

Quote level two
Another block

And three

Inline HTML
You can also use raw HTML in your Markdown, and it'll mostly work pretty well.

Definition list

Is something people use sometimes.

Markdown in HTML

Does *not* work **very** well. Use HTML tags.

Definition list
Is something people use sometimes.
Markdown in HTML
Does *not* work **very** well. Use HTML tags.

Horizontal Rule
Three or more...

---

Hyphens

***

Asterisks

___

Underscores

Three or more...

Hyphens

Asterisks

Underscores
Line Breaks
My basic recommendation for learning how line breaks work is to experiment and discover -- hit <Enter> once (i.e., insert one newline), then hit it twice (i.e., insert two newlines), see what happens. You'll soon learn to get what you want. "Markdown Toggle" is your friend.
Here are some things to try out:
Here's a line for us to start with.

This line is separated from the one above by two newlines, so it will be a *separate paragraph*.

This line is also a separate paragraph, but...
This line is only separated by a single newline, so it's a separate line in the *same paragraph*.

Here's a line for us to start with.
This line is separated from the one above by two newlines, so it will be a separate paragraph.
This line is also begins a separate paragraph, but...
This line is only separated by a single newline, so it's a separate line in the same paragraph.
(Technical note: Markdown Here uses GFM line breaks, so there's no need to use MD's two-space line breaks.)

bump

bump

another bump

bump

bump

bump


[MINOR] Added support for SqlFileBasedTransformer

What is the purpose of the pull request
The current SQLQuery based transformer is limited in functionality, you can't pass multiple Spark SQL statements separated by a semicolon which is necessary if your transformation is complex.
This pull-request adds a new SQLFileBasedTransformer which takes a Spark SQL file as input with multiple Spark SQL statements to support complex transformation logic and applies the transformation to the delta streamer payload.
Jira: https://issues.apache.org/jira/browse/HUDI-1743
Brief change log

Adds Spark SQL File-based transformer to support complex multiline transformations.

Verify this pull request
This pull request is trivial work without any test coverage.
Committer checklist

[x] Has a corresponding JIRA in PR title & commit

[x] Commit message is descriptive of the change

[x] CI is green

[ ] Necessary doc changes done or have another open PR

[ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

Codecov Report

Merging #2747 (ea6236e) into master (aa0da72) will decrease coverage by 3.58%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #2747      +/-   ##
============================================
- Coverage     52.06%   48.47%   -3.59%
+ Complexity     3625     2868     -757
============================================
  Files           479      367     -112
  Lines         22804    17631    -5173
  Branches       2415     1820     -595
============================================
- Hits          11872     8547    -3325
+ Misses         9907     8382    -1525
+ Partials       1025      702     -323

Flag
Coverage Δ
Complexity Δ

hudicli
37.01% <ø> (ø)
0.00 <ø> (ø)

hudiclient
∅ <ø> (∅)
0.00 <ø> (ø)

hudicommon
50.92% <ø> (-0.05%)
0.00 <ø> (ø)

hudiflink
56.01% <ø> (ø)
0.00 <ø> (ø)

hudihadoopmr
33.44% <ø> (ø)
0.00 <ø> (ø)

hudisparkdatasource
?
?

hudisync
?
?

huditimelineservice
?
?

hudiutilities
?
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ
Complexity Δ

...ache/hudi/common/fs/inline/InMemoryFileSystem.java
79.31% <0.00%> (-10.35%)
15.00% <0.00%> (-1.00%)

...e/hudi/common/table/log/HoodieLogFormatWriter.java
78.12% <0.00%> (-1.57%)
26.00% <0.00%> (ø%)

...in/java/org/apache/hudi/utilities/UtilHelpers.java

...g/apache/hudi/utilities/schema/SchemaProvider.java

...he/hudi/utilities/transform/AWSDmsTransformer.java

...udi/utilities/transform/FlatteningTransformer.java

...ache/hudi/hive/HiveMetastoreBasedLockProvider.java

...3/internal/HoodieBulkInsertDataInternalWriter.java

...src/main/java/org/apache/hudi/QuickstartUtils.java

...src/main/scala/org/apache/hudi/DefaultSource.scala

... and 104 more

@yanghua - I've already filled Jira and linked in the description https://issues.apache.org/jira/browse/HUDI-1743
Do you want me to update the title of this pull request?
I will fix this build issue as well.

@vingov Thanks, but the CI has failed, would you please check the reason? If you not sure, you can push an empty commit to retrigger the Travis.

@yanghua - I've fixed the build, can you please merge this code?

@yanghua - I've fixed the build, can you please merge this code?

Thanks, IMO, can you add a unit test for the feature.

@yanghua - I don't see the unit tests for the existing transformers except for two functions, I don't have time now to write unit tests, can I handle it in a separate pull request where I can write unit tests for all transformers?
This is blocking my data pipelines, can we make an exception and merge this pull request?  I'm happy to create a JIRA to track the unit tests for all transformers. thoughts?

@yanghua - I don't see the unit tests for the existing transformers except for two functions, I don't have time now to write unit tests, can I handle it in a separate pull request where I can write unit tests for all transformers?

It's better to follow a unified contribution guide. If we can test it,  we should test it, so that we can make sure the code quality.

This is blocking my data pipelines, can we make an exception and merge this pull request? I'm happy to create a JIRA to track the unit tests for all transformers. thoughts?

You can pick this patch into your inner branch. wdyt?

Codecov Report

Merging #2747 (ef55cd7) into master (aa0da72) will decrease coverage by 0.87%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #2747      +/-   ##
============================================
- Coverage     52.06%   51.18%   -0.88%
+ Complexity     3625     3308     -317
============================================
  Files           479      428      -51
  Lines         22804    20990    -1814
  Branches       2415     2240     -175
============================================
- Hits          11872    10744    -1128
+ Misses         9907     9298     -609
+ Partials       1025      948      -77

Flag
Coverage Δ
Complexity Δ

hudicli
39.53% <ø> (+2.52%)
220.00 <ø> (+25.00)

hudiclient
∅ <ø> (∅)
0.00 <ø> (ø)

hudicommon
50.67% <ø> (-0.31%)
1976.00 <ø> (+1.00)
:arrow_down:

hudiflink
59.11% <ø> (+3.09%)
535.00 <ø> (+33.00)

hudihadoopmr
33.33% <ø> (-0.12%)
198.00 <ø> (+1.00)
:arrow_down:

hudisparkdatasource
73.33% <ø> (+2.46%)
237.00 <ø> (+42.00)

hudisync
46.10% <ø> (+0.63%)
142.00 <ø> (+14.00)

huditimelineservice
?
?

hudiutilities
?
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ
Complexity Δ

.../apache/hudi/sink/compact/CompactionPlanEvent.java
50.00% <0.00%> (-50.00%)
3.00% <0.00%> (ø%)

...pache/hudi/sink/compact/CompactionCommitEvent.java
43.75% <0.00%> (-43.75%)
3.00% <0.00%> (ø%)

...ache/hudi/sink/compact/CompactionPlanOperator.java
58.00% <0.00%> (-21.07%)
8.00% <0.00%> (-1.00%)

.../apache/hudi/sink/partitioner/BucketAssigners.java
33.33% <0.00%> (-16.67%)
2.00% <0.00%> (ø%)

.../org/apache/hudi/sink/utils/NonThrownExecutor.java
66.66% <0.00%> (-11.12%)
4.00% <0.00%> (-1.00%)

...ache/hudi/common/fs/inline/InMemoryFileSystem.java
79.31% <0.00%> (-10.35%)
15.00% <0.00%> (-1.00%)

...hadoop/realtime/RealtimeCompactedRecordReader.java
64.06% <0.00%> (-8.67%)
13.00% <0.00%> (+1.00%)
:arrow_down:

...e/hudi/sink/transform/RowDataToHoodieFunction.java
83.33% <0.00%> (-7.58%)
5.00% <0.00%> (-1.00%)

...java/org/apache/hudi/sink/StreamWriteFunction.java
77.77% <0.00%> (-5.93%)
22.00% <0.00%> (-2.00%)

.../main/scala/org/apache/hudi/HoodieSparkUtils.scala
83.33% <0.00%> (-5.56%)
0.00% <0.00%> (ø%)

... and 124 more

@yanghua - I have added the unit tests, Can you please review and merge?

@yanghua - I have added the unit tests, Can you please review and merge?

thanks for addressing my concerns. @vinothchandar will take over this PR.

@vingov : Did you get a chance to check out my feedback. Once addressed, we can land this.

@nsivabalan - Addressed all the feedback, please review and merge this PR.


Custom attributes for network section (NIC) in VM template

Author Name: Stefan Kooman (Stefan Kooman)
Original Redmine Issue: 4066, https://dev.opennebula.org/issues/4066
Original Date: 2015-10-16

In some cases you want to perform some special action in a VM or on the frontend / hypervisor based on some property of a network adapter. As far as I know it's currently not possible to add a custom attribute to the "NIC=[]" section. It would make life easy if a custom attribute could be added. Example use case: configure a special source based route for the specific nic if LB=TRUE is configured. Or setup special openflow rules on the vnet if OPENFLOW_SPECIAL=TRUE.

Original Redmine Comment
Author Name: Carlos Martín (Carlos Martín)
Original Date: 2015-10-19T08:26:42Z

You can actually add any attribute inside the NIC.
Furthermore, you can even configure in oned.conf if any attribute should be restricted to be used only by oneadmin:
http://docs.opennebula.org/4.14/administration/references/oned_conf.html#restricted-attributes-configuration
Or set the attributes in the vnet template, and copy it to the NICs using that vnet:
http://docs.opennebula.org/4.14/administration/references/oned_conf.html#inherited-attributes-configuration
Can you give it a try to see if your request is already covered?

Seems this issue was implemented. Closing...


feat: implement cost and cc endpoints

Description

not finished yet, pr for tracking purposes

Please provide a brief summary of the changes made in this pull request.
Fixes # (issue reference, if applicable)
Type of change
Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration.

[ ] Locally tested the according endpoints
[ ] Ran E2E-Tests

Checklist:

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation

Cool, jetzt haben wir zwei Analysetools, die uns vollheulen, @930C.  :D

Da müssen wir ja jetzt mal anfangen guten Code zu schreiben :o

Würde das demnächst mergen wollen, damit auf unserem Prod-System bisschen was geht. Wie siehst du das, @930C?

Wäre ready. Habe noch wenigstens ganz einfaches GET costs implementiert.


Standalone Swift - MaaS RabbitMQ Check Failing

In standalone swift deployments, RabbitMQ is installed as part of the shared-infra host group. When installing MaaS, the RabbitMQ check tries to discern the max number of channels. Unfortunately, as RabbitMQ isn't even used, the API returns an empty list, which max() seems to not like.
RabbitMQ Channels API Call
root@9d0ae713-infra1_rabbit_mq_container-f247d7c7:~# http -a  guest:guest http://<IP_ADDRESS>:15672/api/connections?columns=channels
HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Length: 2
Content-Type: application/json
Date: Tue, 07 Jul 2015 21:09:53 GMT
Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)

[]

RabbitMQ MaaS Check
root@9d0ae713-infra1:/usr/lib/rackspace-monitoring-agent/plugins# ./rabbitmq_status.py -H <IP_ADDRESS> -n 9d0ae713-infra1_rabbit_mq_container-f247d7c7
status error e "./rabbitmq_status.py", line 166, in <module>\n    main()\n  File "./rabbitmq_status.py", line 98, in main\n    max_chans = max(connection['channels'] for connection in resp_json if 'channels' in connection)\nValueError: max() arg is an empty sequence\n
Traceback (most recent call last):
  File "./rabbitmq_status.py", line 166, in <module>
    main()
  File "./rabbitmq_status.py", line 98, in main
    max_chans = max(connection['channels'] for connection in resp_json if 'channels' in connection)
ValueError: max() arg is an empty sequence

@byronmccollum I'd like some information about his this deployment was done. When I deploy a stand-alone swift environment, rabbitmq is barely set up. I get 401s because none of the users typically associated with rabbitmq or maas use are present.

@rackertom I'm building an environment, and will let you know if this is still the case...

Yes, this is still failing. The check is slightly different after the RabbitMQ guest user removal, but still failing in the same way...
# ./rabbitmq_status.py -H <IP_ADDRESS> -n 4dbf96e2-infra1_rabbit_mq_container-8d229323 -U maas_user -p ********
status error print_output\n    yield\n  File "./rabbitmq_status.py", line 170, in <module>\n    main()\n  File "./rabbitmq_status.py", line 101, in main\n    max_chans = max(connection['channels'] for connection in resp_json\nValueError: max() arg is an empty sequence\n
Traceback (most recent call last):
  File "./rabbitmq_status.py", line 170, in <module>
    main()
  File "./rabbitmq_status.py", line 101, in main
    max_chans = max(connection['channels'] for connection in resp_json
ValueError: max() arg is an empty sequence

# http -a maas_user:******** http://<IP_ADDRESS>:15672/api/connections?columns=channels
HTTP/1.1 200 OK
Cache-Control: no-cache
Content-Length: 2
Content-Type: application/json
Date: Mon, 05 Oct 2015 16:29:00 GMT
Server: MochiWeb/1.1 WebMachine/1.10.0 (never breaks eye contact)

[]

I can get you access to the environment if you want to further diagnose...

Why is RabbitMQ being installed if it isn't used? It seems like it would be better to tweak the environment so there are no rabbitmq containers.

@git-harry Good question. Seems like there needs to be a swift-shared-infra_hosts group with just Galera, and Memcache. And it looks like the RabbitMQ checks wouldn't even get created if there aren't any RabbitMQ containers.

Given the above, can we recommend a change to the docs for a swift stand-alone to eliminate rmq and close this?

@stevelle What is an appropriate host group(s) that will result in RabbitMQ not getting installed?

I think generally speaking, we're looking to address the root issue upstream in OSA- Don't install RabbitMQ when it isn't necessary. When that is done, I think we can circle back to this issue and confirm that checks do not install for components that are not present.
Objections @byronmccollum @stevelle or @git-harry?

Sounds good to me.

@rackertom make it so IMO

OSA bug filed. https://bugs.launchpad.net/openstack-ansible/+bug/1513132

@rackertom Does #442 affect this in any way? I know the ultimate goal is to just not install RabbitMQ for Stand Alone Swift deployments, but just wondering if there is a nice side effect from the change?

@byronmccollum confirming that #442 should partially address this issue.

osa bug confirms that the issue should be addressed using affinity settings (that is, just don't define a rabbit server in your config, and none of the monitoring stuff will get applied either).
@byronmccollum closing this issue here - please see and add commentary here https://bugs.launchpad.net/openstack-ansible/+bug/1513132 if you have an alternative view

Impacts docs for stand alone swift deployments?


Make repo readme the same as the microsoft/dotnet-framework Docker Hub readme

The current readme is out of date.  With the repo consolidation I think this makes sense and aligns with what is done in https://github.com/dotnet/dotnet-docker.
The only change here is to delete the current README.md and rename README.dotnet-framework.md to README.md.

LGTM
Want to add a line into README.dotnet-framework-build.md that says "this repo will be deleted on 1/1/2019" or something to that effect? Basically, if we ever want to delete the repo, now is the time to communicate that plan.

@richlander - yes - I added your suggestion.


align MPI code instantiation with stan conventions as good as possible

Submission Checklist

[ ] Run unit tests: ./runTests.py test/unit
[X] Run cpplint: make cpplint
[X] Declare copyright holder and open-source license: see below

Summary:
Places code which has to be instantiated once for MPI into stan-standards compliant locations.
Intended Effect:
MPI code needs instantiated code which is now handled following the conventions in stan as good as possible. That is, the cpp file is not inside stan/... and called *_inst.cpp. This is the same logic as done for libstanc, for example. However, there is one notable difference: While in stan the bin/libstanc.a static library is formed and then linked during compilation, this does not work for the MPI code (for reasons not clear to me). However, what does work is to directly link in the object file. This is likely due to how references to shared libraries are handled. The proposed change works and is reasonable as there is no gain at the moment to link either the static library or directly the object file.
How to Verify:
Run the MPI tests:
test/unit/math/rev/mat/functor/map_rect_mpi_test.cpp
test/unit/math/prim/mat/functor/map_rect_mpi_test.cpp
test/unit/math/prim/mat/functor/mpi_parallel_call_test.cpp
test/unit/math/prim/arr/functor/mpi_cluster_test.cpp

Side Effects:
Documentation:
Copyright and Licensing
Please list the copyright holder for the work you are submitting (this will be you or your assignee, such as a university or company): Sebastian Weber
By submitting this pull request, the copyright holder is agreeing to license the submitted work under the following licenses:

Code: BSD 3-clause (https://opensource.org/licenses/BSD-3-Clause)
Documentation: CC-BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

@bob-carpenter I hope this change is in line as you had in mind. This PR eliminates the need for a src directory.


Limitation issue about inputs in output nodes

Hi,
i've seen there is a limitation about inputs I can choose from the web interface in output nodes, but I can bypass this limitation editing the config file of the nodes. I thin this is a web interface limitation.
Thank you,
Giorgio

Hi @jonnynux,
sorry for the late reply. Yes, it's a way to prevent mistakes on the WebUI. If you click on the "eye" icon on the input dialog, you will switch to expert mode and you can add more inputs.
Thanks,
luigi


Changing C function name creates problems..  Code position in compile?

I know, this code is (probably) an awful way to do C/Swift bridging.  But I'm converting a C project to Swift and want to do it one function at a time, so it's easier than figuring out whatever the "official way" is - if there is one.
But If I even change the name of the C function (changing "hello_c" to "hello_c_ORIG"), it breaks.  It gives errors of either not finding various functions in scope, importing "Foundation.h" inside a "hello_c_ORIG", or failing to emit the pre-compiled bridging header (which references the C header, which has the renamed function)
I assume this is because the renamed function changes the position of things in the code.  How do I fix that?

Okay - I'd just missed a name somewhere.  This is why I'm not really a programmer.


Support request regarding new .nbs format

I'm currently maintaining a plugin for minecraft. But it uses the old .nbs file format.
If possible, could you contact me on discord? (Stefano#7366). That would be very nice!

Sent, I'll help you out there


Benchmarking

Adds the architecture to allow ASV benchmarking on this repo.

Main content is an ASV plug-in deferring environment prep to Nox.

Attempts to integrate ASV into CI. Well see how that goes!

CI Benchmarking Problems
It has so far proven impossible to integrate ASV into our Cirrus-based CI:

Using Nox for environment management relies on the plugin nox_asv_plugin.py.
Testing locally shows ASV is able to import this as intended (from the same directory as asv.conf.json).
However Cirrus-CI encounters a ModuleNotFoundError, despite being in the correct directory, and that directory containing nox_asv_plugin.py - see the trace below.

It seems likely that a setting on the Cirrus machine is breaking an assumption within ASV. It may work fine with other providers, e.g. GitHub Actions, but I am reluctant to move away from Cirrus given the intent to leverage compute credits in future. Hopefully it will be possible to fix within Cirrus, however this will require investigation and testing.
I am therefore going to disable CI benchmarking for now, and limit this PR to just providing the architecture for manual ASV runs.

Section of Cirrus trace
nox > python -c from pathlib import Path; print(Path().cwd()); print(list(Path().iterdir()));
/tmp/cirrus-ci-build/benchmarks
[PosixPath('benchmarks'), PosixPath('nox_asv_plugin.py'), PosixPath('nox_asv.conf.yaml'), PosixPath('asv.conf.json')]
nox > asv machine --yes
Traceback (most recent call last):
  File "/tmp/cirrus-ci-build/.nox/benchmarks-ci_mode-true/bin/asv", line 8, in <module>
    sys.exit(main())
  File "/tmp/cirrus-ci-build/.nox/benchmarks-ci_mode-true/lib/python3.8/site-packages/asv/main.py", line 38, in main
    result = args.func(args)
  File "/tmp/cirrus-ci-build/.nox/benchmarks-ci_mode-true/lib/python3.8/site-packages/asv/commands/__init__.py", line 48, in run_from_args
    plugin_manager.import_plugin(plugin)
  File "/tmp/cirrus-ci-build/.nox/benchmarks-ci_mode-true/lib/python3.8/site-packages/asv/plugin_manager.py", line 42, in import_plugin
    mod = __import__(name, {}, {}, [], level=0)
ModuleNotFoundError: No module named 'nox_asv_plugin'

Another note: even if we were able to integrate ASV into CI (I hope we can in future), I still don't know how to identify the right commit to compare with the current commit.

Running ASV after merging to trunk is easy enough: asv continuous HEAD^1 HEAD --bench=ci.
Running ASV before merging to trunk should replace the HEAD^1 above with the commit where the PR's branch diverged from the trunk, but any way I can find of doing this requires unavailable information:

Knowing the name of the trunk branch. This will not always be main if the PR targets a feature branch.
Having enough of a Git history to access the trunk branch. AFAICT Cirrus only checks out the PR's branch, with no knowledge of other branches.

Nice work, @trexfeathers ! To your points:
i. I thought we could get this from the github (REST) api, and that would work, but I think it is even easier. Have a look at environment variables in cirrus, specifically the table in that section. There you will find CIRRUS_BASE_BRANCH and CIRRUS_BRANCH. This, together with a git merge-base should be enough to figure it out. Perhaps even CIRRUS_BASE_SHA is enough without merge-base, but it wasn't immediately clear to me if the SHA refers to the merge-base commit or to the head of the base branch.
ii. The document you link actually offers a solution to that: Just overwrite the cloning procedure by putting a clone_script: section in the config.

Thanks @zklaus that's really helpful! @stephenworsley decided to not implement CI right now to avoid further delays, but I'll certainly work with your suggestions when we come back to this later.

CI Benchmarking Problems
It has so far proven impossible to integrate ASV into our Cirrus-based CI:

Using Nox for environment management relies on the plugin nox_asv_plugin.py.
Testing locally shows ASV is able to import this as intended (from the same directory as asv.conf.json).
However Cirrus-CI encounters a ModuleNotFoundError, despite being in the correct directory, and that directory containing nox_asv_plugin.py - see the trace below.

It seems likely that a setting on the Cirrus machine is breaking an assumption within ASV. It may work fine with other providers, e.g. GitHub Actions, but I am reluctant to move away from Cirrus given the intent to leverage compute credits in future. Hopefully it will be possible to fix within Cirrus, however this will require investigation and testing.
I am therefore going to disable CI benchmarking for now, and limit this PR to just providing the architecture for manual ASV runs. 49d556c
Section of Cirrus trace
nox > python -c from pathlib import Path; print(Path().cwd()); print(list(Path().iterdir()));
/tmp/cirrus-ci-build/benchmarks
[PosixPath('benchmarks'), PosixPath('nox_asv_plugin.py'), PosixPath('nox_asv.conf.yaml'), PosixPath('asv.conf.json')]
nox > asv machine --yes
Traceback (most recent call last):
  File "/tmp/cirrus-ci-build/.nox/benchmarks-ci_mode-true/bin/asv", line 8, in <module>
    sys.exit(main())
  File "/tmp/cirrus-ci-build/.nox/benchmarks-ci_mode-true/lib/python3.8/site-packages/asv/main.py", line 38, in main
    result = args.func(args)
  File "/tmp/cirrus-ci-build/.nox/benchmarks-ci_mode-true/lib/python3.8/site-packages/asv/commands/__init__.py", line 48, in run_from_args
    plugin_manager.import_plugin(plugin)
  File "/tmp/cirrus-ci-build/.nox/benchmarks-ci_mode-true/lib/python3.8/site-packages/asv/plugin_manager.py", line 42, in import_plugin
    mod = __import__(name, {}, {}, [], level=0)
ModuleNotFoundError: No module named 'nox_asv_plugin'

@trexfeathers have you tried setting the module name in the asv config file to `".nox_asv_plugin"?  See special handling for relative imports in the asv plugin manager here: https://github.com/airspeed-velocity/asv/blob/ef016e233cb9a0b19d517135104f49e0a3c380e9/asv/plugin_manager.py#L36-L40

@trexfeathers have you tried setting the module name in the asv config file to ".nox_asv_plugin"?

Yes, although only locally. And in the end I didn't need it. Guess that's something to go back to, thanks 🙂
TBH I'm struggling to see through the ~40 hours I spent getting the plugin to work 😂

Thanks @stephenworsley!


flawhud.json --- different values (editor / hud itself)

https://github.com/CriticalFlaw/TF2HUD.Editor/blob/master/src/TF2HUD.Editor/JSON/flawhud.json#L622
https://github.com/CriticalFlaw/flawhud/blob/master/resource/ui/scoreboard.res#L455
For both BluePlayerList and RedPlayerList, the value is 210 in the editor (when false) but 204 in the HUD itself.

Corrected in 2b81fb301336b974afb6419776f3788106313f03. Thanks.


Add opentrends

What this PR does / Why we need it
Add opentrends logo
Some questions

[x] I have read the contributing guidelines
[x] I abide by this repository Code of Conduct

Noticed, I'll check it when I reach home. :smiley:


59564 - [Newsroom] Common Modules - Global Footer - Secondary Footer links should not be highlighted when hovering and missing tooltips

Severity - 3 - Medium
Description of the issue: When hovering on the secondary footer links, it should not be highlighted and tooltip is missing.
Legacy page:
homepage - https://newsroom.accenture.com/
search page - https://newsroom.accenture.com/search.cfm
Migrated page:
homepage - https://main--accenture-newsroom--hlxsites.hlx.live/
search page - https://main--accenture-newsroom--hlxsites.hlx.live/search
Steps:

Access the Newsroom Legacy and Migrated page.
Verify elements  for Footer in both domains.
Hover on the Footer links to check user interactions.

Expected Result:
Secondary Footer links have tooltips and not highlighted when hovering.

Actual:

Azdo link : https://dev.azure.com/accenturecio26/WebContentDigAssetMgmt_PR1093/_workitems/edit/59564
cc: @ravkiran @amol-anand @raymond-h-dev

this is invalid, closing the bug.


fix the source for search domains

What this PR does / why we need it:
This PR fixes the source of DNS search domains, to retrieve the value from IPPool annotations.
Which issue(s) this PR fixes:
Fixes #19

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: abhinavnagaraj
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [abhinavnagaraj]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: abhinavnagaraj, jzhoucliqr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [abhinavnagaraj,jzhoucliqr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/lgtm


Type assertions in setup are not narrowed in template

After upgrading to vue-tsc 2.0.19 previous code no longer works:
const value: number | undefined = 1

if (!value) {
	throw new Error('Undefined')
}

<Component must-be-number={value} />

I get error
'__VLS_ctx.value' is possibly 'undefined'.

Verified working version 2.0.7

especially annoying when injecting a value and throwing if undefined in setup - still getting is possibly 'undefined' for template checks.
const value = inject('value');
if (value === undefined)
  throw new Error('value is undefined');


Use non-isometric smiles

Codecov Report

Merging #237 into master will not change coverage.
The diff coverage is 50%.

@@          Coverage Diff           @@
##           master    #237   +/-   ##
======================================
  Coverage    81.9%   81.9%
======================================
  Files         108     108
  Lines        5249    5249
======================================
  Hits         4299    4299
  Misses        950     950


16444 Use Keycloak roles for LaunchDarkly update

Issue #: bcgov/entity#16444
Description of changes:

app version = 5.2.7
cleaned up a couple of dialogs
renamed some state variables and getters/setters
get user's roles from Keycloak token
used proper roles in LaunchDarkly update
misc cleanup
updated unit tests

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the bcrs-entities-create-ui license (Apache 2.0).

The roles as recorded in LaunchDarkly (and "staff" is one of them):

Thanks for your reviews Karim, Ketaki and Hongjing!


Verify ArgoCD works with "main" branch

I made a pointless change to an endpoint in resources-staging to return a 401 if someone tries to go to the /faketestdestination endpoint. If argocd syncs as expected I will remove this change.

closed, circleci needs to be updated first


Feature/removeDnsReadyCode

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Harter, Nicholas (Contractor) seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

Codecov Report

Merging #42 into master will increase coverage by 1.65%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master      #42      +/-   ##
==========================================
+ Coverage   51.04%   52.69%   +1.65%
==========================================
  Files           6        6
  Lines         478      463      -15
==========================================
  Hits          244      244
+ Misses        223      208      -15
  Partials       11       11

Impacted Files
Coverage Δ

src/caduceus/caduceus_type.go
100% <ø> (+30%)
:arrow_up:

src/caduceus/caduceus.go
0% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f19501d...d5a1c05. Read the comment docs.


[World] Re-order world tabs with drag and drop

Yay!!!!! Thanks for working on drag and drop. Hopefully this is the first of many :)

https://github.com/feenkcom/gtoolkit/assets/249595/171395b9-5230-45cb-b727-d7ba349c66fd

https://github.com/feenkcom/gtoolkit/assets/249595/2a912ef8-cfaa-43dc-a94f-a93194f7a192