Agent: Introduce a minimal DB storing a safe block number to fetch from

Rationale
With increasing chain history, agents will have to fetch a longer and longer list of events. Although this should happen quite rarely (only on sync), especially the increased load on the RPCs can cause trouble (Ratelimiting, increasing cost).
However, most of the events will likely be discarded right away upon processing events and states. In fact, whenever a request is finalized, which means there are all claim stakes withdrawn and deposit is withdrawn as well, then the list of events for this request will effectively do no change in state.
Currently, all state is replayed though. This problem is extensively described in #760 .
While #760 describes the idea to filter after fetching the events, the load on the RPC will stay unaffected. This issue focusses more on fetching from a right point in time and effectively filtering old events. This issue will automatically overrule #760 and thus #760 can be closed.
Solution
Introduction minimal database
The agent needs to store a block number from which all old events can be ignored. Currently the block number is fetched from the deployment folder. Could this value be overwritten or should it be stored somewhere else?
Safe Point Algorithm
Upon sync and fetching all old events the agent could additionally find the block number from which it is safe to start from.
This would mean that every startup will update the block number to fetch from for the next start up.
Optionally, this algorithm could be moved into the runtime environment and updated whenever the oldest request is discarded.
Finding the safe point
We need to think a bit more thoroughly but most likely the safe point is the block number with the oldest not yet withdrawn RequestCreated event where it is still possible to claim or there are active claims ongoing. Most likely, if the agent saw it's own ClaimStakeWithdrawn event, the claim can be ignored and thus the request if it is not possible to claim anymore.
Additional events
For additional events we need to fetch from the beginning, but we could filter for the specific topics. This could be i.e. finalityPeriod events or at a later point fee changes.
Partial request events
Most likely a safe point will result in ongoing requests being split up (RequestCreated event comes before the safe point, rest of the events will be fetched).
The agent needs to be able to handle such a situation upon sync.

Thanks for creating this.

The agent needs to store a block number from which all old events can be ignored. Currently the block number is fetched from the deployment folder. Could this value be overwritten or should it be stored somewhere else?

I think the deployment files should be considered immutable i.e. read-only. So I would rather store the block number
somewhere else.

Upon sync and fetching all old events the agent could additionally find the block number from which it is safe to start from.
This would mean that every startup will update the block number to fetch from for the next start up.
Optionally, this algorithm could be moved into the runtime environment and updated whenever the oldest request is discarded.

I'm not sure what you mean by "runtime environment" (perhaps "everything after all chains are synced"?), would like to know more about "moving the algorithm" part.

For additional events we need to fetch from the beginning, but we could filter for the specific topics. This could be i.e. finalityPeriod events or at a later point fee changes.

Hm, yes. I wonder if we could somehow get around filtering everything.
In case an event holds entire state data, one could do "backwards windowed search" like

let start block number = S, end block number = E, window size = W (W << S - E)
initialize END = E
events = filter_by_topic(END - W, W) # find all events in blocks [END - W, W]
if events nonempty, return events
else, set END = END - W - 1 and go to 3

However, I do not know enough about RPC filter performance in order to say whether something like this would be
worth the complexity -- it could be that filtering is really fast. In any case, this could probably only be helpful for
huge block ranges.
Another, much better idea, is to just get the contract state directly -- e.g. the agent knows which tokens it needs
so may look up the fee info directly in the contract. I imagine that would be the cheapest and simplest solution
for such cases. Similar for finality period. Not sure what other events are relevant.

Most likely a safe point will result in ongoing requests being split up (RequestCreated event comes before the safe point, rest of the events will be fetched).
The agent needs to be able to handle such a situation upon sync.

Good point. Can we just ignore such events? Assuming a good definition of a safe point, we should not care about those events anyway, right?

Finding the safe point

Eh, I feel this should have its own issue. (Joking, but also maybe not.... :sweat_smile:)

We need to think a bit more thoroughly but most likely the safe point is the block number with the oldest not yet withdrawn RequestCreated event where it is still possible to claim or there are active claims ongoing. Most likely, if the agent saw it's own ClaimStakeWithdrawn event, the claim can be ignored and thus the request if it is not possible to claim anymore.

Let me outline my thoughts.
In order for a safe point algorithm $A$ to be useful, I think we need two things:

$A$ needs to be correct, in the sense that an agent that started syncing from a safe point determined by the algorithm cannot lose money due to not seeing events that occurred before the safe point
$A$ needs to guarantee progress, in the sense that, given a safe point $S_1$ generated at time $t_1$, there is a finite constant $dt$ such that $A(t_1 + dt) > S_1$

I think I understand how your above proposal satisfies 1., but it is not obvious to me that it satisfies 2.


Nuxt app is not recompiling in Docker container on Windows in dev mode

Versions

nuxt: ex: v2.15.3
node: lts-alpine docker image (Currently Node 14.16)

Reproduction
Code is here: https://github.com/dschonholtz/NuxtPersonalWebsite
It is a pretty simple version of the blog with nuxt example, based on the nuxt content example.
https://nuxtjs.org/blog/creating-blog-with-nuxt-content/
The only differences really should be some upgraded versions, my own amateurish styling, and content.
I took that I added a Dockerfile, Dockerfile.dev and docker-compose.yml all of which are pretty simple and work great on my linux desktop.
When it try to do development on my windows laptop and run docker-compose up and I go into the associated container. I can see file changes propagate properly so the volumes are working.
However, when I change a file, the app is not recompiled despite the change happening in the docker container.
This leads to no hot-reload which means the only way to test changes on my windows environment is to run node and everything else outside of the docker container (Which is not something I am particularly interested in)
There are a couple of issues dealing with similar problems:
#2481 - Hot Reloading doesn't work in docker container on windows.
Which recommends that you put a watchers element in the top level of the nuxt.config.js https://github.com/nuxt/nuxt.js/issues/2481
I tried all of the variants described their and read the associated nuxt and webpack documentation and didn't get any closer to a working solution.
Another thread that seemed relevant was https://github.com/nuxt/nuxt.js/issues/5550
However, their the app was recompiling just not refreshing the page so I believe it is different.

Additional Details

[Nuxt Config](https://github.com/dschonholtz/NuxtPersonalWebsite/blob/main/nuxt.config.js)
[Dockerfile.dev](https://github.com/dschonholtz/NuxtPersonalWebsite/blob/main/Dockerfile.dev)

Steps to reproduce

On a windows machine with docker installed
git clone https://github.com/dschonholtz/NuxtPersonalWebsite
docker build -t volume_tester -f Dockerfile.dev .
run run --rm -it -p 3000:3000 -v ${pwd}:/app volume_tester
Change a file locally
Go into the dockerfile and verify the file changed
Cry deeply on observing the page not updating

What is Expected?
The change should cause the app to recompile and hot reload neither of which happens
What is actually happening?
The app should recompile and hot reload.
Finally, thank you so much for reading.
I am trying to follow the instructions to the letter since I'm new to the community and a relative novice with Docker and Nuxt.
So if I am making any obvious mistakes please let me know. I would greatly appreciate any help and I hope it helps someone else!

I'm running into the same exact issue, but I'm using DDEV and running Docker on a macOS (10.15). If I run dev within the Docker environment it doesn't recompile or HMR, but when I run it from the OS, it works just fine.

I'm pretty sure I had tried that fix and it hadn't worked for me. Regardless, I have been doing all of my dev on a linux box recently so I haven't been hitting this. Thanks for the follow up.

Same issue here, Windows too

same problem, in windows, docker and nuxt.
windows: win10
docker: Docker Desktop 4.4.4
docker image: nodejs: 16.15.0 , nuxt: 2.15.8

Same problem, in Ubuntu, docker and nuxt.

Same issue, in WSL2 Ubuntu, docker and nuxt.

+1

thank me later guys
add this to nuxt.config.ts
 //fix issue for docker not reflecting changes
  vite: {
    server: {
      watch: {
        usePolling: true,
        interval: 100, // Poll files every 100ms
      },
    }
  }

@lafllamme thanks its working fine for me :)


Improve Buffer and Slice:

Make Buffer constructible from a Slice
Fix self-move-assignment in Buffer
Add [c]begin/[c]end

Current coverage is 64.24% (diff: 100%)

Merging #1841 into develop will decrease coverage by 0.03%

@@            develop      #1841   diff @@
==========================================
  Files           704        704
  Lines         50786      50788     +2
  Methods           0          0
  Messages          0          0
  Branches          0          0
==========================================
- Hits          32645      32629    -16
- Misses        18141      18159    +18
  Partials          0          0

Powered by Codecov. Last update 87756b9...25097bf

Other than the one question you raised I didn't spot any issues.
Just since no good deed should go unpunished, would it be worth adding unit tests for Buffer and Slice?  Not insisting, just wondering.

I'd be happier about this if there were some Buffer and Slice unit tests, particularly since a lot of the new code is uncovered. Codecov appears to ignore those unused functions instead of red-lining them.

https://codecov.io/gh/ripple/rippled/src/25097bf0da016a60c07467a77d2326562612ac16/src/ripple/basics/Buffer.h (Copy ctor and assignment, Slice ctor and assignment, begin, & end.)

Otherwise, looks good. :+1:

begin and end are lame on memory buffers

They make them usable with std algorithms. We can remove them, but the alternative is uglier, imo:
std::find(b.data(), b.data()+b.size(), ...);

No range-for, etc.

When would you use std::find on a memory buffer?

It was an example. There are other things one may want to do - most of the algorithms in <algorithm> are very useful and most require "iterators". Only few have _n variants.
Since the start and end pointers are frequently needed, and this adds exactly zero overhead compared to getting a pointer and a size, why not have them?
Range-for iterating a buffer is frequently done. Certainly PublicKey and SecretKeydo it a lot when converting to different formats, encoding, decoding, etc.

Suggested discussion topic: Small-Buffer optimization
Curious about Buffer sizes, I ran the unit tests with Buffer constructor dumping the size.
(This does not capture the sizes of default constructed Buffers which are assigned afterwards.)
(I removed test ripple.protocol.STObject parse json array which constructs Buffer sizes 0...999.)
The most common size is 33 bytes.
The vast majority of sizes are less than 72 byte with a very few at size 255, 256 or 257.
Is this likely to be true of rippled production runs?
If so then this would suggest a Small Buffer optimization with static storage of 72 bytes
(or maybe a few sizes, 36, 72, 260 and remove the default constructor).
The current Buffer is just a sized handle to dynamic allocated storage, good for large buffers with move. It also handles varying size simply via assignment or reallocation (I don't see the alloc member function used anywhere).
Small Buffers are cheap to copy and with RVO some copies can be elided. Size can be varied up to a static capacity limit. Growth beyond the static limit is only possible by falling back to allocation, a-la small string optimization, but it looks like this complexity is not necessary as the current design has no high level support for growth. So it looks like a static limit is acceptable in practice.

Small Buffers could be based on (or just alias) boost::static_vector, Docs a useful array wrapper.
This gives the illusion of being a vector up to the static capacity, providing most vector API including begin,end (c.f. the review comments above on using algorithms on Buffers).
The results are extracted and plotted in this notebook:
https://gist.github.com/willwray/f888fdb3ac48b13bd9f7a4a3fda2499d
Here's a plot from the notebook (is this viewable here?)

@vinniefalco: I took your suggestion and removed begin and end from both Slice and Buffer.
@ximinez: I took your suggestion and added some unit tests for Buffer; it should have 100% coverage now.

Love the buffer tests.  Creative solution with the assert on buffer assignment from slice as well.  Thanks.  You may choose to fix up a few comments in the new tests, but not a killer.  :+1:

@ximinez: I've added the extra unit test coverage for clear and  and operator(0)

:+1:

Rebased & squashed.

Merged as 4b0d8b630c71215a8721685c685f76c1124df246


Send ChannelUpdate with Disabled bit if channel inactive for 20 minutes

Fixes #1605
Note:
server is not very unit testable at the moment. Should consider extracting some of the logic behind interfaces to make it easier to test additions like these.

Will start to run this on the faucet!

This doesn't build as is.

Compilation issue fixed, and squashed.

Needs a rebase, now has a switch conflict.

Rebased.

Testing this right now on the faucet, and it continually tries to disable channels when their status hasn't yet changed. Running with an inactive time of 1m, so it'll check every 20s:
5 2018-08-15 23:39:28.227 [DBG] SRVR: Announcing channel(f9db18a030f69efc97bfb176fbd41df24542330576ebb54911dac069e329d769:0) disabled=true
  4 2018-08-15 23:39:28.228 [ERR] SRVR: unable to disable channel: edge not found
  3 2018-08-15 23:39:28.231 [DBG] SRVR: Announcing channel(a404288329a5b0dac1d9bb0749647f8fd741e43cd7d0c52f1df6d3f21c685add:0) disabled=true
  2 2018-08-15 23:39:28.232 [ERR] SRVR: unable to disable channel: edge not found
  1 2018-08-15 23:39:28.232 [DBG] SRVR: Announcing channel(010d364d2de9c26f643f16b3b6c5a3070c0961a7935990440a2f3693cf3ddd46:0) disabled=true
  0 2018-08-15 23:39:28.233 [ERR] SRVR: unable to disable channel: edge not found
  1 2018-08-15 23:39:28.233 [DBG] SRVR: Announcing channel(c5c1e475c6ff73b8a96e7fbfe765b7f0e88e21ac902590bdb4094cded48bb385:0) disabled=true
  2 2018-08-15 23:39:28.233 [ERR] SRVR: unable to disable channel: edge not found
  3 2018-08-15 23:39:43.164 [DBG] SRVR: Announcing channel(f9db18a030f69efc97bfb176fbd41df24542330576ebb54911dac069e329d769:0) disabled=true
  4 2018-08-15 23:39:43.164 [ERR] SRVR: unable to disable channel: edge not found
  5 2018-08-15 23:39:43.165 [DBG] SRVR: Announcing channel(a404288329a5b0dac1d9bb0749647f8fd741e43cd7d0c52f1df6d3f21c685add:0) disabled=true
  6 2018-08-15 23:39:43.165 [ERR] SRVR: unable to disable channel: edge not found
  7 2018-08-15 23:39:43.165 [DBG] SRVR: Announcing channel(010d364d2de9c26f643f16b3b6c5a3070c0961a7935990440a2f3693cf3ddd46:0) disabled=true
  8 2018-08-15 23:39:43.165 [ERR] SRVR: unable to disable channel: edge not found
  9 2018-08-15 23:39:43.165 [DBG] SRVR: Announcing channel(c5c1e475c6ff73b8a96e7fbfe765b7f0e88e21ac902590bdb4094cded48bb385:0) disabled=true
 10 2018-08-15 23:39:43.165 [ERR] SRVR: unable to disable channel: edge not found
 19 2018-08-15 23:39:52.325 [DBG] HSWC: ChannelLink(bdb32cc0d987bde6da8332ecb786b5e5bbccc4f48c117b1f4a10fc4ff4859140:1): sampled fee rate for 3 block conf: 125000 sat/kw
 20 2018-08-15 23:39:58.234 [DBG] SRVR: Announcing channel(a404288329a5b0dac1d9bb0749647f8fd741e43cd7d0c52f1df6d3f21c685add:0) disabled=true
 21 2018-08-15 23:39:58.235 [ERR] SRVR: unable to disable channel: edge not found
 22 2018-08-15 23:39:58.235 [DBG] SRVR: Announcing channel(010d364d2de9c26f643f16b3b6c5a3070c0961a7935990440a2f3693cf3ddd46:0) disabled=true
 23 2018-08-15 23:39:58.235 [ERR] SRVR: unable to disable channel: edge not found
 24 2018-08-15 23:39:58.235 [DBG] SRVR: Announcing channel(c5c1e475c6ff73b8a96e7fbfe765b7f0e88e21ac902590bdb4094cded48bb385:0) disabled=true

Those attempts to disable channels appear to be for non-advertised channels. We should skip these all together. But then a greater question is why it's unable to find an edge for those. Since the recent migration, we're a bit stricter about expectations w.r.t edge policies being in the database. It seems as though the migration should've also picked these up and morphed them into unknown edge policies? cc @joostjager

Other than those two things, I see it properly disabling inactive channels.

Actually scratch that, for those channels they have a chan ID of zero, which means that they were somehow never inserted into the graph at all. Possibly an issue with the funding manager not reliably sending the announcements to the gossiper.

We should def find the greater issue here, but one work around would be to have the funding manager check for channel open on start up, or which have recently transitioned state, and re-insert them into the graph if we find that they're unknown.

Hit this panic while running the patch:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0xcf4c1a]

goroutine 2528 [running]:
main.(*server).fetchLastChanUpdateByOutPoint.func1(0x0, 0xc421183650, 0x21, 0x21)
        /root/gocode/src/github.com/lightningnetwork/lnd/server.go:2942 +0x2a
main.(*server).fetchLastChanUpdateByOutPoint(0xc42058c340, 0x56e8d11296769b2, 0x13753c248299d43f, 0xbc354e16e1bb81a4, 0x5b5ecbfcac30b18f, 0x0, 0xc425c93840, 0xc4202aa9a0, 0x4e)
        /root/gocode/src/github.com/lightningnetwork/lnd/server.go:2958 +0x14c
main.(*server).announceChanStatus(0xc42058c340, 0x56e8d11296769b2, 0x13753c248299d43f, 0xbc354e16e1bb81a4, 0x5b5ecbfcac30b18f, 0x100000000, 0x0, 0x0)
        /root/gocode/src/github.com/lightningnetwork/lnd/server.go:2877 +0x262
main.(*server).watchChannelStatus(0xc42058c340)
        /root/gocode/src/github.com/lightningnetwork/lnd/server.go:3079 +0x615
created by main.(*server).Start
        /root/gocode/src/github.com/lightningnetwork/lnd/server.go:999 +0x2dc

I took another look at the gossiper code, and there're a few areas where forget to send an error back via the main channel. Generally I think we should discourage sending errors via channels back to callers within a method, in favor of instead returning the error, and having the caller pass back that error to its caller in a single place.

@Roasbeef I had a quick look at
[ERR] SRVR: unable to disable channel: edge not found
Could reproduce this on simnet. Open private channel and then force close.
I don't think it has to do with the db migration / unknown edge policies. The error message maybe somewhat misleading. In my test it originates here:
https://github.com/lightningnetwork/lnd/blob/90cdc9da8f60dd94d99d61fd9e86a7ffd39b3b1b/channeldb/graph.go#L2347
Mapping from outpoint to channel id is unavailable.

Turned out the reason we were trying to disable channels where edges were not found in the DB was because closed channels would be lingering around in the status map. Fixed by only copying the status of open channels on each tick.

@joostjager thanks for that report! Led me to finding an existing bug which is now fixed in #1739.

Alrighty, with the latest fixups, giving this another spin on the faucet. Will report back with any comments/reports, then likely green light it for inclusion, as this should dramatically reduce the number of routing failures on mainnet/testnet once folks update.

I think the faucet (which is suuper old) is in a weird state itself (those channels have a chanID of zero). Will try this on a newer node. However, the minor issue of it continually re-trying to disable those channels when it's unable to actually read them is still present.


Issue with variable overrides

Overriding some variable default only affect that variable, and does not affect other variables using that variable.
For example here's a snippet from _variables.scss
$primary:       $blue !default;
.
.
.
$link-color:    theme-color("primary") !default;

Now if I override the variable $primary then it will not reflect in $link-color.
This will not work
@import "bootstrap/scss/functions"; // Required
@import "bootstrap/scss/variables"; // Required
@import "bootstrap/scss/mixins"; // Required

//
// Override Bootstrap here
//
$primary: red;

Direct override will work
@import "bootstrap/scss/functions"; // Required
@import "bootstrap/scss/variables"; // Required
@import "bootstrap/scss/mixins"; // Required

//
// Override Bootstrap here
//
$primary: red;
$link-color: red; // All other elements using $primary will still appear $blue

Issue appears to be inherent in bootstrap, and not with this starter template.

theme-color() functions aren't using the variables, they're pulling from the $theme-color map. It does appear that Sass doesn't update a map's values if the default value of one of its items is updated. Wonder if this is a bug in our implementation in Bootstrap 4, or just a limitation of Sass. Either way, we changed this in v5 so that it's easier to modify things (no more theme-color() function usage), but for now, we might be a little stuck with it in v4.x.
Reproducing the problem here: https://www.sassmeister.com/gist/89886b1f1917dfd5307df1c315e8ef67

Experimenting more, it seems the solution is to remove the default primary key from the original Sass map, then add a new one via map-merge().
Check this for now: https://www.sassmeister.com/gist/8e0efb7fc7c6d6d2d55df87afe4e30c2

This isn't just the case with just maps, this is happening with variable values as well
For example
$font-size-base:              1rem !default;
$input-btn-font-size:         $font-size-base !default;
$btn-font-size:               $input-btn-font-size !default;
$input-font-size:                       $input-btn-font-size !default;

Changes in $font-size-base is not reflecting in other variables (Tested on v5 branch)

Drop the !default from those replacement values and they should work.

Sorry that was just the unchanged snippet from _variables.scss to show that no map is being used for $font-size-base
Here's the updated starter.scss
@import "bootstrap/scss/functions"; // Required
@import "bootstrap/scss/variables"; // Required
@import "bootstrap/scss/mixins"; // Required

//
// Override Bootstrap here
//
$font-size-base: 2rem;
// $btn-font-size: 2rem; // Will not work without directly overriding it

The button font size remains unaffected

So it turns out the required imports should be after the overrides
//
// Override Bootstrap here
//
$font-size-base: 2rem;

// Required
@import "bootstrap/scss/functions"; // Required
@import "bootstrap/scss/variables"; // Required
@import "bootstrap/scss/mixins"; // Required

So I guess it's a matter of documentation.
starter.scss and bootstrap docs https://getbootstrap.com/docs/5.0/customize/sass/#importing needs to be updated.

Your right, the required imports makes more sense to be after the overrides. I haven't had any issues with v5 doing this. Adding or removing maps do need to come after though since Bootstrap doesn't use map-merge after checking if a map already exists with !default.
The docs do seem to be a bit confusing. I initially thought the point of !default is to define a variable with a set value if it doesn't already exist or equal null. So don't know why any other reason our custom variables need to be included after Bootstrap's unless you want to reference Bootstrap's variables in your own but then other variables don't get updated.
Sass doesn't update variables if the one it's linking/referencing to changes as they are not actually referenced. It's more like they are cloned or copied values rather than shared values. For example:
// defaults
$a: 111 !default;
$b: $a !default;

// Variable Override
$a: 999;

.mycss-A { width: $a; } // 999
.mycss-B { width: $b; } // 111

As seen here, $b was not updated when $a changed and still returns 111 even though $a had been overridden.
In Bootstrap's case, the problem starts in @import "bootstrap/scss/variables" this is happening:
// @import "bootstrap/scss/variables"
$blue:  #0d6efd !default;
$primary: $blue !default; // => #0d6efd
$link-color: $primary !default; // => #0d6efd
$font-size-base: 1rem !default;
$input-btn-font-size: $font-size-base !default; // => 1rem
$btn-font-size: $input-btn-font-size !default; // => 1rem

// Custom variables
$primary: red;
$font-size-base: 2rem;

// @import "bootstrap/scss/buttons"
.btn {
.
.
  @include button-size($btn-padding-y, $btn-padding-x, $btn-font-size, $btn-border-radius);
}

// @import "bootstrap/scss/reboot"
a {
  color: $link-color;
}

// CSS Output
.btn {
.
.
   font-size: 1rem;
}

a {
  color: #0d6efd;
}

Here, $btn-font-size was not updated because it was already set to 1rem before $font-size-base was overridden. The same goes for $link-color.
If you want to include custom variables after Bootstrap's variables like the docs state, then you'll need to update every other variable that uses $font-size-base so the changes are reflected and remember to remove all !default tags but doesn't that make the code more messier and less manageable?
$font-size-base: 2rem; // set new font size

// update every other variable that uses $font-size-base
$font-size-sm:  $font-size-base * .875;
$font-size-lg: $font-size-base * 1.25;
.
.
.
$input-btn-font-size: $font-size-base;
$btn-font-size: $input-btn-font-size;
.
.
.

The other option is to include your custom variables after the functions so you can use them in your code but before  Bootstrap's defaults as these defaults will only apply if they haven't already been set in your custom variables. For example:
@import "bootstrap/scss/functions"; // Required

//
// Override Bootstrap defaults here
//
$font-size-base: 2rem;
$primary: red;

@import "bootstrap/scss/variables"; // Required ($primary and $font-size-base will replace the default)
@import "bootstrap/scss/mixins"; // Required

//
// Make any changes to Bootstrap's maps here (add / remove)
//
$custom-color: "#fe669e";
$theme-colors: map-merge(
  (
    "custom": $custom-color
  ),
  $theme-colors
);

!defaults in custom variables will still replace Bootstraps defaults and that makes it easier for others to make child themes without altering the main theme built on Bootstrap.
There doesn't seem to be an issue with Bootstrap, just the way Sass reads variables rather than across the entire project before compiling the CSS. Something I think LESS does? There doesn't seem to be a better way around it.

There's another issue with required imports after variable overrides in v5 (works fine on v4).
btn-outline-* shows black text color on hover.
starter.scss
// Configuration
@import "bootstrap/scss/functions";

//
// Override Bootstrap here
//
$primary: red;

@import "bootstrap/scss/variables";
@import "bootstrap/scss/mixins";
@import "bootstrap/scss/utilities";

index.html
<button type="button" class="btn btn-outline-primary" data-bs-toggle="modal" data-bs-target="#exampleModal">Click me!</button>

Without Hover
With Hover

I haven't tested much with required imports after variable overrides, so more issues are expected.

That issue with btn-outline-* showing red with black text seems to be an issue with the color-contrast($color) function not the required import.
Testing this code shows the issue:
$my-color: red;

.my-text {
  background-color: $my-color;
  color: color-contrast($my-color); // => #000
}

The color-contrast function is for some reason returning black. You could lower the contrast ratio but I know there'll be some people who would be against that probably. Setting $min-contrast-ratio to 3.9 or lower instead of the default 4.5 makes the text white. Black on red is certainly not readable.
In v5.0.0-alpha2 the color contrast ratio was changed from 3:1 to 4.5:1
https://getbootstrap.com/docs/5.0/migration/#colors
// Configuration
@import "bootstrap/scss/functions";

//
// Override Bootstrap here
//
$primary: red;
$min-contrast-ratio: 3.9;

@import "bootstrap/scss/variables";
@import "bootstrap/scss/mixins";
@import "bootstrap/scss/utilities";

Issue with color-contrast at "bootstrap/scss/mixins/_buttons.scss" Line 77 $color-hover: color-contrast($color)
Seems to work with other colours like blue but not red. I don't know if this is something that should be reported as an issue on the main Bootstrap repository?

Thank you for the explanation. Yes I don't think this needs to be reported, overriding color-contrast or updating custom theme colors is simple. But the documentation needs to be updated.

I think the only solution would be to split the default variable settings into dependency levels and then let clients insert their overrides at the right place.
One way to do this would be to have separate scss files / imports:
// overrides for base variables ($white, $blue, ..., $spacer, ...)
@import "variables";
// overrides for dep-1 variables ($primary, $secondary, ..., $blue-100, ..., $spacers, ...)
@import "variables-dep-1";
// overrides for dep-2 variables ($link-color, ...)
@import "variables-dep-2";
// overrides for dep-3 variables ($link-hover-color, ...)
@import "variables-dep-3";

With the !default flag, there is some leeway here:
An assignment like $red:     #c00; works before or after $red:     #dc3545 !default; , but it must come before any reference to $red.
Another solution would be to provide hook functions in _variables.scss that allow clients to insert their tweaks in the right place:
...
$cyan:    #0dcaf0 !default;
// scss-docs-end color-variables

@use "sass:meta";
@if meta.function-exists("variables-hook-1") {
  $dummy: meta.call(meta.get-function("variables-hook-1"));
}

Clients would then define their overrides like this:
@function variables-hook-1() {
  $red:     #c00 !global;
  $primary:       $red !global;
  $danger:        $orange !global;
  @return null;
}
@import "bootstrap";

Unfortunately, this spews a compile-time message, and I have not found a way to silence it:
DEPRECATION WARNING: As of Dart Sass 2.0.0, !global assignments won't be able to
declare new variables.

I'm currently working on a project where a modified $red is the primary color, and $orange the danger color. With v5.0.0-beta2, I had to duplicate the assignment of $orange:  #fd7e14;:
$red:     #c00;
$orange:  #fd7e14; // same as default, but need to pre-define here, so we can use it to override $danger below
$primary:       $red;
$danger:        $orange;
@import "bootstrap";

Unfortunately, using @use At-Rules is only supported by compilers using the latest Dart Sass. It will not work with other compilers. Introducing @use in Bootstrap would break a lot of peoples code.

Here, we @use bootstrap with the namespace bootstrap as we want to keep them separate and don't want any conflicts. We also want to use bootstraps own variables, functions and mixins in our own too. Example: bootstrap.$purple-500 or bootstrap.shift-color()
We then override the defaults with our own before importing bootstraps required imports.
However, like I mentioned. Not all compilers support the @use At-Rules. Only Dart Sass currently supports it. Applications such as Koala, Visual Studio and Hugo, ones I have tested and regularly use, don't work. Others include, Visual Studio or any that use LibSass and RubySass don't work either.
Node.js does support this, however.
Due to its lack of compatibility, it's not a solution for everyone, even if it looks cleaner.
https://sass-lang.com/documentation/at-rules/use

@StudioKonKon @use work only in dart ((
problem in current architecture:

"all in" configuration
hidden references between variables/components, missed arguments at calling functions using generated values - I change option and see changes in an unexpected places
current vars don't used on maximum - I see $space / 2 instead get-map($spaces, 2), $border-* ignored  and etc.
missed options/variables - $enable-shadows does not affect box-shadows used for focus states, but no $enable-shadows-on-select to disable in any case
attribute values ignored in conditions  - I set $enable-shadows: true; and $*-text-decoration: none; for some elements, but  conditions don't check it and generate warnings

Right way:

Separate packages

Required global package with common mixins, variables like as $enable-* and @import optional packages.
// re-declared variables

@import "global/variables";
@import "global/mixins";

$packages: typography, ... !default;
@each $package from $packages {
    @import "packages/#{$package}/package";
}

Packages contains optional components/parts, can use global variables/mixins, contain config and documentation,
Package "Typography":

typography/

_variables.scss - common variables, all variables must be with package prefix (typography-)

// `$%package%-components` variable is required and contain list of components to import
$typography-components: reboot, ... , figures !default;
...
$typography-var: null !defined;

_mixins.scss - inner mixins
_package.scss - @import variables, mixins and components by $typography-components($%package%-components), contain info about required packages and globals
docs/ - documentation and examples
components/ - components generates styles

_reboot.scss - component "reboot"
_figures.scss - component "figures"

"step" variables with increment values:

// global/_variables.scss
$step-size: .25 !default;
$step-size-sm: .125 !default;
$step-size-lg: .5 !default;

$font-size: 1rem !default;
$font-size-sm: $font-size - $step-size !default;
$font-size-lg: $font-size + $step-size !default;

// typography/_variables.scss
$typography-font-size-base: $font-size + 1 !default;
$typography-font-size-step: $step-size-sm !default;

// typography/_headers.scss
@for $i from 1 through 6 {
  h#{$i}, .h#{$i} {
    font-size: $typography-font-size-base + ($typography-font-size-step * $i);
  }
}

Oh! my goodness!
I have spent hours today tinkering with customizing variables like blue primary and everytime i come across some or other error.
i tried following bootstrap theming guide option A and option B thinking that is the correct way, none of them.
finally when i do something like
$blue : green;
$colors: ( "blue" : $blue);
$theme-colors: (
  "custom-color": #900
);
@import 'node_modules/bootstrap/scss/bootstrap';

it works but i am keep going to back to recommended options only to get compile errors...finally this thread saved me :D 🤦

After reading this issue I am still not sure if there is / which is the right solution for overriding variable defaults in Bootstrap 5.0…
I've noticed that in the docs (https://getbootstrap.com/docs/5.0/customize/sass/#variable-defaults) the example doesn't reflect what is written in the paragraph above it.
Which one should I follow?
Thank you!

@dgobnto v5's docs will be updated with the next release, which is coming in the next week or two.

@mdo And in the meanwhile, should I follow the code example (functions > overrides > variables > mixins) or the description (functions > variables > mixins > overrides)?

still not yet updated docs? Which one we should follow, starter kit or docs? docs however said two different things.

v5 docs were updated when v5 stable shipped: https://getbootstrap.com/docs/5.0/customize/sass/#variable-defaults. v4.x may need more updates but we've prioritized v5 patches.

I'm sorry if it's just me being dumb but from what I can see, the docs remain confusing.
Although the text says

Variable overrides must come after our functions, variables, and mixins are imported

the example code just bellow follows another order, namely: functions > variable overrides > variables > mixins.

Oh no, you're not being dumb, I can see the text is wrong, but the code snippet was updated. We updated the snippets, but missed the text in the docs: https://github.com/twbs/bootstrap/commit/6ed439ff7326f3912d9cbb0db9c5ec9e29dc9381#diff-c7298b94458b9d5c93aee66865a7492e32457939be4317ea16958b47a43b732e.
I'll get a fix in for v5.0.2.

@mdo Thanks for fix, I mean that part was confusing.
Anyway why this starter kit set custom variables before functions?
https://github.com/twbs/bootstrap-npm-starter/blob/main/scss/starter.scss#L28-L35


cellxgene-schema must rename ethnicity and ethnicity_ontology_term_id and add "multiethnic"

Also see cellxgene-schema must perform basic 2.0.0 -> 3.0.0 migration for updating existing datasets.
This is a simple rename in the code base:

ethnicity -> self_reported_ethnicity
ethnicity_ontology_term_id -> self_reported_ethnicity_ontology_term_id

with additional validation and labeling requirements for a new value that is allowed - "multiethnic":
self_reported_ethnicity_ontology_term_id

Key
self_reported_ethnicity_ontology_term_id

Annotator
Curator

Value
categorical with str categories. If organism_ontolology_term_id is "NCBITaxon:9606" for Homo sapiens, this MUST be either a HANCESTRO term, "multiethnic" if more than one ethnicity is reported, or "unknown" if unavailable. Otherwise, for all other organisms this MUST be "na".

self_reported_ethnicity

Key
self_reported_ethnicity

Annotator
Data Portal

Value
categorical with str categories. This MUST be "na", "multiethnic", or "unknown" if set in self_reported_ethnicity_ontology_term_id; otherwise, this MUST be the human-readable name assigned to the value of self_reported_ethnicity_ontology_term_id.

Pending question of whether to add "unreported" as a value - differentiate real “unknowns” vs non-reported ethnicity or ancestry. Currently unknown means unreported for ~90% of the cases.

Here's the test cases I ran for this field, and it looks good to me:

Test Case
Expected Result
Obtained Result
Test File
Needs attention

have "multiple" as ethnicity for human
Fail validation
Failed validationLogging: ERROR: 'multiple' in 'self_reported_ethnicity_ontology_term_id' is not a valid ontology term id of 'HANCESTRO'. When 'organism_ontology_term_id' is 'NCBITaxon:9606' (Homo sapiens), self_reported_ethnicity_ontology_term_id MUST be either: a term id of 'HANCESTRO', 'multiethnic' if more than one ethnicity is reported, or 'unknown' if unavailable.
adata_ethnicity_multi.h5ad

have "multiethnic" as ethnicity for human
Pass validation
Pass validation
adata_ethnicity_multiethnic.h5ad

have "multiethnic" as ethnicity for mouse
Fail validation
Failed validationLogging: ERROR: 'multiethnic' in 'self_reported_ethnicity_ontology_term_id' is not a valid value of 'self_reported_ethnicity_ontology_term_id'. When 'organism_ontology_term_id' is NOT 'NCBITaxon:9606' (Homo sapiens), self_reported_ethnicity_ontology_term_id MUST be 'na'.
adata_ethnicity_mouse.h5ad


Username and Password never used!

While creating qBitTorrentClient constructor asks for url, username and password but username and password is never used in code. Especially in this.auth.login() method needs to use those in order to login qBitTorrent Web UI API. I assume Author only tested within localhost and Bypass authentication for clients on localhost toggled on.

Yes, your assumption is entirely correct 😬
A bit silly that I never actually implemented this. I'll update the code ASAP.

Fixed in<EMAIL_ADDRESS>


Events broken

This has been broken since 0.2.0, it works in 0.1.3
use bevy::{app::ScheduleRunnerPlugin, app::AppExit, prelude::*};
use std::time::Duration;

fn main() {
    App::build()
      .add_plugin(ScheduleRunnerPlugin::run_loop(Duration::from_secs_f64(1.0)))
      .add_system(update.system())
      .run()
}

pub fn update(mut shutdown: ResMut<Events<AppExit>>) {
  println!("exiting...");
  shutdown.send(AppExit);
}

What should happen:
The app should exit
What happens:
The ScheduleRunnerPlugin stops controlling frame intervals, and the app doesn't exit.

Thanks for reporting! This is fixed on master branch #536

oh nice! I missed that. Thanks!


Nuget package for 6.0.8 is reporting wrong version

When adding package from nuget in Visual Studio the package is reporting v<IP_ADDRESS> causing the solution not to compile.

Are you sure you're not adding one of the 7.0 betas?

It's a nuget on Visual Studio 2015 RC Bug. I'll report to it's team. In VS 2013 Everything is working as expected. :)
Thank you.


Cannot connect to Huawei WS7100

I'm trying to integrate with my Huawei WiFi AX3 Dual-core WS7100
It is set up on <IP_ADDRESS>
I can easily go by that address, type my password and log in.
But providing the same password, url default username admin and default port results for me in "Can not authenticate." message.

Hi!
Is your Home Assistant server connected directly to the AX3 router?

Actually, I use a similar connection scheme and also have to solve a number of issues :)
My Home Assistant server is connected to a Mikrotik router. Huawei Mesh 3 routers are also connected to the same router.
Without using a clever trick, I would also not have access to the Huawei router from Home Assistant (the Huawei router perceives the Home Assistant server as something in the WAN, and, accordingly, does not allow access to admin panel).
The tricky trick is that the Raspberry Pi 3 is connected to the Huawei router, on which the reverse proxy server is configured (HAProxy, but you can use anything). The port forwarding is configured on the router itself so that requests from the "WAN" (in fact, from the Mikrotik router and other devices, including the smart home server) are redirected to the Raspberry Pi.

HAProxy on Raspberry Pi redirects incoming requests to it back to the Huawei router, but this time the router already believes that these requests are coming from a trusted LAN network, and allows access to admin panel.
When setting up integration, you need to specify the address of the Huawei router from the point of view of the network in which the Home Assistant server itself is located.
So, for example, my Huawei router has a LAN address of <IP_ADDRESS>, but in the network to which the Home Assistant server is connected, its WAN address is <IP_ADDRESS>.
In the component configuration, I will have to specify exactly <IP_ADDRESS>.
Visually, the connection diagram looks like this:

Part of the HAProxy configuration:
frontend http-80
  mode http
  bind *:80

  acl IsMeshRouter hdr_reg(host) ^mesh\.router\.home$ # Home DNS name
  acl IsMeshRouter hdr_reg(host) ^192\.168\.0\.42$     # WAN router address
  acl IsMeshRouter hdr_reg(host) ^192\.168\.3\.1$      # LAN router address (not required)

  use_backend Mesh if IsMeshRouter

backend Mesh
  mode http
  balance leastconn
  http-request set-header Host <IP_ADDRESS> # Strongly required!
  server router <IP_ADDRESS>:80

Yes, this interaction scheme looks somewhat cumbersome, but, unfortunately, the configuration capabilities of Huawei routers are very limited. An alternative option would be to connect the smart home server directly to the Huawei router (to its internal LAN network).


JSON Schema validation https failed without details (768)

Issue Type: Bug
"$schema": "https://my-intranet-server:8081/schema/myschema.json" is underlined with error "Unable to load schema from 'https://my-intranet-server:8081/schema/myschema.json': Request vscode/content failed unexpectedly without providing any details.(768)"
Importantly the logs on my server do not show any connection from the Https request from VS Code!!
I do not use a proxy. The VS Code client and IIS Server are on different Windows 10 PCs on same intranet (HTTP is port 8080 and HTTPS is port 8081) but I can ctrl+click from VS Code on this URI to open a webpage showing the schema file (it opens in Edge browser and Chrome browser) - the IIS Server Logs show connection GET and 200 response fromweb browsers (I have added the self-signed certificate to 'Trusted Root Certification Authorities' on Local Computer).
"$schema": "http://my-intranet-server:8080/schema/myschema.json" is working fine. IIS Server Logs show VS Code request for this.
It is as if VS Code is not sending a request when it reads in the $schema with a https uri that is sent to my intranet server (no logs on server)
I have created "myschema-catalog.json" (for use with Visual Studio), this contains "$schema": "https://json.schemastore.org/schema-catalog.json" and that is working correctly!
If I change myschema.json to contain:
"properties": {
    "$schema": {
        "description": "Link to https://my-intranet-server:8081/schema/myschema.json",
        "type": "string",
        "enum": ["https://my-intranet-server:8081/schema/myschema.json"]
    },

Then when I write a json file and refer to this schema using:
"$schema": "http://my-intranet-server:8080/schema/myschema.json" I get the correctly reported error: "Value is not accepted. Valid values: "https://my-intranet-server:8081/schema/myschema.json".(1)" and the remaining JSON is validated correctly with popups in VS Code. As soon as I change it back to https:... then it fails with the original error.
I have no idea how to proceed?!?!
VS Code version: Code 1.57.1 (507ce72a4466fbb27b715c3722558bb15afa9f48, 2021-06-17T13:28:07.755Z)
OS version: Windows_NT x64 10.0.19043
Restricted Mode: No

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz (8 x 4251)

GPU Status
2d_canvas: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onoop_rasterization: enabledopengl: enabled_onrasterization: enabledskia_renderer: enabled_onvideo_decode: enabledvulkan: disabled_offwebgl: enabledwebgl2: enabled

Load (avg)
undefined

Memory (System)
31.94GB (25.55GB free)

Process Argv
--crash-reporter-id afcfd922-20fb-4764-b0c9-db755715f2fd

Screen Reader
no

VM
0%

Extensions (10)

Extension
Author (truncated)
Version

csharpextensions
jch
1.3.1

docomment
k--
0.1.20

csharp
ms-
1.23.12

vscode-pylance
ms-
2021.6.3

jupyter
ms-
2021.6.999662501

remote-ssh-edit
ms-
0.65.7

cpptools
ms-
1.4.1

powershell
ms-
2021.6.2

markdown-preview-enhanced
shd
0.5.21

win-ca
uko
3.3.0

(1 theme extensions excluded)

A/B Experiments
vsliv368:30146709
vsreu685:30147344
python383:30185418
vspyt653:30270858
pythonvspyt602:30300191
vspor879:30202332
vspor708:30202333
vspor363:30204092
pythonvspyt639:30300192
pythontb:30283811
pythonvspyt551cf:30311713
vspre833:30321513
pythonptprofiler:30281270
vshan820:30294714
pythondataviewer:30285071
vscus158:30321503
pythonvsuse255:30323308
vscorehov:30309549
vscod805cf:30301675
pythonvspyt200:30331937
binariesv615:30325510
aatestflight:30331711

I do not believe the above links are same issue. I have confirmed that the validation is working with Visual Studio 2019 16.10.3. Unfortunately I want to start using "$schema": "http://json-schema.org/draft-07/schema" and Visual Studio only supports draft-04. VS Code seems to support draft-07 just not via https as described above.

Very likely related to the certificate. I will make sure the error message is not swallowed.

To confirm what is happening: The whole json schema functionality (Draft-04) is working fine with Visual Studio 2019 16.10.3. I have added my own catalog schema which uses https://json.schemastore.org/schema-catalog.json - note the https! This lists my schema including https://my-intranet-server:8081/schema/myschema.json". But when I move to VS Code it fails as explained. I am confused why Visual Studio can download and USE the schema but not VS Code. As I move to Draft-06 and -07 I wanted to use VS Code (as those are unsupported in Visual Studio). They work for http: but as the https://json.schemastore.org/schema-catalog.json requires https^ I am in a catch-22!!

I'm experiencing a similar issue,
https://schema.abaplint.org/schema.json loads without problems in a browser, and also works from https://playground.abaplint.org where the monaco editor fetches and use the schema without problems

guess my above error is due to https://github.com/microsoft/vscode/issues/134244
but still it would be nice if it showed more error information

works in 1.61.0, so my scenario was probably due to the certificates

I can confirm that it works for me as well.

but still it would be nice if it showed more error information

What I fixes that the error has the detail.
It will not fix any issues you have with proxy configurations, but hopefully the message will give an indication what the problem is.
The fix is for the next release.


OLH-2198-defra-dangerous-dog-index

Proposed changes

OLH-2198 Onboard Defra Dangerous Dox Index
What changed

Onboarded new service Defra Dangerous Dog Service
Checklists

Environment variables or secrets

[x] No environment variables or secrets were added or changed

moving to blocked awaiting prod id


Handle annotation processors

Some libraries can be used only w/ their annotation processor (e.g. Room).
Implement necessary logic to add them with their annotation processors, handle kapt vs annotationProcessor configurations.

Is there any way to predict that library uses annotation processor?


Minute to hour transformation: failure

Current version of trady (3.2.9?) does not do the minute to hour transformation.
First of all - following part of code in IOhlcvDataExtension was changed previously:
        var tempCandles = new List<IOhlcv>();
        for (int i = 0; i < orderedCandles.Count; i++)
        {
            var indexTime = orderedCandles[i].DateTime;
            if (indexTime >= periodEndTime)
            {
                periodStartTime = indexTime; // previously: periodEndTime
                periodEndTime = periodInstance.NextTimestamp(periodStartTime);

                AddComputedCandleToOutput(outputCandles, tempCandles);
                tempCandles = new List<IOhlcv>();
            }

            tempCandles.Add(orderedCandles[i]);
        }

The code with the old periodEndTime somehow works better :) After transformation you actually get smaller amount of candles. With indexTime you get the same exactly amount of candles as you had before the transformation. This is issue 1.
So, the problem I am describing next - is actually using the old version of code (with periodStartTime = periodEndTime). You can skip it, if issue 1 is not an issue but a feature?:
        string dataString = File.ReadAllText("data.txt");
        var data = JsonSerializer.Deserialize<List<Candle>>(dataString);
        var transformedTradyCandles = data.Transform<PerMinute, Hourly>();

data.txt
p.s. In order for deserialization to work - you have to have argument-less Candle constructor.
The data is that I passing to transformation is in time frame: 8:40 - 10:02, total of 83 records.
Expected result - 3 candles:

at 9:00 (aggregated 8:40 - 9:00)
at 10:00 (aggregated 9:01 - 10:00)
at 11:00 (aggregated 10:01 - 10:03)

Actual result - 4 candles, wrong time components:

8:40
8:41
9:00
10:00

I have not actually compared the amount of Volume. But from the looks of it - candles already have incorrect time component.. This is issue 2
So, as I mentioned, you can skip issue 2 if issue 1 is the one that needs fixing. In either way - you can use the data that I have provided and the sample code to verify either of those 2 issues.
using VS 2019, Current version of trady (3.2.9?, i.e. master branch), .Net core 3.1

After some digging through the code I see at least 1 issue that could lead to above problems. In IOhlcvDataExtension:
        var periodStartTime = orderedCandles[0].DateTime;
        var periodEndTime = periodInstance.NextTimestamp(periodStartTime);

The date that is a start date is in format: {12/2/2017 8:40:00 PM +00:00}
And the end date is: {12/2/2017 9:00:00 PM +02:00}
According to how I understand it - the framework should have just added 1 period to start datetime, i.e. 1 hour. But in addition to adding 1 hour + it also changed the timezone.

Alright, a question:

Is the a reason why after the transformation - we take the first candle date-time for the resulting candle datetime? Isn't a candle - a summary of the previous period? So in that sense - candle date-tome is always the latest datetime, not the datetime of the first candle...
In either case - I don't think that earliest candle datetime shall be used. It's either the period start or the end :) Convince me otherwise.

With that idea in mind - I have applied some fixes to the code:
public static IReadOnlyList Transform<TSourcePeriod, TTargetPeriod>(this IEnumerable candles)
where TSourcePeriod : IPeriod
where TTargetPeriod : IPeriod
{
// ....
        DateTimeOffset periodStartTime = orderedCandles[0].DateTime;
        DateTimeOffset periodEndTime = periodInstance.NextTimestamp(periodStartTime);

        var tempCandles = new List<IOhlcv>();
        for (int i = 0; i < orderedCandles.Count; i++)
        {
            var indexTime = orderedCandles[i].DateTime;
            if (indexTime >= periodEndTime)
            {
                // CORRECTED order
                AddComputedCandleToOutput(outputCandles, tempCandles, periodEndTime); // CORRECTED
                tempCandles = new List<IOhlcv>();

                periodStartTime = indexTime;
                periodEndTime = periodInstance.NextTimestamp(periodStartTime);
            }

            tempCandles.Add(orderedCandles[i]);
        }

        if (tempCandles.Any())
            AddComputedCandleToOutput(outputCandles, tempCandles, periodEndTime); // CORRECTED

        return outputCandles;
    }

and
    private static void AddComputedCandleToOutput(List<IOhlcv> outputCandles, List<IOhlcv> tempCandles, DateTimeOffset periodEndTime) // CORRECTED
    {
        var computedCandle = ComputeCandles(tempCandles, periodEndTime);
        if (computedCandle != null)
            outputCandles.Add(computedCandle);
    }

and
    private static IOhlcv ComputeCandles(IEnumerable<IOhlcv> candles, DateTimeOffset periodEndTime) // CORRECTED
    {
        if (!candles.Any())
            return null;

        var dateTime = periodEndTime.UtcDateTime; // CORRECTED !!!
        var open = candles.First().Open;
        var high = candles.Max(stick => stick.High);
        var low = candles.Min(stick => stick.Low);
        var close = candles.Last().Close;
        var volume = candles.Sum(stick => stick.Volume);
        return new Candle(dateTime, open, high, low, close, volume);
    }

This, plus the fix applied in previous response:
    protected override DateTimeOffset ComputeTimestampByCorrectedPeriodCount(DateTimeOffset dateTime, int correctedPeriodCount)
    {
        var sth = TimeSpan.FromSeconds(NumberOfSecond);
        var sth2 = dateTime.UtcDateTime.Truncate(sth); // FIX here
        var sth3 = sth2.AddSeconds(correctedPeriodCount * NumberOfSecond);

        return sth3;
    }

New actual result:

at 9:00 (aggregated 8:40 - 9:00)
at 10:00 (aggregated 9:01 - 10:00)
at 11:00 (aggregated 10:01 - 10:03)

Tested all, seems to work well nice job. Will probably need a thorough test but also fixes my issue you tagged above.


CSV::from_file does not work with remote URLs

I want to be able to extract documents stored on S3, so I have registered the stream wrapper:
$client = $sdk->createS3();
$client->registerStreamWrapper();

And then I attempt to load the file:
(new Flow())
    ->extract(CSV::from_file(
        file_name: 's3://my-bucket-name/test.csv',
        header_offset: 0,
    ))
    // ...
    ->run();

But the following exception is thrown:
 [Flow\ETL\Exception\InvalidArgumentException]
  File s3://my-bucket-name/test.csv not found.

I think the check for file existence should be modified to allow remote URLs:
if (! (str_contains($file_name, needle: '://') || is_file($file_name))) { /* throw exception */ }

OR add another method to handle resources, eg:
$extractor = CSV::from_resource(
    resource: fopen('s3://my-bucket-name/test.csv'),
    header_offset: 0,
);

hey @shadowhand literally today morning I started looking into this but let me move this issue into the https://github.com/flow-php/etl and convert it into a discussion since it's a longer topic that affects all adapters.


Status indication

Implemented a status indicator to satisfy issue-#12
To prevent AddressExtractor from becoming a god class and jamming in monitoring things, I created an AddressExtractorMonitor and enclosed the existing logging messages and Stopwatch into it.
I also made it virtual so that you could theoretically easily override the Logging method (Also with a custom timespan)
The default logging interval is every minute, but I expanded the SingleSmallFile.txt to 500,000 lines and it finished in ~2,199ms. So I was testing it in 1s intervals. YMMV
public class MyMonitor : AddressExtractorMonitor
{
    public MyMonitor(): base(TimeSpan.FromSeconds(12)) {}

    public override void Log() {
        // Do something
    }
}

Technically my loop method needs work, as the Addresses are only Union'ed once a file has been fully read. So the Status Indicator isn't fully up to date at all times
May try and tweak this myself

Addresses are now added to the Addresses Set as soon as they are Matched, instead of at the end of each file.


Plugin needs tests

It would be great if the plugin had RSpec tests, ideally run on each PR, to make sure that contributors aren't breaking anything (and to make it easier than testing the change manually). The cocoapods-amicable plugin has some good examples of what tests could look like.
Thanks!

Hi @sethfri.
Thanks for the suggestion, we will look into it and will let you know of any updates.


Update docfx.json

Adding header

:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docfx.json
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.

Seems to have worked https://review.docs.microsoft.com/en-us/graph/training/ruby-tutorial?branch=jthake-msft-patch-1

Thanks buddy!


Open-Source Science Initiative

Discuss the best way to enable science with OSS, including:
Interest Groups in various disciplines, such as

Chemistry
Climate and Sustainability
Healthcare and Life Science

Interest Groups for cross-cutting concerns, such as

Reproducibility
Map of Science

Thanks!
Map of Science is the same one mentioned in #14?

Happened 2022-09-23 2 pm.


Material guidelines for checkbox/radio-buttons

Question
I was wondering if you could explain the reasoning behind nesting mat-checkbox and mat-radio inside of mat-form-field, my understanding was that the material team recommended against doing that.
I love the product, I currently use my own custom material implementation but was hoping to move  to your implementation
The following Angular Material components are designed to work inside a <mat-form-field>

<input matNativeControl> & <textarea matNativeControl>
<select matNativeControl>
<mat-select>
<mat-chip-list>

Thanks!

see https://github.com/angular/components/issues/7891

Thank you, my googlefu failed me when trying to find the reasoning.  Awesome piece of software you all created


Enatega SingleVendor Admin Allowing Gap in Rider Username and Accepting Negative Values in Phone Number Field

Describe the bug
I have identified two critical issues in the Enatega SingleVendor Admin system within the rider section. Firstly, the system currently allows users to create rider usernames with gaps, potentially leading to confusion and errors. Secondly, the system accepts negative values in the phone number field, which can impact the ability to contact and manage riders effectively.
To Reproduce
Steps to reproduce the behavior:
Issue 1 (Gap in Rider Username):

Log in to the Enatega SingleVendor Admin system.
Navigate to the rider management section.
Create or edit a rider profile.
Attempt to set a username with a gap (e.g., "john doe" instead of "johndoe").

Issue 2 (Negative Value in Phone Number Field):

Log in to the Enatega SingleVendor Admin system.
Navigate to the rider management section.
Create or edit a rider profile.
Attempt to input a negative value in the phone number field.

Expected behavior
Expected Behavior - Issue 1:
The system should validate and restrict users from creating usernames with gaps, ensuring consistency and preventing potential issues with user identification.
Expected Behavior - Issue 1:
The system should validate and restrict users from creating usernames with gaps, ensuring consistency and preventing potential issues with user identification.
Screenshots

Desktop:

Browser : chrome

Additional context
Please investigate and address these issues promptly to enhance the reliability and functionality of the Enatega SingleVendor Admin system.
Thank you for your attention to these matters.

Hi, @Inshirah23,
Thanks for spotting this issue! We fixed these bugs, now UserName will not allow spaces and other valies will not be less then zero in Enatega's SingleVendor admin.
@ninjas-code-official/ninjas-code
@beKashif


Support configuring default ipynb format or change to a new minimum version

Applies To

[X] Notebooks (.ipynb files)
[ ] Interactive Window and/or Cell Scripts (.py files with #%% markers)

What happened?
Whenever you are creating new .ipynb files, you will by default create the nbformat_minor version as 2. Can't seem to change this behaviour.
VS Code Version
1.71.0
Jupyter Extension Version
v2022.8.1002431955
Jupyter logs
No response
Coding Language and Runtime Version
No response
Language Extension Version (if applicable)
No response
Anaconda Version (if applicable)
No response
Running Jupyter locally or remotely?
No response

Thanks for filing this issue, currently there's no way to change this.

What would you like this changed to?
Out of curiosity why do you want a different value?

Thanks for filing this issue, currently there's no way to change this.

What would you like this changed to?
Out of curiosity why do you want a different value?

I am using nbformat to read a notebook string into a NotebookNode and it is facing a validation problem (as given here). The fix is to change the minor version to 5. But when someone will author a new notebook, it will again be with minor version 2.
So ideally we want it to be changed to 5.

thanks, however I'm still struggling to understand this issue.
please could you provide a sample code  with instructions and with the versions of the packages used so i can repro this issue at my end

Sure!
Steps to repro this issue:

Use nbformat 5.4.0 version
Create a notebook with major version  = 4 and minor version =2
Call nbformat.read(...) on the notebook string and it will generate a Notebook JSON failed error

Please let me know if you need my help in the repro.

Please let me know if you need anymore info from my side.

Sorry, forgot to get back to you, thanks I'll take a look at this..

@DonJayamanne  I see that something has been merged for this. Is it now possible to specify the nbformat version by default ?

Unfortunately we haven't fixed this yet.

[NbConvertApp] ERROR | Notebook JSON is invalid: Additional properties are not allowed ('id' was unexpected),
id is simply not known for notebooks with nbformat below version 4.5.
In my case
"nbformat": 4,
"nbformat_minor": 2
becomes
"nbformat": 4,
"nbformat_minor": 5

@Younthing Please can you file a new issue for what you are running into.
We do not upgrade versions from 4.2 to 4.5, unless your notebook already has an id property or the like (I suspect it has the cell id).
Either way, please file a new issue with simple steps to repro, e.g. create a new notebook with 4.2 as version and how it gets changed to 4.5 (do you save, etc)


[#OSF-7232]Adding affiliations from components does not affiliate nested components (grandchildren)

Purpose
Fix the problem that adding affiliations from components does not affiliate nested components (grandchildren)

QA notes:
What to test:

Create a project (1) and add a component (2)
Within the component, create a nested component (3)
Within 3, create another nested component (4), and then another within 4 (5), and another within 5 (6)
Go to Component 2 and go to Settings
Click to Add Affiliation and in the modal that appears, select to add the affiliation to 2 and every component within it
After affiliation has been added, check the components and nested components
Before change:
Only a few of components child nodes has the correct affiliation information. Usually it will be component 2, 3, 5 has but 4,6 doesn't
After change:
All child nodes of component 2 should have the correct affiliation information no matter it is add or remove affiliation.

Side effects

Ticket

https://openscience.atlassian.net/browse/OSF-7232

@chennan47 Can you rebase onto master, please? They'd like this as a hotfix.

closed in favor of https://github.com/CenterForOpenScience/osf.io/pull/6982


Validate CPE 2.2 and 2.3 identifiers

This also generate a link to https://services.nvd.nist.gov for CPE 2.3 identifiers.
Relates to #5356.

Example: https://deploy-preview-5481--endoflife-date.netlify.app/linux.


[spark] Fix alter table column type change nullability

Purpose

Spark catalog TableChange.UpdateColumnType should only change the data type, but paimon always changes the nullable to true. This pr keeps the original field nullability.
To fix the issue:
CREATE TABLE t (a INT, b BIGINT NOT NULL) USING paimon;
ALTER TABLE t CHANGE COLUMN b TYPE string;

SHOW CREATE TABLE t; => b STRING NOT NULL

Tests

add test
API and Format

no
Documentation

All Spark tests passed, the failed test seems irrelevant:
2024-08-15T03:51:37.1011203Z [ERROR] Tests run: 37, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 797.345 s <<< FAILURE! - in org.apache.paimon.hive.Hive23CatalogITCase
2024-08-15T03:51:37.1078168Z [ERROR] org.apache.paimon.hive.Hive23CatalogITCase.testDropPartitionFromBranchLocationInProperties  Time elapsed: 45.84 s  <<< ERROR!
2024-08-15T03:51:37.1080812Z org.apache.flink.table.api.TableException: Could not execute ALTER TABLE my_hive.test_db.t DROP PARTITION (pt=2)

cc @JingsongLi thank you

All Spark tests passed, the failed test seems irrelevant:
2024-08-15T03:51:37.1011203Z [ERROR] Tests run: 37, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 797.345 s <<< FAILURE! - in org.apache.paimon.hive.Hive23CatalogITCase
2024-08-15T03:51:37.1078168Z [ERROR] org.apache.paimon.hive.Hive23CatalogITCase.testDropPartitionFromBranchLocationInProperties  Time elapsed: 45.84 s  <<< ERROR!
2024-08-15T03:51:37.1080812Z org.apache.flink.table.api.TableException: Could not execute ALTER TABLE my_hive.test_db.t DROP PARTITION (pt=2)

cc @JingsongLi thank you

@ulysses-you
Your repo is passed. https://github.com/ulysses-you/paimon/tree/nullable

Your repo is passed. https://github.com/ulysses-you/paimon/tree/nullable

oh, interesting, it is the double check!


Need maglev/debugger

I tried to run master branch, but got:
ERROR 2702 , a RubyLoadError occurred (error 2702), no such file to load -- maglev/debugger (LoadError)
topaz 1> exit
rake aborted!

Fixed.


latest r-sf not available yet for conda

Hi!
I am not sure this is the right place for question, but we are glad if the latest r-sf 7.7 will be available in conda at somepoint.
thanks for considering!
Koichi

Issue:

Environment (conda list):

$ conda list

Details about  conda and system ( conda info ):

$ conda info

I am also wondering this.

Once someone can get the thing to build it'll be available: https://github.com/conda-forge/r-sf-feedstock/pull/29


Add corrector for ways that have Speed Limit, but are missing Max Speed

The KeepRight layer flags highways that are missing MaxSpeed. This isn't an attribute that's automatically shown in the modification left panel, you have to manually add it. When committing things w/ issues like outdated tags for crosswalks that you've touched, you're alerted before you save the commit and basically given a 1 click button to fix certain things like outdated tags in addition to whatever changes you made.
It would be nice if there's a similar quickfix button for roads where there's a SpeedLimit, but not a MaxSpeed so this KeepRight issue can be more easily caught and cleaned by anyone who touches these ways in iD.

@sanecito I understood your suggestion as deprecating tags like speed_limit and speedlimit. If you meant something else then let us know!


Add briggySmalls fork to list of alternatives

Hi.
Thanks for the great tool.
I've been dogfooding my own variant for a couple of projects, and I've now plucked up the courage to submit it!
You can have a look at the project at github.com/briggySmalls/cookiecutter-pypackage.
The differences are:

Dependency tracking using pipenv
Linting provided by both pylint and flake8 [executed by Tox]
Formatting provided by yapf and isort [checked by Tox]
Autodoc your code from Google docstring style (optional)
All development tasks (lint, format, test, etc) wrapped up in a python CLI by invoke

Hopefully you agree to adding it to the repo.
Thanks again! ✌

The failure doesn't seem to be my change? It appears the installation of pip in python 2.7 failed.
This is a known, transient failure? Perhaps the CI job can be rerun?

The failure was caused by some changes in tox and has now been fixed. Your changes will be merged.

Thanks!


PDF reader issue

Hi! I downloaded mouse-dictionary and tried to use on pdf.
I opened my Pdf file from mozilla.github but seems like the dictionary won't show any word even i point out words with mouse.
Hope there is a way to fix it! Thanks a lot for creating this amazing tool!

@shatano-htn
Hi, thank you for your repon.
Could you share the PDF file?
My Mouse Dictionart + PDF files works well.

Thanks for the reply!
Please find the pdf below.
https://classroom.upeace.org/elearning/pluginfile.php/14437/course/section/2151/Ramsbotham 2016 Contemporary Conflict Resolution Ch 2 pp 38-67.pdf

I cannot log in.

And, if your Mouse Dictionart works on other PDF files but doesn't work on one the specific PDF file, it's likely to be due to the file itself.

I am sorry that I have sent you the URL which you cannot log in.
Please find the attached document.
Yes I can use Mouse-dictionary for other PDF files.
I am so amazed of this tool, will be so helpful  to use with my studying!
Ramsbotham 2016 Contemporary Conflict Resolution Ch 2 pp 38-67-圧縮済み.pdf

I am so amazed of this tool, will be so helpful to use with my studying!

Thank you for your feedback!
I've check the PDF file and I found it had no text data.
Probably, this PDF file was created by just scanning paper,
so virtually this is just an image file that has no text information.
In this case, it's impossible to use Mouse Dictionary on it.
You have to find or make text data first.

Thank you for checking the PDF file and responded me promptly.
Ok I got it. I will probably try to convert it to text data then will use the Mouse Dictionary.
Thanks again for your support!
Looking forward to seeing more updates in near future:)


problems with batch >1

ok with 1 image (wf in the image)
with a batch of 3:

happen with ckps xl turbo and normals

the problem seems condition_zero_ot the positive

same problem

I've the same problem, too.


Override Timestamp hashCode for consistent hashing

Timestamp's hashCode was based on the parameter dateTime which might have
a timezone != UTC. The consequence:
val t = Timestamp.now
t.hashCode != Timestamp(t.toString).hashCode
This patch overrides hashCode to use the UTC normalized time variable of the
Timestamp case class.
This probably fixes #1082

Was wondering whether it would be cleaner to move the UTC conversion into the apply constructors and turn the current UTC time into the case class parameter.
case class Timestamp(utcDateTime: DateTime) extends Ordered[Timestamp]

object Timestamp {
   def apply(time: String): Timestamp = Timestamp(
      DateTime.parse(time).toDateTime(DateTimeZone.UTC)
   )
}

Then we can probably get rid of most of the overrides in the case class.

@sttts, we could map the time zone in the companion's apply method.  We'd also have to make the constructor private to disallow invalid instances from being built.  IMO, it is quite clear as written.  Let's leave a refactor out of this bugfix for a subsequent PR.

Thanks! :shipit:

:confetti_ball: :tada:


[Feature Request]: Improve view more page, Add same footer to all pages

Is there an existing issue for this?

[X] I have searched the existing issues

Feature Description
Footer not same in all pages
UI of view more in packages section hyperlink, does not have a good UI
Use Case
For any user who want to check out packages
Benefits
No response
Add ScreenShots
No response
Priority
High
Record

[X] I have read the Contributing Guidelines
[X] I'm a GSSOC'24 contributor
[ ] I'm a VSOC'24 contributor
[X] I have starred the repository

can you assign me this @apu52 ?

can you assign me this @apu52 ?

I have assigned u .now u can start contributing

i want to contribute add me

it's already assigned to me. you should raise some other issue


Codeunit 7000 Sales Price Calc. Mgt./CopySalesPriceToSalesPrice procedure

Currently, it shows
local procedure CopySalesPriceToSalesPrice(var FromSalesPrice: Record "Sales Price"; var ToSalesPrice: Record "Sales Price")
begin
with ToSalesPrice do
if FromSalesPrice.FindSet then
repeat
ToSalesPrice := FromSalesPrice;
Insert;
until FromSalesPrice.Next = 0;
end;
Is it possible to add an EVENT in the beginning of the procedure with FromSalesPrice,  ToSalesPrice and 'IsHandled' parameters so we could use our own procedure and skip the base BC procedure?
Thanks.

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. We will update this issue with information about availability.


Updates version to be in sync with tags

I noticed the version number was no longer in sync with the tags, so I figured to submit a PR <3

fixed in 37c26102498dfb6e57206fc9eaef6456d771872a


Remove deprecated gateway external name

Fix warnings

Can we get testing instructions to reproduce the issue and then demonstrate the fix? Thanks.

Looks like the format has changed so we should, set external: true like so:
gateway:
  external: true
  name: gateway

Our current method was deprecated in version: "3.5". In all our docker-compose files we have  version: "3.7"
It looks like external: true must be specified for it to work as it was previously.

Looks like we don't need to specify name either if it matches the same as the network configuration.
docker network ls                          Sun 17 Oct 2021 12:54:49 UTC
NETWORK ID     NAME              DRIVER    SCOPE
eb0abd23fac3   bridge            bridge    local
1df64990806d   host              host      local
4943545796cd   isle-buildkit     bridge    local
225f8df1f00d   isle-dc_default   bridge    local
85f08ed21a62   isle-dc_gateway   bridge    local
e540e0060f15   none              null      local

So the following would be fine.
gateway:
  external: true

@DonRichards if you're happy with my changes I'll go ahead and merge it.


Coupon code

First, i wanto say thank you for making this greate project. I couldn't find coupon code form in the cart or in the check out form. Usually ecommerce has a coupon code form that reduce the final total if the code is correct.
Would you mind if add this feature?

@samsularifin it is in roadmap and currently this feature does not exist.

Thank you. I am glad to hear that and i am waiting for the features update.

@samsularifin I also need a coupon code feature.

+1

@sonufrienko i want to get started working on registration and coupon feature, i will suggest if we could have a skype/hangout session along with some other developers so that will be great.

+1

+1

I also need a coupon code feature.
tnx Cezerin dev team


Provide a way to remove listeners

I could not find any method in the beta that provides a way to remove a listener.. i have something like this:
RequestAllTasksListener { hadErrors ->
      if (hadErrors) {
        cancel(true)
      } else {
        download()
      }
    }.also {
      WendyConfig.addTaskRunnerListener(it)
      Wendy.shared.runTasks(null)
    }

And every time i need to call the function with this piece of code a listener is added.. one could say that i should only put a listener one single time, but for our needs this is not true, we need to listen to events, see if there's anything wrong and 'unsubscribe' after that..

What are you using the task runner listener for? The use case that I created it was for UI purposes. If you wanted to show in the UI a status of the task runner such as a progress bar saying that your data is being saved.
With my use case described, you would only need 1 task listener attached to the UI and that's it.
What are you using the listener for if you don't mind me asking to get a better idea on how to solve this problem?

P.S. I didn't even think to add a remove listener function. Because Wendy uses weak references that solved the problem for my use case. After I get a better idea of what you are trying to do, I'll see what I can do to improve the library.

Our app here is kind of old, I'm implementing new stuff here while maintaining some of the old
code (this complicate things a bit XD) ... and the data synchronization part is one of this old codes, on the synchronization part we send all our new data to the server (one request at time), if no error occur we download a package (basically a snapshot of the remote DB at a given time) and update/delete the entities based on this package.
So basically i need to run all tasks and check:

on error: show error and not download the package
on success: download the package

That's basically why i need to listen and unsubscribe after that..
I was reading the wendy's source and i know that it uses weak references, but if i understand correctly, the code i sent on my first comment should be 'cleaned' after being used.. i don't even make a reference for the newly created object (ok, there's the it, but that's the only time).. Anyway, again, if understand correctly how WeakReferences should work, the code below should solve the problem:
    var listener: RequestAllTasksListener?

    listener = RequestAllTasksListener { hadErrors ->
      if (hadErrors) {
        cancel(true)
      } else {
        download()
      }
      listener = null
    }

    WendyConfig.addTaskRunnerListener(listener!!)
    Wendy.shared.runTasks(null)

I will test it soon, but a removeListener i would trust more than explicit setting the variable to null
Here's the task RequestAllTasksListener code:
class RequestAllTasksListener(
    private val onFinish: (hadErrors: Boolean) -> Unit
) : TaskRunnerListener {
  private var hasRecordedErrors = false

  override fun allTasksComplete() {
    onFinish(hasRecordedErrors)
  }

  override fun errorRecorded(task: PendingTask, errorMessage: String?, errorId: String?) {
    hasRecordedErrors = true
  }

  override fun errorResolved(task: PendingTask) {
  }

  override fun newTaskAdded(task: PendingTask) {
  }

  override fun runningTask(task: PendingTask) {
  }

  override fun taskComplete(success: Boolean, task: PendingTask) {
  }

  override fun taskSkipped(reason: ReasonPendingTaskSkipped, task: PendingTask) {
  }
}

ok, that doesn't work, but at least i tried. I think that the weak reference will be freed only when the system decides that is needed.. so basically i will need to control on the listener itself if it has been runned once

From what you stated above, "the synchronization part we send all our new data to the server (one request at time), if no error occur we download a package (basically a snapshot of the remote DB at a given time) and update/delete the entities based on this package." I have a possible idea.
Instead of having a listener that checks when the task runner is complete with all tasks, you can instead use Wendy.shared.getAllTasks().isEmpty after each API call and when it is empty, you can download the whole package.
That's a thought that might make things a little less complex and will fix your issue with the current Wendy code base.

However, I do agree there needs to be a way to remove a listener. That is just something that should be added anyways. Thank you for answering my question about what you use the listener for so I could understand your use case behind using it.
In the end, use whatever method works best for you. We should have a remove listener function anyways.
If the idea I stated in this comment works for your purpose, go ahead and use that to get by. If it will not work for you, you can make a PR or I'll be sure to make a fix for you quick so you can continue on with your app 😄.
Thanks for the PR. We do need a remove listener function.


"Specified key was too long" when upgrading to 1.5.1

Gitea version (or commit ref):
Git version: 1.5.1
Operating system: Ubuntu 16
Database (use [x]):

[ ] PostgreSQL
[x] MySQL
[ ] MSSQL
[ ] SQLite

Can you reproduce the bug at https://try.gitea.io:

[ ] Yes (provide example URL)
[ ] No
[x] Not relevant

Log gist:

Description
When upgrading from 1.4.x to 1.5.1, the migration does not seem to work:
2018/09/17 15:35:53 [...itea/routers/init.go:60 GlobalInit()] [E] Failed to initialize ORM engine: migrate: do migrate: Sync2: Error 1071: Specified key was too long; max key length is 767 bytes
Thanks

Oooops duplicates #4694


Fixed references to qiniu/api in project to qiniu/api.v6

For whatever reason the developer at https://github.com/qiniu/api removed all the files and recreated them at https://github.com/qiniu/api.v6 and https://github.com/qiniu/api.v7
This caused switch to stop running so I just added the api.v6 to all the locations and tested with
gopm run switch.go and everything works.

Awesome, thanks!


Validate RSS links before saving them

Closes #1128
Closes #920

Cannot delete old urls, can you check this @lucasvanhalst?

Right I forgot to test that


Daily project caching

This pull request adds the feature requested in #7 . An artisan command was created with the command projects:save. This can be called daily to store the projects current data.  A new issue has been created for the graphing functionality #8 .

@sbine @darkboywonder
Hey, can you guys spot any super obvious errors on my part. Anything from formatting to docs. :)

@mattstauffer @sbine,
Thank you both for the once over! The branch has been updated to reflect all comments. Matt, please let me know if those git commits look a little better.

There's too much here that has already been requested to be changed. Please read through your PRs in the GitHub UI before tagging someone to review them.

Definitely will improve my GitHub UI flow.

I think I want a little higher attention to detail on these PRs from you before you tag for review.
Absolutely, thanks Matt.

@anthonyterrell holler when it's ready for review again


Customizable query for himalaya notify

It would be helpful for himalaya notify to be able to query for emails that are, e.g. not seen rather than new, since in some (tragic) situations the \Recent flag might not be set on incoming email.

I add a new config option notify-query which takes a valid IMAP query string (the same as the query command, see https://tools.ietf.org/html/rfc3501#section-6.4.4). It will be shipped with the next release, stay tuned!

Could the wiki be updated with examples of how to use this and himalaya watch? I'm not getting notifications but I'm probably not doing things correctly. (I have watch-cmds = ["echo Mailbox change"] as a test and I guess I expected "Mailbox change" to be echoed whenever something happens, but... nothing.)

The wiki has been updated with the new release I just published: https://github.com/soywod/himalaya/wiki/Configuration:config-file. About the watch-cmds, I'm not surprise that you see nothing, because commands are run in a separated thread. Do you have the libnotify installed? You can try sth like watch-cmds = ["notify-send subject body"].

No libnotify - it requires over 500 dependencies and I'm keeping my system minimal.

I see. Maybe you can try to create a script shell that writes sth in a file, sth like:
#!/bin/bash

echo test >/tmp/test

and set up watch-cmds = ["your-script.sh"]

Example for others:
notify-query = "not seen"


Several forms decoupled

What does this PR do and why?
Provide a description of what this PR does. Link to any relevant GitHub issues, Notion tasks or Discord discussions.
Screenshots or screen recordings
If applicable, provide screenshots or screen recordings to demonstrate the changes.
Acceptance checklist

[ ] I have evaluated the Approval Guidelines for this PR.
[ ] I have tested this PR in all supported browsers.
[ ] I have tested this PR in dark mode and light mode (if applicable).

Current dependencies on/for this PR:

main

PR #1714

PR #1715

PR #1716

PR #1717

PR #1718

PR #1719

PR #1720   👈

This comment was auto-generated by Graphite.


Small changes in order to work with build 1809

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.:x: gerardlopez sign nowYou have signed the CLA already but the status is still pending? Let us recheck it.


Get a handle to TemplateApp for eframe 0.19

I am attempting to duplicate an existing project but upgrading from eframe 0.16 to 0.19, but having trouble getting a handle to the TemplateApp struct from main().  How is this done?
The project I am referencing is here:
https://github.com/andy31415/rs-value-plotter
The basic approach in this design is to have a struct
struct TemplateApp {
    buffer : Arc<Mutex<VecDeque<egui::plot::Value>>>
}

and in a std::thread within main(), clone the handle to the TemplateApp and introduce values into the buffer.
Now with eframe 0.19 I want to use a native GUI versus a wasm application. So the impl for as_any_mut for wasm32 is not helping in this case, as far as I can tell.

You cannot get a handle to the app, but you can easily set up some shared state before calling run_native that you pass to your app. That is, create an Arc<Mutex<…>>, clone it, and pass the clone to your app, and modify the other handle.


Retriving colors used in QR code

For a project I'm working on, it would be helpful to know the colors that the QR code I'm detecting is made out of (particularly the background). This seems like it might be a fairly useful and easy thing to add; assuming I end up using jsQR, I might be up for trying to implement it myself. If I do, any pointers for how/ where to start?

Currently jsQR discards colors as the first step of the process, specifically the binarize step. As such I think the best way to approach this would be to take the final result (with the location of the QR code identified) and use that to sample colors from the original image. extract gives back a mappingFunction which takes in x,y values for the QR code (0,0 is the top left, and if the QR is 21x21 then 21,21 would be the bottom right, etc) and returns the pixel coordinates in the original image.
Using that mapping function you could iterate through all the modules (squares) of the QR code and look up the color in the original image data. Then you could average out the values associated with "white" squares and the values associated with "black" squares to get the original colors.
I think this would be pretty effective. One thing to keep in mind however is that this isn't the primary use case of the library, and does add additional computational complexity to the scan (something which for standard uses needs to be efficient. As such if we're adding it to the library we need to make it something users can opt in to so if you don't need color data you don't incur the computational overhead. That said that's a bridge we can cross when it comes up.
Please let me know if this was helpful/if there is something I can clarify further.
Thanks!

Thanks, all of that's really helpful! For the API, what do you think of adding a default false parameter to the scan function and optional property(ies) to the QRCode interface? If that sounds good, any naming preferences? Maybe something along the lines of retrieveColors, backgroundColor, and qrColor? Also, it seems like there can be some variation to QR codes (e.g. rounded finder patterns and logos in the center), any thoughts about how to sample colors or restricting sampling to certain areas?

Also, any preferences on how to average together colors? For now I'll just do a arithmetic mean on the components.

I've gotten it working (here)[https://github.com/david476/jsQR/blob/master/src/index.ts], let me know if there's anything you'd like done differently. How do you think tests should be implemented? I assume you'd want to test both with and without retrieving color, it seems like either having separate output JSON files for color or modifying the format of the existing output files to include both are the most reasonable two options... Alternatively, I could set up a separate set of test images for color retrieval so that testing time isn't basically doubled.

For the API, what do you think of adding a default false parameter to the scan function and optional property(ies) to the QRCode interface

If we're adding more params to the function I think it should be an options object so we can extend it in the future with other options. That said lets get an implementation we're happy with first before thinking about how we want to integrate it. For example some other options could be keeping it on a fork of the library, or as a separate scan method we export. I think which we choose depends on the solidity of the results (I don't think we'd want to mainline the change if the quality we were seeing was low for example), and also how common of a need we foresee this becoming (if it's a literal one off it might not be worth adding a ton of complexity to the API, so a separate function etc could be ideal).

Also, any preferences on how to average together colors? For now I'll just do a arithmetic mean on the components.

Averaging colors, and generally doing arithmetic on RGP values is fraught with hidden complexity. For example to greyscale RGB you can't just average the three values. Rather you have to do this because of how humans perceive colors. You can learn more about why and also how those magic constants were derived here.
All that is to say - We should probably do better than an basic average. It seems like the simplest is to square the values and then sqrt the result. However this has some flaws (#ff0000 (red) and #0000ff (blue) creates a pink, instead of the potentially expected darker purple). You can also go way down the rabbit hole but that is probably unneeded (depending on your needs/expectations).

One more thing you may want to think about is how we want to handle images like this (or this or this). Specifically QR codes with graphics inserted. These will throw off the average, but unclear if that matters for your use case. If it does you could probably only average modules that weren't error corrected, but that would require some information being pulled out of the decode step.

Hope I didn't overwhelm with this info dump, just wanted to make sure you have enough context as you need, since you understand your needs more than I do. As for testing I'd say the first step should be just dumping out some easy to read information (maybe write a 10x10png of each color?) so that for the test suite images you can check to make sure you're getting the kinda results you expect?

Info dump is great, makes it easier to reference back to :)

If we're adding more params to the function I think it should be an options object so we can extend it in the future with other options.

Will do. Assuming typescript doesn't give me any problems, I'll make an interface with optional retrieveColors property and Object.assign whatever's passed in to a constant set of defaults.

Averaging colors, and generally doing arithmetic on RGB values is fraught with hidden complexity.

Yeah... I'll poke around to see if there's anything simple and little bit fancier than a geometric mean. Otherwise I'll swap to that.

One more thing you may want to think about is how we want to handle images like this (or this or this).

Yeah, it doesn't affect me but I thought I'd bring it up. Maybe leave it simple for now and see if anyone wants it later? Also, this one you liked to doesn't have an image, were you talking about the shadow? That's an interesting additional layer of variation on what you might want the library to return...

As for testing I'd say the first step should be just dumping out some easy to read information (maybe write a 10x10png of each color?)

Seems fine to be to just print out the RGB values into JSON; it makes the testing code that much simpler and is pretty easy to check with basic knowledge of RGB or an image editor... Any thoughts on whether/ how to run separate tests for color vs. normal scanning?

So Object.assign doesn't seem to be a thing in jsQR's environment... It's not a problem right now but making default scan options work better may be something needed in the future. I was able to implement averaging in the CIELab color space (still untested) relatively easily, it might not be worth it but whatever.

I just went ahead and implemented tests for color retrieval by changing the format of the output.jsons to:
{
  default: <output of a run with defaults>,
  color: <output of a run with color>
}

I can't seem to get jest to actually run the tests without getting 'heap out of memory' (I don't think this is related to my changes) errors but running generate-test-data didn't take too long and, poking around in the output, color mixing seems to work quite well. In case you're happy with it as is, I'm going to make a pull request. If not, just let me know and I'll change stuff!


Client doesn't work for GetAll with custom GetAllResponseWrapper

If an API uses GetAllResponseWrapper to return a custom response type, the included Client no longer correctly decodes the response. The easy fix is to require configuring the Client with the correct type, but ideally there is an option that can be done automatically, either using the ResponseWrapper function, or refactoring the feature to use a different implementation of custom responses that already works on the client side.

Hi @blinkeye! Thank you for the feedback and for being a user of babyapi!
I figured out a better way to handle HTML template wrappers after creating this example, so #59 updates it to be compatible with the Client by default. I'll hopefully get around to fixing the underlying issue soon, but this should help make the example more usable in the meantime.
Thanks again!

Excellent, I can confirm the example is now working with the CLI as well. Much appreciated!

Merged #60 and released v0.11.0 which fixes this issue. This updates the CLI to use more generic request methods from the Client since the CLI just needs to print the output and doesn't care about types.


ERROR: unable to select packages: unrar

Hi,
my docker container is not working anymore after I upgraded to 1.7.2
Looks like the unrar package is not possible to load.
[cont-init.d] jdownloader2.sh: executing...
[cont-init.d] jdownloader2.sh: installing requested package(s)...
[cont-init.d] jdownloader2.sh: installing 'unrar'...
[cont-init.d] jdownloader2.sh: fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
[cont-init.d] jdownloader2.sh: fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/community/x86_64/APKINDEX.tar.gz
ERROR: unable to select packages:
[cont-init.d] jdownloader2.sh: unrar (no such package):
[cont-init.d] jdownloader2.sh: required by: world[unrar]
[cont-init.d] jdownloader2.sh: exited 1.
[services.d] stopping services
[services.d] stopping s6-fdholderd...
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] syncing disks.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.

found out: unrar moved to non-free ...
https://wiki.alpinelinux.org/wiki/Release_Notes_for_Alpine_3.15.0#unrar_moved_to_non-free
should bsdtar  work also as an exchange?

Yes I think bsdtar is an alternative.  p7zip is another one.

Closing this issue, please re-open if needed.


add strict flag for assert

Addresses Issue

Thank you so much @susarlanikhilesh 🎉 🎉
The tests are failing because some of our tests expect a panic to occur, and if the flag is not set (eg: the code is run via a test rather than cli) I believe viper is returning the default boolean value of false. Is there a way to check if the flag is not set and, if not, set strict to true instead?

Yes the strict flag is false, I did set when panic is true in the test cases

I would prefer for strict=true in cases where not explicitly set. Our philosophy here is to panic by default when an unexpected event occurs, this way we can protect ourselves from being in an invalid state.
viper.GetBool("strict") // if not set needs to be defaulted to true

An easier way to do it might be to invert the name of the flag to something like ignore-asserts that we can default to false which makes things a little easier to implement.


user.py not working

/start T:3 I tried this and its giving an error that is "No function loaded". I am using python code on windows 7. And its not working properly. I am unable to find a brief guide about this code.

Hello,
The README file gives some examples of use and it explains how it works. You may have not selected a plugin -- i.e. what you want to do with EEG readings. For example if you want to simply print the raw values, you can use something like: python user.py -p /dev/ttyUSB0 --add print.
Hope it helps ;)

When I try to print eeg readings it gives me error "createfdContiguous failCorresponding SD file OBCI_8D.TXT".

I've never seen anything like that, and I don't see why a TXT file is involved. Can you please share all the details: exact command you type, board and version of the firmware you are using, etc.?


add yaml-cpp support for file config

This PR adds the yaml-cpp to support yaml format for file config which gives the alias feature to reuse config.
welcome any suggestion of yaml library.

Regarding the library choice, yaml-cpp seems to be the most popular, but rapidyaml seems to be a good alternative. rapidyaml supports resolving anchors automatically, which would simplify our parse implementation (see here).