Cuda Fortran (CUF) backend implementation including tests

CUDA Fortran
CUDA Fortran Kernels

the thread index for each dimension starts at one
kernel invocation: kernel<<<grid size, thread block size>>>

call kernel<<<grid,block[,bytes][,streamid]>>>(arg1,arg2,...)
e.g. using the dim3 derived type: type(dim3) =:: blocks, threads

blocks = dim3(n/256, n/16, 1), threads = dim3(16, 16, 1)

device subprograms are not allowed to have the SAVE attribute or with data initialization

subroutine/function qualifier
description

attributes(host)
to be executed on the host

attributes(global)
to be executed on the device, called using a kernel call

attributes(device)
to be executed on the device, called from a subprogram with global or device attribute

attributes(host,device)
either host or device

attributes(grid_global)
like global, but threads within the grid group are guaranteed to be co-resident on the devic, allowing a grid synchronization (on cc70 hardware and greater)

predefind variables (cudafor) in device subprograms

variable
description

threadidx, threadidx%x and/or threadidx%z
thread index within its thread block (for one or two dimensional thread blocks)

blockdim
dimension of the thread block, same value for all thread blocks in the same grid

blockidx, blockidx%x and/or blockidx%z
block index within the grid

griddim
dimension of the grid

warpssize
constant integer, threads are executed in groups of warpsize (currently 32)

! Kernel definition
attributes(global) subroutine ksaxpy( n, a, x, y )
    real, dimension(*) :: x,y
    real, value :: a
    integer, value :: n, i
    i = (blockidx%x-1) * blockdim%x + threadidx%x
    if( i <= n ) y(i) = a * x(i) + y(i)
end subroutine

! Host subroutine
subroutine solve( n, a, x, y )
    real, device, dimension(*) :: x, y
    real :: a
    integer :: n
    ! call the kernel
    call ksaxpy<<<n/64, 64>>>( n, a, x, y )
end subroutine

Kernel Loop directive

automatic kernel generation and invocation from a region of host code containing one or more tightly nested loops

!$cuf kernel do[(n)] <<< grid, block  [optional stream] >>>
or !$cuf kernel do[(n)] <<< grid, block, 0, streamid >>>
or !$cuf kernel do[(n)] <<< grid, block, stream=streamid >>>

scalar reduction operation, such as summing the values in a vector or matrix, are handled by the compiler, e.g. generation of the final reduction kernel, inserting synchronization into the kernel as appropriate

e.g. (grid shape is computed by the compiler)
!$cuf kernel do(2) <<< (*,*), (32,4) >>>
do j = 1, m
  do i = 1, n
    a(i,j) = b(i,j) + c(i,j)
  end do
end do

e.g. (automatic) reduction
rsum = 0.0
!$cuf kernel do <<< *, * >>>
do i = 1, n
    rsum = rsum + x(i)* y(i)
end do

e.g. explicit reduction

allowed reductions

Fortran integer type: +, *, max, min, iand, ior, and ieor
Fortran real type: +, *, max, min
Fortran complex type: +
Fortran logical type: .and., .or.

value = 0.0
!$cuf kernel do <<< *, * >>> reduce(+:value)
do i = 1, n
    a(i) = real(int(a(i) * 100.0 - 50.0),kind=4)
    if (a(i) .ge. 0.0) then
        value = value + a(i)
    else
        value = value + a(i) + 50.0
    end if
end do

the directive specifying n must be followed by at least that many tightly-nested DO loops
tighly-nested DO loops must have invariant loop limits
the invariant loop limits cannot be a value from an array expression, unless those arrays have the managed attribute
no GOTO or EXIT statements within or between any loops that have been mapped onto the grid and block configuration values

body of the loops may contain assignment statements, IF statements, loops, and GOTO statements

only CUDA Fortran data types allowed
device-specific intrinsics such as the syncthreads and other warp or block-level cooperating, syncing, or barrier functions should be avoided except in very limited situations
subroutine and function calls to attributes(device) subprograms are allowed if they are in the same module as the code containing the directive
implicit loops and F90 array syntax are not allowed within the directive loops

Variable Qualifiers

by default all variables are alllocated in the host main memory

qualifier
description

attributes(device)
allocated in the device global memory

attributes(managed)
may be used in both host and device code

attributes(constant)
device constant variable, allocated in the device constant memory (modifiable by the host but not by the device)

attributes(shared)
may not be data initialized, allocated in the device shared memory for a thread block, a write in one thread only guaranteed to be visible by another thread in the thread block after SYNCTHREADS()

attributes(pinned)
a pinned variable must be an allocatable array, when allocated it will be allocated in host pagelocked memory allowing for faster copies

attributes(texture)
a texture variable must be an F90 pointer (real or integer), read-only

members of derived type may not have the device attribute unless they are allocatable
device variables

device variables and arrays may be passed as actual argumnts to host and device subprograms

subprogram interface must be explicit and the matching dummy argument must also have the device attribute

device variables and arrays declareed in a host subprogram cannot have the save attribute unless they are allocatable

shared variables

shared data may not have the pointer, target or allocatable attributes
members of derived type may not have the shared attribute
real, shared :: x(*) is allowed but memory for this assumed-siye shared memory needs to be defined by the kernel invocation

dummy arguments (like in Fortran) are passed by default by reference, therefor the actual argument must be stored in device global memory

scalar arguments can be passed by value adding the value argument

### F90 pointers

(recently added support for) F90 pointers that point to device data

limited to pointer declared at module scope
accessed through module assocation, or can be passed in to global subroutines
associated() function is also supported in device code

module devptr
! currently, pointer declarations must be in a module
  real, device, pointer, dimension(:) :: mod_dev_ptr
  real, device, pointer, dimension(:) :: arg_dev_ptr
  real, device, target,  dimension(4) :: mod_dev_arr
  real, device, dimension(4) :: mod_res_arr
contains
  attributes(global) subroutine test(arg_ptr)
    real, device, pointer, dimension(:) :: arg_ptr
    ! copy 4 elements from one of two spots
    if (associated(arg_ptr)) then
      mod_res_arr = arg_ptr
    else
      mod_res_arr = mod_dev_ptr
    end if
  end subroutine test
end module devptr

Compilation

CUDA Fortran is enabled in compilation specifying -cuda or renaming the file to have a .cuf or .CUF extension
target architecture e.g. -Mcuda=cc35
specifying emulation mode: -Mcuda=emu
target CUDA toolkit version e.g. -Mcuda=cuda5.0
-Mcuda=fastmath enables faster intrinsics
-Mcuda-nofma disables fused multiply-add
-Mcuda=maxregcount:<n> limits registr usage per thread
-Mcuda=ptxinfo prints memory usage per kernel
-Mcuda=rdc (relocatable device code) allows to call device functions between modules

Conditional compilation

pre-processing is enabled by either the -Mpreprocess option or by using capital letters in the filename extension

then _CUDA macro is defined

e.g.
program p
!@cuf use cudafor
real a(1000)
!@cuf attributes(managed) :: a
real b(1000)
!@cuf real, device :: b_dev(1000)
b = 2.0
!@cuf b_dev = b
!@cuf associate(b=>b_dev)
!$cuf kernel do(1) <<<*,*>>>
do i = 1, 1000
    a(i) = real(i) * b(i)
end do
!@cuf end associate
#ifdef _CUDA
print *,"GPU sum passed? ",sum(a).eq.1000*1001
#else
print *,"CPU sum passed? ",sum(a).eq.1000*1001
#endif
end program

## Device memory

dynamically allocated in host buprograms using the allocate statement, if it does not have the save attribute, it will be automatically deallocated when the subprogram returns

also applicable to scalars

real, allocatable, device :: b(:)
allocate(b(5024),stat=istat)
...
if(allocated(b)) deallocate(b)

automatic and local arrays, having the lifetime of the subprogram (as usual)

subroutine vfunc(a,c,n)
    real, device :: adev(n)
    real, device :: atmp(4)
    ...
end subroutine vfunc   ! adev and atmp are deallocated

allocating device memory using runtime routines (C-like behaviour)

real, allocatable, device :: v(:)
istat = cudaMalloc(v, 100)
...
istat = cudaFree(v)

using NV_CUDAFOR_DEVICE_IS_MANAGED=1 all device data allocation will be managed meory instead of device memory (potentially without coding changes)

Memory transfer

copy H2D (and vice-versa) is possible through simple assigment statements in the host program (using CUDA stream zero)

unless cudaforSetDefaultStream was used to associate one or more device and managed variables to a particular stream

an assignment with device variables on both sides copies the ata between two device variables or arrays
elemental transfers are supported (but perform poorly)
array slices are also supported, whereas the performance is dependent on the size of the slice, the amount of contiguous data in the slices and the implementation

further following expressions are legal:
a = adev
adev = a
b = a + adev
c = x * adev + b

## Synchronization

syncthreads - barrier synchronization for all threads in a single thread block

syncthreads_count - additionally evaluates the integer argument int_value for all threads of the block and returns the number of threads for which int_value evaluates to non-zero

syncthreads_and(int_value) - additionally evaluates the integer argument int_value for all threads of the block and returns non-zero if and only if int_value evaluates to non-zero for all of them

synthreads_or(int_value) - additionally evaluates the integer argument int_value for all threads of the block and returns non-zero if and only if int_value evaluates to non-zero for any of them

syncwarp - cause all executing threads within a warp, and specified in the mask argument, to reach a barrier

threadfence - all global and ahared memory accesses by the calling thread prior to threadfence() are visible to:

all the threads in the thread block for shared memory accesses
all the threads in the device for global memory accesses

threadfence_block - visible to all the threads in the thread block

threadfence_system

warp-vote operations:

allthreads: if( allthreads(a(i)<0.0) ) allneg = .true.
anythread: if( anythread(a(i)<0.0) ) allneg = .true.
ballot(int_value): evaluates the argument int_value for all threads of the warp and returns an integer whose Nth bit is set if and only if int_value evaluates to non-zero for the Nth thread of the warp
activemask - returns a 32-bit integer mask of all the curently active threads
...

Device intrinsics

Fortran intrinsics allowed in device subprograms

abs, aimag, aint, anint, ceiling, cmplx, conjg, dim, floor, int, logical, max, min, mod, modulo, nint, real, sign	* ...

CUF additions by NVIDIA:

...

## Runtime APIs

device management:

cudaChooseDevice
cudaDeviceGetAttribute
cudaDeviceGetCacheConfig
cudaDeviceGetLimit
cudaSetDevice
...

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.Michael Staneker seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.

Many thanks. Two quick remarks before I take a proper look:

You may need to add your ECMWF email address (assuming it is the one your commits are referencing) to your Github profile so the CLA check succeeds. Github allows to add additional email adresses, so no need to replace your current one.
Can you please rebase the branch on top of main to include the test configuration? This should then trigger the test workflow. (Same for the other PR, please)

Codecov Report

Merging #2 (b404306) into main (5eca276) will decrease coverage by 0.00%.
The diff coverage is 97.29%.

@@            Coverage Diff             @@
##             main       #2      +/-   ##
==========================================
- Coverage   90.58%   90.57%   -0.01%
==========================================
  Files          79       80       +1
  Lines       13929    13960      +31
==========================================
+ Hits        12617    12644      +27
- Misses       1312     1316       +4

Flag
Coverage Δ

lint_rules
97.34% <ø> (-0.02%)
:arrow_down:

loki
90.82% <97.29%> (+<0.01%)
:arrow_up:

transformations
82.73% <ø> (-0.32%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

loki/sourcefile.py
88.31% <83.33%> (-0.79%)
:arrow_down:

loki/backend/cufgen.py
100.00% <100.00%> (ø)

loki/ir.py
91.65% <100.00%> (+0.03%)
:arrow_up:

loki/program_unit.py
96.53% <100.00%> (+0.04%)
:arrow_up:

transformations/transformations/derived_types.py
98.46% <0.00%> (-1.54%)
:arrow_down:

...nsformations/transformations/single_column_claw.py
54.10% <0.00%> (-0.69%)
:arrow_down:

lint_rules/lint_rules/ifs_coding_standards_2011.py
97.34% <0.00%> (-0.02%)
:arrow_down:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.


Add field id word count docids database

This PR introduces a new database, field_id_word_count_docids, that maps the number of words in an attribute with a list of document ids. This relation is limited to attributes that contain less than 11 words.
This database is used by the exactness criterion to know if a document has an attribute that contains exactly the query without any additional word.
Fix #165
Related to specifications:#36

bors try

bors try

BTW, @LegendreM doesn't it fixes #196?


optionally add "Reviewed-by:" comments to merge

It would be great if hub had an option to add "Reviewed-by:" comments to the merge commit log.
For example, if I ran hub merge https://github.com/ceph/ceph/pull/13503 --reviewed-by, hub could use the reviewers API to discover who had approved the PR, and add the following lines to the merge commit log:
Reviewed-by: Casey Bodley<EMAIL_ADDRESS>Reviewed-by: Radoslaw Zarzynski<EMAIL_ADDRESS>
... or if it's too hard to discover the user's email, even just the github usernames would work.
Reviewed-by: @cbodley
Reviewed-by: @rzarzynski

This would make it a lot easier for us to record who approved a GitHub pull request in Git itself.

This is a very interesting feature idea; thank you.
I'm not yet sure whether this would be a good fit for hub core but I'm going to leave this open so I can think about it for a while. Other people have also requested that we improve the merge command so it can do merge by squash/rebase as per GitHub web interface. If that gets added, this feature would make a nice addition too.

If course this is also true for merging PRs on github.com !


Add Dark mode

The user should be able to choose their preference from the settings.
Here is the basic idea.

This UI is not implemented yet.

please assign this to me I can do it quickly and I would love to work on it

Sure. Let us know if you have any questions. Note: TailwindCSS for UI.

@gaurav-iitb Let me know if you need any help with the task.

Hey, if this is up for a re-assignment, I'd love to work on it!

@devkiran i would like to work on it.

@devkiran I would like to work on this issue.

Thanks for your interest. Let me collect the requirements for this feature and get back to you.

Thanks for your interest. Let me collect the requirements for this feature and get back to you.

Okay

I can contribute with the theme can you assign it to me

@devkiran can I upgrade the project with the shadcn-ui for the darkmode and various other components

No, you can't use shadcn-ui for this project at the moment. We might consider it in the future, but not right now.

@yacineahmaich assigning to you.


next/release

:tada: This PR is included in version 1.24.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


We need to add test run warning if browser provider doesn't support browser manipulation

Blocking on #573. Extracted from #671

I believe it was implemented in context of browser providers? \cc @AndreyBelym

Yes, it was implemented: https://github.com/DevExpress/testcafe/blob/master/src/browser/provider/plugin-host.js#L81

Closed via #573


The Function Apps seem to be automatically updated to the latest version of Master rather than the latest release version

As shown in the screenshot we are on 1.0.11090.0 (~1) but the latest release is 1.0.11075

This is correct, we didn't publish a release for that patch. We'll update the repo soon.


Streaming support for gRPC?

Hey,
thanks for creating this library! Is streaming (client streaming, server streaming, and bidirectional streaming) supported by this library?
I'm trying to create a streaming response, but it doesn't seem to be working.

Hey, thanks for the reply!
I'm quite new to Rust and gRPC, but I'll give it a try and see qhat I can do.


Add --no-deps flag to pip-sync, which is simply passed through to pip install

--no-deps can now be provided to pip-sync, whereby --no-deps will be passed through to pip install, and un-locked deps of locked requirements will be either removed or not installed in the first place, in order to match the lockfile exactly, despite breaking package dependencies.
Changelog-friendly one-liner: pip-sync now accepts --no-deps, to match locked txts exactly, ignoring any unlocked dependencies
Contributor checklist

[x] Provided the tests for the changes.
[x] Gave a clear one-line description in the PR (that the maintainers can add to CHANGELOG.md on release).
[ ] Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

@AndydeCleyre
Just wondering, why do you prefer to mark tasks this way:
[✓] Provided the tests for the changes.

instead of
[x] Provided the tests for the changes.

wich is the current Github's markdown syntax? See the doc.
Thie first one we get this progress bar on PRs list:

The second one marks tasks correctly:

Just wondering, why do you prefer to mark tasks this way?

Just because I never read those docs, and generally think of a check as "yes" and an x as "no," but thank you, now I know better.

@AndydeCleyre
There is an issue #321 the implementation of which can supersede this PR. Let me know what you think of it.

@atugushev If --no-deps functionality is enabled by an implementation for #321 that's great and should be merged instead. My only interest here is that there be an acceptable interface for people who don't want #907 behavior, so we can fix #896 and its sub-issue of sync behavior inconsistency.
Inviting @ulope and @Zac-HD to comment

Close this in favor of #1080.


WIP - STS-2144: Added forms not supported in v1 content creation error

Work In Progress - Do not merge

Why not set it as a draft PR?

But why did you do this one? The error code 40923 may now be taken for another error since it's the clear successor to be picked. 40050 and 40051 are "protected" by later codes. These error codes are supposed to be universal if they are exposed to partners and senders. They are also documented publicly here. Not saying this is all perfect though but in general we want one error code to only mean one thing.

But why did you do this one? The error code 40923 may now be taken for another error since it's the clear successor to be picked. 40050 and 40051 are "protected" by later codes. These error codes are supposed to be universal if they are exposed to partners and senders. They are also documented publicly here. Not saying this is all perfect though but in general we want one error code to only mean one thing.

This one was for removing errors that the forms service returns. This PR is for an error that kivra core returns.


Add exp2 op

#20498

Codecov Report
Attention: Patch coverage is 63.41463% with 15 lines in your changes missing coverage. Please review.

Project coverage is 60.01%. Comparing base (1354fde) to head (64e781b).

Files with missing lines
Patch %
Lines

keras/src/backend/jax/numpy.py
28.57%
5 Missing :warning:

keras/src/backend/tensorflow/numpy.py
28.57%
5 Missing :warning:

keras/src/backend/torch/numpy.py
16.66%
5 Missing :warning:

:exclamation:  There is a different number of reports uploaded between BASE (1354fde) and HEAD (64e781b). Click for more details.
HEAD has 6 uploads less than BASE

Flag
BASE (1354fde)
HEAD (64e781b)

keras
4
1

keras-torch
1
0

keras-tensorflow
1
0

keras-jax
1
0

Additional details and impacted files
@@             Coverage Diff             @@
##           master   #20505       +/-   ##
===========================================
- Coverage   82.10%   60.01%   -22.10%
===========================================
  Files         515      515
  Lines       47670    47711       +41
  Branches     7468     7474        +6
===========================================
- Hits        39141    28634    -10507
- Misses       6721    17325    +10604
+ Partials     1808     1752       -56

Flag
Coverage Δ

keras
60.01% <63.41%> (-21.95%)
:arrow_down:

keras-jax
?

keras-numpy
60.01% <63.41%> (+<0.01%)
:arrow_up:

keras-tensorflow
?

keras-torch
?

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Inconsequent use of Configuration(s) in menu, breadcrum and tabs

Describe the bug
Sometimes GUI 3.0 is using Configuration and sometimes Configurations.
The menu shows:
Configuration
Show Configuration
Manage Configurations
The breadcrum shows:
Configurations > Show (also add Configurations here? Breadcrum should reflect menu items?)
Configurations > Manage (also add Configurations here? Breadcrum should reflect menu items?)
At the top of the body page (below the tabs) you see:
Configuration (also add Show here?)
Manage Configurations
Category

Frank Console

To Reproduce
Steps to reproduce the behavior:

Run any recent framework version. For example clone and start Frank!Runner.

Expected behavior
Menu:
Configurations
Show Configuratios
Manage Configurations
Breadcrum:
Configurations > Show Configurations
Configurations > Manage Configurations
At the top of the body page (below the tabs):
Show Configurations or remove this line as it is already in the breadcrum. Hmm, I see other pages have this too but the names are not 1 on 1 the same as in the menu. For example "Test Pipeline" vs "Test > PipeLine" vs "Test a Pipeline". Would be better to sync menu breadcrum and page title.
Manage Configurations
Screenshots

Environment:

Application server: all
Browser: all
IAF Version: I think already a lot of old version but at least 7.5-20200214.142106

Additional context
None

For now, show configuration has been edited so the breadcrum is consistent with the menu item. The unnecessarily top line has also been removed.


C++ transpiler should not translate Val Int into C++ int

Val Int is 64 bits, while C++ int is 32 bits (on most modern plarforms).
Moreover, we shall translate Val Int into a C++ structure.

To be precise, although the current implementation picks 64-bit unconditionally, Int is supposed to be as large as the machine's word.


Rippled "Current ledger is unavailable"

Hi!
I'm running a MainNet Ripple Node (v0.90.1) on an Azure VM. (B4MS, 4 cores, 16gb memory).
Last night it seems it wasn't up with the latest ledger anymore. It kept returning { "error":"noCurrent","error_code":16,"error_message":"Current ledger is unavailable." }. I looked at the logs and noticed it was spamming "Validations:WRN Unable to determine hash of ancestor from ledger hash=[Random hashes here]".
I am running using the node_size set to huge, using nudb and the db is stored on a seperate SSD drive for performance reasons. It is being run as a docker image. The node isn't out of memory afaik, usually doesn't use more than 8gb
The problem resolved after i restarted the docker container (ie, the rippled process got restarted). +/- 8-9 hours later the problem occurred again.
Any idea what could be wrong or what i could do next time to see what is wrong?

The warning message you saw suggests your node was significantly behind the rest of the network. Causes could be network connectivity issues or an overloaded local server (slow disks are a frequent cause). It is not fatal though, and the node should automatically catch up with the network when the issue resolves.
It sounds like your server is adequate for the configuration you have. Does the issue occur regularly? Are there any other processes on the server that might be a cause? If you rule those out, the typical starting point for investigating rippled itself is to run [server_info](https://ripple.com/build/rippled-apis/#server-info) and [peers](https://ripple.com/build/rippled-apis/#peers) commands. I suggest running those 5-10 minutes after your server is started and running "normally" and keep them on hand to compare if you have trouble again.

Check your SSD i/o rates, eg with fio.  When running in a virtualised environment, depending on your provider, an "SSD" can sometimes have an i/o performance closer to HDD (and as such, insufficient for nudb).

Spinned up another node to experiment with. This one doesn't have any rpc load.
Had the problem again, this time on both servers. The first one had a gap in the ledgers. It got fixed after i requested the first missing ledger. Then after a minute or so it didn't have a gap at all anymore.
The second server had more gaps. When i requested the first missing ledger it gave my a response to that request saying it's not on the current ledger.
Here is a part of the logs of the first node (which currently has no gap(s)):
2018-Apr-26 07:13:32 LoadMonitor:WRN Job: processLedgerData run: 2114ms wait: 0ms
2018-Apr-26 07:13:32 LoadMonitor:WRN Job: acceptLedger run: 1729ms wait: 0ms
2018-Apr-26 07:13:32 LedgerConsensus:WRN By the time we got 0B56CDEDF3D7B11A3437B539E10734C51D598E37E2EB6428F1110550AF98FE72 no peers were proposing it
2018-Apr-26 07:13:33 LoadMonitor:WRN Job: advanceLedger run: 1057ms wait: 0ms
2018-Apr-26 07:13:39 LoadMonitor:WRN Job: acceptLedger run: 2056ms wait: 1ms
2018-Apr-26 07:13:39 LedgerConsensus:WRN By the time we got C15AC5507C13AF9F005C2F3D3EB0636E145962D6681B037BA9F3DFE8D6D472AB no peers were proposing it
2018-Apr-26 07:13:39 LedgerConsensus:WRN By the time we got 28194F3AC7687F74E32A61089E7DE4C288ED77087E207E5DA425FBCD52F23102 no peers were proposing it
2018-Apr-26 07:13:39 LedgerConsensus:WRN By the time we got C31660F191CCAE9F6A12316A7683DD892553B75206EF8B6B9946447649026844 no peers were proposing it
2018-Apr-26 07:13:43 LoadMonitor:WRN Job: processLedgerData run: 1183ms wait: 0ms

I would assume that many Warnings isn't a good thing and probably indicates an io issue. Will see if i can get some info regarding the i/o rates

After installing fio, you could try running something like this to get an idea:
./fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
If it helps to have some other figures for comparison, on Digital Ocean using the above, an external SSD Volume benchmarked 15/5MB/s r/w, and 3500/1000 IOPS r/w.  This is in the region of HDD for i/o (when RAID'd). When running rippled on such a configuration the i/o was much too slow and reported errors similar to yours.  Conversely, the local SSD boot drives benchmark 10-20x times those figures.  I've no experience with Azure VM but I'm guessing you may have a similar issue.


KMM State

A TCA-compatible State implementation in Kotlin multiplatform and an adapter to use with TCA in Swift.
interface State<S : State<S, Action, Environment>, Action : Any, Environment> {
    fun reduce(action: Action, environment: Environment): Pair<S, Effect<Action>>
}

This PR includes a SampleState with associated SampleStateView and SampleStateTests to demonstrate usage in Swift.
let state: SampleState = ..
let environment: SampleEnvironment = ..
let store: Store<SampleState, SampleAction> = Store(
    initialState: state,
    reducer: Reducer(SampleState.reduce),
    environment: environment
)

var body: some View {
    WithViewStore(store) { viewStore in
        HStack {
            Button("−") { viewStore.send(.DecrementButtonTapped()) }
            Text("\(viewStore.count)")
            Button("+") { viewStore.send(.IncrementButtonTapped()) }
        }
    }
}

Happy to see this!


When adding support for executable metadata formats, consider gray-matter style eval option

Basically a way to make the plugin safe to use on untrusted site data.

Eh, I think I'm actually going to hold off on this for now, mainly because I'm not sure how a system to make Metalsmith safe to use on untrusted site data would even work in the first place.
If the untrusted site has control of the metalsmith.json file, it could simply set the eval option on this plugin, rendering the currently proposed implementation useless. And if they don't have control of the metalsmith.json file, then you don't need this option at all; just don't enable the executable metadata formats.


Calculation of power coefficient using FAST

Hi,
In Simulink, I've just implemented a PMSG outside the FAST block. Now to calculate the produced power I need to have the exact value of c_p. However, I found an approximated dynamic of c_p in some literature. I wonder if I can get the c_p through the OutList. I am new to FAST.
Best regards,
Ehsan ASLMOSTAFA

Hi @EhsanASL92,
Are you using AeroDyn v15?  You can output the rotor aerodynamic power coefficient from the OutList of AeroDyn v15 via output RtAeroCp.
Best regards,

Dear @jjonkman ,
Thanks for your reply. I get the outList through the outData in Fast. It is an 831 array which I attached below. I don't know exactly how can I access the OutList of AeroDyn15! since in the 831 array I cannot find RtAeroCp?
matlab.zip

Dear @EhsanASL92,
I do see other AeroDyn v15 OutList parameters in this area, e.g., RtArea.  You can add RtAeroCp to the AeroDyn OutList within the AeroDyn input file and rerun you simulation.
Best regards,

Note that these outputs have recently been renamed RtFld* (RtFldCp for instance)

Thank you so much dear @jjonkman and @ebranlard,
Is it possible to explain how to add RtAeroCp to the AeroDyn OutList?
Best regards

Hi @EhsanASL92 , here's an example of AeroDyn input file where you can see different channels added to the OutList:
https://github.com/OpenFAST/r-test/blob/main/modules/aerodyn/ad_BAR_RNAMotion/OpenFAST_BAR_00_AeroDyn15.dat#L111
(between the line with the keyword OutList and the line with "END")

Thank you so much @ebranlard,
I tried to add "RtFldCp" to the outputs, but I get this while I tried to run FAST!
FAST_InitializeAll:AD_Init:SetParameters:SetOutParam:RtFldCp is not an available output channel.
Best regards

You must be using a version prior to 3.4, then you need "RtAeroCp" instead.

Thank you so much @ebranlard,
I am not sure about the version I am using. But I changed "RtAero" to "RtAeroCp" and see no difference in the output!
Is the power coefficient calculated for OC3 5MW NREL before so I can use the values for different pitch angles?
Best regards


feat(jsonpath-plus): Update to 7.1.0 => performance upgrade

Updated the version of jsonpath-plus to 7.1.0 to benefit from performance upgrades in JSONPath library. Just take a look at this pull request: https://github.com/JSONPath-Plus/JSONPath/pull/175 => You can see time reduction up to 50%, I personally even reached reduction up to 75%.
Would be great if could just release a new version on NPM with this change. Thanks :)

Thanks!

I'll still need to release it, but hoping to do it today or tomorrow.


Remove emulator testing code from gax.

We have replaced the emulator testing with the automated testing framework.

LGTM. RIP

Current coverage is 70.10% (diff: 100%)

Merging #134 into master will increase coverage by 8.18%

@@             master       #134   diff @@
==========================================
  Files            48         44     -4
  Lines          2227       1967   -260
  Methods           0          0
  Messages          0          0
  Branches        329        281    -48
==========================================
  Hits           1379       1379
+ Misses          771        511   -260
  Partials         77         77

Powered by Codecov. Last update ca69095...1968628


Feature - Add a new form layout

Just a form layout for a profile settings. This closes #395

I saw the reviews and will modify the file accordingly.
The layout kinda look like something from TailwindUI indeed, but this kind of grid layout is popular when designing a profile settings page.

Thanks for making the changes! I'm gonna pull it down and have a play with some ideas, tahnks!


update and set null: There are no changes to save. This query cannot be built

Which versions of Rust and Diesel are you using?
rustc 1.19.0-nightly (06fb4d256 2017-04-30)
diesel 0.12
Which feature flags are you using?
postgres, chrono, serde
What are you trying to accomplish?
generate a query to update fields to null
e.g.
UPDATE "tablename" SET "value" = NULL WHERE "id" = ?

What is the full error you are seeing?
panicked at 'called `Result::unwrap()` on an `Err` value: Error(DbResult(QueryBuilderError(StringError("There are no changes to save. This query cannot be built"))), State { next_error: None, backtrace: None
 })', /checkout/src/libcore/result.rs:859

Because None is a noop:
https://github.com/diesel-rs/diesel/blob/e52d1710d98a05c3aa4663017ec47dbb40e1d747/diesel/src/query_builder/update_statement/changeset.rs#L45
And struct with only None values can not pass this:
https://github.com/diesel-rs/diesel/blob/d44c50f0d95f28561971d5f73e45d2ed9a337652/diesel/src/query_builder/update_statement/mod.rs#L58
How can we reproduce this?
define a struct with only Option fields and derive AsChangeset
  #[derive(AsChangeset)]
  #[table_name="tablename"]
  struct Update {
    value: Option<i32>,
  }

Then update using this struct
  diesel::update(tablename::table.filter(dsl::id.eq(id)))
    .set(&Update {
        value: None
    })
    .execute(conn)?;

Please quote as much of your code as needed to reproduce (best link to a public repository or a Gist)
N/A
Please post as much of your database schema as is relevant to your error
N/A

http://docs.diesel.rs/diesel/query_builder/trait.AsChangeset.html

By default, any Option fields on the struct are skipped if their value is None. If you would like to assign NULL to the field instead, you can annotate your struct with #[changeset_options(treat_none_as_null = "true")].


hand dirCacheTime through again

I got confused for a moment when testing the google-drive-poll-changes patch why the dir cache was not honored. This is a very simple patch which might not needed depending on how many further changes are planned. In any case, it's a bandaid for people who use HEAD.

That looks fine.
There is now quite a bit of duplicated code in mount/cmount - most of the flags handling which makes me think we should factor it into mountlib.
I plan to use mountlib for serving other filesystem like things like webdav in due course.


Neta test cleanup features

Move integration tests to feature specs,  except:
javascript_test.rb - the teat is skipped so there is no point to transform it to a spec
search_for_user_test.rb - tests are commented out
Also, after merging all MR- the categories.yml, medialinks.yml, profiles.yml and test_helper.rb should be deleted

thx for your great work @netagonen <3 🌈🎉

finally merged! Thank you @netagonen and @zaziemo for the work!


Allow setting namespace in helm template without a kubeconfig

What this PR does / why we need it:
If you don't have a working kubeconfig, helm template does not respect the --namespace option, instead silently ignoring it and using "default". It gets set on the settings object, but does not get returned by the Namespace() method. helm template should not require a working kubeconfig or any communication with a kubernetes cluster, as it is only generating templates.
This PR adds a fallback to return the already set namespace field, rather than ignoring the user's intentions and returning "default".
Closes https://github.com/helm/helm/issues/12124
Special notes for your reviewer:
If applicable:

[ ] this PR contains documentation
[ ] this PR contains unit tests
[ ] this PR has been tested for backwards compatibility

@joejulian I copied their fix and added a unit test here https://github.com/helm/helm/pull/12979. I confirmed that without the fix being present the test fails.


The scrollable area for the Response is greatly reduced

@jemerald We will have to re-think how we display large responses. The change to not show the entire response because of concerns with the performance of multi-megabyte sized responses, has lead to decreased utility for the general case:

Yes, this looks particularly bad when the authentication tab is selected and token text is displayed (my current token text takes up to 35 lines), this why my layout change PR uses a fixed proportion between request and response.
One thing I can think of, is to automatically reduce the entire request section when API response is received allowing better response viewing, then user can hover/click it to bring it back?

Yeah, in the case where you have a large auth token (or request body, or headers), you could quite easily not be able to see the response at all:

This might be of use (recording for investigation later): http://stackoverflow.com/questions/198656/nested-scroll-areas/1571972#1571972

I'm going to temporarily revert so there is a scroll viewer around everything, at the expense of performance for very large responses.
I'll set up the next priority issue to make the Request and Response sections separate tabs, so that the response body can use AvalonEdit's virtualizing functionality better.


Black not configured in pre-commit to find src files?

I'm playing around with pre-commit and black together, and looked through scikit-hep/cookie for inspiration (with the now-recommended hatch backend), but when pre-commit runs, black doesn't check any files. I've gone through black's docs and looked at other projects a bit and can't seem to see how black is figuring out the files / directory to search. Is it possible that the the generated pyproject.toml in this instance doesn't pass black enough information? Otherwise is there something else I've missed?

Sheepishly closing this - it was a hasty issue creation. The commits I was experimenting with had no modified code for black to check. Apologies for the noise!


Add Kindergarten Garden Exercise

We didn't agree on this directly on this, but I mentioned that I would pick one to work on, so hope you will still accept it.
http://forum.exercism.org/t/implement-some-missing-php-exercises-for-48in24/9865/11

Thanks again for the contribution! [joke] I wonder how much you could do if you actually had time to add an exercise every two weeks [/joke].

haha.. As said in the forum multiple times, personal reasons. Perhaps I will tell you about it, some day, but not in public. :)


Add libc6-compat to base image

What would you like to be added:
$ apk add libc6-compat

to onecloud-base and host-base image.

Use bundle_libraries build image.


Use 'address' property of SocketAddress object

Bun's server.requestIP returns an instance of SocketAddress. This creates inconsistency between types string (returned when getting IP by headers) and SocketAddress (returned by requestIP method.)
This PR modifies the returned object of the Elysia derivation so that it uses the address property to return a string of the connecting user's IP address, consistent with the header-based check.

SocketAddress might have more information too, such as the ip type ipv4/ipv6 https://github.com/oven-sh/bun/blob/39101d16c3eee21ef732a61800f16c7c48a8fba3/packages/bun-types/bun.d.ts#L2221
if we didn't provide that, it may make the user install separate package for detecting the type of IP

Would it be better to then return a SocketAddress-like object containing the ipv4/ipv6 type? It should be trivial to add this,  could use a simple indexOf to find : and then set the IP type.

Either way, a string that is being returned from the headers doesn't contain the IP type, hence required the user to detect the type of IP, when that should be managed by the plugin.

i think it would be better to have something like dynamic types if typescript supports it
maybe something like this:
type ReturnedIP = 'Bun' extends keyof typeof globalThis
  ? SocketAddress
  : string | null | undefined;

I tried but couldn't get it to work, It always returned SocketAddress as the type


skipped adding usage when there is no leaf  domain available

What type of PR is this?
/kind bug

What this PR does / why we need it:
this pr adds a skip step if there is no leaf domain available so that panic is not thrown when a not is deleted and a new workload is scheduled
Which issue(s) this PR fixes:

Fixes #3693
Special notes for your reviewer:
Does this PR introduce a user-facing change?

/ok-to-test
I believe the code fixes the issue, but we need to have a test also. I think it could be a new test case here.

FYI I opened another PR to make sure this issue is addressed before the 0.10 release: https://github.com/kubernetes-sigs/kueue/pull/3715.

/close


Error in PosConsensusFeature.InitializeAsync

The code below uses Single but is is possible that the collection contains no elements:
        /// <inheritdoc />
        public override Task InitializeAsync()
        {
            NetworkPeerConnectionParameters connectionParameters = this.connectionManager.Parameters;

            // Replace CMB.
            bool oldCMBRemoved = connectionParameters.TemplateBehaviors.Remove(connectionParameters.TemplateBehaviors.Single(x => x is ConsensusManagerBehavior));
            Guard.Assert(oldCMBRemoved);
            connectionParameters.TemplateBehaviors.Add(new ProvenHeadersConsensusManagerBehavior(this.chain, this.initialBlockDownloadState, this.consensusManager, this.peerBanning, this.loggerFactory, this.network, this.chainState, this.checkpoints, this.provenBlockHeaderStore, this.connectionManagerSettings));

            return Task.CompletedTask;
        }

It leads to the following error:

ye, rised this question in #sidechain channel, now I'll take a look at their code, maybe it's the order of the injected features or something like that

Closing.


The Control+Enter key combination to run every tests in freecodecamp is not generic

Challenge Waypoint: Use CSS Selectors to Style Elements has an issue.
User Agent is: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/600.7.12 (KHTML, like Gecko) Version/8.0.7 Safari/600.7.12.
Please describe how to reproduce this issue, and include links to screenshots if possible.
The key combination provided to run every test in freecodecamp is not generic as I observed the fact that by pressing Ctrl+Enter in Mac OS X , will not lead to any result. I believe the key's must be generic in nature so as to neutralize the usage of the machine in which we are running the test. I felt this is a bug ,so raised

There was an issue with my laptop keyboard. Re-starting and re-trying the command worked!. Closing the issue.


SIL: fix memory behavior of bind_memory and rebind_memory

Since dead-store-elimination is now more accurate, it's important that the bind_memory instructions are also defined to read memory. Otherwise they are not considered as barriers for dead-store-elimination.
Fixes a miscompile.
rdar://112150142

@swift-ci test

@swift-ci benchmark


Allow tasks to recur for x times

⚠ Please check that this feature request hasn't been suggested before.

[X] I searched previous Ideas in Discussions didn't find any similar feature requests.
[X] I searched previous Issues didn't find any similar feature requests.
[X] I am only requesting a single feature. Multiple changes should be split into individual requests, with links between them.
[X] I believe my requested feature will be generally applicable to other users, not just me: it is not uniquely tied to my personal workflow.

🔖 Feature description
Hi, I've been really enjoying Tasks plugin. Thank you all for making it so powerful. I have a lot of recurring tasks in my daily workflow and a lot of them don't keep repeating forever. It would be great if Tasks can only recur for a certain occurences.
Workflow as below.

A user can specify a task that recurs "every day for 5 times".
When user checks the task to done, a new task is generated with "every day for 4 times" and an updated date
When user checks a task that recurs "every day for 1 time", a new task is not generated.

✔ Solution
I have read on documentation that Tasks plugin doesn't currently support recur tasks for x times because the plugin doesn't know how often it recurs. Rrule supports this function using parameter "COUNT".
So my proposal is, when creating the "next" task, check if "COUNT" exists,

if exists,

check if current COUNT=1

if so, don't generate "next" task
otherwise, update the rrule string with COUNT minus 1

else, continue with current logic

❓ Alternatives
No response
📝 Additional Context
No response

Thanks for checking the docs. Yes, we are tracking this one already in:

#1729

So closing as a duplicate.


A huge developer time saver, J="" make archives

This gives warnings about -j1 being forced some places, but
it still successfully builds all archvies in parallel. The
resulting archives have been tested on the board. It is
disabled by default so no functional change unless someone
adds J="" or J=8 etc. in front of the make archives.

I tried the more logical solution of just removing the -j1 flag and doing make archives -j, but this causes some issues in the build that are not remedied by the NOT_PARALLEL tag for the nuttx targets.

Applied via cherry-pick.


Update Extension

Steps to reproduce
Can you pls update the sekaikomik extension
Expected behavior
sekaikomik.pw is the newest version
Actual behavior
p
Mangayomi version
beta 0.29
Device
asus rog strix
Other details
No response
Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[X] I have written a short but informative title.
[X] If this is an issue with an extension, I should be opening an issue in the extensions repository.
[X] I have updated all installed extensions.

Steps to reproduce
Can you pls update the sekaikomik extension
Expected behavior
sekaikomik.pw is the newest version
Actual behavior
p
Mangayomi version
beta 0.29
Device
asus rog strix
Other details
No response
Acknowledgements

[x]  I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open or closed issue.
[x]  I have written a short but informative title.
[x]  If this is an issue with an extension, I should be opening an issue in the extensions repository.
[x]  I have updated all installed extensions.

Pls add sekaikomik extension


MWPW-145574 - Dynamic Nav

Adds code to add the gnav source to sessionStorage to allow for the same nav experience in subsequent clicks
entry pages will set gnavSource to sessionStorage is a gnav source meta tag is present
on pages will check for a gnavSource in sessionStorage, and if one is found, either modify the gnav-source meta tag, or create one
pages without the dynamic-nav metadata will do nothing or remove gnavSource from sessionStorage

Context:
The wiki in the ticket provides more context, but it was desired by our stakeholders that this only be a single tab experience.
Resolves: MWPW-145574
Test URLs:

Before: https://main--milo--adobecom.hlx.page/drafts/slavin/dynamic-nav/hugo-boss-case-study-entry/?martech=off
After: https://dynamic-nav--milo--jasonhowellslavin.hlx.page/drafts/slavin/dynamic-nav/hugo-boss-case-study-entry/?martech=off

To test:

Go to the after link. On the Hugo Boss page
Observe the nav (pricing and resources)
Right click "watch now" and open in a new tab
Observe the nav is the default milo nav ("Explore")
From the Hugo Boss page left click "Watch Now" to open in the same tab
Observe the pricing and resources nav loads

@JasonHowellSlavin - I was under the impression that this would go into bacom and not Milo. If we want to add this feature in Milo we should support more use-cases but at this point there aren't any requirements.
The custom code that you've built can be added in in scripts.js just before loadArea, this way you ensure that all the logic is executed before the navigation is actually loaded.

@JasonHowellSlavin - I was under the impression that this would go into bacom and not Milo. If we want to add this feature in Milo we should support more use-cases but at this point there aren't any requirements. The custom code that you've built can be added in in scripts.js just before loadArea, this way you ensure that all the logic is executed before the navigation is actually loaded.

I suppose no other teams replied to the discussion thread  so perhaps it is better to put this functionality in Bacom at the moment so bytes are not put in front of other consumers. Do you think it is preferable to move this code to bacom?
I chose to put it in checkForPageMods because there is similar code there checking for meta tags and running code based on those tags and properties. This is called before decorateDocumentExtras so I thought it would be fine, but if you see that there may be an issue with execution timing, I am happy to move it to scripts.js.
Thank you for your help and insight!

@JasonHowellSlavin Can you fix the lighthouse error? It'll be interesting to see the performance impacts.

@JasonHowellSlavin - but if we use the current approach, wouldn't a bacom nav be displayed on cc or the other way around if the meta is on?

I am concerned that the session storage will be passed on from project to another (e.g. bacom to cc) and the dynamic nav will be wrongfully inherited.

Hmm. That is a good point. I suppose we were assuming that the User Journey would not lead them out of the current project's pages, but the chance is not none.
Perhaps we could check the config. Not all configs have coderoot or imsClientId but all the ones I checked (cc/dc/bacom/milo/express/helpx) had one or both. We could use those values in sessionStorage to check against to ensure the same project.
I am a bit weary of adding something to the config as an identifier or something that authoring needs to author, like an additional metadata tag. What do you think?

@JasonHowellSlavin - I don't think that ims client ID or codeRoot are enough, we might have requirements where different set of pages in a project get different dynamic navigations. I think we need to add something else to distinguish between configs if this feature is going into Milo and can be used by all consumers.

@JasonHowellSlavin - I don't think that ims client ID or codeRoot are enough, we might have requirements where different set of pages in a project get different dynamic navigations. I think we need to add something else to distinguish between configs if this feature is going into Milo and can be used by all consumers.

I have now added a requirement for dynamicNavKey to be set in a project's config. This key can be the project's shorthand name, in our instance bacom. The dynamic nav will now check for that key, and if it does not find it, stop executing. I also moved the code to it's own file in order to keep things cleaner. Let me know what you think, and thank you.

Closing this PR in favor of https://github.com/adobecom/milo/pull/2180.
@narcis-radu I updated using your last comment here on the above PR. Thank you.


Add PGX to tokenlist

Add PGX to tokenlist file

This contract is not verified

Please verify
Add token logo to this PR

Thanks for the doing above. However, the PR's got all the history from another PR (#945) on its commit history which makes it seem very confusing.
Can you close this PR and open a new one with only the changes to PGX pls?


Resolve hosts so that we properly randomize any round robin RRsets

When using a round-robin DNS record set for a Zookeeper ensemble (i.e. a single A record that points to all the IP addresses of the servers in the ensemble), we found that Kazoo was always connecting to the same server. It appears that the underlying socket calls that Kazoo uses to set up connections do not rotate the addresses in a round robin properly (if at all).
The simplest solution to this is to use a call to socket.getaddrinfo in the collect_hosts method when compiling the list of hosts from the connect string. This allows any hostnames to be dereferenced and provides a full set of IP addresses that can then be randomized by the client. This works regardless of whether a hostname, an IPv4, or an IPv6 address is provided, as socket.getaddrinfo handles it properly and returns the tuples needed for the socket calls.

The test failure appears to be unrelated to anything I changed. Closing and reopening to try a fresh build

Continuing transient failures - a different set this time.

It's unclear to me why the build is failing. The errors change on each run, and they appear to be something inherent to either Travis CI or the test suite, not the changes I have made. The initial failure was due to a test that needed to be changed for the new code, but that's it.

OK, now that this is passing, I think it can be merged!

Thanks @rgs1. That's a good catch. All fixed now with explicit mapping of the tuple.

+1 on this, but only after the CHANGES.rst is updated with a short description of the fix

It would seem that CHANGES.rst is more appropriate to be modified when the next version of kazoo is cut by the kazoo team. Which may or may not include just this PR.
Can we get this merged?

So what I would like and it doesn't seem to hard is to add the following to the CHANGES.rst file.
2.X.Y (TBD)
-----------

Features
********

Bug Handling
************

Documentation
*************

Add your change under the appropriate section and then it doesn't become the kazoo approvers job to track all the things that go into a release (and saves the volunteers that maintain this library some work).
Seem doable?

Done, @harlowja, and managed to get it to pass travis-ci again.

Thanks much :+1:

I ran into this issue recently, and would love to see this merged in!

Just one comment that I'd like cleared up, now that I am thinking about what this means for people that were giving kazoo a CNAME and altering (after the fact) the A records under that CNAME, it seems like doing this resolution of A records at set_hosts call time would basically lost the advantage of having a CNAME in the first place right?

Maybe it's something like:
def _expand_hosts(host, port):
    for rhost in socket.getaddrinfo(host.strip(), port, 0, 0, socket.IPPROTO_TCP):
        yield (rhost[4][0], rhost[4][1])

def _connect_loop(self, retry):
    # Iterate through the hosts a full cycle before starting over
    status = None
    hosts_ports = []
    for host, port in self.client.hosts:
        hosts_ports.extend(_expand_hosts(host, port))
    for host, port in hosts_ports:
        if self.client._stopped.is_set():
            status = STOP_CONNECTING
            break
        status = self._connect_attempt(host, port, retry)
        if status is STOP_CONNECTING:
            break
    if status is STOP_CONNECTING:
        return STOP_CONNECTING
    else:
        raise ForceRetryError('Reconnecting')

Yeah, I can definitely see where this can cause problems. While it may be fine for short-lived use cases that connect and go away, anything else would have problems with changes to the underlying DNS (which is one of the reasons we use RRsets in the first place). I like the idea of doing it in the connection handler upon each connection attempt. That's more in line with when DNS resolution is done now.
Given that this is still a PR, I'm going to drop this PR and create a new one with the changes. No reason to have speculative changes in the history.

Resubmitted at PR #372


The directive is not re-rendered to show datetime selection when configuration options is changed

When I changed from 'minView': 'day' to 'hour' in the controller the directive is not giving the time selection.
I have this. When i change the configuration from controller
<datetimepicker data-ng-model="data.start_date"  data-datetimepicker-config="{startView:'day',  minView:'day'}"/>
<datetimepicker data-ng-model="data.start_date"  data-datetimepicker-config="{startView:'day',  minView:'hour'}"/>
But I am not able to select time?

Please create a fiddle or plunker that demonstrates the problem.

Closing due to lack of response from OP


BigQuery KMS should not be used on VIEWS

Describe the bug
If I add one kms key for the entire project that should be used for all models, like below. It will blow up, because you can't add KMS encryption to views. ( Unless I'm missing something here )
models:
  test:
    +kms_key_name: 'projects/<project>/locations/<location>/keyRings/key-ring/cryptoKeys/key'

Steps To Reproduce
Add kms_key_name to the entire project or specific folder that contains views.
Expected behavior
It should skip views, and only add the encryption key to the tables.
System information
Which database are you using dbt with?

[ ] postgres
[ ] redshift
[x] bigquery
[ ] snowflake
[ ] other (specify: ____________)

The output of dbt --version:
dbt --version
installed version: 0.20.0
   latest version: 0.20.0

Up to date!

Plugins:
  - bigquery: 0.20.0
  - snowflake: 0.20.0
  - redshift: 0.20.0
  - postgres: 0.20.0

The operating system you're using:
MAC
Additional context
Should probably be in its own issue, but I would like to be able to add default encryption key to the entire dataset that DBT creates. Maybe this is possible, but I didn't find anything about it :)
Cheers!

Thanks for the report @AndreasTA-AW! I agree, dbt-bigquery should be smarter here about which configs can apply to which materialization types.
We have an adapter method called get_table_options:
https://github.com/dbt-labs/dbt/blob/159e79ee6ba86c852f4bf067dd0f1b6356135383/plugins/bigquery/dbt/adapters/bigquery/impl.py#L771-L805
That's called in a Jinja macro, bigquery_table_options, which templates out the appropriate string:
https://github.com/dbt-labs/dbt/blob/159e79ee6ba86c852f4bf067dd0f1b6356135383/plugins/bigquery/dbt/include/bigquery/macros/adapters.sql#L31-L40
Then that stringified version is included in the create view as statement:
https://github.com/dbt-labs/dbt/blob/159e79ee6ba86c852f4bf067dd0f1b6356135383/plugins/bigquery/dbt/include/bigquery/macros/adapters.sql#L61-L70
I can think of two possible resolutions here:

Add conditional logic within the get_table_options method that checks config.get('materialization') for properties like kms_key_name that won't work on views.
Create a new method, get_view_options, that serves the same purpose as get_table_options but excludes any configs that are unsupported on views. Then the bigquery__create_view_as macro would call bigquery_view_options --> adapter.get_view_options.

I'm leaning toward the second option. The first obfuscates differences, and it won't account for custom materializations. By contrast, mapping in the second (create view as --> get_view_options, create table as --> get_table_options) feels right. The only challenge will be implementing get_table_options + get_view_options in a way that doesn't duplicate a bunch of logic.
This feels like a straightforward and self-contained change, so I'm going to mark it a good first issue. Would you be interested in contributing the fix for it? :)

Thanks for the quick response! :)
I looked at the code before I submitted the PR and it looks like it worked before a major refactor of the code. Before, I think it only triggered for tables, but now when it’s in python, I think i triggers for both tables and views. I do agree with the second solution though, to keep them isolated.
I will check if I can put some time to submit a PR, I’m a consultant and a father and all that :P but I can try to find some time.
Also, wanna let u know that DBT is a great tool and it makes life a lot easier. Ofc it got some flaws, like everything, but it’s getting there!
cheers!


LineGraph Graph Fix

Line's bug fixed, now it's shown without problem.

The reason that there is a flag to update the points is to prevent updating the points more than once per frame, calling it like this directly is going to cause unnecessary updates. This code is not going to perform well in big graphs.

What is this PR fixing exactly?

No longer required, fixed by https://github.com/GlistEngine/GlistEngine/pull/928


Gauge init clobbers existing .gitignore

Context: I've been trying out gauge for adding acceptance tests to an existing project.
Expected behavior
gauge init should add to an existing .gitignore if present
Actual behavior
gauge init clobbers existing .gitignore
Steps to reproduce

Create a repository with a .gitignore that has some content and commit
Run gauge init java
See the .gitignore's previous content has been clobbered:

--- a/.gitignore
+++ b/.gitignore
@@ -1,2 +1,4 @@
-/build/
-
+.gauge
+logs
+gauge_bin
+reports

Gauge version
Gauge version: 0.9.1

Plugins
-------
java (0.6.4)

Thanks for reporting this. I can replicate this.
But I noticed something:

If the target directory is not empty, but is a gauge project, gauge detects that and errors out with the message that This is already a Gauge Project...
However gauge does not check if the target directory is actually empty. Whereas gauge tempates assume so, since they write to files but not append or skip if they exist.

So question is - what should be the behaviour?
/cc @getgauge/core

Fixed through PR #788. Should be available in nightly >= 30 Aug 2017.

New line should be added after gauge directories in .gitignore, so it is easier to differentiate between user's ignored files and ones added by gauge.
Current file has
# gauge reports generated by reporting plugins
reports
myIgnoredDir

Should be
# gauge reports generated by reporting plugins
reports

myIgnoredDir

Also, .gauge has more than just failures, can change # failures metadata dir to just # failures metadata dir

@nehashri The required changes have been done in language templates.

Comments are not shown for gauge initialization without internet, gauge initialization of csharp template.
initializing maven template is not appending to existing .gitignore, gauge contents are not added.

Initialization of Maven templates is done by mvn, it fetches the project files from archetype an put in the target dir hence it overwrites the existing files.
Required changes for offline installation are done in language runners.
Fix should be available in nightly >= Gauge.05-09-2017, gauge-java.05-09-2017 and gauge-ruby.05-09-2017


Update docfx.json

The directories for the versioned packages prototype did not have entries for ms.topic or ms.service so they were showing up as blank.  Added appropriate entries.

Docs Build status updates of commit f4e6cdb:
:white_check_mark: Validation status: passed

File
Status
Preview URL
Details

docfx.json
:white_check_mark:Succeeded

For more details, please refer to the build report.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Warning about no README when readme = false

🐛 Bug description
I have some crates nested as subdirectories, one of which is the crate I'm using with wasm-pack. When building it, it reports:
[WARN]: :-) origin crate has no README

It similarly warned about having no license until I changed my Cargo.toml to the following (relevant to both points above):
[package]
readme = false
license-file = "../../LICENSE.txt"

This stopped the complaints about the missing license, at least, which I have only in my repo/workspace root.
🤔 Expected Behavior
While not a critical problem, it would be nice if the Cargo-standard setting of package.readme = false meant that wasm-pack didn't try looking for it.
👟 Steps to reproduce
My repo (simplified) is set up like this:
README.md
LICENSE.txt
Cargo.toml <- workspace.members = [ "crates/foo", "crates/bar" ]
crates/
  foo/
    Cargo.toml <- package.readme = false, package.license-file = "../../LICENSE.txt"
  bar/ <- the wasm project
    Cargo.toml <- package.readme = false, package.license-file = "../../LICENSE.txt"

🌍 Your environment
wasm-pack version: 0.10.3
rustc version: 1.66.1 (90743e729 2023-01-10)

For triage / discussion purposes, I wanted to mention here in the issue that the 2 tests for readme.rs were doing exactly the same thing, so I also fixed that. This revealed that wasm-pack wasn't actually honoring an authored package.readme location. So this change fixes both that, as well as allowing for a boolean value per documentation.


Update Adobe CMAP resource github page URL

I was trying to make a clone of PDF Miner on Go that is why I was looking into source code. The link given in the docstring was broken. Updated with current CMAP github resource page.
Pull request
Please remove this paragraph and replace it with a description of your PR. Also include the issue that it fixes.
How Has This Been Tested?
Please remove this paragraph with a description of how this PR has been tested.
Checklist

[x] I have read CONTRIBUTING.md.
[x] I have added a concise human-readable description of the change to CHANGELOG.md.
[x] I have tested that this fix is effective or that this feature works.
[x] I have added docstrings to newly created methods and classes.
[x] I have updated the README.md and the readthedocs documentation. Or verified that this is not necessary.

Top! Thanks for your contribution!


Build upon default config - Closes #1248

Yep, I just wanted to open this as soon as possible to get some feedback on the new flags. I think they are good and necessary :).
However I also think about deprecated failFast inside the config. This should be a explicit flag and not just an option somewhere hiding in the config?

Hm, the new flags are not usable from the gradle plugin until the next version ...
So split this pr for cli and gradle?


Add Management Group Registration Capabilities

Add MG registration api
Swagger PR link

Please provide a link to the swagger PR in the comments. To release this change, you will need to update the version (in the .csproj and assemblyinfo.cs files). Also, if this change involves a new api-version, then you will need to update the Az.Sdk.RP.props file in this directory to update the api-version tag

This is not a new api-version. The assemblyinfo.cs and .csproj files are now updated


convert pkg/util/tagger and comp/core/tagger/types to module

What does this PR do?

Converts pkg/util/tagger and comp/core/tagger/types to go module
Motivation

OTEL-1995: this is a dependency for creating a comp/otelcol/otlp/components/processor/infraattributesprocessor module
Additional Notes

Possible Drawbacks / Trade-offs

more modules to maintain in modules.py, unit tests, etc.
Describe how to test/QA your changes

run invoke unit and integration tests (through pipeline)

/merge


How can you use this to cluster textual data

Hello,
I'am interested to know is there any provision to cluster text data, I see that you are taking coordinates as input. Can you explain how to extend it for clustering a set of text documents.

The usual way to cluster text documents is made as follows:
1 - the data is transformed using the TF-IDF (https://en.wikipedia.org/wiki/Tf–idf) normalization
2 - the proper clustering technique (as DBSCAN can be) is applied to the transformed data, by using the cosine distance measure
As far as the TF-IDF transformation is regarded, you've to implement it by your own or to look to existing implementations. Instead, for the DBSCAN you can use this fork from me (https://github.com/speedymrk9/spark_dbscan) which extends @alitouka's repo allowing the usage of the cosine distance measure, which is the one you need.
Hope I've been clear, I'm at your disposal for any doubt.

but if the cosine distance is found the memory complexity would be O(n^2) right. I meant is there any way one can convert the text data into the coordinates your package requires. So that I can use your efficient implementation.

Sorry but I have not understood what you've said. May you explain me? The way to convert text documents into coordinates is the TFIDF transformation.
----- Messaggio originale -----
Da: "D. Rajeev. Reddy"<EMAIL_ADDRESS>Inviato: 01/09/2015 17:13
A: "alitouka/spark_dbscan"<EMAIL_ADDRESS>Cc: "Marco Gaido"<EMAIL_ADDRESS>Oggetto: Re: [spark_dbscan] How can you use this to cluster textual data (#7)
but if the cosine distance is found the memory complexity would be O(n^2) right. I meant is there any way one can convert the text data into the coordinates your package requires. So that I can use your efficient implementation.
—
Reply to this email directly or view it on GitHub.


SAK-2907 PR zfm2

Changed CONTRIBUTING.md and the licence header template a really useless way. Awaiting job offer from Red Hat.

NOTE: I rebased so there is no history of multiple commits.


Hour of Code Landing Page

Hour of Coding Landing page looks great!
Just a couple things:

[ ] Can we make all the links pop open in a New Tab by default?
[ ] Let's use an aka link for the Asphodel game.  We can change the target later - http://aka.ms/asphodel
[ ] Change text for Play and Code descriptions: Play - "Place the arrows in the maze to help Asphodel get to her trophy", Code - "Follow the tutorial to code your own Shark  Attack game!"
[x] Missing Design Tutorial - https://github.com/microsoft/pxt-arcade/issues/2524
[x] Missing Certificate (I can come up with something if you need)

[ ] links color are an unfortunate mix

[ ] can we add a makecode.com/hour-of-code redirect page -- it's getting confusing to figure out where the hour-code is.

I made an aka link - http://aka.ms/arcadeHOC

Design game is I'd say 70% done :D the bones are all there, I just gotta pretty it up and hide the blocks and make more example costumes

Quick Request @ganicke -

[ ] Can you link the Certificate button to https://1drv.ms/b/s!AqsgsTyHBmRBjyzY4tqc7dhNDJTX
[ ] Can you switch out the Design activity image to

OK, I'll add those.

These "action-buttons" overlay some of text in the panel...

Fixed.


[Do not merge] Add ruby 2.4.0 as an available rbenv ruby

needs https://github.com/alphagov/packager/pull/127 to be merged and the resulting built package to put in our apt repo


Cannot import database: language "plpgsql" does not exist

Hello fellow people,
I'm trying to import an existing database with the following config:
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: postgresql-tbnext-test
  namespace: test
spec:
  instances: 1

  imageName: ghcr.io/cloudnative-pg/postgresql:14.5

  postgresql:
    parameters:
      shared_buffers: 32MB
      pg_stat_statements.max: '10000'
      pg_stat_statements.track: all
      auto_explain.log_min_duration: '4s'

  bootstrap:
    initdb:
      owner: superuser
      import:
        type: microservice
        databases:
          - postgres
        source:
          externalCluster: postgres-test-10

  externalClusters:
    - name: postgres-test-10
      connectionParameters:
        # Use the correct IP or host name for the source database
        host: postgresql.test.svc.cluster.local
        user: admin
        dbname: postgres
        sslmode: disable

      password:
        name: db-test-secret-new
        key: password

  storage:
    size: 1Gi
    storageClass: pd-standard

  resources:
    requests:
      memory: "128Mi"
      cpu: "0.1"
    limits:
      memory: "128Mi"
      cpu: "0.1"

Dumping the data from the old database now runs smooth, but recovering into the new db runs into an issue. Some key functionalities of our application are written in plpgsql. Somehow the current container does not accept these (ERROR:  language "plpgsql" does not exist") with the remark to use the 'CREATE EXTENSION' method. But at this stage this is impossible for me.
How can I solve this issue?
Here are the logs:
{"level":"info","ts":1665755637.0472665,"logger":"pg_restore","msg":"pg_restore: from TOC entry 437; 1255 1157999 FUNCTION abc_method(integer, date, integer, integer, uuid) admin","pipe":"stderr","logging_pod":"__"}
{"level":"info","ts":1665755637.047276,"logger":"pg_restore","msg":"pg_restore: error: could not execute query: ERROR:  language \"plpgsql\" does not exist","pipe":"stderr","logging_pod":"postgresql-tbnext-test-1"}
{"level":"info","ts":1665755637.0472832,"logger":"pg_restore","msg":"HINT:  Use CREATE EXTENSION to load the language into the database.","pipe":"stderr","logging_pod":"__"}
{"level":"info","ts":1665755637.0472946,"logger":"pg_restore","msg":"Command was: CREATE FUNCTION sdfkj_method(integer,  date,  integer,  integer,  uuid) RETURNS TABLE(id integer)","pipe":"stderr","logging_pod":"__"}

Thanks @jacobfederer for posting this. We'll look into it.

I ran into this today as well


No version tags/releases

Hello!
I have noticed that after version v0.5.8 there are no tags, even though we are currently in version v0.5.11
Since I'm part of a team trying to maintain a fork, it would be great help to actually have tags on the actual version commits. That way we can base off a version and update as we go.
Thank you in advance

You are absolutely correct -- this should not happen again.  Apologies.

Really appreciate it :) thank you

Thanks for bringing this up.  I will be following up on this.  I'm currently working on a defined codeline / release plan.  I'd like to hold this open until that plan is in place and agreed upon, then we'll close this issue.


Create release v1.2.0

Create an official v1.2.0 release.
Review and update release docs as needed.
Dependencies:

https://github.com/hyperledger/indy-node/issues/1828
https://github.com/hyperledger/indy-plenum/issues/1645
https://github.com/sovrin-foundation/token-plugin/pull/459

Several updates were needed to complete the release:
Sovrin

https://github.com/sovrin-foundation/sovrin/pull/360
https://github.com/sovrin-foundation/sovrin/pull/359
https://github.com/sovrin-foundation/sovrin/pull/357
https://github.com/sovrin-foundation/sovrin/pull/356
https://github.com/sovrin-foundation/sovrin/pull/355
https://github.com/sovrin-foundation/sovrin/pull/354

Token Plugins

https://github.com/sovrin-foundation/token-plugin/pull/459
https://github.com/sovrin-foundation/token-plugin/pull/458
https://github.com/sovrin-foundation/token-plugin/pull/457
https://github.com/sovrin-foundation/token-plugin/pull/456
https://github.com/sovrin-foundation/token-plugin/pull/455
https://github.com/sovrin-foundation/token-plugin/pull/454
https://github.com/sovrin-foundation/token-plugin/pull/453
https://github.com/sovrin-foundation/token-plugin/pull/452

Indy-test-automation

https://github.com/hyperledger/indy-test-automation/pull/138
https://github.com/hyperledger/indy-test-automation/pull/136


Set up a screenshots server

in the vein of https://screenshots.debian.net/
We need a screenshots server for the AppCenter app. Ideally this ties into AppHub eventually.

What about d3 ?

Screenshots should be trivially small, I'm not sure we need to outsource them immediately, just make sure they are abstracted.

If you#re using AppStream, you pretty much just need to copy the media/ output directory to some location accessible via HTTP(S) and set the MediaBaseUrl in your data accordingly.
This looks like https://appstream.debian.org/media/ on Debian.
For the whole Debian archive (all active suites & sections), this is currently 340MB.

Ok good to know, but this doesn't seem related to MVP right ? maybe we could create another repo or something ?

You are correct, this should be separated from mvp, as it will most likely grow with more features.

This is a bit more than a little script to write, we need to choose the techs, the format... (API ?)... We should speak about that on Slack first I guess.

I'm interested in working on this part if you guys could share some thing i can start with ?

A $5 digital ocean droplet with a volume attached should work fine for this.

Probably the more interesting/difficult part is figuring out how AppCenter gets the screenshots and/or how Houston gets those screenshots to this server. Probably would be convenient to discuss API with Corentin in Paris

The developer sets the url links in the appstream file (reference).  Then when you run appstream-generator it copies the links to a local directory that is used to serve them.  If this is correct then the screenshot server should already basically be setup when we setup the repo server. I am not 100% sure what I just said was accurate, as I have yet to test anything on it.

@btkostner is correct ;-)
asgen will download and resize the screenshots and then publish them in some user-defined location and set a user-defined link in the resulting collection AppStream metadata.
(so, you will get a bunch of image files in a defined directory structure which you can place on any server you want - you just need to tell asgen where they will be reachable from the web)

Yeah we want to bypass that because:

There's no validation.  If we have a web interface we can at least verify that they are a sane size and/or make screenshot checking part of the review process.
It's kind of complicated. Developers still have to 'host' the screenshots somewhere for us to copy them from.

So really this is an elementary/infrastructure issue for the server and an elementary/houston issue for the front-end.

@danrabbit Well, the whole point of having screenshots in AppStream was to give upstream projects contol over them, and not users - because the screenshots on services like screenshots.debian.org sucked really bad for many applications.
But you obviously can replace any screenshots you want - at some point in future asgen will grow some screenshot override logic anyway, because we want this in Debian (so we can veto upstream-provided screenshots that may contain copyrighted material).

@ximion Yeah I just want to make sure we (as the distributor) have the final say so we can, like you said, veto upstream screenshots that might contain copyrighted content or just be of low quality

Houston still needs to validate the app before it makes it to the main repo, so it doesn't bypass the validation step. We just need to add an image viewer step validation thing.
I was thinking about this a bit some time ago. One solution I got was hosting the images in git, with the code. Then redirecting the urls in appstream code to a raw.github proxy (probably hosted with houston) to fetch the images. This has the advantage of being easy for the developer. Bad thing being is that we would probably have to break appstream spec and use absolute paths in the appdata file.

It is not recommended to serve anything from raw.github, it purposefully doesn't return correct content type. You want us to proxy them rather than store them? With some caching, what's the difference?

@btkostner Yeah just including them in the git repo would probably be easy. I still think it would be nice to get immediate feedback and a bit of hand-holding though. Otherwise you have to go through the submission process to find out your screenshots are too small or a weird aspect ratio and submit a new version of your app just to fix it

The proxy is to fix the content-type header for asgen. Past that everything would continue as it normally would with asgen grabbing the file.

@btkostner I don't see any reason why this would require any black magic in AppStream.
Only the generator would need to special-case Github URLs, but we can do that with less effort - patches welcome ;-)

@ximion I was going to have houston rewrite the urls when it clones the code and builds it.
I'll keep you posted when I start working on the asgen screenshots to make sure we can get this done the best way possible.


Honda block lane change on blindspot alert

Working on 2017 CR-V
BSM radars output a message on B-CAN with a bit that indicates whether the BSM indicator is on. A white panda acts as gateway to forward B-CAN messages to a CAN bus that openpilot has access to.
The fake ethernet port is the easy way to get messages from B-CAN to openpilot because it is already run down to the OBD port which is close to several connectors with B-CAN.  However, the fake ethernet connector has a CAN0 which on Honda vehicles with a Bosch radar is the radar bus.  This means B-CAN is being forwarded to the radar bus, which is annoying, but does work with the addition of another CAN parser in openpilot.
Is this an acceptable approach for integrating B-CAN in general?  The BSM alert integration is the first of multiple features that would be a nice additions (controlling blinkers/hazards, locking doors through athena, etc...)
B-CAN DBC
BO_ 318291879 BSM_STATUS_RIGHT: 8 XXX
 SG_ BSM_ALERT : 4|1@0+ (1,0) [0|1] "" XXX
 SG_ BSM_MODE : 6|2@0+ (1,0) [0|3] "" XXX

BO_ 318291615 BSM_STATUS_LEFT: 8 XXX
 SG_ BSM_ALERT : 4|1@0+ (1,0) [0|1] "" XXX
 SG_ BSM_MODE : 6|2@0+ (1,0) [0|3] "" XXX

VAL_ 318291879 BSM_MODE 2 "blind_spot" 1 "cross_traffic" 0 "off";
VAL_ 318291615 BSM_MODE 2 "blind_spot" 1 "cross_traffic" 0 "off";

example dbc
gateway panda setup:

CAN0 on gateway panda connected to fake ethernet CAN0
CAN1 on gateway panda connected to B-CAN
flash with firmware that forwards B-CAN to fake ethernet CAN0 example code

easy places to connect to B-CAN:

6 pin accessory lighting connector (may be present on many Honda vehicles)

image from here
Blue 36 pin body control module connector (may be specific to CR-V and Civic)

image from here

depends on https://github.com/commaai/opendbc/pull/286

You need to add the dbc file to the release file list for the tests to pass. https://github.com/commaai/openpilot/blob/master/release/files_common#L522

Updated opendbc and bumped master, so this PR doesn't need to touch the opendbc submodule.
Can you add a comment with a link to this PR somewhere in the carState code? Then people know how to place the proxy panda.

changes complete, tests now passing


Microsoft Teams schema

Hi,
We're trying to figure out what is the purpose of the Microsoft Teams schema (https://docs.microsoft.com/en-us/office/office-365-management-api/office-365-management-activity-api-schema#microsoft-teams-schema).
The description is quite generic,

"Extends the Common schema with the properties specific to all Microsoft Teams events."

and we can't find any content on our tenant.
Does anybody know what it's supposed to contain/ be used for ? Is there any roadmap of adding Teams content to the API ?
Regards

@JulienRavet The Teams schema is used to return auditing events related to Microsoft Teams. See:
https://docs.microsoft.com/en-us/microsoft-365/compliance/search-the-audit-log-in-security-and-compliance#microsoft-teams-activities
https://docs.microsoft.com/en-us/microsoftteams/audit-log-events

Thank you for the link, very useful.
Do you know if having the Teams user activity (message, like, etc..) in the API can be expected in the future ?


added fragment for about page

added AboutFragment and removed about dialog

Can you add a shortcut for containers to be used as a widget to lunch the game


chronos getting stuck

Hi,
We are trying to get to using chronos in the production environment but while testing it, every so often chronos stops scheduling jobs on mesos. Our environment is Centos 6.4 running Linux 2.6.32-358.el6.x86_64. The logs dont point to anything obvious and both the mesos and chronos UI's are still up. Any pointers?
Thanks guys!

Here is some info from the zookeeper log which seems a little off:
2015-04-01 20:21:15,997 [myid:] - INFO  [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when processing sessionid:0x14c76cd82100014 type:create cxid:0x551c6c28 zxid:0x1bff txntype:-1 reqpath:n/a Error Path:/chronos/state/state Error:KeeperErrorCode = NodeExists for /chronos/state/state

Sorry I should have probably included this earlier itself.

java -Xmx512m -Djava.library.path=/usr/local/lib:/usr/lib64:/usr/lib -Djava.util.logging.SimpleFormatter.format=%2$s %5$s%6$s%n -cp /usr/bin/chronos org.apache.mesos.chronos.scheduler.Main --zk_hosts <IP_ADDRESS>:2181 --master zk://<IP_ADDRESS>:2181/mesos --http_port 4400
java version "1.7.0_45"
OpenJDK Runtime Environment (rhel-<IP_ADDRESS>.el6-x86_64 u45-b15)
OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode)

Could you try removing the chronos state in zookeeper by running the zkCli with the command "rmr /chronos/state"?

Same problem here, I tried removing /chronos/state and it is still stuck. It registered as new framework because the frameworkId is stored under /chronos/state/state/frameworkId, now I have 2 chronos registered but only 1 really running.

Could you try running curl -d "frameworkId=YOUR_FRAMEWORK_ID" -X POST http://YOUR_MESOS_URL:5050/master/shutdown to shutdown your original Chronos framework?

The old framework is listed as terminated framework now. thanks.
$ curl http://head1:5050/master/shutdown -d frameworkId=20141202-035938-1405352852-5050-24734-0001 -XPOST -v
> POST /master/shutdown HTTP/1.1
> User-Agent: curl/7.35.0
> Host: head1:5050
> Accept: */*
> Content-Length: 54
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 54 out of 54 bytes
< HTTP/1.1 200 OK
< Date: Thu, 16 Apr 2015 20:30:30 GMT
< Content-Length: 0
<
* Connection #0 to host head1 left intact

I can confirm It started working after reseting /chronos/state and removing old chronos frameworkid from mesos.

Great, any objections to me closing this issue?

none from me, @vikhyath ?

@elingg thanks! does this make it a permanent solution or will chronos get stuck again after a while? Is there any insight you can give us, as to why this might be happening?

it stopped working for me once I started the second chronos master.

I have 2 servers running chronos using the same zookeeper uri, both can connect to zookeeper and mesos. Chronos UI at port 4400 show the same list of jobs on both, so I am pretty sure they are configured to use same zookeeper server and mesos master.
ii  chronos                                     2.3.2-0.1.20150207000917.u amd64                      Fault tolerant job scheduler for Mesos which handles dependencies and ISO8601 based schedule
ii  mesos                                       0.21.1-1.1.ubuntu1404      amd64                      Cluster resource manager with efficient resource isolation

@dangra so you ran into this chronos being stuck state only because you have 2 chronos masters? My situation was chronos with 1 master just stopped scheduling jobs on mesos.

With 2 Chronos masters it makes sense that this could cause an issue. You would want to store your zookeeper state separately. With 1, I'm not clear on how your ZK State got corrupted. Any idea of what triggered this?

@elingg: not sure why I was under the impression Chronos can run in HA mode wiht one master active and the other in standby, if that is not true then I will look at running Chronos with Marathon instead.

@dangra you should be able to run two chronos at once. The parameter you should specify is a different framework name for each chronos instance to make sure your state doesn't get corrupted, see https://github.com/mesos/chronos/blob/79a5459744ab1270d024944cbb4850f8ad30629e/src/main/scala/org/apache/mesos/chronos/scheduler/config/SchedulerConfiguration.scala#L100.

@elingg we run into this on a constant basis. 1 chronos + 1 master + zookeeper on the same VM (and  about 10 other VM's acting as mesos slaves). Our environment is Centos6.4. Looks like many folks out these use Ubuntu.

I am after another lead here, even with 1 master it is hanging but always after one of my jobs fails. Our original 2 master setup has been running for 3 months without issues until recently that we configured chronos to send emails and one job started failing. Every time I restart chronos it works until this job run and fails. To be clear the mesos task for this job is started fine but the process returns non-zero. As additional info I am using docker containerizers for all my mesos tasks.

Did you try exit(1) in the process when something goes wrong?

With 2 Chronos masters it makes sense that this could cause an issue. You would want to store your zookeeper state separately.

But by using different paths per chronos master to store state in zookeeper will defeat the point of running 2 Chronos masters for HA purposes. Isn't it?
They will register with different frameworkId and considered as different mesos frameworks too, so if one master is down the other can't takeover looking after the jobs of the former.
When I launch 2 masters using the same ZK uri (zk://host1,host2,host3/chronos) I can see both registering as "candidates":
[zk: localhost:2181(CONNECTED) 14] ls /chronos/state/candidate
[_c_1339d35c-93f5-4728-95ef-dc7e63b7798a-latch-0000000012, _c_3de858b3-f3de-436b-85ff-dd6ae339c1c7-latch-0000000011]
[zk: localhost:2181(CONNECTED) 15] get /chronos/state/candidate/_c_1339d35c-93f5-4728-95ef-dc7e63b7798a-latch-0000000012
head2.mydomain.com:4400
cZxid = 0x800045ea8
ctime = Fri Apr 17 19:36:21 CEST 2015
mZxid = 0x800045ea8
mtime = Fri Apr 17 19:36:21 CEST 2015
pZxid = 0x800045ea8
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x24cb997ff630052
dataLength = 31
numChildren = 0
[zk: localhost:2181(CONNECTED) 16] get /chronos/state/candidate/_c_3de858b3-f3de-436b-85ff-dd6ae339c1c7-latch-0000000011
head1.mydomain.com:4400
cZxid = 0x800033ae1
ctime = Fri Apr 17 04:38:33 CEST 2015
mZxid = 0x800033ae1
mtime = Fri Apr 17 04:38:33 CEST 2015
pZxid = 0x800033ae1
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x14cb998a1ef01d6
dataLength = 31
numChildren = 0

Is this setup possible and the only setting I need to set differently on each master is framework name even if both masters reuses frameworkId stored in zookeeper? I will try although it doesn't fix the original issue.

Did you try exit(1) in the process when something goes wrong?

@vikhyath  Why should I try exit(1) if the process is already failing with non-zero?

Oops, read it as zero.

@dangra, you are correct that you need to use the same Zookeeper path. I would recommend using different framework names. It was one possibility for the corrupt zookeeper state.
@vikhyath, there was an issue with Chronos jobs running docker stuck in staging, which was recently fixed in Mesos. It seems like you have a different issue, but does this help at all, https://issues.apache.org/jira/browse/MESOS-2583? You might also want to try with a new version of Mesos that has this fix.

In my case Chronos hangs when it can't send emails on job failure. My SMTP server wasn't correctly configured, it was accepting chronos connection but rejecting email sending for the domain I set in my job definition.
I think you can still consider it a bug in Chronos that other healthy jobs doesn't run when SMTP server rejects its emails.
@elingg: thank a lot for the hint on mesos_framework_name, I am using it now.


chore(main): release 20.1.0

:robot: I have created a release beep boop
20.1.0 (2024-08-21)
Features

Added custom checkbox states (ff96638), closes #117

This PR was generated with Release Please. See documentation.

:robot: Created releases:

v20.1.0
:sunflower:


Fix PSI exception

Describe the bug
The issue appears whenever you run PSI test from stats tests, on two distributions, where you have long tails and one distribution is significantly shifted from the other.
The error that is raised appears in the following line:
    if len(d1) != len(d2):
        raise ValueError("Distributions do not have the same number of bins.")

It appears, because after binning the two distributions, one has more bins than the other. Therefore, the underlying issue might be in binning or quantile binning classes.
To Reproduce
I don't have the code to reproduce it, however, one needs to generate different distributions and find when this bug appears.
Expected behavior
The error should not appear

The issue is that when you compute quantile bins and you have a lot of duplicate values in the data, e.g. 0s, there is a chance that multiple boundaries of quantile bins are in exactly same location. In that case, the two bins are merged into one.
If this situation happens in any distribution and not the other, the error above has been thrown. Unfortunately, this issue cannot be fixed,  because that is how the quantile bucketing works. However the #116 PR adds an informative warning:
                ApproximationWarning(
                    f"Unable to calculate quantile bins for this feature, because possibly "
                    f"there is too many duplicate values.Approximated quantiles, as a result,"
                    f"the multiple boundaries have the same value. The number of bins has "
                    f"been lowered to {boundaries-1}. This can cause issue if you want to "
                    f"calculate the statistical test based on this binning. We suggest to "
                    f"retry with max number of bins of {boundaries-1} or apply different "
                    f"type of binning e.g. simple"
                )

In case you ever see this warning, either rerun bucketing with less bins for that feature, or modify the bucketing method to simple.