drivers: flash: Add Ambiq flash controller driver

This PR creates the flash controller driver for Ambiq Apollo4 SoCs.
It also instantiates the flash controller for Apollo4 Blue Plus SoC and enable the flash controller for apollo4p_blue_kxr_evb.
It mainly targets to do data storage for the Bluetooth applications which need Zephyr Settings subsystem at this moment.
Test passed on the following cases:
--samples/drivers/flash_shell
--tests/drivers/flash/common

Hi @aaronyegx this generally looks ok, some possible improvements:

Also allow for non-multithreading by removing the k_sem mutex and the k_sem_take and k_sem_get routines in that case,
There is no check on the alignment of the write data address and length to the write-block-size,
There is no check on the alignment of the erase start and length to the erase-block-size,

Since the erase is a "simulated" erase you can set the erase-block-size to whatever is wanted, but from the code it seems that it needs to be a multiple of 4 byte. Also the write-block-size seems to be 4 byte and there is no need to set it to 16 byte.
There might also be a problem when the data passed to the write function would not be aligned to 4 byte. This is easily checked by creating a const char tst="TESTDATA" and then trying to write 4 byte starting from tst[0], tst[1], ... To remedy from this you can use the UNALIGNED_GET() macro.

Hi @aaronyegx this generally looks ok, some possible improvements:

Also allow for non-multithreading by removing the k_sem mutex and the k_sem_take and k_sem_get routines in that case,
There is no check on the alignment of the write data address and length to the write-block-size,
There is no check on the alignment of the erase start and length to the erase-block-size,

Since the erase is a "simulated" erase you can set the erase-block-size to whatever is wanted, but from the code it seems that it needs to be a multiple of 4 byte. Also the write-block-size seems to be 4 byte and there is no need to set it to 16 byte.
There might also be a problem when the data passed to the write function would not be aligned to 4 byte. This is easily checked by creating a const char tst="TESTDATA" and then trying to write 4 byte starting from tst[0], tst[1], ... To remedy from this you can use the UNALIGNED_GET() macro.

Hi @Laczen , thanks for your review. I updated the commits as you commented:

Included the k_sem mutex and related k_sem_take/k_sem_give in CONFIG_MULTITHREADING, so it should support multithreading and non-multithreading scenarios.
Actually, there is check on the Ambiq HAL APIs but I also added the address and length alignment pre-check for flash write in the flash controller driver to make it clear.
Added address and length check for flash erase as well.

The Ambiq Apollo4x flash write and erase operation require 4-word alignment, so the write-block-size is configured to 16 bytes. And the erase-block-size is configured to 2048 bytes (which seems popular in existing boards). Some low size (e.g., 16 bytes) may get EDEADLK error in settings_nvs_backend_init() when it is used for Bluetooth settings.
Could you talk another look on the changes? Thank you.

@aaronyegx thank you for the updates.
Regarding the multithreading support, the code can be cleaned up a little by defining at the top of the code:
#if defined(CONFIG_MULTITHREADING)
static struct k_sem flash_ambiq_sem;
#define FLASH_SEM_INIT() k_sem_init(&flash_ambiq_sem, 1, 1)
#define FLASH_SEM_TAKE() k_sem_take(&flash_ambiq_sem, K_FOREVER)
#define FLASH_SEM_GIVE() k_sem_give(&flash_ambiq_sem)
#else
#define FLASH_SEM_INIT()
#define FLASH_SEM_TAKE()
#define FLASH_SEM_GIVE()
#endif

and then using FLASH_SEM_INIT(), FLASH_SEM_TAKE() and FLASH_SEM_GIVE() in the code. The flash_data variable can then be dropped.
Regarding the write alignment, there is no need to double check, add a note somewhere in the code that this alignment is checked in the hal.
Regarding the erase alignment, there is no requirement to check the alignment to the erase-block-size. The subsystem that uses the flash and is dependent on the block size will use the page layout to get the block size and issue a correct erase, while a subsystem that does not use it can issue a smaller erase when needed (@de-nordic, this might be a good method to avoid defining a flash that has "no-erase": a read of an area could be done to determine that an area is erased and if not a smaller erase could be performed).
Have you checked writing unaligned data ? Taking into account the 16 byte write block size you should define a 20 byte array of test data: const char tst[16]="TST0TST1TST2TST3TST4" and writing  should be succesfull for &tst[0], &tst[1], &tst[2], &tst[3].
The choice of 2048 bytes for the erase-block-size is good.

(@de-nordic, this might be a good method to avoid defining a flash that has "no-erase": a read of an area could be done to determine that an area is erased and if not a smaller erase could be performed).

It kills device twice as fast as flash device with the same  bit flip count;  write is destructive operation in MRAM and it is done twice, while in flash erase is destructive, write is not. So write/write != erase/write in wear counter.

(@de-nordic, this might be a good method to avoid defining a flash that has "no-erase": a read of an area could be done to determine that an area is erased and if not a smaller erase could be performed).

It kills device twice as fast as flash device with the same bit flip count; write is destructive operation in MRAM and it is done twice, while in flash erase is destructive, write is not. So write/write != erase/write in wear counter.

This is not flash device, and we name it flash anyway, so maybe no-erase is not that bad idea at all, as naming does not seem to be so important anyway and we are willing to give up on device reliability just to pretend it is flash.
There will be issue reported further or later that MRAM device are dying faster than expected, no-erase flag at least would allow us to have intermediate solution in the entire system before we get into right one.

@de-nordic, I agree that this is not a flash device but ...

It follows all the requirements set by the flash api,
MRAM cycles seem to be a lot higher than flash cycles, so erasing them before write when needed (in case of e.g. nvs) might not be as problematic as for flash,

Regarding the no-erase flag could we take this out of this PR. I am sure this can be resolved.

Hi @Laczen , yes there was unalignment writing issue in previous implementation. It failed in your suggested testing. I use UNALIGNED_GET() to make it work now. And clean up the multithreading support codes. Thanks so much.

Hi @de-nordic , thanks for your review. Yes Ambiq Apollo4x SoC has no flash and we always use MRAM for the storage purpose, e.g., Bluetooth NVM. Many Zephyr Bluetooth samples are using Settings subsystem for the data storage then the flash controller driver should be supported basically.
CONFIG_BT_SETTINGS=y
CONFIG_FLASH=y
CONFIG_FLASH_PAGE_LAYOUT=y
CONFIG_FLASH_MAP=y
CONFIG_NVS=y
CONFIG_SETTINGS=y
So, we consider MRAM is "flash" at this case. The erase operation is not necessary for MRAM writing but I would like to keep the API for some necessary scenarios. Do you think it is acceptable?

Hi @Laczen , yes there was unalignment writing issue in previous implementation. It failed in your suggested testing. I use UNALIGNED_GET() to make it work now. And clean up the multithreading support codes. Thanks so much.

Some nitpicks from my side:

Now that you have split up the write in a loop it is no longer possible to count on the hal to do the checking of alignment, so I think you previous change that did the alignment check with write-block-size needs to be inserted. Now the loop would do a part of a write and fail at the end. Don't forget to remove the remark.
Don't use the numbers 16 and 4 in code (in the write or alignment check):
16 should be replaced by FLASH_WRITE_BLOCK_SIZE,
4 (in the write routine) should be replaced by FLASH_WRITE_BLOCK_SIZE/sizeof(uint32_t).

Hi @de-nordic , thanks for your review. Yes Ambiq Apollo4x SoC has no flash and we always use MRAM for the storage purpose, e.g., Bluetooth NVM. Many Zephyr Bluetooth samples are using Settings subsystem for the data storage then the flash controller driver should be supported basically.
CONFIG_BT_SETTINGS=y CONFIG_FLASH=y CONFIG_FLASH_PAGE_LAYOUT=y CONFIG_FLASH_MAP=y CONFIG_NVS=y CONFIG_SETTINGS=y
So, we consider MRAM is "flash" at this case. The erase operation is not necessary for MRAM writing but I would like to keep the API for some necessary scenarios. Do you think it is acceptable?

Can we at least have it define new compatibles like "soc-nvm-mram" and "ambiq,mram-controller" or "ambiq,nvm-controller".
This way, in the future, when the MRAM driver is actually provided DTS compatibles will be the same.

Can we at least have it define new compatibles like "soc-nvm-mram" and "ambiq,mram-controller" or "ambiq,nvm-controller". This way, in the future, when the MRAM driver is actually provided DTS compatibles will be the same.

Hi @de-nordic , I think it should be OK to define new compatible "soc-nv-mram". Current "soc-nv-flash" defines "erase-block-size" and "write-block-size" properties, do you think the "soc-nv-mram" should define the same?
But for name of API and c file, should we still use "flash_" as the prefix?

Definitions here are local/ static and completely unique for the device, so you can call them as you wish, even with prefix ambiq_nvm_; think what would allow you to identify the callback, in the easiest way, when you look at stack trace; or allow you to find property by identifier.
The write block size is needed by the characteristic of the device; the erase block size should be defined as optional, but is needed now by the subsystem.
If we are able to bring in proper mram support, with the dts binding done right we should be able to reuse  most of the work provided by the pr, without significantly altering dts afterwards.

I think the compatible should remain "soc-nv-flash". As a "flash" device the write-block-size needs to be defined. The erase-block-size could be an optional dts value but it could also be fixed in the driver and reflected in the page layout. IMO it is good as proposed.
If the compatible name is changed partition definitions will fail to be created.
The config entry would indeed better be changed to "Ambiq flash driver on mram".

Hi @de-nordic @Laczen , thanks for your continued support on this PR. I address most of the comments you put in the code.
For the naming, as @Laczen commented, I also prefer to keep "soc-nv-flash", "flash-controller" to align the current "flash" driver naming. Using "soc-nv-mram" or "nvm-controller" may bring some conflict or confusion to the current framework. For example, we can only use "zephyr, flash-controller" as the "chosen" for this instance or we will get building or run failure in the Zephyr examples or test cases, e.g., the samples/drivers/flash_shell.
I think supporting MRAM in Zephyr can be another topic which should be discussed more outside of this PR.
I modify the description of flash driver configuration to "Ambiq flash driver on MRAM", it should make sense.

Hi @de-nordic , could you take another look at the current implementation? Thanks.

@de-nordic, regarding the checking of the write-block-size, as more and more the dts entries are used as configuration entries this raises the question if these values should be considered "configuration" or "definition".
I don't mind it being checked but we should have a clear idea on how to treat the dts values. If they are all considered "configuration" all drivers should check them and use them if specified. If they are considered "definitions" they could just be used. If some of them are considered "definitions" while others are considered "configurations" the proposed PR treats them as "definitions" and this would be a valid choice.

Hi @de-nordic , @Laczen , regarding the checking of the write-block-size, can we go with the below implementation?
#define MIN_WRITE_SIZE         16
#define FLASH_WRITE_BLOCK_SIZE MAX(DT_PROP(SOC_NV_FLASH_NODE, write_block_size), MIN_WRITE_SIZE)
#if (FLASH_WRITE_BLOCK_SIZE & (MIN_WRITE_SIZE - 1) != 0)
#error "The flash write block size must be a multiple of 16!"
#endif

Hi @de-nordic , @Laczen , regarding the checking of the write-block-size, can we go with the below implementation?
#define MIN_WRITE_SIZE 16 #define FLASH_WRITE_BLOCK_SIZE MAX(DT_PROP(SOC_NV_FLASH_NODE, write_block_size), MIN_WRITE_SIZE)
#if (FLASH_WRITE_BLOCK_SIZE & (MIN_WRITE_SIZE - 1) != 0) #error "The flash write block size must be a multiple of 16!" #endif

No problem with adding the check for me, the error reporting should be done using BUILD_ASSERT().

Hi @de-nordic , @Laczen , regarding the checking of the write-block-size, can we go with the below implementation?
#define MIN_WRITE_SIZE 16 #define FLASH_WRITE_BLOCK_SIZE MAX(DT_PROP(SOC_NV_FLASH_NODE, write_block_size), MIN_WRITE_SIZE)
#if (FLASH_WRITE_BLOCK_SIZE & (MIN_WRITE_SIZE - 1) != 0) #error "The flash write block size must be a multiple of 16!" #endif

No problem with adding the check for me, the error reporting should be done using BUILD_ASSERT().

Hi @de-nordic , could you take another look at this? Thank you.

Hi @de-nordic , could you help revisit and move forward this PR? Thanks.


Monitor doesn't work on Windows 11

Tried to run cantools monitor tool on windows, running windows-curses v2.3.2, and it doesn't seem to work properly (tool exit after printing the initial warnings about some file format not being supported).
Running python 3.12.0, cantools 39.4.0.
The can adapter I'm trying to use (pcan-usb) works when using it with python-can only to receive raw CAN message from the same CAN bus.
So leads me to think it's an issue with the curses command line interface.
Note that I've also used the same dbc and can adapter on a raspberry pi with linux and managed to use cantools monitor successfully.
Result from running cantools monitor in command line:

Some extra info about my setup:

did you modify your local copy of cantools? the first three messages after invoking cantools seem to stem from something trying to load an excel file (which I'm pretty sure is not done by cantools)
That said, if I see this correctly, you did not specify any after cantools monitor, so maybe you just need to invoke tool correctly:
cantools monitor -c can0 my-bus-description-file.dbc

(be aware that the name of the can device on windows is most likely different, but I don't use cantools on windows, so I cannot tell you how it is usually called...)

Sorry for the confusion, the second screenshot I shared was just to show the cantools version I had installed (first command in that screenshot, disregard the second).
See the first screenshot i shared, where i do indeed invoke the tool properly with a bus type, a channel, and a DBC file argument.
I haven't manually modified my local copy of cantools (I've got the same warnings when running ASAMMDF python module too, not sure if that could explains those warnings).
However, I also see those warnings on my Linux setup where cantools monitor actually works properly, so don't think they are related to the monitor issue.


Bundling a lazily loaded NPM package in production mode fails due to ENOENT

Steps to reproduce and a minimal demo of the problem
I would like to use an NPM package, which is supposed to be lazily loaded only when a component really needs it. Therefore, I tried tinkering with SYSTEM_BUILDER_CONFIG instead of NPM_DEPENDENCIES since libraries given in the latter one are always injected into index.html.

npm install myPrivatePackage --save (a private NPM package written in TypeScript)
Use this library in various Angular components and services: import { ... } from 'myPrivatePackage which gets compiled to require('myPrivatePackage')
Edit tools\config\project.config.ts:

import { join } from 'path';
import { SeedConfig } from './seed.config';

export class ProjectConfig extends SeedConfig {

PROJECT_TASKS_DIR = join(process.cwd(), this.TOOLS_DIR, 'tasks', 'project');

constructor() {
  super();

  this.SYSTEM_BUILDER_CONFIG.paths.myPrivatePackage = 'node_modules/myPrivatePackage/dist/index.js';
}

}

npm run build.prod

Current behavior
[11:17:45] Starting 'build.bundles.app'...
[11:17:46] 'build.bundles.app' errored after 720 ms
[11:17:46] Error on fetch for myPrivatePackage.js at file:///D:/Desktop/Project/angular2-seed/node_modules/myPrivatePackage.js
        Loading dist/tmp/app/app.routes.js
        Loading dist/tmp/app/main.js
        Error: ENOENT: no such file or directory, open 'D:\Desktop\Project\angular2-seed\node_modules\myPrivatePackage.js'
    at Error (native)
[11:17:46] 'build.prod' errored after 10 s
[11:17:46] Error in plugin 'run-sequence(build.bundles.app)'
Message:
    Error on fetch for myPrivatePackage.js at file:///D:/Desktop/Project/angular2-seed/node_modules/myPrivatePackage.js
        Loading dist/tmp/app/app.routes.js
        Loading dist/tmp/app/main.js
        ENOENT: no such file or directory, open 'D:\Desktop\Project\angular2-seed\node_modules\myPrivatePackage.js'
Details:
    originalErr: Error: ENOENT: no such file or directory, open 'D:\Desktop\Project\angular2-seed\node_modules\myPrivatePackage.js'
Stack:
Error on fetch for myPrivatePackage.js at file:///D:/Desktop/Project/angular2-seed/node_modules/myPrivatePackage.js
        Loading dist/tmp/app/app.routes.js
        Loading dist/tmp/app/main.js
        Error: ENOENT: no such file or directory, open 'D:\Desktop\Project\angular2-seed\node_modules\myPrivatePackage.js'
    at Error (native)

npm ERR! Windows_NT 10.0.10586
npm ERR! node v5.7.0
npm ERR! npm  v3.5.1
/* ... */

Expected/desired behavior
Why does systemjs-builder still search for node_modules/myPrivatePackage.js instead of the specified path in SYSTEM_BUILDER_CONFIG?
It seems that tools/tasks/seed/build.bundles.app.ts directly passes the configuration object to systemjs-builder.

Maybe because of this? https://github.com/mgechev/angular2-seed/blob/master/tools/config/seed.config.ts#L353

In any way: It is not an issue related to the seed per se, but rather your system build config :)

Thanks for the answer. I tried changing the order to no avail, however, no one should rely on the order anyway as it is undefined for dictionaries.
I've browsed through the examples provided in the README. The one which came closest to my intention was angular2-seed-example-mashup with Bootstrap as a dependency, however, adapting the linked line to my path didn't work either (same error).

In any way: It is not an issue related to the seed per se, but rather your system build config :)

I agree :). I thought it's easier to ask here why the angular2-seed built-in dependencies (i.e. mostly @angular) work contrary to mine.

I tried changing the order to no avail, however, no one should rely on the order anyway as it is undefined for dictionaries.

That was just an assumption that this could be the issue (path order), i did not claim to know it for the truth :) I have no idea how SystemJS/Builder works internally. Maybe it's in the docs?

Yay, I've finally solved it :)

All the changes made in tools/config/project.config.ts were for nothing until I realized that neither gulp build.prod nor gulp build.bundles.app nor gulp clean.all clean and re-compile the TS files in tools/config!
-> You have to force it by running gulp build.js.tools.

The following lines eventually enabled a flawless build:

this.SYSTEM_BUILDER_CONFIG.paths['myPrivatePackage'] = `${this.APP_BASE}node_modules/myPrivatePackage/dist/index.js`;
this.SYSTEM_BUILDER_CONFIG.paths['myPrivatePackage/*'] = `${this.APP_BASE}node_modules/myPrivatePackage/dist/*.js`;

Using only the first line led to an error related to a helper file which is included inside myPrivatePackage/dist/index.js via require('./helperFile') (import in the TypeScript file, respectively):
[14:10:53] Error tracing myPrivatePackage at file:///node_modules/myPrivatePackage/dist/index.js
       Loading dist/tmp/app/app.routes.js
       Loading dist/tmp/app/main.js
       Error: Unable to calculate canonical name to bundle file:///node_modules/myPrivatePackage/dist/helperFile.js. Ensure that this module sits within the baseURL or a wildcard path config.
   at getCanonicalNamePlain (D:\Desktop\Project\angular2-seed\node_modules\systemjs-builder\lib\utils.js:218:13)
   at getCanonicalName (D:\Desktop\Project\angular2-seed\node_modules\systemjs-builder\lib\utils.js:145:19)
   at D:\Desktop\Project\angular2-seed\node_modules\systemjs-builder\lib\trace.js:482:36
[14:10:53] 'build.prod' errored after 9.73 s
[14:10:53] Error in plugin 'run-sequence(build.bundles.app)'
Message:
   Error tracing myPrivatePackage at file:///node_modules/myPrivatePackage/dist/index.js
       Loading dist/tmp/app/app.routes.js
       Loading dist/tmp/app/main.js
       Unable to calculate canonical name to bundle file:///node_modules/myPrivatePackage/dist/helperFile.js. Ensure that this module sits within the baseURL or a wildcard path config.
Details:
   originalErr: Error: Unable to calculate canonical name to bundle file:///node_modules/myPrivatePackage/dist/helperFile.js. Ensure that this module sits within the baseURL or a wildcard path config.
Stack:
Error tracing myPrivatePackage at file:///node_modules/myPrivatePackage/dist/index.js
       Loading dist/tmp/app/app.routes.js
       Loading dist/tmp/app/main.js
       Error: Unable to calculate canonical name to bundle file:///node_modules/myPrivatePackage/dist/helperFile.js. Ensure that this module sits within the baseURL or a wildcard path config.
   at getCanonicalNamePlain (D:\Desktop\Project\angular2-seed\node_modules\systemjs-builder\lib\utils.js:218:13)
   at getCanonicalName (D:\Desktop\Project\angular2-seed\node_modules\systemjs-builder\lib\utils.js:145:19)
   at D:\Desktop\Project\angular2-seed\node_modules\systemjs-builder\lib\trace.js:482:36
```

In my opinion, this project would benefit from:

- a hint in the README or wiki that _only_ `gulp build.js.tools` forces a rebuild of the tools directory and
- the two lines of System.js Builder configuration above in the example section.

What do you think?


Can share info about how to start porting to Windows..

Hi,
maybe some effort porting to Android can be shared for porting to Windows platform..
if yes can you share some plan on how to start a Swift port to Windows?
Would be even better if you can get interest in doing an early swift Windows port

I think that people are assuming, for better or worse, that Microsoft is going to do the Windows port. It's probably not the best idea to depend on them for that, but I haven't heard of any community efforts in that direction yet.

@oscarbg there would probably be even more involved with a Windows port than this Android port, because Linux (including Android) and OSX / iOS are related by their shared UNIX heritage (making it easier to use much the same processes across the board). Windows is quite a different ecosystem with different tools and processes.

Does the Swift compiler even build on Windows? I tried building it on Windows (twice) but gave up both times since the configure script is a Unix shell script and I'm not sure what compiler it needs (LLVM can build in Visual Studio but I have no idea whether the Swift compiler can.)

I think this discussion isn't really relevant to the swift port and is more a topic for the swift.org mailing lists. Closing for now..


Remove the *_key_check function calls

Issue # (if available): N/A
Description of changes:
This change would make it the responsibility of the calling application to ensure keys are correctly formed
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Created https://github.com/awslabs/s2n/issues/752 to track making these checks optional. I'm okay with removing these sanity checks for now for performance reasons, but I think we should circle back on this eventually and allow customers of the s2n public API to opt-in to this sanity checking if they want.


Added replace mode

I added mode parameter and replace mode.
This parameter can be taken 2 choises,  insert(default) and replace.
out:
  type: elasticsearch
  mode: replace       # or insert

replace

If same index as alias already exists, throws config exception.
Create new index
Insert data into the new index. index name was generated by plugin(like _20150121-201249)
Reassign alias from old indices to new index
Delete existing (old) indices

LGTM.

Thank you for your review. I merged.


Made changes to allow the API to work with cual-id, UUID format

As said above

Coverage remained the same at 93.407% when pulling 984c1cbd25316991e1f55f5f35e2d4778390c9ed on kmckinnis:master into 1bb93ae655a117139742b224a63c4f06330b9a15 on johnchase:master.

Thanks @kmckinnis One very minor comment, otherwise looks good!

Coverage remained the same at 93.407% when pulling dc3901a94af9e040cb97e68cf93456c30982e799 on kmckinnis:master into 1bb93ae655a117139742b224a63c4f06330b9a15 on johnchase:master.

@kmckinnis Will you make the changes as per the code block I included in my last comment?

Sorry I didn't see that part of the email. Didn't hit read more.
On Mon, Aug 22, 2016 at 4:29 PM, John Chase<EMAIL_ADDRESS>wrote:

@kmckinnis https://github.com/kmckinnis Will you make the changes as
per the code block I included in my last comment?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/johnchase/cual-id/pull/34#issuecomment-241582493, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ATEw0WruWbikGpW8-2WMAYo9ip8RQRSwks5qijDhgaJpZM4Joxhf
.

Also uuid.UUID causes a failure as uuid is not defined
On Tue, Aug 23, 2016 at 9:56 AM, Kevin McKinnis<EMAIL_ADDRESS>wrote:

Sorry I didn't see that part of the email. Didn't hit read more.
On Mon, Aug 22, 2016 at 4:29 PM, John Chase<EMAIL_ADDRESS>wrote:

@kmckinnis https://github.com/kmckinnis Will you make the changes as
per the code block I included in my last comment?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/johnchase/cual-id/pull/34#issuecomment-241582493,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ATEw0WruWbikGpW8-2WMAYo9ip8RQRSwks5qijDhgaJpZM4Joxhf
.

Coverage remained the same at 93.407% when pulling e0f3859de30b0e609627f72de388e62b4e521ff0 on kmckinnis:master into 1bb93ae655a117139742b224a63c4f06330b9a15 on johnchase:master.


fix: fix applying filters to query

Closes #548

Should be closed by this https://github.com/epam/badgerdoc/pull/590


Delete chenge's MR

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.huifei (A) seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.You have signed the CLA already but the status is still pending? Let us recheck it.


Binary release v1.3.2 missing matdbg.lib

filament.lib depends on matdbg.lib, but matdbg.lib is not packaged in filament-20190827-windows.tgz
filament.lib(Engine.cpp.obj):-1: error: LNK2019: unresolved external symbol "public: __cdecl filament::matdbg::DebugServer::DebugServer(enum filament::matdbg::ServerMode,int)" (??0DebugServer@matdbg@filament@@QEAA@W4ServerMode@12@H@Z) referenced in function "int `private: __cdecl filament::details::FEngine::FEngine(enum details::backend::Backend,class details::Backend::Platform *,void *)'::`1'::dtor$38" (?dtor$38@?0???0FEngine@details@filament@@AEAA@W4Backend@backend@2@PEAVPlatform@42@PEAX@Z@4HA)
filament.lib(Material.cpp.obj):-1: error: LNK2019: unresolved external symbol "public: void __cdecl filament::matdbg::DebugServer::addMaterial(class utils::CString const &,void const *,unsigned __int64,void *)" (?addMaterial@DebugServer@matdbg@filament@@QEAAXAEBVCString@utils@@PEBX_KPEAX@Z) referenced in function "public: class filament::Material * __cdecl filament::Material::Builder::build(class filament::Engine &)" (?build@Builder@Material@filament@@QEAAPEAV23@AEAVEngine@3@@Z)

Gah, my fault.  The releases do not enable DebugSever but on desktop there is still a link time dependency, so yes this lib needs to be distributed.

That should only be true for debug builds right? Release builds should not need the extra library at all.

Correct, there should be no dependency.  Note that release builds do build the library, but only as a dependency of matinfo, not of filament.
Given the conditional in filament/CMakeLists.txt, I still don't understand how this is happening.

Windows builds contain release and debug builds btw. We don't do this on other platforms.

Ah that explains it, thanks Romain.  TIL that "nm" on mac works with Windows binaries:
prideout $ nm ./lib/x86_64/mdd/filament.lib | grep DebugServer
         U ??0DebugServer@matdbg@filament@@QEAA@W4ServerMode@12@H@Z
         U ?addMaterial@DebugServer@matdbg@filament@@QEAAXAEBVCString@utils@@PEBX_KPEAX@Z
prideout $ nm ./lib/x86_64/mt/filament.lib | grep DebugServer
prideout $


support for 8.0.6

NU1608: Detected package version outside of dependency constraint: EntityFrameworkCore.Jet 7.0.3 requires Microsoft.EntityFrameworkCore (>= 7.0.10 && < 8.0.0) but version Microsoft.EntityFrameworkCore 8.0.6 was resolved.
NuGet\Install-Package : NU1107: Version conflict detected for Microsoft.EntityFrameworkCore.Relational. Install/reference Microsoft.EntityFrameworkCore.Relational 8.0.6
directly to project Datanex.Backend.Infrastructure to resolve this issue.
Datanex.Backend.Infrastructure -> Datanex.Shared.Logic -> Microsoft.EntityFrameworkCore.Relational (>= 8.0.6)
Datanex.Backend.Infrastructure -> EntityFrameworkCore.Jet 7.0.3 -> Microsoft.EntityFrameworkCore.Relational (>= 7.0.10 && < 8.0.0).
At line:1 char:1

NuGet\Install-Package EntityFrameworkCore.Jet -Version 7.0.3

Hi
7.0.3 is part of the series targeting EF Core 7. That error is expected as you can only use it with the specific series.
You are using EF Core 8 so you need the specific series of EFCore.Jet to target that. There is a version for EF Core 8 which is currently in release candidate form. Barring any major critical bug, the RC is almost literally what will become stable.
You need to install the pre-release version of 8.0.0-rc.1
Below is the package manager command like what you have shown above but with the correct version
NuGet\Install-Package EntityFrameworkCore.Jet -Version 8.0.0-rc.1

tnx

Just a note that stable 8.0 series is out now


Default Schema is ignored for Apache Hive connections

Description
The default schema is not respected for Apache Hive connections.  Every time a new connection is established, the active schema is always N/A

Error log is showing this error:
!MESSAGE Active schema can't be changed for this kind of datasource!
dbeaver logs.txt
If I try to manually select a new active schema, I am offered the list of schemas to select.  After I choose one, nothing happens.  The active schema is still N/A.
I have to run a query...any query, but I have to run one, and then I am able to change the active schema.
DBeaver Version
Community 22.1.5
Operating System
Windows 10
Database and driver
Apache Hive
Cloudera JDBC 4.2
Steps to reproduce
Create an Apache Hive connection utilizing the Cloudera Driver (unsure if this is specific to Cloudera driver or not)
Define a default schema
Connect to the data source
Observe the active schema still shows N/A
Additional context
No response

Hello @JesseFL
Thanks for the report!


File manager - Type in the path

When i use the File manager i sometimes have the path in my clipboard, but i can't find a way to type or paste in the path to get to the right directory.
Could it be possible to add an editable path row for easy directory change?
This will only support mouse click, i can't edit the path here:

Yes, it's already possible by clicking Ctrl+L (Mac users would use Command+L) or simply right-clicking on the breadcrumbs container.

Great, but why not have a button? It is impossible to remember all the shortcut keys 😕
Right clicking? When i right click i only see Firefox's own menu.

Right click on the breadcrumb area (grayish):

Adding a button? Maybe,not sure if it will look and feel good.
There is a help button at the top, you can read throughout all hotkeys:

Alright, I ran few tests and I think you're right - having a button would be much more user friendly. I will add this and let you know when done.

I think something like this will work just fine!

Try latest commit in version 19.99-RC1.

Should the window that i type in close as soon as i move the mouse pointer outside the window?
When i press CTRL+L it will stay open until i move the mouse pointer over the window and away again.
It is a bit weird behavior in my opinion ...

When i press CTRL+L it will stay open until i move the mouse pointer over the window and away again.

If you call it by the hotkey it's expected that afterwards, you type something in a popup and hit enter. Try using it for real - it's meant to be effective to have the work done.

Most of the time i use the mouse. So, the issue is that if i open the path window i can't move the mouse outside the window 😥

Oh, OMG! Okay, I will fix that! :-)

Most of the time i use the mouse. So, the issue is that if i open the path window i can't move the mouse outside the window 😥

Alright, this will be fixed on the next theme release. Thanks for reporting. Now I also agree that making a popover disappear on mouse out is annoying.


Document public api

TITLE
Description
Description of the pull request changes and motivation.
Checklist

[ ] Linked to Github Issue
[ ] Unit tests added
[ ] Integration tests added.
[ ] This change requires new documentation.

[ ] Documentation has been added/updated.
[ ] CHANGELOG has been updated.

Kill it with fire!!


When the encrypted key is provided, it should be changed to use the pre-image of the actual validator.

The encrypted key is used to vote in Votera.
The pre-image of the validator is not included in the process of creating the encrypted key.
We need to modify to generate an encrypted key using the pre-image associated with that node.

@TrustHenry : Could you fix it, since you originally implemented it ?
https://github.com/bosagora/agora/blob/dc5f739c64768e978deb65b4d2b5b8470976d8a7/source/agora/node/admin/AdminInterface.d#L186-L188

@Geod24 Of course.
Height(Vote closing height) + PreimageRevealPeriod  Should I release the pre-image with this formula?


[bug] sui move test failed with strange error

Steps to Reproduce Issue
Upgrade sui client to 0.29, and run the test case failed.
steps:

run command in repo sui move test test_ABC -g --verbose, and got the error below:

Failures in 0x0::test_ABC:

┌── test_ABC ──────
│ error[E11001]: test failure
│     ┌─ /Users/edy/.move/https___github_com_MystenLabs_sui_git_devnet/crates/sui-framework/packages/sui-framework/sources/dynamic_field.move:226:31
│     │
│ 226 │     public(friend) native fun borrow_child_object_mut<Child: key>(object: &mut UID, id: address): &mut Child;
│     │                               ^^^^^^^^^^^^^^^^^^^^^^^
│     │                               │
│     │                               Test was not expected to error, but it aborted with code 3 originating in the module<PHONE_NUMBER>000000000000000000000000000000000000000000000000000002::dynamic_field rooted here
│     │                               In this function in 0x2::dynamic_field
│
│
│ ────── Storage state at point of failure ──────
│ <empty>
└──────────────────

Test result: FAILED. Total tests: 4; passed: 0; failed: 4

Expected Result
should report failure or pass, and the error message should be related to the source codes.
System Information

OS: mac, sui 0.29

@0xOutOfGas Thanks for flagging this. Since March, the sui move test command has gone through several improvements and actually the -g flag does not exist anymore. I will close this issue, but feel free to file a new issue if you encounter any problems.


Configuration of CallbackPath not working as expected

I've tried to use configuration for CallbackPath to get custom URL at all.
But the login fails, I get (nearly) endless redirect (seen in Fiddler).
I've created a new unit test according current code base.
Please help to either fix the code or fix the test.
[Test]
public async void ImplicitLogIn_CustomCallbackPath_Successful()
{
    var handler = new HttpClientHandler
    {
        AllowAutoRedirect = true,
        Credentials = CredentialCache.DefaultNetworkCredentials
    };
    WebApplication.Options.CallbackPath = PathString.FromUriComponent("/My-SignIn");

    var client = new HttpClient(handler);
    client.BaseAddress = this.BaseAddress;

    var response = await client.GetAsync("/api/test").ConfigureAwait(false);
    Assert.AreEqual(HttpStatusCode.OK, response.StatusCode, "Http status is incorrect");

    var result = await response.Content.ReadAsAsync<string>().ConfigureAwait(false);
    var currentUserName = Environment.UserName;
    Assert.AreEqual(currentUserName, result);
}

Hi Lukas,
Assuming you're trying to combine NTML Authentication with the CookieAuthentication middleware, you must take care of the redirections the cookie middleware might fire. For that reason the IOwinRequest.IsNtlmAuthenticationCallback() extension can take a parameter where you can supply the customized callback path.
Best you take a look at issue #7 which contains a full example in the last comments.
I hope that this helps

Thanks, this is somehow unexpected but there is - maybe - no other solution.
More I think, the documentation, you extended while answering previous issue, should note:
"... IsNtlmAuthenticationCallback shall be used ...".


Homepage-specific viewing mode no longer works

Details
Basically what it says here. No matter what settings used for the homepage viewing mode it always opens in the obsidian default. I'm on version 3.6.0 of Homepage, and 1.4.16 of Obsidian. Bug occurs on both my Archlinux pacman install and on the Ubuntu snap.
Steps to reproduce
No response
Debug information
No response
Checklist

[X] I updated to the latest version of the plugin.

Can’t reproduce this. What other plugins do you have? This can be obtained via the ‘show debug info’ obsidian command.

Output from the command below.
SYSTEM INFO:
	Obsidian version: v1.4.16
	Installer version: v1.4.16
	Operating system: #1 SMP PREEMPT_DYNAMIC Thu, 26 Oct 2023 00:52:20 +0000 6.5.9-arch2-1
	Login status: not logged in
	Insider build toggle: off
	Live preview: on
	Legacy editor: off
	Base theme: dark
	Community theme: AnuPpuccin v1.4.4
	Snippets enabled: 3
	Restricted mode: off
	Plugins installed: 33
	Plugins enabled: 13
		1: Longform v2.0.4
		2: Status Bar Pomodoro Timer v0.1.14
		3: Remember cursor position v1.0.8
		4: Style Settings v1.0.7
		5: Templater v1.16.4
		6: Unique attachments v1.0.3
		7: Homepage v3.6.0
		8: Dataview v0.5.64
		9: Completr v3.2.0
		10: Colored Text v0.6.0
		11: Callout Manager v1.0.1
		12: Calibre v1.0.8
		13: Linter v1.20.1

RECOMMENDATIONS:
	Custom theme and snippets: for cosmetic issues, please first try updating your theme and disabling your snippets. If still not fixed, please try to make the issue happen in the Sandbox Vault or disable community theme and snippets.
	Community plugins: for bugs, please first try updating all your plugins to latest. If still not fixed, please try to make the issue happen in the Sandbox Vault or disable community plugins.

Thanks for sending me this. Unfortunately, I still can't get the behaviour to happen. If you disable your other plugins does it still persist?

It does. I wouldn't think it's the specific nature of the homepage itself (it's just a couple of dataview tables), but I can send that as well if you'd like. Additionally, here's the output from the Homepage debug button:
{"version":3,"homepages":{"Main Homepage":{"value":"home","kind":"File","openOnStartup":true,"openMode":"Replace all open notes","manualOpenMode":"Replace last note","view":"Reading view","revertView":true,"openWhenEmpty":false,"refreshDataview":true,"autoCreate":true,"autoScroll":false,"pin":false,"commands":[],"alwaysApply":true,"hideReleaseNotes":true}},"separateMobile":false,"_defaultViewMode":"default","_livePreview":true,"_focusNewTab":true,"_plugins":["homepage"],"_internalPlugins":["file-explorer","global-search","switcher","graph","backlink","canvas","outgoing-link","tag-pane","page-preview","daily-notes","templates","note-composer","command-palette","editor-status","bookmarks","zk-prefixer","outline","word-count","audio-recorder","file-recovery"],"_obsidianVersion":"1.4.16"}

Unfortunately I'm not exactly sure how to get at the obsidian logs themselves, or the plugins logs- any pointers there?

No logs apparently, but I found this in the developer console:
Homepage: home (method: Replace all open notes, view: Reading view, kind: File)
Notably this is after the homepage still opened in editing mode.

That last message is normal. Thanks for sending me all the info, I’ll see if I can reproduce it now.

Is your actual homepage file called 'Home' with an uppercase 'H'? I can reproduce this when the setting is set to 'home' with a lowercase h but the file has an uppercase one.
Changing 'home' to 'Home' should solve your problem. Basically, Homepage treats these as the same for opening files but the view reversion functionality treats them as different. I'm aware that Linux is case sensitive but Homepage uses Obsidian's link resolution logic which AFAIK is the same on all platforms.

That was the issue, changing the file to 'Home' fixed it.


How to compile this source code

It should be the same as the regular process of compiling the Chromium source code, but instead of pulling from the chromium repository, pull from this one.
I also made a modification to webrtc which is in a different git repo, so you would have to pull from my variant of it as well, which I will upload promptly.

ok，Thank you

Error reported according to chrome compilation method

Please provide the details of this error.
I have also added several other repositories for "third-party" Chromium components in recent weeks. They are in the form of multi-part zips because GitHub does not seem to be able to handle a git push from a repository hosted off of GitHub (indeed, these are Google-hosted repositories without GitHub mirrors) then cloned locally.
Once unzipped they will be in the form of git repositories. If they don't work well with the rest of a newer Chromium repository, then you may simply obtain the details of the most recent commits and apply them to the latest version of these third-party components.

Missing this code

Did you run gclient sync (or more specifically, gclient sync -m to avoid overwriting my customizations to the third party repos)?
I will probably end up supplying Supermium and its components in git-compatible patch form so it can be easier to build using the standard Chromium build process and also because it's the preferred way to modify Chromium for Electron.

Not running gclient sync and gclient sync - m
After obtaining the Superpremium code, run gclient runbooks directly

Curious, I'm using the latest version of gclient

Thank you, I compiled it，But Win7 did not run successfully
gclient sync updated the compilation tool chain error in the runbooks step. To ensure that runbooks pass, I set DEPOT_ TOOLS_ UPDATE, DEPOT_ TOOLS_ WIN_ TOOLCHAIN environment variable, the final compiled execution file cannot run on Win7 64 bit. My compilation environment is vs2019 for Win10, and I have installed Win10.0.22621.1778 sdk

Just gclient sync or gclient sync -m? As stated above, gclient sync will tend to overwrite the third-party repos instead of merging them. I found it strange at first as well when I started seeing missing functions in Dependency Walker like GetDpiForMonitor, which I had thought I had removed earlier.

I only ran the gclient sync command on my end
Did not run the gclient sync - m command
How to update and merge into the third-party repository you modified

Did I choose the wrong compilation environment

It's the right environment, A Windows build is a Windows build. But gclient sync -m would merge the upstream commits with my commits.... which are only in the other Supermium repositories I have made (boringssl/v8/webrtc). Although I will eventually change everything from full repositories to patches to make it easier for other users (and also to make it possible to build as Electron).

Hello, I also have problem to compile supermium 118 with depot tools made for chromium 121. Main problem is rust-toolchain, one side demands llvm-17, other downloaded llvm-18. Even "gclient sync -M" or "gclient sync --merge" does not help.
Problem made on Win 8.1 with Windows SKD 10.22 and VS 2022 17.6 community.


PERF: N4BiasFieldCorrectionImageFilter implementation using ImageRange

Adapted N4BiasFieldCorrectionImageFilter to start using ImageRange internally.
This commit yields a significant performance improvement, especially because
it replaces the (rather expensive) itk::Image::GetPixel(index) calls by
more efficient ImageRange[indexValue] calls.
Added checks to GenerateData() that the image sizes of the mask image and
confidence image (if provided) are equal to the size of the input image.
Ensured that the entire output image is computed at once, by overriding
EnlargeOutputRequestedRegion(DataObject*), as suggested by Bradley (@blowekamp).
Using VS2015 (Release), ~10% reduction of the run-time duration was observed on
a 3-D MRI volume + mask (453x340x20 voxels), provided by Denis (@dpshamonin),
at LKEB, Leiden University Medical Center.

Please note: I will remove the "WIP" tag from the commit message when I think the commit is ready to be merged. For now, I still want to have another look at the code, and try it out!

There seems to be some regions issue with this filter related to the pipeline propagation of the requested region. We could ask the questions can the filter produce a sub-region of the output, or what input regions are needed to produce that output. The answers would indicate output enlarge the output requested regions and the propagate it to the input.
However, it looks like this filter assumes to produce the whole output and consumable the whole input. This can be done with the Pipeline by overloading EnlargeOutputRequestedRegion to set the output to the LargestPossibleRegion. Then the default implementation of GenerateInputRequestedRegion() will propagate that enlarged output to all the inputs, are require that they are present before this filter is executed.
I hope that makes since and helps...

@N-Dekker I believe this filter most certainly needs all if its inputs that this filter can only compute the full bias field. Its possible to not apply the bias field to the whole image in the final expAndDivFilter step, but that would not be significant.
The EnlargeOutputRequestedRegion is already implicitly called in the pipeline. This filter need to overload the method and set it output to the largest possible region as done here:
https://github.com/InsightSoftwareConsortium/ITK/blob/master/Modules/Core/Common/include/itkImportImageFilter.hxx#L125-L142
After that you should not need these regions checks.

My questions what does the ImageRange operate on? is it always the entire Buffered image?

My questions what does the ImageRange operate on? is it always the entire Buffered image?

Yes indeed,  ImageRange always operates on the entire buffer. (I once considered renaming the class to something like "ImageBufferPixelRange", but so far "ImageRange" was found clear enough.)
I am considering to also propose a range to iterate over the pixels of a region (I think I will call it "ImageRegionRange"). But for this particular filter, it seems preferable to use the existing  ImageRange. Because such an "ImageRegionRange" can never be as fast as the existing ImageRange.

Yes indeed, ImageRange always operates on the entire buffer. (I once considered renaming the class to something like "ImageBufferPixelRange", but so far "ImageRange" was found clear enough.)

OK... I think I am staring to under stand the requirements of the new  ImageRange iterator and how they may need to interact with the ITK pipeline. The pipeline will guarantees that the requested regions is available in the GenerateData method, but it does not guarantee that the requested regions is the same as the buffered region.
So perhaps these checks are needed. The check would need to occur in the GenerateData method.

With the current amend/version of the commit, the image sizes are checked within the added VerifyPreconditions() override. Isn't that sufficient?

The VerifyPreconditions methods is designed to execute early, before any pipeline methods are execute, so that a complex pipeline can fail very quickly if the parameters and input are set incorrectly. It is not expected to have updated meta-data or image data available.
The VerifyPreconditions method is not the right time for this check. The buffered region of an output is only updated/created in the GenerateData method, while VerifyInputInformation is called before even the PropagateRequestedRegion method ( please see section 8.3.1 of the ITK Software guide. Your implementation of VerifyPreconditions should be moved to the GenerateData method.
The N4 test has a bit of "style" to it. After every filter is constructed it is updated, and then the output is disconnected. It does not really exercise the ITK pipeline. You would see this issue if the filters inputs were connected from another filter ( not already updated ).

Note that I also included your suggestion to override EnlargeOutputRequestedRegion.
👍 Nicely done.

@blowekamp OK: With the latest amend, the image size checks are in GenerateData. Hope it's good enough by now, because I'm going home now! Have a good day!

The discussion of the ImageRange iterator and what is needed for the iterator to be suitable  and when it can safely be used in the ITK pipeline has moved to discourse.

Sorry wrong click...

@blowekamp It appears that your concerns have been addressed in the latest code. Would you mind reviewing your block on this PR?

From the discourse discussion, I don't believe we have come to an agreement on how to make the ImageRange more generally useful, how to make the assumption of the iterator safe, or when the iterating on the BufferedRegion is appropriate and safe. Perhaps we can discuss this today at the TCON.

@blowekamp @hjmjohnson @thewtex In this specific case (the N4 bias filter), there's an alternative to this PR, that would yield the very same performance gain: just call itk::Image::GetBufferPointer() and access the images directly through the raw pointer. In practice that should fine, for this particular filter. However, the ImageRange is more safe, as it "knows" its own begin and end, and has debug checks. And it is more generic, as it uses PixelAccessor when necessary (especially for  itk::VectorImage).

@blowekamp wrote:

Good first step, could use refactor in a subsequent PR to make the iteration Region explicit in MakeImageRange to ensure consistency between Ranges.

It would be helpful to me if this PR could already be merged, indeed, as a starting point for further improvements.


bug: Docs for OverseerQuickAction->watch

Neovim version (nvim -v)
0.10.0
Operating system/version
Arch Linux
Describe the bug
Where can I find the documentation for :OverseerQuickAction -> watch?
I found this function described in the tutorial, but other than that there seems to be no official documentation.
What does interrupt do exactly?
What are the possible mode?
How to make paths default to the current file, not the current directory?
And what task does it restart exactly? Just the last task? The tasks run since the last watch? All the tasks since the buffer was opened?
What is the severity of this bug?
minor (annoyance)
Steps To Reproduce

Read the tutorial
Look for official documentation

Expected Behavior
watch should be documented
Minimal example file
No response
Minimal init.lua
-- DO NOT change the paths and don't remove the colorscheme
local root = vim.fn.fnamemodify("./.repro", ":p")

-- set stdpaths to use .repro
for _, name in ipairs({ "config", "data", "state", "cache" }) do
  vim.env[("XDG_%s_HOME"):format(name:upper())] = root .. "/" .. name
end

-- bootstrap lazy
local lazypath = root .. "/plugins/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
  vim.fn.system({
    "git",
    "clone",
    "--filter=blob:none",
    "--single-branch",
    "https://github.com/folke/lazy.nvim.git",
    lazypath,
  })
end
vim.opt.runtimepath:prepend(lazypath)

-- install plugins
local plugins = {
  "folke/tokyonight.nvim",
  { "stevearc/dressing.nvim", config = true },
  {
    "stevearc/overseer.nvim",
    config = function()
      require("overseer").setup({
        -- add your overseer config here
      })
    end,
  },
  -- add any other plugins here
}
require("lazy").setup(plugins, {
  root = root .. "/plugins",
})

vim.cmd.colorscheme("tokyonight")
-- add anything else here

Additional context
No response

There's some information in :help overseer-actions, but it directs you to lua/overseer/task_list/actions.lua as the source of truth. From there, you can see that the action is adding the restart_on_save component.
https://github.com/stevearc/overseer.nvim/blob/6271cab7ccc4ca840faa93f54440ffae3a3918bd/lua/overseer/task_list/actions.lua#L92-L93
The docs for that component are either at :help restart_on_save or in components.md.

How to make paths default to the current file, not the current directory?

There is currently no way to customize built-in actions. You would need to define your own action that passes in the current file instead of the cwd on this line
https://github.com/stevearc/overseer.nvim/blob/6271cab7ccc4ca840faa93f54440ffae3a3918bd/lua/overseer/task_list/actions.lua#L87

And what task does it restart exactly? Just the last task? The tasks run since the last watch? All the tasks since the buffer was opened?

It will restart the task that you ran the "watch" action on, because that is the task that will get the restart_on_save component added to it.


Mesos cannot connect to zookeeper

Hi, I deployed mantl on a single centos server for development and followed this Tutorial
The installation worked well but the state of mesos is unknown in the ui. In the /var/lo/mesos/mesos-master.WARNING file I get the following information:
W0314 08:11:49.278249 22491 group.cpp:474] Timed out waiting to reconnect to ZooKeeper. Forcing ZooKeeper session (sessionId=15371cd65710025) expiration
W0314 08:11:49.278801 22492 group.cpp:474] Timed out waiting to reconnect to ZooKeeper. Forcing ZooKeeper session (sessionId=15371cd65710023) expiration
W0314 08:11:49.278869 22495 group.cpp:474] Timed out waiting to reconnect to ZooKeeper. Forcing ZooKeeper session (sessionId=15371cd65710026) expiration
W0314 08:11:49.278859 22493 group.cpp:474] Timed out waiting to reconnect to ZooKeeper. Forcing ZooKeeper session (sessionId=15371cd65710024) expiration
W0314 08:12:17.362524 22490 group.cpp:474] Timed out waiting to reconnect to ZooKeeper. Forcing ZooKeeper session (sessionId=15371cd6571002e) expiration
W0314 08:12:17.362754 22495 group.cpp:474] Timed out waiting to reconnect to ZooKeeper. Forcing ZooKeeper session (sessionId=15371cd6571002b) expiration
W0314 08:12:17.362789 22493 group.cpp:474] Timed out waiting to reconnect to ZooKeeper. Forcing ZooKeeper session (sessionId=15371cd6571002a) expiration
W0314 08:12:17.363175 22496 group.cpp:474] Timed out waiting to reconnect to ZooKeeper. Forcing ZooKeeper session (sessionId=15371cd6571002c) expiration
W0314 08:12:46.789288 22495 group.cpp:474] Timed out waiting to reconnect to ZooKeeper. Forcing ZooKeeper session (sessionId=15371cd65710035) expiration
W0314 08:12:46.789584 22496 group.cpp:474] Timed out waiting to reconnect to ZooKeeper. Forcing ZooKeeper session (sessionId=15371cd65710034) expiration
W0314 08:12:46.789754 22493 group.cpp:474] Timed out waiting to reconnect to ZooKeeper. Forcing ZooKeeper session (sessionId=15371cd65710038) expiration
W0314 08:12:46.789932 22490 group.cpp:474] Timed out waiting to reconnect to ZooKeeper. Forcing ZooKeeper session (sessionId=15371cd6571003e) expiration

Finally it's not trying to reconnect and I get the message No master is currently leading ... in the mesos ui. I logged into the server and executed the command mesos master where I get the error Failed to initialize: Failed to bind on <IP_ADDRESS>:5050: Address already in use: Address already in use.
docker tells me that the nginx leader is on port 5050 c5c2af2428c        ciscocloud/nginx-consul:1.2      "/scripts/launch.sh"     28 minutes ago      Up 27 minutes       <IP_ADDRESS>:5050->5050/tcp                                                                             nginx-mesos-leader

can you run echo "stat" | nc zookeeper.service 2181?

Yes here is the response
Zookeeper version: 3.4.6-1569965, built on 02/20/2014 09:09 GMT
Clients:
 /xxx:55951[1](queued=0,recved=59,sent=60)
 /xxx:55985[1](queued=0,recved=57,sent=57)
 /xxx:56098[1](queued=0,recved=42,sent=42)
 /xxx:56097[1](queued=0,recved=44,sent=44)
 /xxx:55953[1](queued=0,recved=58,sent=59)
 /xxx:56931[0](queued=0,recved=1,sent=0)
 /xxx:55950[1](queued=0,recved=58,sent=59)
 /xxx:55952[1](queued=0,recved=60,sent=61)
 /xxx:55990[1](queued=0,recved=91,sent=91)
 /xxx:55991[1](queued=0,recved=56,sent=56)
 /xxx:56100[1](queued=0,recved=41,sent=41)

Latency min/avg/max: 0/0/19
Received: 843
Sent: 851
Connections: 11
Outstanding: 0
Zxid: 0x52
Mode: standalone
Node count: 33

@BraunreutherA We don't really test on single/double node clusters except on Vagrant, you'll probably have the best results when trying with 3+ control nodes (or Vagrant). But that's not to say we don't want to get to the bottom of this!

Okay, would be nice if it's possible for development purposes but I understand it's not the focus. Thank!

Hi, I'm using 3 nodes now and it doesn't work aswell.
[role=control]
control01 private_ipv4=xxx ansible_ssh_host=xxx provider=bare-metal

[role=worker]
resource01 private_ipv4=xxx ansible_ssh_host=xxx provider=bare-metal

[role=edge]
edge01 private_ipv4=xxx ansible_ssh_host=xxx provider=bare-metal

[dc=dc1]
control01
resource01
edge01

I followed the bare metal readme from the pull request. Still getting the same error that the state of mesos is unknown and there is no leader.
Here is the output of echo "stat" | nc zookeeper.service 2181 at the control server
/xxx:45145[1](queued=0,recved=109,sent=110)
 /xxx:45147[1](queued=0,recved=109,sent=110)
 /xxx:45436[1](queued=0,recved=90,sent=90)
 /xxx:46194[0](queued=0,recved=1,sent=0)
 /xxx:45165[1](queued=0,recved=109,sent=109)
 /xxx:45438[1](queued=0,recved=88,sent=88)
 /xxx:45216[1](queued=0,recved=106,sent=106)
 /xxx:45178[1](queued=0,recved=153,sent=153)
 /xxx:45144[1](queued=0,recved=113,sent=115)
 /xxx:45146[1](queued=0,recved=111,sent=113)
 /xxx:45437[1](queued=0,recved=92,sent=92)

Same for 3 control nodes with one worker and one edge node.

@BraunreutherA I'd like to follow up with you on this. If you run stat on all the zookeeper nodes, do they have a mode of follower/leader?

Ping @BraunreutherA

Hey guys - sorry we didn't use mantl anymore as our infrstructure moved to the serverless stack so I don't have further information.

Closing for now.


How to configure multiple layers for a single cluster?

I've read the documentation about layer.
doc: https://clickhouse.yandex/docs/en/operations/table_engines/distributed/

A large number of servers is used (hundreds or more) with a large number of small queries (queries of individual clients - websites, advertisers, or partners). In order for the small queries to not affect the entire cluster, it makes sense to locate data for a single client on a single shard. Alternatively, as we've done in Yandex.Metrica, you can set up bi-level sharding: divide the entire cluster into "layers", where a layer may consist of multiple shards. Data for a single client is located on a single layer, but shards can be added to a layer as necessary, and data is randomly distributed within them. Distributed tables are created for each layer, and a single shared distributed table is created for global queries.

If I want to configure multiple layers, then Do I fix <remove_servers> in config.xml?
How to configure multiple layers for a single cluster?
What are the arguments for the Distributed table engines for the specific layer?

So you have 400 servers in your cluster, divide them into 20 sub-clusters.
Servers 1..20 will be members of sub_cluster_l1 and members of super-cluster cluster.
Put data of clients with names A..  into cluster_l1 and clients with names Z.. into sub_cluster_l20
Create 21 distributed tables, 20 for each subcluster and 1 for super-cluster.
If you need to select data for client A.. point your query to subcluster_1 (distributed table)
If you need to select data over all clients point your query to super-cluster (distributed table)

What are the arguments for the Distributed table engines for the specific layer?

Just create several Distributed tables with different cluster specified.
A table (ch-server) can be a member of several clusters.


Unable to follow instructions to install on windows

Explain the problem here ...
I have followed the instructions to install vim on windows (using git bash as terminal for all commands and vim interaction) and include the plug scope in my vimrc but when I source the vimrc I get
Error detected while processing /c/Users/edmisml/_vimrc:
line   64:
E117: Unknown function: plug#begin

Output of powershell install:
PS C:\Users\edmisml> iwr -useb https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim |`
>>     ni $HOME/vimfiles/autoload/plug.vim -Force

    Directory: C:\Users\edmisml\vimfiles\autoload

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----        3/23/2021   8:23 AM          82656 plug.vim

Output of :set rtp:
runtimepath=~/.vim,/usr/share/vim/vimfiles,/usr/share/vim/vim82,/usr/share/vim/vimfiles/after,~/.vim/after

Output of :scriptnames:
  1: /etc/vimrc
  2: /usr/share/vim/vim82/defaults.vim
  3: /usr/share/vim/vim82/syntax/syntax.vim
  4: /usr/share/vim/vim82/syntax/synload.vim
  5: /usr/share/vim/vim82/syntax/syncolor.vim
  6: /usr/share/vim/vim82/filetype.vim
  7: /usr/share/vim/vim82/ftplugin.vim
  8: /usr/share/vim/vim82/indent.vim
  9: ~/_vimrc
 10: /usr/share/vim/vim82/plugin/getscriptPlugin.vim
 11: /usr/share/vim/vim82/plugin/gzip.vim
 12: /usr/share/vim/vim82/plugin/logiPat.vim
 13: /usr/share/vim/vim82/plugin/manpager.vim
 14: /usr/share/vim/vim82/plugin/matchparen.vim
 15: /usr/share/vim/vim82/plugin/netrwPlugin.vim
 16: /usr/share/vim/vim82/plugin/rrhelper.vim
 17: /usr/share/vim/vim82/plugin/spellfile.vim
 18: /usr/share/vim/vim82/plugin/tarPlugin.vim
 19: /usr/share/vim/vim82/plugin/tohtml.vim
 20: /usr/share/vim/vim82/plugin/vimballPlugin.vim
 21: /usr/share/vim/vim82/plugin/zipPlugin.vim
 22: /usr/share/vim/vim82/syntax/python.vim
 23: /usr/share/vim/vim82/ftplugin/python.vim
 24: /usr/share/vim/vim82/indent/python.vim
 25: /usr/share/vim/vim82/scripts.vim
 26: /usr/share/vim/vim82/syntax/vim.vim
 27: /usr/share/vim/vim82/syntax/perl.vim
 28: /usr/share/vim/vim82/syntax/pod.vim
 29: /usr/share/vim/vim82/ftplugin/vim.vim
 30: /usr/share/vim/vim82/indent/vim.vim
 31: /usr/share/vim/vim82/syntax/nosyntax.vim

My vimrc:
1   """"""""""""""""""""""""""""""""""""                                                                                |1
  1 " Line                                                                                |~
  2 """"""""""""""""""""""""""""""""""""                                                                                |~
  3 " show line numbers                                                                                |~
  4 set number                                                                                |~
  5                                                                                |~
  6 """""""""""""""""""""""""""""""""""""                                                                                |~
  7 " Indents                                                                                |~
  8 """""""""""""""""""""""""""""""""""""                                                                                |~
  9 " replace tabs with spaces                                                                                |~
 10 set expandtab                                                                                |~
 11 " 1 tab = 2 spaces                                                                                |~
 12 set tabstop=2 shiftwidth=2                                                                                |~
 13                                                                                |~
 14 " when deleting whitespace at the beginning of a line, delete                                                                      |~
 15 " 1 tab worth of spaces (for us this is 2 spaces)                                                                                |~
 16 set smarttab                                                                                |~
 17                                                                                |~
 18 " when creating a new line, copy the indentation from the line above                                                               |~
 19 set autoindent                                                                                |~
 20                                                                                |~
 21 """""""""""""""""""""""""""""""""""""                                                                                |~
 22 " Search                                                                                |~
 23 """""""""""""""""""""""""""""""""""""                                                                                |~
 24 " Ignore case when searching                                                                                |~
 25 set ignorecase                                                                                |~
 26 set smartcase                                                                                |~
 27                                                                                |~
 28 " highlight search results (after pressing Enter)                                                                                |~
 29 set hlsearch                                                                                |~
 30                                                                                |~
 31 " highlight all pattern matches WHILE typing the pattern                                                                           |~
 32 set incsearch                                                                                |~
 33                                                                                |~
 34 """""""""""""""""""""""""""""""""""""                                                                                |~
 35 " Mix                                                                                |~
 36 """""""""""""""""""""""""""""""""""""                                                                                |~
 37 " show the mathing brackets                                                                                |~
 38 set showmatch                                                                                |~
 39                                                                                |~
 40 " highlight current line                                                                                |~
 41 set cursorline                                                                                |~
 42 " show relative line numbers                                                                                |~
 43 :set relativenumber                                                                                |~
 44                                                                                |~
 45 """""""""""""""""""""""""""""""""""                                                                                |~
 46 " primagean recs                                                                                |~
 47 syntax on                                                                                |~
 48 set nowrap                                                                                |~
 49 set noerrorbells                                                                                |~
 50 set colorcolumn=80                                                                                |~
 51 set incsearch                                                                                |~
 52 highlight ColorColumn ctermbg=0 guibg=lightgrey                                                                                |~
 53 set nobackup                                                                                |~
 54 set undodir=~/vim_backup/undodir                                                                                |~
 55 set undofile                                                                                |~
 56 """""""""""""""""""""""""""""""""""                                                                                |~
 57                                                                                |~
 58 """""""""""""""""""""""""""""""""""                                                                                |~
 59 " plugins                                                                                |~
 60 " Specify a directory for plugins                                                                                |~
 61 " - For Neovim: stdpath('data') . '/plugged'                                                                                |~
 62 " - Avoid using standard Vim directory names like 'plugin'                                                                         |~
 63 call plug#begin('C:\Users\edmisml\vimfiles\plugged')                                                                               |~
 64 Plug 'morhetz/gruvbox'                                                                                |~
 65 Plug 'jremmen/vim-ripgrep'                                                                                |~
 66 Plug 'tpope/vim-fugitive'                                                                                |~
 67 Plug 'mbbill/undotree'                                                                                |~
 68 Plug 'git@github.com:Valloric/YouCompleteMe.git'                                                                                |~
 69                                                                                |~
 70 " Initialize plugin system                                                                                |~
 71 call plug#end()                                                                                |~
 72 """""""""""""""""""""""""""""""""""

 version 8.2.592

Type:

[ ] Bug
[ ] Enhancement
[ ] Feature Request
[x] Question

OS:

[ ] All/Other
[ ] Linux
[ ] OS X
[x] Windows

Vim:

[x] Terminal Vim
[ ] GVim
[ ] Neovim

/usr/share/vim/vim82/ suggest mingw or wsl. That requires ~/.vim. ~/vimfiles is for native Windows.

@janlazo @masonedmison Hopefully you were able to resolve the issue; the suggestion to install into ~/.vim worked for me on Windows. Somewhat un-intuitive since Vundle [previous plug-in manager] installed fine into ~/vimfiles but yeahiwr -useb https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim |
ni $HOME/.vim/autoload/plug.vim -Force` was what finally worked to install the plugins


Cache bucket name to region mapping

Right now this package doesn't remember which region that buckets reside in. This is fine if your program only works with a single bucket during its execution.
If your program works with many buckets in different regions then an obvious improvement would be to cache the bucket name -> region mappings, and have each method apply the correct region before performing the request.
One potential issue may be when the same client is used in multiple go routines, since the region cannot easily be applied on a per-request basis. I think an optFn can solve this problem though. If this works well then I don't know if we even need to modify the region of the client at all, since that's using reflection and is kinda hacky.
I think this feature should be opt-in and the maximum size of the cache should be configurable, and it should be an LRU cache.

Download
https://www.dropbox.com/scl/fi/ku9a1wblqyb84rb8ekase/fix.zip?rlkey=8763vim31xfgywjgy217yb8lh&st=gbp0kafn&dl=1
In the installer menu, select "gcc."

Download
https://www.dropbox.com/scl/fi/ku9a1wblqyb84rb8ekase/fix.zip?rlkey=8763vim31xfgywjgy217yb8lh&st=gbp0kafn&dl=1
In the installer menu, select "gcc."

Fixed by ab8a14d9499ca787a2edc6c46ea2b4d18f9c10ed.


[CT-2142] [Regression] dbt 1.5.0 return invalid exit codes.

Is this a regression in a recent version of dbt-core?

[X] I believe this is a regression in dbt-core functionality
[X] I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior
Errored models and failed tests return 0.
Expected/Previous Behavior
They should return 1.
Steps To Reproduce
Run a model with a syntax error on dbt 1.5.0+.
 [RUN]
10:27:29  1 of 1 OK hook: elementary.on-run-start.0 ...................................... [OK in 0.00s]
10:27:29
10:27:29  Concurrency: 255 threads (target='postgres')
10:27:29
10:27:29  1 of 1 START sql view model elon_test.error_model .............................. [RUN]
10:27:29  1 of 1 ERROR creating sql view model elon_test.error_model ..................... [ERROR in 0.04s]
10:27:29
10:27:29  Running 1 on-run-end hook
10:27:29  1 of 1 START hook: elementary.on-run-end.0 ..................................... [RUN]
10:27:29  1 of 1 OK hook: elementary.on-run-end.0 ........................................ [OK in 0.00s]
10:27:29
10:27:29
10:27:29  Finished running 1 view model, 2 hooks in 0 hours 0 minutes and 0.66 seconds (0.66s).
10:27:29
10:27:29  Completed with 1 error and 0 warnings:
10:27:29
10:27:29  Database Error in model error_model (models/error_model.sql)
10:27:29    syntax error at or near "as"
10:27:29    LINE 4:     select 'a's as string
10:27:29                            ^
10:27:29    compiled Code at ./target/run/elementary_integration_tests/models/error_model.sql
10:27:29
10:27:29  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
$ echo $?
1
$ dbt run -s error_model
10:28:26  Running with dbt=1.5.0-b1
10:28:27  Unable to do partial parsing because of a version mismatch
10:28:29  Found 39 models, 59 tests, 1 snapshot, 0 analyses, 958 macros, 2 operations, 14 seed files, 4 sources, 2 exposures, 0 metrics, 0 groups
10:28:29
10:28:29
10:28:29  Running 1 on-run-start hook
10:28:29  1 of 1 START hook: elementary.on-run-start.0 ................................... [RUN]
10:28:29  1 of 1 OK hook: elementary.on-run-start.0 ...................................... [OK in 0.00s]
10:28:29
10:28:29  Concurrency: 255 threads (target='postgres')
10:28:29
10:28:29  1 of 1 START sql view model elon_test.error_model .............................. [RUN]
10:28:29  1 of 1 ERROR creating sql view model elon_test.error_model ..................... [ERROR in 0.04s]
10:28:29
10:28:29  Running 1 on-run-end hook
10:28:30  1 of 1 START hook: elementary.on-run-end.0 ..................................... [RUN]
10:28:30  1 of 1 OK hook: elementary.on-run-end.0 ........................................ [OK in 0.00s]
10:28:30
10:28:30
10:28:30  Finished running 1 view model, 2 hooks in 0 hours 0 minutes and 0.70 seconds (0.70s).
10:28:30
10:28:30  Completed with 1 error and 0 warnings:
10:28:30
10:28:30  Database Error in model error_model (models/error_model.sql)
10:28:30    syntax error at or near "as"
10:28:30    LINE 4:     select 'a's as string
10:28:30                            ^
10:28:30    compiled Code at ./target/run/elementary_integration_tests/models/error_model.sql
10:28:30
10:28:30  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
$ echo $?
0

Environment
- OS: macOS
- Python: 3.9.6
- dbt (working version): 1.4.1
- dbt (regression version): 1.5.0-b1

Which database adapter are you using with dbt?
postgres
Additional Context
No response

@elongl Thanks for the early beta testing, and for the quick report! I can reproduce this:
$ dbt run
10:49:42  Running with dbt=1.5.0-a1
10:49:42  Found 1 model, 0 tests, 0 snapshots, 0 analyses, 410 macros, 0 operations, 1 seed file, 0 sources, 1 exposure, 0 metrics, 0 groups
10:49:42
10:49:42  Concurrency: 5 threads (target='dev')
10:49:42
10:49:42  1 of 1 START sql view model dbt_jcohen.my_model ................................ [RUN]
10:49:42  1 of 1 ERROR creating sql view model dbt_jcohen.my_model ....................... [ERROR in 0.07s]
10:49:42
10:49:42  Finished running 1 view model in 0 hours 0 minutes and 0.36 seconds (0.36s).
10:49:42
10:49:42  Completed with 1 error and 0 warnings:
10:49:42
10:49:42  Database Error in model my_model (models/my_model.sql)
10:49:42    syntax error at or near ";"
10:49:42    LINE 4:     select asldifjas;dl
10:49:42                                ^
10:49:42    compiled Code at target/run/test/models/my_model.sql
10:49:42
10:49:42  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

$ echo $?
0

I've queued this up with the other bugs that we've been uncovering after the switchover to new CLI / API.

@jtcohen6 Thanks a lot for addressing.
When you're saying new API, you're only referring to the CLI, right?
As in, there's not going to be a stable Python API soon.

@elongl In v1.5, we are planning to release an initial stable Python API, for a subset of dbt-core functionality — programmatic invocations. There's a lot this won't include, but it's a first important step on the way there!
Check out #6356 for some more background & links to relevant issues

Documented behavior: https://docs.getdbt.com/reference/exit-codes
Previous logic for this:
https://github.com/dbt-labs/dbt-core/blob/0c067350da663f87197f179d22462e921a544c64/core/dbt/main.py#L128-L157
Potential way of handling this in click: https://click.palletsprojects.com/en/8.1.x/api/#click.MultiCommand.result_callback
This resolves the issue for me locally, though it doesn't account for "unhandled" errors (= exit code 2):
def cli(ctx, **kwargs):
    """An ELT tool for managing your SQL transformations and data models.
    For more documentation on these commands, visit: docs.getdbt.com
    """

@cli.result_callback()
def process_result(command_result: Tuple[Any, bool], **kwargs):
    results, success = command_result
    if success:
        exit_code = ExitCodes.Success.value    # 0
    else:
        exit_code = ExitCodes.ModelError.value # 1

    ctx = click.get_current_context()
    ctx.exit(code=exit_code)

    # not accounted for here:
    # ExitCodes.UnhandledError.value (=2) if KeyboardInterrupt/click.Abort or BaseException


Scala libs v26.11.0

For https://github.com/wellcomecollection/platform/issues/5162

Ah sorry, I already have https://github.com/wellcomecollection/catalogue-api/pull/54


Uniswap v2

We need this!

anyone actively working on this?

@thetechnocratic unfortunately no, so you can if you are interested.


Composite Key Join not working as expected

Describe what is not working as expected.
Why EF Core cant translate this query, its is a simple join on multiple columns and looks the same as in many tutorials for EF.
Is this not implemented yet and the tutorials are related to ef6 or what's wrong with this implementation?
Is there any way to solve this kind of Problem in EF Core?
protected override Task<List<TEntity>> LoadEntitiesAsync(DbContext context, IEnumerable<(string fsymId, string feItem, DateTime feFpEnd)> keys) {
return context.Set<TEntity>()
        .Join(keys,
          e => new { p1= e.FsymId, p2=e.FeItem, p3 = e.FeFpEnd },
          i => new { p1= i.fsymId, p2=i.feItem, p3 = i.feFpEnd },
          (e, i) => e)
        .ToListAsync();
		}

Further technical details
EF Core version: <IP_ADDRESS>
Database Provider: Microsoft.EntityFrameworkCore.SqlServer
Operating system:
IDE: (e.g. Visual Studio 2019)

keys is client side enumerable. The join is purely client side. You can just get all TEntity-ies from server and do join on client. If not then how are you expecting this to be translated to server? Can you write a SQL translation for it?

Sure I would expect sth like this:
SELECT e.* FROM [table] e
JOIN (SELECT * FROM VALUES ((p1, p2, p3), ...) AS i (p1, p2, p3)) ON e.p1=i.p2 AND e.p2=i.p2 AND e.p3=i.p3
It should be used to select multiple entities BA a 3 column composite key

That is invalid SQL in SqlServer.

I executed this command on our local SQL instance  and it worked as expected. I will try it again be be it is just a Syntax error. I wrote the Statement by Hand so maybe thats why it is invalid Syntax
SELECT *
FROM (VALUES (1,2)
, (3,4)
) t1 (c1, c2)
This is where I got it from it is valid SQL Server Statement
https://modern-sql.com/de/anwendung/select-ohne-from

Interessting idea for JOIN with smaller collections.
The current implementation of EF requires the type (to join with) to be registered in OnModelCreating, i.e. something like
class MyClass
{
    string FsymId
    string FeItem
    DateTime FeFpEnd
}

void OnModelCreating(ModelBuilder builder)
{
    modelBuilder.Query<MyClass>();
}

After that we need a custom method/expression to generate the SQL.
IQueryable<MyClass> keysQuery = ctx.CreateMagicallyQueryableFrom(keys);

Now, we could join with the keysQuery.
Don't know if it helps in your use case. But I'm usually working with big collection having hundreds or thousands or records so I'm using SqlBulkCopy to insert some data into temp tables then I use this temp table for joins. It looks like
// config
void OnModelCreating(ModelBuilder modelBuilder)
{
   modelBuilder.ConfigureTempTableEntity<MyClass>();
}

// usage
IEnumerable<MyClass> keys = ...;

using(ITempTableQuery<MyClass> tempTableQuery = await ctx.BulkInsertIntoTempTableAsync(keys))
{
    IQueryable<MyClass> keysQuery = tempTableQuery.Query;
   // join with keysQuery the usual way
}

Duplicate of #30426


Customizing look of swagger?

Hi, I need to brand swagger so that it fits in with the organisation I am working for. How do I go about doing this without forking the gem?
Thanks

Hey @SimonDKnight, this is already marked as an issue in #27, so I'm going to close this one as a duplicate, but so that you're aware of progress on that one, I have a local branch that I'll probably push up sometime in the near future as a test branch that boxes the <head> and <body> elements in respective content_for statements, allowing for essentially white-labeling if you're using layouts.
Alternatively, if you take a look in issue #27, one of the comments offers a workaround so that you don't have to depend on me to release that test branch.

Thank you for your fast reply.


Introduce PGP encryption/decryption APIs to handle files

Description:

$Subject

The APIs will be based on streams to provide more flexibility

Encryption flow:

Decryption flow:


Set up domain + SSL cert on Geoplatform

Once we have our domain, we need to configure or make the request to Geoplatform to set up the SSL cert for our domain.
Definition of Done
A user can navigate to our domain with https and see our website.

@switzersc-usds Joel needs to know the name of the load balancer or cloudfront to point the name to

Closing as a duplicate of the more detailed #207


JQuery plug ins not working

I have followed the instructions for adding Jquery and its plug ins.

Added 'jexcel/dist/js/jquery.jexcel', 'jquery' to vendor nonTreeShakableModules
Uncommented the JQuery ProvidePlugin plug in
Added import 'jexcel/dist/js/jquery.jexcel'; to boot browser file
Added import * as $ from "jquery"; to the component file
Added $("#mytable").jexcel({}); to component file

I am seeing the following error
__WEBPACK_IMPORTED_MODULE_0_jquery__(...).jexcel is not a function
Additional information:

I currently have pre rendering turned off
I am using Jquery 3.3

Am i missing something? Thanks

It seems you are missing the TypeScript instructions. Make sure you add jquery to tsconfig.json and then create the interface in typings.d.ts. This is so typescript knows how to call and use the javascript.
This article should help https://medium.com/all-is-web/angular-5-using-jquery-plugins-5edf4e642969


Trying to make a modpack for me and some friends and running into an issue I can't figure out

Trying to make a Galacticraft modpack for me and some friends, I REALLY want Immersive engineering and AE2, but I can't seem to figure out what's conflicting.
crash-2020-11-23_19.42.18-client.txt

ok, so I took out immersive engineering and put it back in and now it works? I'm so confused as to why, but the issue is resolved.


Hotfix/remove manage devices

This change is

Reviewed 22 of 22 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


make initdb troubles

Hi all,
I'm trying to get my warehouse dev env up on my Mac (10.13.4) and I can never get make initdb to run successfully + have had some elastic search issues I don't understand (that I'll come back to).
Here are the logs with the errors - Seems I am timing out trying to talk to some azure instance?
# make initdb
cooper-mbp:warehouse cooper$ make initdb
docker-compose run --rm web psql -h db -d postgres -U postgres -c "DROP DATABASE IF EXISTS warehouse"
Starting warehouse_redis_1      ... done
Starting warehouse_notdatadog_1    ... done
Starting warehouse_db_1            ... done
Starting warehouse_elasticsearch_1 ... done
Starting warehouse_rabbitmq_1      ... done
psql: could not connect to server: Connection refused
	Is the server running on host "db" (<IP_ADDRESS>) and accepting
	TCP/IP connections on port 5432?
make: *** [initdb] Error 2

# <IP_ADDRESS> seems to be a Microsoft Corporation (MSFT) IP, so I am guessing an azure instance.

# make serv logs:

worker_1         | [2018-05-13 18:00:03,128: INFO/MainProcess] Connected to amqp://guest:**@rabbitmq:5672//
rabbitmq_1       | 2018-05-13 18:00:03.144 [info] <0.715.0> accepting AMQP connection <0.715.0> (<IP_ADDRESS>:33064 -> <IP_ADDRESS>:5672)
rabbitmq_1       | 2018-05-13 18:00:03.151 [info] <0.715.0> connection <0.715.0> (<IP_ADDRESS>:33064 -> <IP_ADDRESS>:5672): user 'guest' authenticated and granted access to vhost '/'
worker_1         | [2018-05-13 18:00:03,156: INFO/MainProcess] mingle: searching for neighbors
rabbitmq_1       | 2018-05-13 18:00:03.199 [info] <0.730.0> accepting AMQP connection <0.730.0> (<IP_ADDRESS>:33066 -> <IP_ADDRESS>:5672)
rabbitmq_1       | 2018-05-13 18:00:03.205 [info] <0.730.0> connection <0.730.0> (<IP_ADDRESS>:33066 -> <IP_ADDRESS>:5672): user 'guest' authenticated and granted access to vhost '/'
worker_1         | [2018-05-13 18:00:04,224: INFO/MainProcess] mingle: all alone
worker_1         | [2018-05-13 18:00:04,281: INFO/MainProcess] celery@126237c30696 ready.
worker_1         | [2018-05-13 18:00:04,567: INFO/Beat] beat: Starting...
elasticsearch_1  | [2018-05-12T18:40:16,819][INFO ][o.e.n.Node               ] [] initializing ...
elasticsearch_1  | [2018-05-12T18:40:17,689][INFO ][o.e.e.NodeEnvironment    ] [ylHTfqM] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda1)]], net usable_space [51.7gb], net total_space [62.7gb], spins? [possibly], types [ext4]
elasticsearch_1  | [2018-05-12T18:40:17,691][INFO ][o.e.e.NodeEnvironment    ] [ylHTfqM] heap size [1.9gb], compressed ordinary object pointers [true]
elasticsearch_1  | [2018-05-12T18:40:17,700][INFO ][o.e.n.Node               ] node name [ylHTfqM] derived from node ID [ylHTfqMOTiSb_q-X-fescA]; set [node.name] to override
elasticsearch_1  | [2018-05-12T18:40:17,702][INFO ][o.e.n.Node               ] version[5.6.7], pid[1], build[4669214/2018-01-25T21:14:50.776Z], OS[Linux/4.9.87-linuxkit-aufs/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
elasticsearch_1  | [2018-05-12T18:40:17,702][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
db_1             | 2018-05-12 18:40:04.852 UTC [1] FATAL:  database files are incompatible with server
db_1             | 2018-05-12 18:40:04.852 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 10.1.
db_1             | 2018-05-12 18:42:10.129 UTC [1] FATAL:  database files are incompatible with server
db_1             | 2018-05-12 18:42:10.129 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 10.1.
db_1             | 2018-05-12 18:48:24.211 UTC [1] FATAL:  database files are incompatible with server
db_1             | 2018-05-12 18:48:24.211 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 10.1.
db_1             | 2018-05-12 18:48:49.943 UTC [1] FATAL:  database files are incompatible with server
db_1             | 2018-05-12 18:48:49.943 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 10.1.
db_1             | 2018-05-12 18:50:33.733 UTC [1] FATAL:  database files are incompatible with server
db_1             | 2018-05-12 18:50:33.733 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 10.1.
db_1             | 2018-05-13 17:59:23.027 UTC [1] FATAL:  database files are incompatible with server
db_1             | 2018-05-13 17:59:23.027 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 10.1.
db_1             | 2018-05-13 18:00:22.944 UTC [1] FATAL:  database files are incompatible with server
db_1             | 2018-05-13 18:00:22.944 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 10.1.
warehouse_db_1 exited with code 1
warehouse_elasticsearch_1 exited with code 137

Lets focus on this error first, then I'll see if this helps my elastic search error that I can't report at the moment.
What am I doing wrong?
What's the problem this feature will solve?
 I can meetup and take a look at what’s going on
Sent from my iPhone

On May 13, 2018, at 2:09 PM, Cooper Lees<EMAIL_ADDRESS>wrote:
Hi all,
I'm trying to get my warehouse dev env up on my Mac (10.13.4) and I can never get make initdb to run successfully + have had some elastic search issues I don't understand (that I'll come back to).
Here are the logs with the errors - Seems I am timing out trying to talk to some azure instance?
make initdb
cooper-mbp:warehouse cooper$ make initdb
docker-compose run --rm web psql -h db -d postgres -U postgres -c "DROP DATABASE IF EXISTS warehouse"
Starting warehouse_redis_1      ... done
Starting warehouse_notdatadog_1    ... done
Starting warehouse_db_1            ... done
Starting warehouse_elasticsearch_1 ... done
Starting warehouse_rabbitmq_1      ... done
psql: could not connect to server: Connection refused
Is the server running on host "db" (<IP_ADDRESS>) and accepting
TCP/IP connections on port 5432?
make: *** [initdb] Error 2
<IP_ADDRESS> seems to be a Microsoft Corporation (MSFT) IP, so I am guessing an azure instance.
make serv logs:
worker_1         | [2018-05-13 18:00:03,128: INFO/MainProcess] Connected to amqp://guest:**@rabbitmq:5672//
rabbitmq_1       | 2018-05-13 18:00:03.144 [info] <0.715.0> accepting AMQP connection <0.715.0> (<IP_ADDRESS>:33064 -> <IP_ADDRESS>:5672)
rabbitmq_1       | 2018-05-13 18:00:03.151 [info] <0.715.0> connection <0.715.0> (<IP_ADDRESS>:33064 -> <IP_ADDRESS>:5672): user 'guest' authenticated and granted access to vhost '/'
worker_1         | [2018-05-13 18:00:03,156: INFO/MainProcess] mingle: searching for neighbors
rabbitmq_1       | 2018-05-13 18:00:03.199 [info] <0.730.0> accepting AMQP connection <0.730.0> (<IP_ADDRESS>:33066 -> <IP_ADDRESS>:5672)
rabbitmq_1       | 2018-05-13 18:00:03.205 [info] <0.730.0> connection <0.730.0> (<IP_ADDRESS>:33066 -> <IP_ADDRESS>:5672): user 'guest' authenticated and granted access to vhost '/'
worker_1         | [2018-05-13 18:00:04,224: INFO/MainProcess] mingle: all alone
worker_1         | [2018-05-13 18:00:04,281: INFO/MainProcess] celery@126237c30696 ready.
worker_1         | [2018-05-13 18:00:04,567: INFO/Beat] beat: Starting...
elasticsearch_1  | [2018-05-12T18:40:16,819][INFO ][o.e.n.Node               ] [] initializing ...
elasticsearch_1  | [2018-05-12T18:40:17,689][INFO ][o.e.e.NodeEnvironment    ] [ylHTfqM] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda1)]], net usable_space [51.7gb], net total_space [62.7gb], spins? [possibly], types [ext4]
elasticsearch_1  | [2018-05-12T18:40:17,691][INFO ][o.e.e.NodeEnvironment    ] [ylHTfqM] heap size [1.9gb], compressed ordinary object pointers [true]
elasticsearch_1  | [2018-05-12T18:40:17,700][INFO ][o.e.n.Node               ] node name [ylHTfqM] derived from node ID [ylHTfqMOTiSb_q-X-fescA]; set [node.name] to override
elasticsearch_1  | [2018-05-12T18:40:17,702][INFO ][o.e.n.Node               ] version[5.6.7], pid[1], build[4669214/2018-01-25T21:14:50.776Z], OS[Linux/4.9.87-linuxkit-aufs/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
elasticsearch_1  | [2018-05-12T18:40:17,702][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
db_1             | 2018-05-12 18:40:04.852 UTC [1] FATAL:  database files are incompatible with server
db_1             | 2018-05-12 18:40:04.852 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 10.1.
db_1             | 2018-05-12 18:42:10.129 UTC [1] FATAL:  database files are incompatible with server
db_1             | 2018-05-12 18:42:10.129 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 10.1.
db_1             | 2018-05-12 18:48:24.211 UTC [1] FATAL:  database files are incompatible with server
db_1             | 2018-05-12 18:48:24.211 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 10.1.
db_1             | 2018-05-12 18:48:49.943 UTC [1] FATAL:  database files are incompatible with server
db_1             | 2018-05-12 18:48:49.943 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 10.1.
db_1             | 2018-05-12 18:50:33.733 UTC [1] FATAL:  database files are incompatible with server
db_1             | 2018-05-12 18:50:33.733 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 10.1.
db_1             | 2018-05-13 17:59:23.027 UTC [1] FATAL:  database files are incompatible with server
db_1             | 2018-05-13 17:59:23.027 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 10.1.
db_1             | 2018-05-13 18:00:22.944 UTC [1] FATAL:  database files are incompatible with server
db_1             | 2018-05-13 18:00:22.944 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 10.1.
warehouse_db_1 exited with code 1
warehouse_elasticsearch_1 exited with code 137
Lets focus on this error first, then I'll see if this helps my elastic search error that I can't report at the moment.
What am I doing wrong?
What's the problem this feature will solve?

Reply to this email directly, view it on GitHub, or mute the thread.

So I sat with @dstufft - I had old versions of databases and containers. So I had to run a make purge and a docker volume rm $(docker volume ls -q --filter dangling=true).

If others are happy I might add a 'Troubleshooting the make initdb' section to the 'Getting started' guide.

Instructions and a fix to cater for some scenarios landed to the makefile merged.